How do people install their bots into toolforge? In the past, I've just done a "git pull" to the bastion host and run with the code directly out of the source tree, but I'm not sure that's a good plan.
My current project is a more complicated, with a bot that I'm going to run on krb as a toolforge cron job https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Creating_scheduled_jobs_(cron_jobs), a web service that's going to run via "webservice --backend=kubernetes python3.9 start" and also some command-line scripts that I'll want to run interactively. All this code will be in the same git repo. So I'm thinking of building a wheel and installing it that way. So, I guess that means building the wheel locally with setuptools, scp the wheel over to the bastion, shell into a krb node to do "pip install" into my virtualenv.
The stumbling block with that plan seems to be that toolforge insists on your webapp being installed at $HOME/www/python/src/app.py, which isn't going to play well with a wheel installation.
You would have to make a stub app.py, https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Python#Using_a_uWSGI_app_with_a_default_entry_point_that_is_not_app.py.
I've considered doing PEP 517 builds with Poetry on toolforge before, but didn't find a good way to automate it. So I'm sticking with git pull and micropipenv (to install dependencies from Poetry.lock) at the moment.
ACN
On Wed, Dec 28, 2022 at 1:31 PM Roy Smith roy@panix.com wrote:
How do people install their bots into toolforge? In the past, I've just done a "git pull" to the bastion host and run with the code directly out of the source tree, but I'm not sure that's a good plan.
My current project is a more complicated, with a bot that I'm going to run on krb as a toolforge cron job, a web service that's going to run via "webservice --backend=kubernetes python3.9 start" and also some command-line scripts that I'll want to run interactively. All this code will be in the same git repo. So I'm thinking of building a wheel and installing it that way. So, I guess that means building the wheel locally with setuptools, scp the wheel over to the bastion, shell into a krb node to do "pip install" into my virtualenv.
The stumbling block with that plan seems to be that toolforge insists on your webapp being installed at $HOME/www/python/src/app.py, which isn't going to play well with a wheel installation. _______________________________________________ pywikibot mailing list -- pywikibot@lists.wikimedia.org To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org
Hi,
On 12/28/22 10:31, Roy Smith wrote:
How do people install their bots into toolforge? In the past, I've just done a "git pull" to the bastion host and run with the code directly out of the source tree, but I'm not sure that's a good plan.
This is what I do. Each bot/webapp has a requirements.txt with a mix of pinned and unpinned dependencies, I create the virtualenv, install requirements.txt. For webapps, this is automated with the `webservice-python-bootstrap` script[1]. Whenever I git pull, manually check to see if I need to reinstall/update dependencies.
I personally find this to be the simplest way of enabling straightforward local development and minimal work to deploy, usually it's just `git pull && webservice restart`.
The stumbling block with that plan seems to be that toolforge insists on your webapp being installed at $HOME/www/python/src/app.py, which isn't going to play well with a wheel installation.
Right. From a Toolforge perspective, if there's some other, better way to do it, filing a bug would be a good first step to figuring out how to add that. T229172[2] contains some discussion of PEP 517 which ACN mentioned, but doesn't discuss the requirement of app.py.
[1] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Python#Creating_a_vir... [2] https://phabricator.wikimedia.org/T229172
-- Kunal / Legoktm
Well, thanks to everybody who offered suggestions, but after spending a few days exploring all sorts of fascinating python packaging technology, I've decided to keep doing it the way I've always done it, i.e. just use git pull on the bastion. The final straw was when I realized I would break my little bit of automation which lets my tools introspect the git repo so they can log the branch and commit id they're running:
2022-12-21T00:59:55 INFO dykbot [1671584395] git: main (4f0ef2870376b71b70d08397b273a1ffaf3c3344)
I'm sure this is some horrible layering violation, but it's really handy.
On Dec 29, 2022, at 3:43 AM, Kunal Mehta legoktm@debian.org wrote:
Hi,
On 12/28/22 10:31, Roy Smith wrote:
How do people install their bots into toolforge? In the past, I've just done a "git pull" to the bastion host and run with the code directly out of the source tree, but I'm not sure that's a good plan.
This is what I do. Each bot/webapp has a requirements.txt with a mix of pinned and unpinned dependencies, I create the virtualenv, install requirements.txt. For webapps, this is automated with the `webservice-python-bootstrap` script[1]. Whenever I git pull, manually check to see if I need to reinstall/update dependencies.
I personally find this to be the simplest way of enabling straightforward local development and minimal work to deploy, usually it's just `git pull && webservice restart`.
The stumbling block with that plan seems to be that toolforge insists on your webapp being installed at $HOME/www/python/src/app.py, which isn't going to play well with a wheel installation.
Right. From a Toolforge perspective, if there's some other, better way to do it, filing a bug would be a good first step to figuring out how to add that. T229172[2] contains some discussion of PEP 517 which ACN mentioned, but doesn't discuss the requirement of app.py.
[1] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Python#Creating_a_vir... [2] https://phabricator.wikimedia.org/T229172
-- Kunal / Legoktm _______________________________________________ pywikibot mailing list -- pywikibot@lists.wikimedia.org To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org