I've found it very useful with php tools to use the instructions here:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#PHP
where it says "You may run the container on your *local* computer (not on Toolforge servers) by executing a command like this: ..."
which allows testing changes locally in a nice way before deploying them.
However there doesn't seem to be anything equivalent for the python images - I can see what's available here:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Python_(uWSGI)
and I can see I should use the image docker-registry.tools.wmflabs.org/toolforge-python37-web (by the way, docker-registry.toolforge.org does NOT work right now when fetching images!)
but I can't figure out how to tell the image where to find the python app in the docker start command. Any hints out there?
Arthur
On Tue, Jun 2, 2020 at 7:28 AM Arthur Smith arthurpsmith@gmail.com wrote:
and I can see I should use the image docker-registry.tools.wmflabs.org/toolforge-python37-web (by the way, docker-registry.toolforge.org does NOT work right now when fetching images!)
docker-registry.toolforge.org is a tool for browsing the registry, not a registry itself. The registry is found at https://docker-registry.tools.wmflabs.org as shown in the image name that you referenced.
but I can't figure out how to tell the image where to find the python app in the docker start command. Any hints out there?
These containers do not really work the way that a typical Docker image would. The `webservice` command manipulates them at runtime in various ways for boring legacy compatibility reasons. For a Python runtime, the `webservice-runner` component which is started inside the container runs /usr/bin/uwsgi with a lot of configuration [0].
You could figure out something that works for you to run the uwsgi container in a different manner. The community contributed documentation that you have found for running lighttpd really has no relation to how the PHP containers are actually used by `webservice`.
[0]: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/tools-web...
Bryan
Depending on which web framework you’re using, there might be better options than exactly reproducing the Toolforge environment. (Unless you’re debugging issues specific to that environment, I suppose.) For instance, Flask https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool has a development mode which automatically reloads source files as soon as they change on disk, and offers you an in-browser debugger when an error occurs. It’s amazing, I can’t recommend it enough.
Cheers, Lucas
Am Di., 2. Juni 2020 um 15:28 Uhr schrieb Arthur Smith < arthurpsmith@gmail.com>:
I've found it very useful with php tools to use the instructions here:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#PHP
where it says "You may run the container on your *local* computer (not on Toolforge servers) by executing a command like this: ..."
which allows testing changes locally in a nice way before deploying them.
However there doesn't seem to be anything equivalent for the python images
- I can see what's available here:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Python_(uWSGI)
and I can see I should use the image docker-registry.tools.wmflabs.org/toolforge-python37-web (by the way, docker-registry.toolforge.org does NOT work right now when fetching images!)
but I can't figure out how to tell the image where to find the python app in the docker start command. Any hints out there?
Arthur _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Depending on which web framework you’re using, there might be better options than exactly reproducing the Toolforge environment. (Unless you’re debugging issues specific to that environment, I suppose.) For instance, Flask https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool has a development mode which automatically reloads source files as soon as they change on disk, and offers you an in-browser debugger when an error occurs. It’s amazing, I can’t recommend it enough.
Right, I've done that, but I've found it's helpful to try to as accurately as possible reproduce the production environment in development if possible - for example in the php case there was a tricky problem I ran into at one point related to how the server invoked subprocesses. Anyway, thanks for the pointers, the uwsgi startup mentioned looks like it might work for me!
Arthur
On Tue, Jun 2, 2020 at 1:27 PM Lucas Werkmeister < lucas.werkmeister@wikimedia.de> wrote:
Depending on which web framework you’re using, there might be better options than exactly reproducing the Toolforge environment. (Unless you’re debugging issues specific to that environment, I suppose.) For instance, Flask https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool has a development mode which automatically reloads source files as soon as they change on disk, and offers you an in-browser debugger when an error occurs. It’s amazing, I can’t recommend it enough.
Cheers, Lucas
Am Di., 2. Juni 2020 um 15:28 Uhr schrieb Arthur Smith < arthurpsmith@gmail.com>:
I've found it very useful with php tools to use the instructions here:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#PHP
where it says "You may run the container on your *local* computer (not on Toolforge servers) by executing a command like this: ..."
which allows testing changes locally in a nice way before deploying them.
However there doesn't seem to be anything equivalent for the python images - I can see what's available here:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Python_(uWSGI)
and I can see I should use the image docker-registry.tools.wmflabs.org/toolforge-python37-web (by the way, docker-registry.toolforge.org does NOT work right now when fetching images!)
but I can't figure out how to tell the image where to find the python app in the docker start command. Any hints out there?
Arthur _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
-- Lucas Werkmeister (he/er) Full Stack Developer
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0 https://wikimedia.de
Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Just to follow up, I did get this working. The main thing seems to be setting up a uwsgi.ini file that works, then the docker run command just needs to call something like "sh -c "/usr/bin/uwsgi --ini <uwsgi.ini file>"; setting up the virtualenv is also needed as a preceding step, which can be done with similar docker run command(s) to install and run pip etc. Do you think it's worth adding this to the wikitech docs?
Arthur
On Tue, Jun 2, 2020 at 2:36 PM Arthur Smith arthurpsmith@gmail.com wrote:
Depending on which web framework you’re using, there might be better
options than exactly reproducing the Toolforge environment. (Unless you’re debugging issues specific to that environment, I suppose.) For instance, Flask https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool has a development mode which automatically reloads source files as soon as they change on disk, and offers you an in-browser debugger when an error occurs. It’s amazing, I can’t recommend it enough.
Right, I've done that, but I've found it's helpful to try to as accurately as possible reproduce the production environment in development if possible
- for example in the php case there was a tricky problem I ran into at one
point related to how the server invoked subprocesses. Anyway, thanks for the pointers, the uwsgi startup mentioned looks like it might work for me!
Arthur
On Tue, Jun 2, 2020 at 1:27 PM Lucas Werkmeister < lucas.werkmeister@wikimedia.de> wrote:
Depending on which web framework you’re using, there might be better options than exactly reproducing the Toolforge environment. (Unless you’re debugging issues specific to that environment, I suppose.) For instance, Flask https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool has a development mode which automatically reloads source files as soon as they change on disk, and offers you an in-browser debugger when an error occurs. It’s amazing, I can’t recommend it enough.
Cheers, Lucas
Am Di., 2. Juni 2020 um 15:28 Uhr schrieb Arthur Smith < arthurpsmith@gmail.com>:
I've found it very useful with php tools to use the instructions here:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#PHP
where it says "You may run the container on your *local* computer (not on Toolforge servers) by executing a command like this: ..."
which allows testing changes locally in a nice way before deploying them.
However there doesn't seem to be anything equivalent for the python images - I can see what's available here:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Python_(uWSGI)
and I can see I should use the image docker-registry.tools.wmflabs.org/toolforge-python37-web (by the way, docker-registry.toolforge.org does NOT work right now when fetching images!)
but I can't figure out how to tell the image where to find the python app in the docker start command. Any hints out there?
Arthur _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
-- Lucas Werkmeister (he/er) Full Stack Developer
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0 https://wikimedia.de
Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
On Tue, Jun 2, 2020 at 2:36 PM Arthur Smith arthurpsmith@gmail.com wrote:
Just to follow up, I did get this working. The main thing seems to be setting up a uwsgi.ini file that works, then the docker run command just needs to call something like "sh -c "/usr/bin/uwsgi --ini <uwsgi.ini file>"; setting up the virtualenv is also needed as a preceding step, which can be done with similar docker run command(s) to install and run pip etc. Do you think it's worth adding this to the wikitech docs?
I think it would be fine to document as a workflow you have used, but I think that it should be in a separate article from https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes rather than inline there. My reasoning for that is that this is a) not really a matching process to the actual Toolforge runtime and b) most users will not want or need to use the process you have found for testing.
We have a lot more cleanup work to do, but Sarah R and others have been doing a lot of work to try and streamline the main Help pages to make learning about the core workflows in Toolforge easier. We really do not have a strong set of editorial policies yet for how to organize these sorts of useful but less mainstream tips. That is something that we need to work on and would be happy to hear ideas about. Luckily wiki's are pretty good at dealing with splitting and merging content over time, so we do not have to get things perfect the first time. :)
Bryan