Hello,
I just received a dozen emails about grid engine migration. I tried to migrate my personal tool (tool.martin-urbanec) first. This tool currently generates a Jupyter-notebook based report daily.
I do that by calling jupyter nbconvert --to html --execute community_configuration_usage.ipynb from a virtual environment where Jupyter is installed, together with a couple of other Python modules.
I managed to create new virtual environment that works from the new Buster bastion, and it works when executed directly from the bastion, but I can't get it to execute via the k8s-based engine:
tools.martin-urbanec@tools-sgebastion-10 ~ $ cat ~/public_html/growth-reports/community_configuration_usage/update_report_buster.sh #!/bin/bash
source ~/jupyter-buster/bin/activate cd ~/public_html/growth-reports/community_configuration_usage
jupyter nbconvert --to html --execute community_configuration_usage.ipynb tools.martin-urbanec@tools-sgebastion-10 ~ $ toolforge-jobs run k8s-community-configuration-update-report --image 'tf-python37-DEPRECATED' --command 'bash ~/public_html/growth-reports/community_configuration_usage/update_report_buster.sh' --mem '1Gi' --wait [toolforge-jobs] ERROR: job 'k8s-community-configuration-update-report' failed: +------------+-----------------------------------------------------------------------------------------+ | Job name: | k8s-community-configuration-update-report | +------------+-----------------------------------------------------------------------------------------+ | Command: | bash ~/public_html/growth-reports/community_configuration_usage/update_report_buster.sh | +------------+-----------------------------------------------------------------------------------------+ | Job type: | normal | +------------+-----------------------------------------------------------------------------------------+ | Container: | tf-python37-DEPRECATED | +------------+-----------------------------------------------------------------------------------------+ | File log: | yes | +------------+-----------------------------------------------------------------------------------------+ | Emails: | none | +------------+-----------------------------------------------------------------------------------------+ | Resources: | mem: 1Gi, cpu: default | +------------+-----------------------------------------------------------------------------------------+ | Status: | Failed | +------------+-----------------------------------------------------------------------------------------+ | Hints: | Last run at 2022-04-02T12:43:04Z. Pod in 'Pending' phase. State | | | 'waiting'. Reason 'ContainerCreating'. | +------------+-----------------------------------------------------------------------------------------+ tools.martin-urbanec@tools-sgebastion-10 ~ $ cat k8s-community-configuration-update-report.err /data/project/martin-urbanec/public_html/growth-reports/community_configuration_usage/update_report_buster.sh: line 6: jupyter: command not found /data/project/martin-urbanec/public_html/growth-reports/community_configuration_usage/update_report_buster.sh: line 6: jupyter: command not found tools.martin-urbanec@tools-sgebastion-10 ~ $
I tried to execute the job using other images (incl. the non-deprecated tf-python39), but the behavior was the same every time. I think this is happening, because I need to create the virtual environment from within the container, rather than via the bastion, because of differences between environment available in the container and at the bastion. I don't know how to do that. With websevices, I would do a webservice python3.9 shell and create the environment from there, but I can't find a comparable solution with toolforge-jobs. How can I fix this issue?
A related issue is that previously, the HTML version of the notebook looked like this:
[image: image.png]
While after I ran the update_report_buster.sh script, it looked like this:
[image: image.png]
Due to a different HTML structure, the "Show code" / "Hide code" button stopped working :/.
Thanks for any help,
Martin Urbanec
Hi Martin,
I usually write a bash script to set up the venv and run it under toolforge-jobs. I've put a skeleton version at P24028 https://phabricator.wikimedia.org/P24028.
You may be able to use the webservice shell since the images are similar.
Regards, JJ
On Sun, Apr 3, 2022 at 8:14 PM Martin Urbanec martin.urbanec@wikimedia.cz wrote:
I tried to execute the job using other images (incl. the non-deprecated tf-python39), but the behavior was the same every time. I think this is happening, because I need to create the virtual environment from within the container, rather than via the bastion, because of differences between environment available in the container and at the bastion. I don't know how to do that. With websevices, I would do a webservice python3.9 shell and create the environment from there, but I can't find a comparable solution with toolforge-jobs. How can I fix this issue? _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
On 4/2/22 14:53, Martin Urbanec wrote:
Hello,
I just received a dozen emails about grid engine migration. I tried to migrate my personal tool (tool.martin-urbanec) first. This tool currently generates a Jupyter-notebook based report daily.
I do that by calling jupyter nbconvert --to html --execute community_configuration_usage.ipynb from a virtual environment where Jupyter is installed, together with a couple of other Python modules.
I managed to create new virtual environment that works from the new Buster bastion, and it works when executed directly from the bastion, but I can't get it to execute via the k8s-based engine:
Your problem may be related to bootstrapping the venv. See if this information can help you:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python#Kubernetes_python_...
This is very similar to what JMC89 replied in the other email.
Hello everyone,
Thanks for the suggestion. Does this mean that I have to maintain 2 virtual environments (one for bastion and the other one for k8s pod my code will actually run in)?
Martin
po 4. 4. 2022 v 10:06 odesílatel Arturo Borrero Gonzalez < aborrero@wikimedia.org> napsal:
On 4/2/22 14:53, Martin Urbanec wrote:
Hello,
I just received a dozen emails about grid engine migration. I tried to migrate my personal tool (tool.martin-urbanec) first. This tool currently generates a Jupyter-notebook based report daily.
I do that by calling jupyter nbconvert --to html --execute community_configuration_usage.ipynb from a virtual environment where Jupyter is installed, together with a couple of other Python modules.
I managed to create new virtual environment that works from the new Buster bastion, and it works when executed directly from the bastion, but I can't get it to execute via the k8s-based engine:
Your problem may be related to bootstrapping the venv. See if this information can help you:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python#Kubernetes_python_...
This is very similar to what JMC89 replied in the other email.
-- Arturo Borrero Gonzalez Site Reliability Engineer Wikimedia Cloud Services Wikimedia Foundation
You can use the one used for k8s on the bastion too, the thing is that the k8s environment expects it to be in a specific path, that is `$HOME/pyenv/`.
Now, that might not work if the python version in the bastion is too different from the one the k8s job has, so you might have to have two different ones in that case yes (bastions are restricted to the python version of the OS, the k8s jobs can use different ones depending on the image).
On 04/05 11:08, Martin Urbanec wrote:
Hello everyone,
Thanks for the suggestion. Does this mean that I have to maintain 2 virtual environments (one for bastion and the other one for k8s pod my code will actually run in)?
Martin
po 4. 4. 2022 v 10:06 odesílatel Arturo Borrero Gonzalez < aborrero@wikimedia.org> napsal:
On 4/2/22 14:53, Martin Urbanec wrote:
Hello,
I just received a dozen emails about grid engine migration. I tried to migrate my personal tool (tool.martin-urbanec) first. This tool currently generates a Jupyter-notebook based report daily.
I do that by calling jupyter nbconvert --to html --execute community_configuration_usage.ipynb from a virtual environment where Jupyter is installed, together with a couple of other Python modules.
I managed to create new virtual environment that works from the new Buster bastion, and it works when executed directly from the bastion, but I can't get it to execute via the k8s-based engine:
Your problem may be related to bootstrapping the venv. See if this information can help you:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python#Kubernetes_python_...
This is very similar to what JMC89 replied in the other email.
-- Arturo Borrero Gonzalez Site Reliability Engineer Wikimedia Cloud Services Wikimedia Foundation
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
In general, if you are using a Python virtual environment on k8s, you should only modify it from a shell in a matching k8s container. The easiest way to do this is with `webservice --backend=kubernetes python3.9 shell` (or whatever version you are using). If the python version you're using matches what's in use on the Buster bastion, it should work, but using a k8s shell will be the most reliable.
On Tue, Apr 5, 2022 at 5:16 AM David Caro dcaro@wikimedia.org wrote:
You can use the one used for k8s on the bastion too, the thing is that the k8s environment expects it to be in a specific path, that is `$HOME/pyenv/`.
Now, that might not work if the python version in the bastion is too different from the one the k8s job has, so you might have to have two different ones in that case yes (bastions are restricted to the python version of the OS, the k8s jobs can use different ones depending on the image).
On 04/05 11:08, Martin Urbanec wrote:
Hello everyone,
Thanks for the suggestion. Does this mean that I have to maintain 2 virtual environments (one for bastion and the other one for k8s pod my code will actually run in)?
Martin
po 4. 4. 2022 v 10:06 odesílatel Arturo Borrero Gonzalez < aborrero@wikimedia.org> napsal:
On 4/2/22 14:53, Martin Urbanec wrote:
Hello,
I just received a dozen emails about grid engine migration. I tried to migrate my personal tool (tool.martin-urbanec) first. This tool currently generates a Jupyter-notebook based report daily.
I do that by calling jupyter nbconvert --to html --execute community_configuration_usage.ipynb from a virtual environment where Jupyter is installed, together with a couple of other Python modules.
I managed to create new virtual environment that works from the new Buster bastion, and it works when executed directly from the bastion, but I can't get it to execute via the k8s-based engine:
Your problem may be related to bootstrapping the venv. See if this information can help you:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python#Kubernetes_python_...
This is very similar to what JMC89 replied in the other email.
-- Arturo Borrero Gonzalez Site Reliability Engineer Wikimedia Cloud Services Wikimedia Foundation
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
-- David Caro SRE - Cloud Services Wikimedia Foundation https://wikimediafoundation.org/ PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment." _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
I second that. I used to not worry too much about doing it right, but eventually got bitten by my sloppyness. The problem seems to be that the paths which get embedded in your venv executables will be different, which leads to hard-to-debug problems.
On Apr 6, 2022, at 10:12 AM, AntiCompositeNumber anticompositenumber@gmail.com wrote:
In general, if you are using a Python virtual environment on k8s, you should only modify it from a shell in a matching k8s container. The easiest way to do this is with `webservice --backend=kubernetes python3.9 shell` (or whatever version you are using). If the python version you're using matches what's in use on the Buster bastion, it should work, but using a k8s shell will be the most reliable.
On Tue, Apr 5, 2022 at 5:16 AM David Caro dcaro@wikimedia.org wrote:
You can use the one used for k8s on the bastion too, the thing is that the k8s environment expects it to be in a specific path, that is `$HOME/pyenv/`.
Now, that might not work if the python version in the bastion is too different from the one the k8s job has, so you might have to have two different ones in that case yes (bastions are restricted to the python version of the OS, the k8s jobs can use different ones depending on the image).
On 04/05 11:08, Martin Urbanec wrote:
Hello everyone,
Thanks for the suggestion. Does this mean that I have to maintain 2 virtual environments (one for bastion and the other one for k8s pod my code will actually run in)?
Martin
po 4. 4. 2022 v 10:06 odesílatel Arturo Borrero Gonzalez < aborrero@wikimedia.org> napsal:
On 4/2/22 14:53, Martin Urbanec wrote:
Hello,
I just received a dozen emails about grid engine migration. I tried to migrate my personal tool (tool.martin-urbanec) first. This tool currently generates a Jupyter-notebook based report daily.
I do that by calling jupyter nbconvert --to html --execute community_configuration_usage.ipynb from a virtual environment where Jupyter is installed, together with a couple of other Python modules.
I managed to create new virtual environment that works from the new Buster bastion, and it works when executed directly from the bastion, but I can't get it to execute via the k8s-based engine:
Your problem may be related to bootstrapping the venv. See if this information can help you:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python#Kubernetes_python_...
This is very similar to what JMC89 replied in the other email.
-- Arturo Borrero Gonzalez Site Reliability Engineer Wikimedia Cloud Services Wikimedia Foundation
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
-- David Caro SRE - Cloud Services Wikimedia Foundation https://wikimediafoundation.org/ PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment." _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
