Hello,
I just received a dozen emails about grid engine migration. I tried to migrate my personal tool (tool.martin-urbanec) first. This tool currently generates a Jupyter-notebook based report daily.
I do that by calling jupyter nbconvert --to html --execute community_configuration_usage.ipynb from a virtual environment where Jupyter is installed, together with a couple of other Python modules.
I managed to create new virtual environment that works from the new Buster bastion, and it works when executed directly from the bastion, but I can't get it to execute via the k8s-based engine:
tools.martin-urbanec@tools-sgebastion-10 ~ $ cat ~/public_html/growth-reports/community_configuration_usage/update_report_buster.sh #!/bin/bash
source ~/jupyter-buster/bin/activate cd ~/public_html/growth-reports/community_configuration_usage
jupyter nbconvert --to html --execute community_configuration_usage.ipynb tools.martin-urbanec@tools-sgebastion-10 ~ $ toolforge-jobs run k8s-community-configuration-update-report --image 'tf-python37-DEPRECATED' --command 'bash ~/public_html/growth-reports/community_configuration_usage/update_report_buster.sh' --mem '1Gi' --wait [toolforge-jobs] ERROR: job 'k8s-community-configuration-update-report' failed: +------------+-----------------------------------------------------------------------------------------+ | Job name: | k8s-community-configuration-update-report | +------------+-----------------------------------------------------------------------------------------+ | Command: | bash ~/public_html/growth-reports/community_configuration_usage/update_report_buster.sh | +------------+-----------------------------------------------------------------------------------------+ | Job type: | normal | +------------+-----------------------------------------------------------------------------------------+ | Container: | tf-python37-DEPRECATED | +------------+-----------------------------------------------------------------------------------------+ | File log: | yes | +------------+-----------------------------------------------------------------------------------------+ | Emails: | none | +------------+-----------------------------------------------------------------------------------------+ | Resources: | mem: 1Gi, cpu: default | +------------+-----------------------------------------------------------------------------------------+ | Status: | Failed | +------------+-----------------------------------------------------------------------------------------+ | Hints: | Last run at 2022-04-02T12:43:04Z. Pod in 'Pending' phase. State | | | 'waiting'. Reason 'ContainerCreating'. | +------------+-----------------------------------------------------------------------------------------+ tools.martin-urbanec@tools-sgebastion-10 ~ $ cat k8s-community-configuration-update-report.err /data/project/martin-urbanec/public_html/growth-reports/community_configuration_usage/update_report_buster.sh: line 6: jupyter: command not found /data/project/martin-urbanec/public_html/growth-reports/community_configuration_usage/update_report_buster.sh: line 6: jupyter: command not found tools.martin-urbanec@tools-sgebastion-10 ~ $
I tried to execute the job using other images (incl. the non-deprecated tf-python39), but the behavior was the same every time. I think this is happening, because I need to create the virtual environment from within the container, rather than via the bastion, because of differences between environment available in the container and at the bastion. I don't know how to do that. With websevices, I would do a webservice python3.9 shell and create the environment from there, but I can't find a comparable solution with toolforge-jobs. How can I fix this issue?
A related issue is that previously, the HTML version of the notebook looked like this:
[image: image.png]
While after I ran the update_report_buster.sh script, it looked like this:
[image: image.png]
Due to a different HTML structure, the "Show code" / "Hide code" button stopped working :/.
Thanks for any help,
Martin Urbanec
