Hello,

I just received a dozen emails about grid engine migration. I tried to migrate my personal tool (tool.martin-urbanec) first. This tool currently generates a Jupyter-notebook based report daily.

I do that by calling jupyter nbconvert --to html --execute community_configuration_usage.ipynb from a virtual environment where Jupyter is installed, together with a couple of other Python modules.

I managed to create new virtual environment that works from the new Buster bastion, and it works when executed directly from the bastion, but I can't get it to execute via the k8s-based engine:

tools.martin-urbanec@tools-sgebastion-10 ~
$ cat ~/public_html/growth-reports/community_configuration_usage/update_report_buster.sh
#!/bin/bash

source ~/jupyter-buster/bin/activate
cd ~/public_html/growth-reports/community_configuration_usage

jupyter nbconvert --to html --execute community_configuration_usage.ipynb

tools.martin-urbanec@tools-sgebastion-10 ~
$ toolforge-jobs run k8s-community-configuration-update-report --image 'tf-python37-DEPRECATED' --command 'bash ~/public_html/growth-reports/community_configuration_usage/update_report_buster.sh' --mem '1Gi' --wait
[toolforge-jobs] ERROR: job 'k8s-community-configuration-update-report' failed:
+------------+-----------------------------------------------------------------------------------------+
| Job name: | k8s-community-configuration-update-report |
+------------+-----------------------------------------------------------------------------------------+
| Command: | bash ~/public_html/growth-reports/community_configuration_usage/update_report_buster.sh |
+------------+-----------------------------------------------------------------------------------------+
| Job type: | normal |
+------------+-----------------------------------------------------------------------------------------+
| Container: | tf-python37-DEPRECATED |
+------------+-----------------------------------------------------------------------------------------+
| File log: | yes |
+------------+-----------------------------------------------------------------------------------------+
| Emails: | none |
+------------+-----------------------------------------------------------------------------------------+
| Resources: | mem: 1Gi, cpu: default |
+------------+-----------------------------------------------------------------------------------------+
| Status: | Failed |
+------------+-----------------------------------------------------------------------------------------+
| Hints: | Last run at 2022-04-02T12:43:04Z. Pod in 'Pending' phase. State |
| | 'waiting'. Reason 'ContainerCreating'. |
+------------+-----------------------------------------------------------------------------------------+
tools.martin-urbanec@tools-sgebastion-10 ~
$ cat k8s-community-configuration-update-report.err
/data/project/martin-urbanec/public_html/growth-reports/community_configuration_usage/update_report_buster.sh: line 6: jupyter: command not found
/data/project/martin-urbanec/public_html/growth-reports/community_configuration_usage/update_report_buster.sh: line 6: jupyter: command not found

tools.martin-urbanec@tools-sgebastion-10 ~
$

I tried to execute the job using other images (incl. the non-deprecated tf-python39), but the behavior was the same every time. I think this is happening, because I need to create the virtual environment from within the container, rather than via the bastion, because of differences between environment available in the container and at the bastion. I don't know how to do that. With websevices, I would do a webservice python3.9 shell and create the environment from there, but I can't find a comparable solution with toolforge-jobs. How can I fix this issue?

A related issue is that previously, the HTML version of the notebook looked like this:

While after I ran the update_report_buster.sh script, it looked like this:

Due to a different HTML structure, the "Show code" / "Hide code" button stopped working :/.

Thanks for any help,

Martin Urbanec