Hi all,
When I run python3 --version on Toolforge I see version 3.5.3 is installed. Because python 3.5 reached the end of its life in September 2020, pip is really unhappy about that.
Is there a way to use a later version of python3 on Toolforge? If not, are there plans to upgrade the OS and upgrade python with it?
PS: I know from Help:Toolforge/Python https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python that 3.7.3 is available on Kubernetes, but I am dealing with scripts that are submitted via jsub and not k8s, and the overhead of converting them is prohibitive.
Thanks!
The grid and bastions only support Python 3.5; that won't change until they get an OS upgrade (https://phabricator.wikimedia.org/T275864). As you noted, python 3.7 and 3.9 are available on Kubernetes but writing YAML configuration can be complex. There is a project to build a tool similar to jsub for submitting jobs to the Kubernetes cluster. A beta version of that tool is available for testing on the dev-buster bastion, see https://phabricator.wikimedia.org/T285944 for details. In this case, beta means "might not have all the features" and "implementation details might change". Because it's just another way to submit jobs to the existing k8s cluster, it will still be at least as reliable as the grid.
On Sun, Sep 26, 2021 at 9:44 PM Huji Lee huji.huji@gmail.com wrote:
Hi all,
When I run python3 --version on Toolforge I see version 3.5.3 is installed. Because python 3.5 reached the end of its life in September 2020, pip is really unhappy about that.
Is there a way to use a later version of python3 on Toolforge? If not, are there plans to upgrade the OS and upgrade python with it?
PS: I know from Help:Toolforge/Python that 3.7.3 is available on Kubernetes, but I am dealing with scripts that are submitted via jsub and not k8s, and the overhead of converting them is prohibitive.
Thanks! _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
I don't know if it's generally available yet, but I've been using tools-sgebastion-11, which is running Debian Buster with Python 3.7.
On the other hand, I don't actually use the system Python. Quite a while ago, before the Buster machine was available, I built Python 3.7 from source and I've been using that. It sounds scary, but it's not really. Building from source can be a nightmare, but on the Debian boxes it's nothing more than downloading the tarball, running "configure" and "make" and waiting for that to finish. If you've done that kind of thing before, it's a reasonable way to go, but it's not for everybody.
Of course, at this point, even 3.7 is pretty old. At some point, I'll probably break down and build 3.9. I really want the better f-strings that came out in 3.8.
On Sep 26, 2021, at 9:43 PM, Huji Lee huji.huji@gmail.com wrote:
Hi all,
When I run python3 --version on Toolforge I see version 3.5.3 is installed. Because python 3.5 reached the end of its life in September 2020, pip is really unhappy about that.
Is there a way to use a later version of python3 on Toolforge? If not, are there plans to upgrade the OS and upgrade python with it?
PS: I know from Help:Toolforge/Python https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python that 3.7.3 is available on Kubernetes, but I am dealing with scripts that are submitted via jsub and not k8s, and the overhead of converting them is prohibitive.
Thanks! _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
Do you have any links to share about building python from source?
And is it safe to assume that if I build python from source and put it inside of a venv, when I submit to grid it'll use my python for the job?
On Sun, Sep 26, 2021 at 10:12 PM Roy Smith roy@panix.com wrote:
I don't know if it's generally available yet, but I've been using tools-sgebastion-11, which is running Debian Buster with Python 3.7.
On the other hand, I don't actually use the system Python. Quite a while ago, before the Buster machine was available, I built Python 3.7 from source and I've been using that. It sounds scary, but it's not really. Building from source can be a nightmare, but on the Debian boxes it's nothing more than downloading the tarball, running "configure" and "make" and waiting for that to finish. If you've done that kind of thing before, it's a reasonable way to go, but it's not for everybody.
Of course, at this point, even 3.7 is pretty old. At some point, I'll probably break down and build 3.9. I really want the better f-strings that came out in 3.8.
On Sep 26, 2021, at 9:43 PM, Huji Lee huji.huji@gmail.com wrote:
Hi all,
When I run python3 --version on Toolforge I see version 3.5.3 is installed. Because python 3.5 reached the end of its life in September 2020, pip is really unhappy about that.
Is there a way to use a later version of python3 on Toolforge? If not, are there plans to upgrade the OS and upgrade python with it?
PS: I know from Help:Toolforge/Python https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python that 3.7.3 is available on Kubernetes, but I am dealing with scripts that are submitted via jsub and not k8s, and the overhead of converting them is prohibitive.
Thanks! _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
On Sep 28, 2021, at 12:56 PM, Huji Lee huji.huji@gmail.com wrote:
Do you have any links to share about building python from source?
First, download the source tarball. If I were doing this today, I'd use Python 3.9.7, which is at https://www.python.org/downloads/release/python-397/ https://www.python.org/downloads/release/python-397/
Unpack the tarbar and look at the README.rst file. In a nutshell, all you should have to do is:
./configure make make test
and then create a virtualenv using the binaries you just built as a base. If memory serves, the configure step should take a few minutes, and the make is on the order of an hour.
And is it safe to assume that if I build python from source and put it inside of a venv, when I submit to grid it'll use my python for the job?
I don't do much grid work, but, looking at my old jobs, it looks like I was running grid jobs with a venv:
$ cat jobs/get_socks.2019-12-16-01-51-12/job.bash #!/bin/bash
source /home/roysmith/sock-classifier/env/bin/activate
/home/roysmith/sock-classifier/src/utils/get_socks.py \ --archive-dir=/home/roysmith/sock-classifier/data/archives-22618 \ --job-name=get_socks.2019-12-16-01-51-12 \ --log=/home/roysmith/sock-classifier/jobs/get_socks.2019-12-16-01-51-12/get_socks.log
which seemed to work fine. That particular job was done with the stock Python binary, but I can't see how if you built a venv based on a binary your compiled youself it would be any different. If you want to try an experiment, see if you can run /data/project/spi-tools-dev/python-distros/Python-3.7.3-install/bin/python3 (I think the permissions should allow it). And assuming you can, try building a venv against that binary and see if it works for you on the grid.
WARNING: this is just a quick-and-dirty suggestion for testing that it works. I absolutely don't guarantee that I won't blow away that directory tomorrow, so if the experiment works and you want to go this route for your production job, you'll totally want to build your own.
Also, please note, I'm just speaking as a Toolforge user. I can't promise what the folks who run Toolforge think about this. They haven't told me I can't do it, but I don't know if they're totally on board with the idea.
On 9/28/21 8:42 PM, Roy Smith wrote:
Also, please note, I'm just speaking as a Toolforge user. I can't promise what the folks who run Toolforge think about this. They haven't told me I can't do it, but I don't know if they're totally on board with the idea.
I certainly don't like the manual work and duplication between tools involved with it, but at the same time as long as you're compiling it as a grid job (and not on a bastion) I can't quickly think of any ways this would be actively harmful to other tools. (standard disclaimer: I'm not speaking on behalf of other roots here, it's possible that I'm forgetting something.)
Having said that, we're hoping to completely move everything to Kubernetes in the next few years and completely decommission the grid. Among other benefits, this lets us release new containers with newer runtimes much faster than the grid allows, currently we pretty much track newest Debian releases (the grid is oldoldstable at this point) which includes Python 3.9 and others.
Tooling for non-webservices is still lacking, but if you want you can try out a beta version of the new "Toolforge Jobs framework". It still likely has some bugs and missing features, but the basics do usually work. The details are on Phabricator: https://phabricator.wikimedia.org/T285944
Taavi