On Oct 7, 2020, at 4:04 PM, Bryan Davis bd808@wikimedia.org wrote:
On Wed, Oct 7, 2020 at 11:54 AM Roy Smith roy@panix.com wrote:
It would be really good if the bastions could be updated to something more modern, so we could have a uniform python infrastructure everywhere. I think I once opened a phab ticket for this, which was closed as some variation on infeasible.
I ran into exactly the issue described below myself. After trying a bunch of various workarounds, I ended up building my own python 3.7 from source. That's suboptimal in so many ways, but it was the only way I could find to get a consistent setup between my test/dev environment on the bastion and my production environment on kubernetes.
Use a `webservice python3.7 shell` session as your dev/test environment and you will a) get the same python version as the "production" container, and b) move your dev/test workload off of the limited resources of the bastion server and onto the more scalable Kubernetes cluster.
BTDT. It was unusable. People can search the list archives if they're interested in details. At this point, I can't even get in:
WARNING: No explict backend provided. Using default of 'kubernetes' For help refer to https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web runtime: failed to create new OS thread (have 3 already; errno=11) runtime: may need to increase max user processes (ulimit -u) fatal error: newosproc
I understand that k8s is scalable, and the way of the future, and that upgrades need to be prioritized, etc. But the current setup is not an effective development environment.
At some point, this is going to become a more acute problem, since 3.5 is officially end-of-life.
The bastions in Toolforge need to be compatible with the Grid Engine cluster because they act as job submission hosts for the grid. Today, the grid engine cluster is running Debian Stretch as its base operating system [0]. Debian Stretch is a supported release through June 2022. There is currently no scheduled work to rebuild and replace the Debian Stretch instances in Toolforge, but rest assured that this work will happen before the end of life of Debian Stretch. I expect that the Toolforge admin team will start discussing the work needed to rebuild the Toolforge bastions and grid engine instances sometime after the Debian project releases their next stable version, Bullseye [1].
Bryan
Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud