The Toolforge admins would like to invite all Toolforge Kubernetes
users to begin migration of their tools to the 2020 Kubernetes
cluster. Instructions for migration and other details are on Wikitech
[0].
Timeline:
* 2020-01-09: 2020 Kubernetes cluster available for beta testers on an
opt-in basis
* 2020-01-24: 2020 Kubernetes cluster general availability for
migration on an opt-in basis
* 2020-02-10: Automatic migration of remaining workloads from 2016
cluster to 2020 cluster by Toolforge admins
We announced beta testing for this new cluster on 2020-01-09 [1].
Since then more than 70 tools have migrated, with approximately 110
tools now using it [2]. The Toolforge admins have also fixed a few
small issues that our early testers noticed. We are now ready and
excited to have many more tools move their workloads from the legacy
Kubernetes cluster over to the new 2020 Kubernetes cluster.
Thanks to Legoktm, Magnus, and others who helped during the beta
testing phase by trying things out and reporting issues that they
found.
For most tools the migration requires a small number of manual steps [0]:
* webservice stop
* kubectl config use-context toolforge
* alias kubectl=/usr/bin/kubectl; echo "alias
kubectl=/usr/bin/kubectl" >> $HOME/.profile
* webservice --backend=kubernetes [TYPE] start
This could also be a good opportunity for tools to upgrade to newer
language runtimes such as php7.3 and python3.7. See the list on
Wikitech [3] for currently available types. When upgrading to a new
runtime, do not forget to rebuild Python virtual environments, NPM
packages, or Composer packages if you are using them as well.
[0]: https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration
[1]: https://lists.wikimedia.org/pipermail/cloud-announce/2020-January/000247.ht…
[2]: https://tools.wmflabs.org/k8s-status/
[3]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Available_con…
Bryan (on behalf of the Toolforge admins and the Cloud Services team)
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
On Thursday, January 23 at 15:00 UTC we'll be temporarily removing multiple
hypervisors from service to replace failed hardware. This process will
require rebooting the following list of virtual machines as we migrate
workloads to different hypervisors.
Unfortunately, we had several hardware failures happen in a short time and
need to take action as soon as possible.
Full list of instances that will be rebooted during Thursday's maintenance
(shown as <project>: <instance name>):
antiharassment: antiharassment-web1
automation-framework: af-puppetmaster02
bastion: bastion-restricted-eqiad1-01
cloudinfra: cloud-cumin-01
cloudinfra: mx-out01
cyberbot: cyberbot-exec-01
deployment-prep: deployment-cache-upload05
deployment-prep: deployment-elastic05
deployment-prep: deployment-hadoop-test-1
deployment-prep: deployment-kafka-main-2
deployment-prep: deployment-mx02
deployment-prep: deployment-webperf11
extdist: extdist-04
fastcci: fastcci-worker2
hashtags: hashtags-prod
huggle: huggle-wl
iiab: medbox3-iiab
language: language-eg
language: language-mleb-master
library-upgrader: upgrader-05
lta-tracker: tracker1
mobile: apps-talk-pages
mobile: apps-team-tools
monitoring: thanos-be01
monitoring: thanos-prom01
mwoffliner: mwoffliner2
mwoffliner: mwoffliner3
mwstake: mwstake
ores: ores-web-06
packaging: builder01
petscan: petscan3
petscan: petscan4
phabricator: phab-tin
pluggableauth: pluggableauth-server
quarry: quarry-web-01
quarry: quarry-worker-02
reading-web-staging: readingwebstaging
search: wdsearch2
traffic: diffscan
utrs: utrs-database2
wcdo: wcdo
wikibase-registry: wbregistry-01
wikilabels: wikilabels-backups-01
wikitextexp: parsing-qa-01
wm-bot: wm-bot
- WMCS Team
Together with the last Python 2 release from April, 2020, Pywikibot team
will release the **last version that supports Python 2**. We created a
**python2" tag** marking the version, so you can continue running your
Python 2 scripts using this tag, if you really need to.
After that version, Pywikibot is not going to receive any further patches
and bug fixes related to Python 2. Its code is going to be cleaned from
Python 2 specific functions, patches, deprecations and other stuff, so make
sure you'll use this tag if you still want to run Pywikibot using Python 2.
Pywikibot team strongly recommends to migrate your scripts to Python 3. To
make it happen, you can use Python 2to3 script installed by default with
Python 2.6+, see https://docs.python.org/2/library/2to3.html. You can also
just try to run your script using Python 3 (the "-simulate" parameter could
be handy) and fix all the issues. If you encounter problems with the
migration, you can always ask us here:
https://phabricator.wikimedia.org/T242120
Best regards,
Martin Urbanec and Dvorapa
We'll be upgrading the cloud services OpenStack install next Tuesday,
beginning at 12:00 noon UTC
The entire upgrade process may take an hour or two. Early on in the
process, Horizon (and associated OpenStack APIs) will be disabled
(probably for 20 to 30 minutes.) There may also be brief network
interruptions during the upgrade.
Toolforge and existing VMs should be largely unaffected apart from
possible network hiccups.
- Andrew + the WMCS team
I am happy to announce that a new and improved Kubernetes cluster is
now available for use by beta testers on an opt-in basis. A page has
been created on Wikitech [0] outlining the self-service migration
process.
Timeline:
* 2020-01-09: 2020 Kubernetes cluster available for beta testers on an
opt-in basis
* 2020-01-23: 2020 Kubernetes cluster general availability for
migration on an opt-in basis
* 2020-02-10: Automatic migration of remaining workloads from 2016
cluster to 2020 cluster by Toolforge admins
This new cluster has been a work in progress for more than a year
within the Wikimedia Cloud Services team, and a top priority project
for the past six months. About 35 tools, including
https://tools.wmflabs.org/admin/, are currently running on what we are
calling the "2020 Kubernetes cluster". This new cluster is running
Kubernetes v1.15.6 and Docker 19.03.4. It is also using a newer
authentication and authorization method (RBAC), a new ingress routing
service, and a different method of integrating with the Developer
account LDAP service. We have built a new tool [1] which makes the
state of the Kubernetes cluster more transparent and on par with the
information that we already expose for the grid engine cluster [2].
With a significant number of tools managed by Toolforge administrators
already migrated to the new cluster, we are fairly confident that the
basic features used by most Kubernetes tools are covered. It is likely
that a few outlying issues remain to be found as more tools move, but
we have confidence that we can address them quickly. This has led us
to propose a fairly short period of voluntary beta testing, followed
by a short general availability opt-in migration period, and finally a
complete migration of all remaining tools which will be done by the
Toolforge administration team for anyone who has not migrated their
self.
Please help with beta testing if you have some time and are willing to
get help on irc, Phabricator, and the cloud(a)lists.wikimedia.org
mailing list for early adopter issues you may encounter.
I want to publicly praise Brooke Storm and Arturo Borrero González for
the hours that they have put into reading docs, building proof of
concept clusters, and improving automation and processes to make the
2020 Kubernetes cluster possible. The Toolforge community can look
forward to more frequent and less disruptive software upgrades in this
cluster as a direct result of this work. We have some other feature
improvements in planning now that I think you will all be excited to
see and use later this year!
[0]: https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration
[1]: https://tools.wmflabs.org/k8s-status/
[2]: https://tools.wmflabs.org/sge-status/
Bryan (on behalf of the Toolforge admins and the Cloud Services team)
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808