- Cloud-announce - lists.wikimedia.org

Keystone/Horizon sessions will be reset on Tuesday
by Andrew Bogott 16 Feb '20

16 Feb '20

On Tuesday morning I'm going to switch the OpenStack Keystone token engine from UUID tokens to fernet tokens[0]. The changeover[1] will be abrupt and cause all existing sessions to reset (e.g. if you're using Horizon you'll have to log back in, and if you're in the middle of creating a VM that creation will probably fail.) Should you encounter this interruption, just try again in a few minutes and everything should be fine. The new tokens will generally behave the same from a user standpoint, but will allow us to simplify and modernize things on our end a bit. The switch will happen at 15:00 UTC on this Tuesday, 2020-02-18. That's 7AM, Pacific Coast time. [0] https://docs.openstack.org/keystone/pike/admin/identity-fernet-token-faq.ht… [1] https://phabricator.wikimedia.org/T243418

1 0

VM downtimes later today
by Andrew Bogott 13 Feb '20

13 Feb '20

In order to repair some ailing hardware, I'm going to migrate several cloud-vps instances later today. Each will be down for a few minutes (or longer, depending on disk size) and then rebooted. Toolforge users will be unaffected by this change. Affected VMs will be: vanadium cloudstore-client-2 cloudstore-client-1 cn-staging-2 wikispore-test wm-bot traffic-cache-atstext-buster etherpad traffic-cache-atsupload jbond-buster commons-corruption-checker-main codesearch6 canary1022-01 ores-web-06 cloud-cumin-01 petscan4 thanos-prom01 thanos-be01 extdist-04

1 1

(very) Brief WMCS + wikitech outage Monday 2020-02-10, 15:00 UTC
by Andrew Bogott 04 Feb '20

04 Feb '20

On Monday we'll be restarting the database server that supports most WMCS services. During the restart various things will fail: Wikitech pages will fail to load, OpenStack API calls will fail, etc. In all cases if you encounter an issue you can just count to 20 and try again, by which time things will most likely be back up. Active VMs, tools and other things hosted on toolforge or cloud-VPS should be unaffected. The restart will happen at 15:00 UTC on Monday -- that's 7AM Pacific Time.

1 0

[Toolforge] 2020 Kubernetes cluster open for general use
by Bryan Davis 24 Jan '20

24 Jan '20

The Toolforge admins would like to invite all Toolforge Kubernetes users to begin migration of their tools to the 2020 Kubernetes cluster. Instructions for migration and other details are on Wikitech [0]. Timeline: * 2020-01-09: 2020 Kubernetes cluster available for beta testers on an opt-in basis * 2020-01-24: 2020 Kubernetes cluster general availability for migration on an opt-in basis * 2020-02-10: Automatic migration of remaining workloads from 2016 cluster to 2020 cluster by Toolforge admins We announced beta testing for this new cluster on 2020-01-09 [1]. Since then more than 70 tools have migrated, with approximately 110 tools now using it [2]. The Toolforge admins have also fixed a few small issues that our early testers noticed. We are now ready and excited to have many more tools move their workloads from the legacy Kubernetes cluster over to the new 2020 Kubernetes cluster. Thanks to Legoktm, Magnus, and others who helped during the beta testing phase by trying things out and reporting issues that they found. For most tools the migration requires a small number of manual steps [0]: * webservice stop * kubectl config use-context toolforge * alias kubectl=/usr/bin/kubectl; echo "alias kubectl=/usr/bin/kubectl" >> $HOME/.profile * webservice --backend=kubernetes [TYPE] start This could also be a good opportunity for tools to upgrade to newer language runtimes such as php7.3 and python3.7. See the list on Wikitech [3] for currently available types. When upgrading to a new runtime, do not forget to rebuild Python virtual environments, NPM packages, or Composer packages if you are using them as well. [0]: https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration [1]: https://lists.wikimedia.org/pipermail/cloud-announce/2020-January/000247.ht… [2]: https://tools.wmflabs.org/k8s-status/ [3]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Available_con… Bryan (on behalf of the Toolforge admins and the Cloud Services team) -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

Instance downtime on January 23 15:00 UTC
by Jason Hedden 21 Jan '20

21 Jan '20

On Thursday, January 23 at 15:00 UTC we'll be temporarily removing multiple hypervisors from service to replace failed hardware. This process will require rebooting the following list of virtual machines as we migrate workloads to different hypervisors. Unfortunately, we had several hardware failures happen in a short time and need to take action as soon as possible. Full list of instances that will be rebooted during Thursday's maintenance (shown as <project>: <instance name>): antiharassment: antiharassment-web1 automation-framework: af-puppetmaster02 bastion: bastion-restricted-eqiad1-01 cloudinfra: cloud-cumin-01 cloudinfra: mx-out01 cyberbot: cyberbot-exec-01 deployment-prep: deployment-cache-upload05 deployment-prep: deployment-elastic05 deployment-prep: deployment-hadoop-test-1 deployment-prep: deployment-kafka-main-2 deployment-prep: deployment-mx02 deployment-prep: deployment-webperf11 extdist: extdist-04 fastcci: fastcci-worker2 hashtags: hashtags-prod huggle: huggle-wl iiab: medbox3-iiab language: language-eg language: language-mleb-master library-upgrader: upgrader-05 lta-tracker: tracker1 mobile: apps-talk-pages mobile: apps-team-tools monitoring: thanos-be01 monitoring: thanos-prom01 mwoffliner: mwoffliner2 mwoffliner: mwoffliner3 mwstake: mwstake ores: ores-web-06 packaging: builder01 petscan: petscan3 petscan: petscan4 phabricator: phab-tin pluggableauth: pluggableauth-server quarry: quarry-web-01 quarry: quarry-worker-02 reading-web-staging: readingwebstaging search: wdsearch2 traffic: diffscan utrs: utrs-database2 wcdo: wcdo wikibase-registry: wbregistry-01 wikilabels: wikilabels-backups-01 wikitextexp: parsing-qa-01 wm-bot: wm-bot - WMCS Team

1 0

Pywikibot is ending Python 2 support!
by Martin Urbanec 15 Jan '20

15 Jan '20

Together with the last Python 2 release from April, 2020, Pywikibot team will release the **last version that supports Python 2**. We created a **python2" tag** marking the version, so you can continue running your Python 2 scripts using this tag, if you really need to. After that version, Pywikibot is not going to receive any further patches and bug fixes related to Python 2. Its code is going to be cleaned from Python 2 specific functions, patches, deprecations and other stuff, so make sure you'll use this tag if you still want to run Pywikibot using Python 2. Pywikibot team strongly recommends to migrate your scripts to Python 3. To make it happen, you can use Python 2to3 script installed by default with Python 2.6+, see https://docs.python.org/2/library/2to3.html. You can also just try to run your script using Python 3 (the "-simulate" parameter could be handy) and fix all the issues. If you encounter problems with the migration, you can always ask us here: https://phabricator.wikimedia.org/T242120 Best regards, Martin Urbanec and Dvorapa

1 0

cloud-vps maintenance Tuesday, 2019-01-14
by Andrew Bogott 14 Jan '20

14 Jan '20

We'll be upgrading the cloud services OpenStack install next Tuesday, beginning at 12:00 noon UTC The entire upgrade process may take an hour or two. Early on in the process, Horizon (and associated OpenStack APIs) will be disabled (probably for 20 to 30 minutes.) There may also be brief network interruptions during the upgrade. Toolforge and existing VMs should be largely unaffected apart from possible network hiccups. - Andrew + the WMCS team

2 1

[Toolforge] New Kubernetes cluster open for beta testers
by Bryan Davis 10 Jan '20

10 Jan '20

I am happy to announce that a new and improved Kubernetes cluster is now available for use by beta testers on an opt-in basis. A page has been created on Wikitech [0] outlining the self-service migration process. Timeline: * 2020-01-09: 2020 Kubernetes cluster available for beta testers on an opt-in basis * 2020-01-23: 2020 Kubernetes cluster general availability for migration on an opt-in basis * 2020-02-10: Automatic migration of remaining workloads from 2016 cluster to 2020 cluster by Toolforge admins This new cluster has been a work in progress for more than a year within the Wikimedia Cloud Services team, and a top priority project for the past six months. About 35 tools, including https://tools.wmflabs.org/admin/, are currently running on what we are calling the "2020 Kubernetes cluster". This new cluster is running Kubernetes v1.15.6 and Docker 19.03.4. It is also using a newer authentication and authorization method (RBAC), a new ingress routing service, and a different method of integrating with the Developer account LDAP service. We have built a new tool [1] which makes the state of the Kubernetes cluster more transparent and on par with the information that we already expose for the grid engine cluster [2]. With a significant number of tools managed by Toolforge administrators already migrated to the new cluster, we are fairly confident that the basic features used by most Kubernetes tools are covered. It is likely that a few outlying issues remain to be found as more tools move, but we have confidence that we can address them quickly. This has led us to propose a fairly short period of voluntary beta testing, followed by a short general availability opt-in migration period, and finally a complete migration of all remaining tools which will be done by the Toolforge administration team for anyone who has not migrated their self. Please help with beta testing if you have some time and are willing to get help on irc, Phabricator, and the cloud(a)lists.wikimedia.org mailing list for early adopter issues you may encounter. I want to publicly praise Brooke Storm and Arturo Borrero González for the hours that they have put into reading docs, building proof of concept clusters, and improving automation and processes to make the 2020 Kubernetes cluster possible. The Toolforge community can look forward to more frequent and less disruptive software upgrades in this cluster as a direct result of this work. We have some other feature improvements in planning now that I think you will all be excited to see and use later this year! [0]: https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration [1]: https://tools.wmflabs.org/k8s-status/ [2]: https://tools.wmflabs.org/sge-status/ Bryan (on behalf of the Toolforge admins and the Cloud Services team) -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

2 1

unexpected VM reboots
by Andrew Bogott 27 Dec '19

27 Dec '19

1 0

EXTENDED cloud-vps maintenance Thursday, 2019-12-12
by Andrew Bogott 12 Dec '19

12 Dec '19

Most of the upgrade is complete; however the upgrade has caused an unexpected partial network interruption. Work on this issue is ongoing, I'll send another email when the issue is resolved. -Andrew We'll be upgrading the cloud services OpenStack install on Monday, beginning at 06:00 UTC The entire upgrade process may take an hour or two. Early on in the process, Horizon (and associated OpenStack APIs) will be disabled (probably for 20 to 30 minutes.) There may also be brief network interruptions during the upgrade. Toolforge and existing VMs should be largely unaffected apart from possible network hiccups. This is the second upgrade since October; we are trying to speed up our upgrade schedule in order to catch up with the upstream releases. With luck there will be further such upgrades in the coming months. - Andrew + the WMCS team

2 2

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce