Hello everyone,

The Wikimedia Cloud Services team has been working for the past year to upgrade the entire Toolforge infrastructure to the Debian Stretch operating system. You may recognize Toolforge from the "tools.wmflabs.org" domain. We were required to move away from the old operating system, Ubuntu Trusty, as the Ubuntu upstream developers set an end-of-life date of 31 March for Trusty.[0] After this point, we cannot use Ubuntu Trusty on our servers, as it would not be considered secure. The deadline was thus a hard deadline.

A bit on how Toolforge works: the Wikimedia projects have a community of volunteer software developers who work on tools that help Wikimedians in their work. You have probably used at least one such tool. They are so important for our productivity, but they are the best efforts of volunteers, and do not have service level agreements[1] or other guarantees. And sometimes, volunteers move on to other projects (they're volunteers after all) and so the tools basically run themselves until they stop for some reason.

Toolforge is designed to encourage as much collaborative maintenance as possible, including by requiring tools to be open source. We also sent ample notifications to tool maintainers via their email addresses on file including instructions on how to upgrade. In the final two weeks, we sent daily notifications.

Despite our best efforts, there were around 385 holdouts when we shut down the Ubuntu Trusty job grid.[2] They have not been deleted, but they are no longer running. This includes tools that are widely used by members of the community. It could also include tools that no one has used in years and no one will miss. I am not sure which is which, but what I do know is that lots of people use lots of tools on lots of wikis.

I need your help. If you see community members on village pumps, social media, etc. asking about tools that have gone down (especially if they involve "tools.wmflabs.org"), please forward them on to me at jhare@wikimedia.org. You are free to respond yourself using information in this email and anything else you know, of course. If it is at all possible, I would like to build a report on how many complaints there are around this. This will help me advocate for the community, because I think it would be really cool if we could have fewer of these jarring transitions in the future.

Please review the list.[2] Some of these might have already been migrated to the new operating system. If you recognize a tool on the list and think people might want to hear about what happened, do feel free to proactively notify people you think should be notified. I'm also happy to write something for Tech News if you think that would be helpful. Unfortunately, the Cloud Services team is not equipped to resurrect individual tools; we cannot guarantee they will work on the new OS due to operating system runtime differences (different versions of Python, etc).

Thank you for your help, if you are available to provide it.

My best regards,

James Hare

[0] So why Debian Stretch and not Ubuntu Xenial? Because the Wikimedia Foundation decided to consolidate around Debian GNU/Linux in 2014 or so, and the Ubuntu Trusty VMs on Cloud Services were the last holdouts.

[1] https://en.wikipedia.org/wiki/Service-level_agreement

[2] https://tools.wmflabs.org/trusty-tools/