Andrew, this was a great post. I already knew much of this and I enjoyed it immensely.

On Mar 9, 2018 5:42 PM, "Andrew" no-reply@phabricator.wikimedia.org wrote:

Andrew published this post.

I've spent the last few months building new web servers to support some of the basic WMCS web services: Wikitech, Horizon, and Toolsadmin. The new Wikitech service is already up and running; on Wednesday I hope to flip the last switch and move all public Horizon and Toolsadmin traffic to the new servers as well.

If everything goes as planned, users will barely notice this change at all.

This is a lot of what our team does -- running as fast as we can just to stay in place. Software doesn't last forever -- it takes a lot of effort just to hold things together. Here are some of the problems that this rebuild is solving:

- T186288 https://phabricator.wikimedia.org/T186288: *Operating System obsolescence*. Years ago, the Wikimedia Foundation Operations team resolved to move all of our infrastructure from Ubuntu to Debian Linux. Ubuntu Trusty will stop receiving security upgrades in about a year, so we have to stop using it by then. All three services (Wikitech, Horizon, Toolsadmin) were running on Ubuntu servers; Wikitech was the last of the Foundation's MediaWiki hosts to run on Ubuntu, so its upgrade should allow for all kinds of special cases to be ignored in the future.

- T98813 https://phabricator.wikimedia.org/T98813: *Keeping up with PHP and HHVM*. In addition to being the last wiki on Trusty, Wikitech was also the last wiki on PHP 5. Every other wiki is using HHVM and, with the death of the old Wikitech, we can finally stop supporting PHP 5 internally. Better yet, this plays a part in unblocking the entire MediaWiki ecosystem (T172165 https://phabricator.wikimedia.org/T172165) as newer versions of MediaWiki standardize on HHVM or PHP 7.

- T168559 https://phabricator.wikimedia.org/T168559: *Escaping failing hardware*. The old Wikitech site was hosted on a machine named 'Silver'. Hardware wears out, and Silver is pretty old. The last few times I've rebooted it, it's required a bit of nudging to bring it back up. If it powered down today, it would probably come back, but it might not. As of today's switchover, that scenario won't result in weeks of Wikitech downtime.

- T169099 https://phabricator.wikimedia.org/T169099: *Tracking OpenStack upgrades*. OpenStack (the software project that includes Horizon and most of our virtual machine infrastructure) releases a new version every six months. Ubuntu packages up every version with all of its dependencies, and provides a clear upgrade path between versions. Debian, for the most part, does not. The new release of Horizon is no longer deployed through an upstream package at all, but instead is a pure Python deploy starting with the raw Horizon source and requirements list, rolled into Wheels and deployed into an isolated virtual environment. It's unclear exactly how we'll transition our other OpenStack components away from Ubuntu, but this Horizon deploy provides a potential model for deploying any OpenStack project, any version, on any OS. Having done this I'm much less worried about our reliance on often-fickle upstream packagers.

- T187506 https://phabricator.wikimedia.org/T187506: *High availability*. The old versions of these web services were hosted on single servers. Any maintenance or hardware downtime meant that the websites were gone for the duration. Now we have a pair of servers with a shared cache, behind a load-balancer. If either of the servers dies (or, more likely, we need to reboot one for kernel updates) the website will remain up and responsive.

Of course, having just moved wikitech to HHVM, the main Wikimedia cluster is being upgraded from HHVM to PHP 7, and Wikitech will soon follow suit. The websites look the same, but the race never ends.

*POST DETAIL* https://phabricator.wikimedia.org/phame/post/view/87/ running_red-queen-style/

*EMAIL PREFERENCES* https://phabricator.wikimedia.org/settings/panel/emailpreferences/

*To: *Andrew, TerraCodes, Dispenser, Se4598, tom29739, zhuyifei1999, Papuass, Sebastian_Berlin-WMSE, Niedzielski, chasemp, MZMcBride, Arlolra, KartikMistry, Quiddity, PeterBowman, fgiunchedi, Mholloway, Bamyers99, MaxSem, Krenair, Giftpflanze, Ladsgroup, bd808, Jay8g, Dalba, madhuvishy, Framawiki, greg, zeljkofilipin, mmodell, yuvipanda *Cc: *Andrew, Quiddity, 1978Gage2001, aborrero, Chicocvenancio, Tbscho, Freddy2001, JJMC89, srodlund, Luke081515, Gryllida, jayvdb, zhuyifei1999, scfc, coren, Jay8g, bd808, Krenair, chasemp