Small status update: All wikis have now been rolled back to 1.35.0-wmf.16 in order to limit the fallout from T244529. There is a tentative plan for unblocking the train but we will revisit that on Monday.
Thanks to everyone who jumped in to help diagnose issues, restore services and especially for those who helped out on incident documentation.
Have a great weekend folks!
On Fri, Feb 7, 2020 at 7:44 AM Mukunda Modell mmodell@wikimedia.org wrote:
When deploying MediaWiki 1.35.0-wmf.18 to Group 2 wikis on Thursday, there was a serious incident which took down all wikis for several minutes and resulted in a rollback to wmf.16.
My current understanding is that attempting to deploy wmf.18 again is likely to cause another severe outage. The train remains blocked until the root cause of the outage is identified.
Investigation is ongoing. Details and further updates will be posted on the status tracking task for wmf.18[1] and some background on the investigation has been documented in the incident report[2].
[1] https://phabricator.wikimedia.org/T233866 [2] https://wikitech.wikimedia.org/wiki/Incident_documentation/20200206-mediawik...