[Engineering] [Ops] The train will resume tomorrow (was Re: All wikis reverted to wmf.8 last night due to T119736)

[Matthew Flaschen]

On 07/12/2016 08:15 PM, aude wrote:
> This (unbreak now) bug has been open since November.  I wonder how this
> has been allowed to remain open and not addressed for this long?

This has not all been caused by Echo, and it really isn't one bug, just 
one symptom.

There are clearly multiple causes.  The Echo one has been addressed, and 
there are multiple fixes and mitigation on the CentralAuth/core auth 
side, some merged (e.g. https://gerrit.wikimedia.org/r/#/c/298531/ , 
https://gerrit.wikimedia.org/r/#/c/298416/ ), some still being worked 
on/discussed ( https://gerrit.wikimedia.org/r/#/c/297946/ , 
https://gerrit.wikimedia.org/r/#/c/297936/ ), but work is not done.

Matt