On 16/09/16 15:29, Antoine Musso wrote:
I have pushed 1.28.0-wmf.19 on thursday at 19:10 UTC. It quickly got noticed that Wikidata was no more able to dispatch updates to the wikis which is tracked in:
https://phabricator.wikimedia.org/T145819
This Friday, I was busy debugging the issue and did not notice a task about account creation being broken: https://phabricator.wikimedia.org/T145839
As soon as I seen that, I have reverted to 1.28.0-wmf.18 at 12:50 UTC.
Account creation has been impossible from Thursday 19:10 UTC until Friday 12:50 UTC. Please pass the word around as needed.
My deep apologizes for not having found out earlier the impact was so important.
I will write an incident report this afternoon with actionables.
A preliminary incident report that has not been peer reviewed is available at:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20160915-MediaWik...
For the time being, all wikis are staying on 1.28.0-wmf.18.
We will discuss the best course of action on Monday. Most probably get a proper fix then push wmf.19 and shift wmf.20 to later.