Hello,
We are holding the deployment of MediaWiki version 1.28.0-wmf.19 due to a couple of bugs that have surfaced.
The first one is that renaming a user was blocked [T145596]. Reported by K6ka, triaged by MarcoAurelio. The issue is fixed now thanks to Aaron Schulz and Kunal Mehta.
The second blocker is way nastier. I have pushed the upgrade to group1. matanya (a long volunteer with a lot of technical patches) immediately reported the infoboxes on the Hebrew Wiki were on the wrong side, which prompted a rollback. [T145673].
Timo Tijhof and Kunal Mehta have found the root cause. I can speak for them as to who/when we will get a solution.
For now. I am holding the train. Will reassess tomorrow and ideally push group1 at 19:00 UTC then follow with group2 at 20:00UTC.
Stay tuned!
Up-to-date status: https://tools.wmflabs.org/versions/
MW-1.28.0-wmf.19 deployment blockers https://phabricator.wikimedia.org/T143328
[T145596] Renames getting stuck on mediawiki.org (Sept 13, 2016) https://phabricator.wikimedia.org/T145596
[T145673] 1.28.0-wmf.19 broke template RTL placement https://phabricator.wikimedia.org/T145673
On 14/09/16 23:05, Antoine Musso wrote:
<snip> For now. I am holding the train. Will reassess tomorrow and ideally push group1 at 19:00 UTC then follow with group2 at 20:00UTC.
..
Up-to-date status: https://tools.wmflabs.org/versions/
MW-1.28.0-wmf.19 deployment blockers https://phabricator.wikimedia.org/T143328
Hello,
I have pushed 1.28.0-wmf.19 on thursday at 19:10 UTC. It quickly got noticed that Wikidata was no more able to dispatch updates to the wikis which is tracked in:
https://phabricator.wikimedia.org/T145819
This Friday, I was busy debugging the issue and did not notice a task about account creation being broken: https://phabricator.wikimedia.org/T145839
As soon as I seen that, I have reverted to 1.28.0-wmf.18 at 12:50 UTC.
Account creation has been impossible from Thursday 19:10 UTC until Friday 12:50 UTC. Please pass the word around as needed.
My deep apologizes for not having found out earlier the impact was so important.
I will write an incident report this afternoon with actionables.
On 16/09/16 15:29, Antoine Musso wrote:
I have pushed 1.28.0-wmf.19 on thursday at 19:10 UTC. It quickly got noticed that Wikidata was no more able to dispatch updates to the wikis which is tracked in:
https://phabricator.wikimedia.org/T145819
This Friday, I was busy debugging the issue and did not notice a task about account creation being broken: https://phabricator.wikimedia.org/T145839
As soon as I seen that, I have reverted to 1.28.0-wmf.18 at 12:50 UTC.
Account creation has been impossible from Thursday 19:10 UTC until Friday 12:50 UTC. Please pass the word around as needed.
My deep apologizes for not having found out earlier the impact was so important.
I will write an incident report this afternoon with actionables.
A preliminary incident report that has not been peer reviewed is available at:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20160915-MediaWik...
For the time being, all wikis are staying on 1.28.0-wmf.18.
We will discuss the best course of action on Monday. Most probably get a proper fix then push wmf.19 and shift wmf.20 to later.
Thank you for trying to untangle this web. I'm going to comment on the incident report talk page.
Pine
On Fri, Sep 16, 2016 at 7:57 AM, Antoine Musso hashar+wmf@free.fr wrote:
On 16/09/16 15:29, Antoine Musso wrote:
I have pushed 1.28.0-wmf.19 on thursday at 19:10 UTC. It quickly got noticed that Wikidata was no more able to dispatch updates to the wikis which is tracked in:
https://phabricator.wikimedia.org/T145819
This Friday, I was busy debugging the issue and did not notice a task about account creation being broken: https://phabricator.wikimedia.org/T145839
As soon as I seen that, I have reverted to 1.28.0-wmf.18 at 12:50 UTC.
Account creation has been impossible from Thursday 19:10 UTC until Friday 12:50 UTC. Please pass the word around as needed.
My deep apologizes for not having found out earlier the impact was so important.
I will write an incident report this afternoon with actionables.
A preliminary incident report that has not been peer reviewed is available at:
https://wikitech.wikimedia.org/wiki/Incident_documentation/ 20160915-MediaWiki
For the time being, all wikis are staying on 1.28.0-wmf.18.
We will discuss the best course of action on Monday. Most probably get a proper fix then push wmf.19 and shift wmf.20 to later.
-- Antoine "hashar" Musso
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
tl;dr: All wikis are staying at 1.28.0-wmf.18 for now
Last week all wikis were rolled back to MediaWiki version 1.28.0-wmf.18 due to several problems that were spotted on Friday (2016-09-16)[0][1].
The problems with wmf.19 seemed resolved by Monday (2016-09-19). The plan was to roll wmf.19 out to all wikis yesterday afternoon and continue with the wmf.20 branch-cut today as scheduled.
Late yesterday a large performance regression was discovered in wmf.18[2].
We've paused the rollout of wmf.19 and the branching of wmf.20 to allow time to investigate the nature of this performance regression.
After there is a better understanding of the performance regression, we will reevaluate our plans for branching and rollout of wmf.19 and wmf.20.
-- Tyler
[0]. https://phabricator.wikimedia.org/T111441 [1]. https://phabricator.wikimedia.org/T145819 [2]. https://phabricator.wikimedia.org/T146099
What are your plans for when you'll deploy wmf.19, branch wmf.20 and deploy wmf.20?
I ask because I have a patch lined up that I wanted to merge today right after the wmf.20 branch cut, because I want it to spend the full week on beta labs before it goes to production. I can merge that patch today and have it go to beta, and have it go into wmf.20 once it gets cut, but I'd only be willing to do that if I had a guarantee that wmf.20 would not be cut before early next week. Because it's currently unknown when wmf.20 will be cut, I don't know what's safe to do.
Unrelatedly, I also have a patch merged in master that fixes a regression in wmf.19. Since wmf.19 is not deployed right now, the regression doesn't affect production, but if wmf.19 is ever going to be deployed, then the fix needs to be cherry-picked into it. Do you know if wmf.19 is going to be deployed, or is it going to be skipped? Since wmf.19 is not deployed right now, should I submit a cherry-pick to it for SWAT today so that it gets unbroken in the event it ever gets deployed?
I understand that you are investigating and will reevaluate after, but it would be nice to have these questions answered quickly so we have some sense of what is and isn't going to happen, and aren't blocked by this uncertainty for much longer,
On Tue, Sep 20, 2016 at 12:27 PM, Tyler Cipriani tcipriani@wikimedia.org wrote:
tl;dr: All wikis are staying at 1.28.0-wmf.18 for now
Last week all wikis were rolled back to MediaWiki version 1.28.0-wmf.18 due to several problems that were spotted on Friday (2016-09-16)[0][1].
The problems with wmf.19 seemed resolved by Monday (2016-09-19). The plan was to roll wmf.19 out to all wikis yesterday afternoon and continue with the wmf.20 branch-cut today as scheduled.
Late yesterday a large performance regression was discovered in wmf.18[2].
We've paused the rollout of wmf.19 and the branching of wmf.20 to allow time to investigate the nature of this performance regression.
After there is a better understanding of the performance regression, we will reevaluate our plans for branching and rollout of wmf.19 and wmf.20.
-- Tyler
[0]. https://phabricator.wikimedia.org/T111441 [1]. https://phabricator.wikimedia.org/T145819 [2]. https://phabricator.wikimedia.org/T146099
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
I have cut 1.28.0-wmf.20[0] for MediaWiki and extensions as the branch cut was blocking merges to master for developers.
The state of deployed code has not changed – all wikis are running the 1.28.0-wmf.18 branch of MediaWiki and extensions.
Plans for moving forward are still being discussed on Phabricator[1].
-- Tyler
[0]. https://phabricator.wikimedia.org/diffusion/MW/browse/wmf%252F1.28.0-wmf.20/ [1]. https://phabricator.wikimedia.org/T144644
Group0 wikis (mediawikiwiki, test2wiki, testwiki, testwikidatawiki, and zerowiki) are running version 1.28.0-wmf.20 of MediaWiki and extensions. All other wikis are running 1.28.0-wmf.18.
Tomorrow there will be a shortened train schedule in the normal train deployment window during which 1.28.0-wmf.20 will be pushed to all wikis.
Any blockers to this plan are tracked on Phabricator[0].
Thank you for all your help and patience while we get the train schedule[1] back on track!
-- Tyler
[0]. https://phabricator.wikimedia.org/T144644 [1]. https://www.mediawiki.org/wiki/MediaWiki_1.28/Roadmap
wikitech-l@lists.wikimedia.org