The most recent developer satisfaction survey supports the claim that
volunteers find it hard to get code reviews. The hypothesis is that by
having clearer ownership around code, there should be clearer
accountability, and it should be clearer what code lacks ownership that
probably should.
This also applies to Wikimedia staff. It's often not clear when needing to
make changes to core, who is the best person to talk to about that code,
and who should be responsible/accountable/consulted/informed about any
changes to the code.
I am involved in the development of the Desktop improvements project [1]
and when we began the work there was no recognized owner for a lot of the
skin code in core. Since then my team has taken the role of maintainer and
it has allowed us to drive a lot of changes. I'm seeing new skins pop up
and we have started hearing from skin engineers, so I feel this has been a
positive thing. However, as we've navigated other parts of the codebase,
we've had to ask a lot of questions such as "Who maintains the legacy
parser? Are they open to changes there, or should that code be considered
frozen?"; "Who should we talk to about changes relating to the JavaScript
of the watchstar?"
What with the proposed move to Gitlab, I have been very interested in the
code owners file documented on their site [2]
I hypothesize that identifying code owners and getting Wikimedia
engineering to agree to some basic expectations around code ownership might
go a long way to helping set expectations around who to talk to about
making changes, and actually getting those changes carried to completion.
The experiment of adding an OWNERS file to MediaWiki core seems like a
cheap one, given if we find it does nothing the file can easily be removed.
If you are interested in helping me explore this, feel free to join the
conversation on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/745946
Wishing everyone a happy holiday
Jon
[1] mediawiki.org/wiki/Desktop_improvements
[2] https://docs.gitlab.com/ee/user/project/code_owners.html
How’d we do in our strive for operational excellence last month? Read on to find out!
Incidents
6 documented incidents last month. That's above the two-year and five-year median of 4 per month (per Incident graphs <https://codepen.io/Krinkle/full/wbYMZK>).
2021-11-04 large file upload timeouts <https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-11-04_large…>; Impact: For 9 months, editors were unable to upload large files (e.g. to Commons). Editors would receive generic error messages, typically after a timeout. In retrospect, a dozen different distinct production errors had been reported and regularly observed that were related and provided different clues, however most of these remained untriaged and uninvestigated for months. This may be related to the affected components having no active code steward.
2021-11-05 TOC language converter <https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-11-05_TOC_l…>; Impact: For 6 hours, wikis experienced a blank or missing table of contents on many pages. For up to 3 days prior, wikis that have multiple language variants (such as Chinese Wikipedia) displayed the table of contents in an incorrect or inconsistent language variant (which are not understandable to some readers).
2021-11-10 cirrussearch commonsfile outage <https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-11-10_cirru…>; Impact: For ~2.5 hours, the Search results page was unavailable on many wikis (except English Wikipedia). On Wikimedia Commons the search suggestions feature was unresponsive as well.
2021-11-18 codfw ipv6 network <https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-11-18_codfw…>; Impact: For 8 minutes, the Codfw cluster experienced partial loss of IPv6 connectivity for upload.wikimedia.org. This did not affect availability of the service because the "Happy Eyeballs <https://en.wikipedia.org/wiki/Happy_Eyeballs>" algorithm ensures browsers (and other clients) automatically fallback to IPv4. The Codfw cluster generally serves Mexico and parts of the US and Canada. The upload.wikimedia.org service serves photos and other media/document files, such as displayed in Wikipedia articles.
2021-11-23 core network routing <https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-11-23_Core_…>; Impact: For about 12 minutes, Eqiad was unable to reach hosts in other data centers via public IP addresses. This was due to a BGP routing error. There was no impact on end-user traffic, and impact on internal traffic was limited (only Icinga alerts themselves) because internal traffic generally uses local IP subnets which we currently route with OSPF instead of BGP.
2021-11-25 eventgate-main outage <https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-11-25_event…>; Impact: For about 3 minutes, eventgate-main was down. This resulted in 25,000 MediaWiki backend errors due to inability to queue new jobs. About 1000 user-facing web requests failed (HTTP 500 Error). Event production briefly dropped from ~3000 per second to 0 per second.
Incident follow-up
Remember to review and schedule Incident Follow-up work <https://phabricator.wikimedia.org/project/view/4758/> in Phabricator, which are preventive measures and tech debt mitigations written down after an incident is concluded. Read more about past incidents at Incident status <https://wikitech.wikimedia.org/wiki/Incident_status> on Wikitech.
Recently resolved incident follow-up:
Disable DPL on wikis that aren't using it <https://phabricator.wikimedia.org/T287916>
Filed after a July 2021 incident, done by Amir (Ladsgroup) and Kunal (Legoktm).
Create easy access to MySQL ports for faster incident response and maintenance <https://phabricator.wikimedia.org/T291352>
Filed in Sep 2021, and carried out by Stevie (Kormat).
Create paging alert for primary DB hosts <https://phabricator.wikimedia.org/T233684>
Filed after a Sept 2019 incident, done by Stevie (Kormat).
Trends
November saw 27 new production error reports of which 14 were resolved, and 13 remain open and carry over to the next month.
Of the 301 errors still open from previous months, 16 were resolved. Together with the 13 carried over from November that brings the workboard to 298 unresolved tasks.
Figure 1: Unresolved error reports by month <https://phabricator.wikimedia.org/phame/post/view/261/production_excellence…>.
Outstanding errors
Take a look at the workboard and look for tasks that could use your help.
→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/
💡 Did you know:
*To find your team's error reports, use the appropriate ***"Filter" link in the sidebar of the workboard***.*
Issues carried over from recent months:
Apr 2021:
9 of 42 issues left.
May 2021:
16 of 54 issues left.
Jun 2021:
9 of 26 issues left.
Jul 2021:
11 of 31 issues left.
Aug 2021:
10 of 46 issues left.
Sep 2021:
10 of 24 issues left.
Oct 2021:
20 of 49 issues left.
Nov 2021:
13 of 27 new issues <https://phabricator.wikimedia.org/maniphest/query/0W0Nuk9umBDc/#R> are carried forward.
Thanks!
Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!
Until next time,
– Timo Tijhof
🔗 Share or read later via https://phabricator.wikimedia.org/phame/post/view/261/
Hi all,
On Wednesday we will be issuing a security and maintenance release to all
supported branches of MediaWiki.
The new releases will be:
- 1.35.5
- 1.36.3
- 1.37.1
This release includes fixes for multiple high severity authorization
bypasses in MediaWiki core, it is recommended you patch immediately. A
short LocalSettings.php configuration snippet will also be shared to
disable the vulnerable functionality if you are unable to patch right away.
This snippet should work across all vulnerable MediaWiki versions,
including end-of-life ones.
In addition to that, this will resolve other issues in MediaWiki core and
also includes some fixes previously committed to git, including minor
security and hardening patches along with bug fixes included for
maintenance reasons.
It also fixes 2 issues in MediaWiki tarball bundled extensions.
We will make the fixes available in these respective release branches and
master. Tarballs will be available for the above mentioned point releases
as well.
A summary of some of the security fixes that have gone into non-bundled
MediaWiki extensions will also follow later.
- 1.38.0-wmf.12 has been rolled out to all wikis. There are a couple of
low frequency errors that are being logged bug they are being worked on.
- Conductor: Ahmon Dancy (yours truly)
- Backup Conductor: Brennen Bearnes
- Blocker Task: T293953 <https://phabricator.wikimedia.org/T293953>
- Current Status <https://versions.toolforge.org/>
📊 By the Numbers
Sparklines comparing with the last 5 trains.
- 231 Patches █▅▁▅▂
- 1 Rollbacks ▁▃█▆▃
- 0 Days of delay ▁▂▆█▁
- 8 Blockers ▃▃▁█▇
✨ Trainbow Love 🎉Thanks to folks who reported or resolved blockers:
- Amir Sarabadani (WMF)
- Timo Tijhof
- Taavi Väänänen
- EBernhardson
- Daniel Kinzler
- Roan Kattouw
- Jon Robson
- Zabe
- and many more!
-
Hi,
<https://packagist-mirror.wmcloud.org/> is exactly what it sounds like,
a mirror of the metadata on packagist.org. It's very simple, just a
systemd timer and some Apache config. Since setting it up 2 or 3 years
ago I've barely had to touch it. And based on access logs, some people
are actually using it (though we never switched CI over to it, as the
idea/goal once was).
But after being reminded about it by the current Cloud VPS cleanup, I'm
not interested in running this anymore. packagist.org is actually open
source (*looks at npmjs.com in sadness*) and hasn't had any reliability
issues recently AFAIR. There's also a bit of work to do, the VM needs to
be upgraded from stretch, and it should be switched over to the new
composer/mirror script from upstream.
So if someone would like to take it over, comment on
<https://phabricator.wikimedia.org/T296968> and I'm happy to add you and
explain how it works. Otherwise please consider this notice that I
intend to shut it down at the end of the year.
-- Legoktm
Hi,
Myself and Erutuon are working on writing a Rust library[1] that
validates and normalizes MediaWiki page titles, initially for use in
bots and tools, but it has other potential use cases too.
There is some prior art for this, a mediawiki-title npm package[2] is
used by various nodejs services, and there's a mediawiki.Title
ResourceLoader module in core too, all of which reimplement MediaWiki's
title parsing, validation and normalization routines...basically
MediaWikiTitleCodec and what it calls.
One problem area is $wgLegalTitleChars. It's part of a PHP regex that
gets used to check if any invalid characters are present by inverting
it, basically [^{legalchars}]. It's also exposed via the meta=siteinfo
API, which suggests that it's useful for external callers, but in its
current form it's really not.
The current value is:
$wgLegalTitleChars = "
%!\"$&'()*,\\-.\\/0-9:;=?@A-Z\\\\^_`a-z~\\x80-\\xFF+";
It's is a mix of literal characters, some escaped and some not, and
various ranges. Specifically, it's tuned for PHP regexes, like / is
escaped because it delimits regexes (an unnecessary escape and therefore
problematic in Rust), and it uses syntax like \x80-\xFF, while in
JavaScript we want \u0080-\uFFFF. We'd also like to use the Unicode
escape class syntax in Rust, but for now have worked around it to use
the byte class.
In fact, we have functions that parse the byte class and turn it into a
unicode class, Title::convertByteClassToUnicodeClass() in PHP and a
JavaScript version too[3]. This seems entirely unnecessary to me, given
that we could just write this sequence in a more regex-neutral manner to
make it more portable:
$wgLegalTitleCharacters = [
'characters' => " %!\"$&'()*,-./:;=?@\\\\^_`~",
// +
'plus' => true,
// A-Z, a-z, 0-9
'alphanumeric' => true,
// \x80-\xFF
'non-ascii' => true,
];
Characters are literal characters to run through preg_quote or
mw.util.escapeRegExp, and the ranges are to be specified in whatever
format the specific regex engine would like.
But this also opens up the question - is there a valid use case for
customizing $wgLegalTitleChars anymore? We already have a comment that
says "Don't change this unless you know what you're doing". There's also
one that says "In some rare cases you may wish to remove + for
compatibility with old links." - is that still a consideration today?
I would think that for easy importing/exporting across various MediaWiki
wikis we want the set of legal title characters to be rather static. And
if someone wants to ban some character from being used,
Extension:TitleBlacklist (bundled) provides a much less invasive way to
do so.
So to recap:
1. Can we get rid of the ability to customize legal title characters?
2. If #1 is no, any objections to the breaking change of swapping out
$wgLegalTitleChars (string) for $wgLegalTitleCharacters (array)?
Note that extensions can still read from the old global, just it
can't be overridden in LocalSettings anymore.
The actual patch to review is
<https://gerrit.wikimedia.org/r/c/mediawiki/core/+/745386>.
[1] https://gitlab.com/mwbot-rs/mwbot/-/tree/master/mwtitle
[2] https://github.com/wikimedia/mediawiki-title/
[3] https://github.com/wikimedia/mediawiki-title/blob/master/lib/utils.js