📘 (est. 3 minute read)
https://phabricator.wikimedia.org/phame/live/1/post/140/
-------
How’d we do in our strive for operational excellence last month? Read on to
find out!
- Month in numbers.
- Highlighted stories.
- Current problems.
## 📊 *Month in numbers*
* 4 documented incidents in January 2019. [1]
* 16 Wikimedia-prod-error tasks closed. [2]
* 17 Wikimedia-prod-error tasks created. [3]
## *️⃣ *Unable to move certain file pages*
Xiplus reported that renaming a File page on zh.wikipedia.org led to a
fatal database exception. Andre Klapper identified the stack trace from the
logs, and Brad (Anomie) investigated.
The File renaming failed because the File page did not have a media file
associated with it (such move action is not currently allowed in
MediaWiki). But, while handling this error the code caused a different
error. The impact was that the user didn't get informed about why the move
failed. Instead, they received a generic error page about a fatal database
exception.
Brad fixed the code a few hours later, and it was deployed by Roan later
that same day.
Thanks! — https://phabricator.wikimedia.org/T213168
## *️⃣ *DBPerformance regression detected and fixed*
During a routine audit of Logstash dashboards, I found a DBPerformance
warning. The warning indicated that the limit of 0 for “master connections”
was violated. That's a cryptic way of saying it found code in MediaWiki
that uses a database master connection on a regular page view.
MediaWiki can have many replica database servers, but there can be only one
master database at any given moment. To reduce chances of overload,
delaying edits, or network congestion; we make sure to use replicas
whenever possible. We usually involve the master only when source data is
being changed, or is about to be changed. For example, when editing a page,
or saving changes.
As the vast majority of traffic is page views, we have lower thresholds for
latency and dependency on page views. In particular, page views may (in the
future) be routed to secondary data centres that don’t even have a master
DB.
Tchanders from the Anti-Harassment tea) investigated the issue, found the
culprit, and fixed it in time for the next MediaWiki train. Thanks! —
https://phabricator.wikimedia.org/T214735
## *️⃣ *TemplateData missing in action*
Tacsipacsi and Evad37 both independently reported the same TemplateData
issue. TemplateData powers the template insertion dialog in VisualEditor.
It wasn't working for some templates after we deployed the 1.33-wmf.13
branch.
The error was “Argument 1 passed to ApiResult::setIndexedTagName() must be
an instance of array, null given”. This means there was code that calls a
function with the wrong parameter. For example, the variable name may've
been misspelled, or it may've been the wrong variable, or (in this case)
the variable didn't exist. In such case, PHP implicitly assumes “null”.
Bartosz (Matmarex) found the culprit. The week before, I made a change to
TemplateData that changed the “template parameter order” feature to be
optional. This allows users to decide whether VisualEditor should force an
order for the parameters in the wikitext. It turned out I forgot to update
one of the references to this variable, which still assumed it was always
present.
Brad (Anomie) fixed it later that week, and it was deployed the next day.
Thanks! — https://phabricator.wikimedia.org/T213953
## 📈 *Current problems*
Take a look at the workboard and look for tasks that might need your help.
The workboard lists known issues, grouped by the week in which they were
first observed.
→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/
There are currently 188 open Wikimedia-prod-error tasks as of 12 February
2019. (We’ve had a slight increase since November; 165 in December, 172 in
January.)
For this month’s edition, I’d like to draw attention to a few older issues
that are still reproducible:
* [2013; Collection extension] Special:Book fatal error for blocked users.
— https://phabricator.wikimedia.org/T56179
* [2013; CentralNotice] Fatal error when placeholder key contains a space.
— https://phabricator.wikimedia.org/T58105
* [2014; LQT] Fatal error when attempting to view certain threads. —
https://phabricator.wikimedia.org/T61791
* [2015; MassMessage] Warning about Invalid message parameters. —
https://phabricator.wikimedia.org/T93110
* [2015; Wikibase] Warning “UnresolvedRedirectException” for some pages on
Wikidata (and Commons). — https://phabricator.wikimedia.org/T93273
## 💡Terminology
A “Fatal error” (or uncaught exception) prevents a user action. For example
— a page might display “MWException: Unknown class NotificationCount.”,
instead the article content.
A “Warning” (or non-fatal, or PHP error) lets the program continue to
display a mostly page regardless. This may cause corrupt, incorrect, or
incomplete information to be shown. For example — a user may receive a
notification that says “You have (null) new messages”.
## 🎉 Thanks!
Thank you to everyone who has helped by reporting, investigating, or
resolving problems in Wikimedia production. Including: Xiplus‚ Anomie,
Daimona Gilles, He7d3r, Jdforrester, MatmaRex, MModell, Nikerabbit,
Catrope, Tchanders, Tgr, and Thiemo.
Thanks!
Until next time,
– Timo Tijhof
👢*There's a snake in my boot. Reach for the sky!*
-------
Footnotes:
[1] Incidents. –
https://wikitech.wikimedia.org/wiki/Special:AllPages?from=Incident+document…
[2] Tasks closed. –
https://phabricator.wikimedia.org/maniphest/query/COTGbmxGcm_l/#R
[3] Tasks created. –
https://phabricator.wikimedia.org/maniphest/query/DLRuzOg9bSJA/#R
Hi,
Noticed something neat or cool that someone did? Or is someone just
being awesome in general? Take this opportunity to say thanks!
* Thanks to Christian (QChris) for the many years of helping administrate
Gerrit repositories, both during his work as WMF staff, and in the 4 years
since then.
* Thanks to Kunal (Legoktm) for re-starting our Thank you threads as a
weekly occurrence.
Who would you like to thank?
-- Timo Tijhof (Krinkle)
Hi all,
The PE Program team has been preparing for the Wikimedia Foundation midterm
(3 year) planning by reviewing and organizing all of the followup from
TechConf. We published an update of this work on Wiki here:
https://www.mediawiki.org/wiki/Platform_Evolution/Updates/Feb_2019
A list of oiuyr proposed Goals and Outcomes for the next 3 years is here:
https://www.mediawiki.org/wiki/Platform_Evolution/Goals
We are currently looking for feedback on the work we have done so far,
please check out the update for more info.
Thanks everyone!
--
Corey Floyd
Director of Engineering, Core Platform
Wikimedia Foundation
cfloyd(a)wikimedia.org
Reminder: Technical Advice IRC meeting this week **(Wednesday) 4-5 pm UTC**
on #wikimedia-tech.
Question can be asked in English & Persian.
The Technical Advice IRC Meeting is a weekly support event for volunteer
developers. Every Wednesday, two full-time developers are available to help
you with all your questions about Mediawiki, gadgets, tools and more! This
can be anything from "how to get started" over "who would be the best
contact for X" to specific questions on your project.
If you know already what you would like to discuss or ask, please add your
topic to the next meeting:
https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
Hope to see you there!
Michi (for the Technical Advice IRC Meeting crew)
--
Michael F. Schönitzer
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
https://wikimedia.de
Unsere Vision ist eine Welt, in der alle Menschen am Wissens der Menschheit
teilhaben, es nutzen und mehren können. Helfen Sie uns dabei!
https://spenden.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
Hi people ,
This is Kaushik Reddy again.
After a long work, I would like to introduce you to my idea proposal for
the (Wikimedia) GSoC '19.
Here is it:
1) Building an animation to dynamically create popups overlapped on a
geographical map using a real-time API from Wikimedia.
I had found the root of this in here( I have gone through it a bit):
https://meta.m.wikimedia.org/wiki/Research:Data
I would like to work this out as my Google summer of code-19 project for
Wikimedia. May I know who good is it? Also, feel free to comment on this.
With regards,
Kaushik.
TL;DR:
* MediaWiki jobs now run on the same Debian OS version as wmf-production,
which makes make the wmf-quibble PHPUnit tests even more useful.
* The QUnit tests for MediaWiki repos now use Headless Chrome. (For other
repos, this was done last year.)
## Quibble HHVM now on Debian Stretch
Quibble is the Jenkins runner for MediaWiki-related repositories. Its
variants for php70 through php73 were already based on Debian 9
("Stretch"), with a recent Chromium version (Thanks Legotkm and Hashar!).
The HHVM variant, however, was still on Debian 8 ("Jessie"). This caused a
number of side effects:
* wmf-quibble-hhvm no longer matched wmf production (due to a different
operating system version). This could cause subtle differences in backend
behaviour to miss problems in our testing.
* Debian 8 provided an outdated version of Chromium (version 57), which is
no longer supported by Google, and actually had problems with its PKI-NSS
storage layer. These were fixed in later versions of Chromium.
* The Chromium versions differed between the hhvm and php7 jobs, because
the latter is provided by Debian 9 with Chrome 69. This could cause subtle
bugs that affected one job but not the other. It also precluded use of
Headless Chrome.
I've ported the Docker image for quibble-*-hhvm to Debian 9. The tests now
match wmf production, and our QUnit tests and browser tests now
consistently run against the same (recent) version of Chrome. Apart from
one test failure in the Navigation Timing extension, [1] all gated
repositories were verified and still pass.
## Headless Chrome
Since the stable release of Chromium 59 in 2017, this browser now comes
with a headless mode that starts and runs significantly faster than the
desktop variant.
Most of our non-mediawiki repositories only had Jenkins jobs using the
newer Debian 9 (with Chrome 69), and thus were already enjoying the speed
of Headless Chrome.
Now that quibble-hhvm is upgraded to Debian 9, which comes with Chrome 69,
the MediaWiki tests can use Headless Chrome as well. [2]
I've also rebuild the quibble jobs again with another round of Debian 9
package updates, which further upgrades Chromium from 69 to 71.
Happy testing!
-- Timo Tijhof
[1] https://phabricator.wikimedia.org/T215714 –
1 test failing within ext.navigationTiming (temporary disabled).
[2] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/486188/ –
build: Use Headless Chrome for karma-qunit tests.
Hi All,
Here are the minutes from this week's TechCom meeting:
* On Last Call ending 20 February 1pm PST(21:00 UTC, 22:00 CET)
Wikibase Front-End Architecture
<https://phabricator.wikimedia.org/T213318> with the caveat that this
is an opportunity for Wikidata to experiment with this approach while
TechCom discusses out to do Front End Architecture in a more
universal/organized way. Also noting this introduces a circular
dependency that should be resolved at some point.
* IRC Meeting 11pm PST February 13 (February 14 07:00 UTC, 08:00 CET)
in #wikimedia-office Use Github login for mediawiki.org -
<https://phabricator.wikimedia.org/T215046>
* Reopened: RFC: Multiblocks - let admins create multiple, overlapping
blocks on a single user <https://phabricator.wikimedia.org/T202673>
because it was closed due to lack of resourcing rather than it being
something that would never potentially be implemented.
You can also find our meeting minutes at
<https://www.mediawiki.org/wiki/Wikimedia_Technical_Committee/Minutes>
See also the TechCom RFC board
<https://phabricator.wikimedia.org/tag/mediawiki-rfcs/>.
If you prefer you can subscribe to our newsletter here
<https://www.mediawiki.org/wiki/Newsletter:TechCom_Radar>
Thanks,
Kate
--
Kate Chapman
Senior Program Manager, Core Platform
Wikimedia Foundation
kchapman(a)wikimedia.org
Hello Everyone,
I am working on improving developer productivity! Specifically, I am aiming
to improve the local development environment.
Satisfaction with the local development environment got the second lowest
score in the developer satisfaction survey, and we (Release Engineering)
would like to find out which things about it you like, and which are
troublesome. In order to do so, we need your help! I’d love to have the
opportunity to interview you about your local setup and/or observe how you
interact with it.
Please send me an email if you’d like to participate, and I or one of my
colleagues will schedule some time with you.
I’m looking forward to our conversations. This is a chance for you to shape
the local development environment, so please don’t hold back :)
Outcomes resulting from the interviews (non personally identifiable
information) will be made available after analysis has been completed.
--
Jeena Huneidi
Software Engineer, Release Engineering
Wikimedia Foundation