Hello,
Here's what the performance team has been up to.
== Dashboards & instrumentation ==
We spent time instrumenting software and curating displays of performance
data. We have several new dashboards to share with you:
* Global edit rate and save failures (new)
https://grafana.wikimedia.org/dashboard/db/edit-count
* Performance metrics (revamped)
https://grafana-admin.wikimedia.org/dashboard/db/performance-metrics
* Page load performance
https://grafana.wikimedia.org/dashboard/db/navigation-timing
...by continent:
https://grafana.wikimedia.org/dashboard/db/navigation-timing-by-continent
...by country :
https://grafana.wikimedia.org/dashboard/db/navigation-timing-by-geolocation
...by browser :
https://grafana.wikimedia.org/dashboard/db/navigation-timing-by-browser
* We found that certain browsers were reporting wildly inaccurate timing
data and skewing our summary performance metrics, and reacted by validating
browser metric data more strictly against Navigation Timing API specs.
== ResourceLoader ==
ResourceLoader is the MediaWiki subsystem responsible for loading CSS,
JavaScript, and i18n interface messages for dynamic site features. It is
critical to site performance. Changes to ResourceLoader are focused on
reducing backend response time, ensuring we make efficient use of the
browser cache, and reducing time to first paint (the time it takes any
content to appear). This work is led by Timo Tijhof.
* The "/static/$mwBranch" entry point has been deprecated and removed in
favor of wmfstatic - a new multiversion-powered entrypoint accessed via
"/w" (via RewriteRule)
https://phabricator.wikimedia.org/T99096
* Restricting addModuleStyles() to style-only modules (ongoing)
https://phabricator.wikimedia.org/T92459
* Startup module check is now based on a feature test instead of browser
blacklist
https://phabricator.wikimedia.org/T102318
== WebPageTest ==
Page load performance varies by browser, platform, and network. To
anticipate how code changes will impact page performance for readers and
editors, we use WebPageTest (https://wikitech.wikimedia.org/wiki/WebPageTest),
a web performance browser automation tool. WebPageTest loads pages on
Wikimedia wikis using real browsers and collects timing metrics. This work
is led by Peter Hedenskog.
* We now generate waterfall charts for page loads on Firefox. Previously we
were only able to produce them with Chrome.
* We tracked downs two bugs in WebPageTest that caused it to report an
incorrect value for time-to-first-byte and reported them upstream.
https://phabricator.wikimedia.org/T130182https://phabricator.wikimedia.org/T129735
* We upgraded the WebPageTest agent instance after observing variability in
measurements when the agent is under load.
https://phabricator.wikimedia.org/T135985
* We designed a new dashboard to help us spot performance regressions
https://grafana.wikimedia.org/dashboard/db/webpagetest
== Databases ==
The major effort in backend performance has been to reduce replication lag.
Replication lag occurs when a slave database is not able to reflect changes
on the master database quickly enough and falls behind. Aaron Schulz set
out to bring peak replication lag down from ten seconds to below five, by
identifying problematic query patterns and rewriting them to be more
efficient. We are very close to hitting that target: replication lag is
almost entirely below five seconds on all clusters.
https://phabricator.wikimedia.org/T95501
* High lag on databases used to generate special pages no longer stops job
queue processing
https://phabricator.wikimedia.org/T135809
== Multi-DC ==
"Multi-DC" refers to ongoing work to make it possible to serve reads from a
secondary data center. Having MediaWiki running and serving requests in
more than one data center will reduce latency and improve site reliability.
This project is led by Aaron Schulz.
In order for this to be possible, we need to be able to anticipate which
requests will need the master database, so we can route them accordingly.
The plan is to achieve this by making sure that GET requests never require
a master database connection. We've made progress incremental progress
here, most recently by changing action=rollback to use JavaScript to
perform HTTP POST requests.
We also need to be able to broadcast cache purges across data centers. The
major work on this front has been the addition to core of EventBus classes
that relay cache proxy and object cache purges. Stas Malyshev of the
discovery team is assisting with this work.
== Thumbor ==
"Thumbor" is shorthand for the project to factor thumbnail rendering out of
MediaWiki and into a standalone service based on Thumbor (
http://thumbor.org/). This project is led by Gilles Dubuc. The following
list summarizes recent progress:
- Simplified the VCL as much as possible
- Added client throttling with the tbf vmod
- Added progressive JPEG support to ImageMagick engine
- Added configurable chroma subsampling support
- Made SVG detection more robust
- Added multilanguage SVG support
- Reproduced temp folder security mechanism found in MediaWiki for SVG for
all file types
- Swift's rewrite.py ported to vagrant. On Vagrant thumbor now hooks itself
into the same point in the stack it will in production
- Swift storage implemented (shard support left to do)
- Matched Content-Disposition behavior to MediaWiki
- Vastly increased performance on JPEG processing by using a long-running
exiftool process and named pipes to pass commands to it
- Made one instance of thumbor run on each available core on vagrant, since
thumbor is single-threaded
- Debian packaging well under way: https://phabricator.wikimedia.org/T134485
all dependencies covered except one. 14 backports and 17 new packages so
far. Working with Filippo to get as many of these into Debian proper as
possible.
Until next time,
Aaron, Gilles, Ori, Timo, and Peter
With https://gerrit.wikimedia.org/r/288633 some new classes in the Babel
extension were put into a new namespace "MediaWiki\Babel\BabelBox".
Legoktm approved but Thiemo Mättig (WMDE) disagrees. PHPUnit tests are
already in namespace "Babel\Tests".
Hi,
I just released MediaWiki-Codesniffer 0.7.2, which is a bugfix release
for 0.7.1. There was an issue[1] in the SpacyParenthesisSniff that would
sometimes eat the content inside of the parenthesis or brackets. I ran a
script to check and the only extension that was affected and had bad
code committed was ArticlePlaceholder, which has already been fixed.
Thanks to Thiemo and Nikerabbit for reporting, and PleaseStand for the
patch. Additionally, EBernhardson overhauled how our tests run so we now
have the ability to test sniffs that automatically fix code, which will
hopefully prevent regressions like this in the future!
I submitted autogenerated patches to upgrade all extensions that were
already at 0.7.1 to upgrade to 0.7.2 - they should only touch composer.json.
[1] https://phabricator.wikimedia.org/T134857
-- Legoktm
Hello Wikimedia Developers,
We are quickly reaching capacity at the Esino Lario hackathon and will be
closing registration for the hackathon next week on Tuesday, May 31.
If you have not yet registered but plan on attending, now would be a good
time to do so.
Reminder: You need to register for BOTH the hackathon and the Wikimania
main event if you are planning to attend both. We are not able to post a
public list of everyone who has registered so please email the hackathon
team: me, Nick & Simone (CCed) before Monday if you are unsure of your
registration or have questions.
More information about the event itself will go up on the event page linked
below in the coming weeks and we will email all registered participants
with further information as well.
https://wikimania2016.wikimedia.org/wiki/Hackathon
Thanks! & See you in Italy!
Rachel
Hello,
The Discovery Team recently added descriptive text to the Wikipedia.org
page footer in order to give visitors a better idea of what the sister wiki
projects are really all about. Check it out at www.wikipedia.org or view a
mobile screen capture here
<https://www.mediawiki.org/wiki/File:Descriptive-text_sister_projects-wikipe…>
.
We also wrapped up a quick, one question survey that ran for a week on the
portal to determine how visitors arrive at the page. The results showed
that many of the visitors arrive by clicking on a bookmarked link or by
typing in 'wikipedia' in their browser. We had many encouraging and
uplifting comments as well - see the full results here
<https://commons.wikimedia.org/wiki/File:Wikipedia_Portal_Survey_-_May_2016.…>
.
In the coming weeks and months we'll be reaching out to many of those
survey takers who graciously provided their name and email addresses to
engage in deeper conversations on how they use the portal and Wikipedia in
general.
As always, more detailed information is available on wiki for the Wikipedia
Portal <https://www.mediawiki.org/wiki/Wikipedia.org_Portal> and A/B testing
<https://www.mediawiki.org/wiki/Wikipedia.org_Portal_A/B_testing>.
On behalf of our happy little Wikipedia.org Portal team,
Deb
--
Deb Tankersley
Product Manager, Discovery
Wikimedia Foundation
Hello,
Just a quick note: the Discovery Portal team updated the articles by
language statistics on wikipedia.org this morning. I've attached a small
screenshot to this email that displays the new numbers.
Cheers,
Deb
--
Deb Tankersley
Product Manager, Discovery
Wikimedia Foundation
[For your info]
I've spend some time in the last weeks rewriting and cleaning up our
technical Git / Gerrit / Code Review documentation on mediawiki.org.
135 edits later, things feel a bit cleaner. :)
=== Changes ===
There is a central Git/Gerrit "Troubleshooting" page linked from the
most relevant pages, instead of separate sections on three or four
different pages which partially duplicated content (setting up Git and
Gerrit on different operating systems had similar problems and missing
central information).
Some rather specific subpages were merged into existing pages (e.g.
into "Gerrit/Advanced usage").
The entrance page https://www.mediawiki.org/wiki/Gerrit has a
"Prerequisites" section and the sections are now structured by usecase
/ type of user.
Some "too much detail" content was also removed (like the history of
the Git and Gerrit software; you can check Wikipedia as it's irrelevant
if you want just to get your patch done).
Full list of changes: https://phabricator.wikimedia.org/T134256#2317711
=== No changes ===
Some stuff I did not touch (as perfect is the enemy of good).
For example, I'm still wondering about the role of
https://www.mediawiki.org/wiki/Git_for_dummies compared to
https://www.mediawiki.org/wiki/Gerrit/Getting_started and
https://www.mediawiki.org/wiki/Gerrit/Tutorial .
Plus I still consider the screenshots of command line text output
distractive on https://www.mediawiki.org/wiki/Gerrit/Tutorial .
=== Thank you ===
Thank you a lot to everybody who provided feedback, answers to my
Gerrit questions, and helped editing those pages!
=== Next steps ===
After this technical documentation exercise, the next steps will focus
on soft / social+organizational aspects of code review:
* https://phabricator.wikimedia.org/T129068
Improve code contribution guidelines for patch authors by fixing
https://www.mediawiki.org/wiki/Gerrit/Code_review/Getting_reviews etc
* https://phabricator.wikimedia.org/T129067
Agree on + document a structured, standardized approach for
reviewing code contributions by fixing
https://www.mediawiki.org/wiki/Gerrit/Code_review etc.
Needless to say, your help is very welcome.
More information to come soon.
Cheers,
andre
--
Andre Klapper | Wikimedia Bugwrangler
http://blogs.gnome.org/aklapper/
