Hi All, welcome to the monthly MediaWiki Insights email!
We’re starting this email again by celebrating volunteer contributors who got their first patch merged in MW core, WMF deployed extensions or services over the past month:
Many thanks and congrats to Nemoralis, RockingPenny4, Theprotonade, S8321414, Philip and Aram!
Enable more people to know MediaWiki and contribute effectively
We’ve officially hit the baseline https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Contributor_retention_and_growth#Baselines we’ve set to measure the growth in number of people who have submitted patches to MediaWiki core: During the period July 2022 - June 2023 (= last WMF fiscal year), 70 people have submitted more than 5 patches to MW core. This year, we reached this number already in March and are currently observing a 14.5% increase in the number of contributors who have submitted more than 5 patches between July 2023 and March 2024 compared to July 2022 - March 2023.
To achieve the goal of a 20% increase, we will continue with consultancy and code review for teams and volunteers as part of our regular work. We are also planning a dedicated MediaWiki focus area for the upcoming Wikimedia Hackathon https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2024 to help people contribute to MediaWiki. If you are attending the Hackathon and are interested in joining the fun to help others onboard in MediaWiki, want to get started with MediaWiki development or already have a project you’d like to get help on, please monitor the Hackathon channels and workboard over the coming weeks for more communication around this!
Project Snapshots: Sustainability and evolution of the platform requires the efforts of many: Deprecated wfGetDB(), migration to Prometheus and supporting page content in core REST API
The theme of many initiatives that ensure sustainability and evolution of the MediaWiki platform is that it requires the collaborative effort of many. In some cases this may be work done primarily by specific teams, in other cases this is a collective effort of staff across teams and volunteers.
One great example for the collective effort of volunteers and staff is that wfGetDB() - the old global function to get database connections - is now hard-deprecated https://phabricator.wikimedia.org/T273239, thanks to many people who did the migration across many extensions! https://phabricator.wikimedia.org/T273239
With wfGetDB() out of the picture, we no longer rely on global state when accessing database connections. This means that code that wants to access another wiki’s database may not end up accidentally mixing information from two wikis. Besides, the new IConnectionProvider interface - getReplicaDatabase() is hopefully more readable than wfGetDB(DB_REPLICA). This is part of our wider work to modernize MediaWiki's platform so that we can inject services for scale and testing.
Supporting page content in core REST API for fetching HTML
We already talked about RESTBase deprecation and Parser Unification milestones in the last edition https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Reports/February_2024#Project_Snapshots:_Milestones_reached_on_3_major_multi-year_projects of the MediaWiki Insights email. Oftentimes, work on one initiative benefits other initiatives and vice versa. This is very true for the following achievement, which is closely related to Parser Unification and RESTBase deprecation:
The MediaWiki REST endpoints for fetching page HTML https://www.mediawiki.org/wiki/API:REST_API/Reference#Get_HTML now support all kinds of content. This provides an easy way for bots and scripts to get the content of any wiki page just like it would be shown to users when they view an article in the browser (*).
This may seem like a simple and obvious thing, but we did not have the capability until recently: From introduction of these endpoints in 2020 up until the completion of the work on T359426 https://phabricator.wikimedia.org/T359426, they were limited to wikitext pages and would fail on pages containing JavaScript or Lua or Wikidata items.
A lot of changes were necessary “under the hood” to get to a point where the REST API could provide rendered HTML for all kinds of content, while using the Parsoid rendering of wiki pages. Much of the necessary work overlaps with the efforts of sunsetting RESTbase https://www.mediawiki.org/wiki/RESTBase/deprecation and implementing support for Parsoid page views https://phabricator.wikimedia.org/T55784. Some key aspects were:
- Integrating Parsoid with ContentHandler (T311648 https://phabricator.wikimedia.org/T311648) - Populating the cache with Parsoid output when pages are edited (T320534 https://phabricator.wikimedia.org/T320534) - Removing the caching layer for Parsoid output in RESTbase (T344945 https://phabricator.wikimedia.org/T344945) - Implementing variant conversion for the API endpoint (T317019 https://phabricator.wikimedia.org/T317019)
Having support for all kinds of page content in the REST API while using Parsoid for wikitext pages is a major milestone towards supporting Parsoid page views. It demonstrates that Parsoid has been fully integrated with the content rendering and caching infrastructure in MediaWiki. This is the culmination of the efforts of several teams over multiple years. Many thanks to everyone involved!
Note that, while the REST endpoints now support all kinds of content, they are not quite yet ready for prime time: they still lack proper integration with the edge caches https://www.mediawiki.org/wiki/Manual:Varnish_caching that will ensure good performance when a lot of clients start using these APIs. To address this issue and to improve version management for API endpoints, we are considering changing the canonical URLs of the endpoints.
(*) We still use the old parser of page views though. See the last MediaWiki Insights email https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Reports/February_2024#Project_Snapshots:_Milestones_reached_on_3_major_multi-year_projects for where we’re at on the roadmap.
MediaWiki metrics to Prometheus migration: Status and call to action
The Observability team is currently working on migrating MediaWiki metrics to Prometheus https://prometheus.io/docs/introduction/overview/, utilizing StatsLib https://www.mediawiki.org/wiki/Manual:Stats, an internally developed, Prometheus-capable metrics interface. We have been using Prometheus in Wikimedia production for several years as it offers several benefits over Graphite https://prometheus.io/docs/introduction/comparison/. Migrating ensures we stay ahead with a supported, scalable metrics platform for a more effective, multidimensional metrics analysis and storage engine.
We are closing in on about 8% of total metrics emitted to graphite migrated over to Prometheus https://grafana.wikimedia.org/d/nCxX65cSk/mediawiki-statslib-migration?orgId=1 and are now ready to invite more people in to help contribute to this effort! Your expertise can help drive the success of this migration and support in migrating your component’s metrics to StatsLib (T350592) https://phabricator.wikimedia.org/T350592:
- Look up your component, extension, or module and follow the examples/docs in the task to migrate your metrics to the new metrics interface. - Help deprecate and clean up/remove outdated metrics not in use (or graphed in dashboards). - Collaboration in testing and feedback for a seamless transition.
A more detailed announcement and call to action will follow.
Many thanks to the Observability team for their leadership on this initiative, Derrick, Timo and Larissa from the MediaWiki Platform team for their consultancy and help in converting MediaWiki metrics so far; Kavitha, Giuseppe, Janis, and Clement from Service Ops for infrastructure support; and the Search Platform team for their recent involvement!
Annual Plan 2024/25: Key Result drafts published
The Wikimedia Foundation has recently published the draft “key results” https://en.wikipedia.org/wiki/Objectives_and_key_results by the Product & Technology department for the upcoming annual plan. While this is not yet at the project/initiatives level (“hypotheses”), the draft KRs give insights in focus areas for the next year. The most relevant objectives for MediaWiki platform work and services for developers are WE5 (“Knowledge Platform I”) and WE6 (“Knowledge Platform II”). Input and questions on these drafts https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2024-2025/Product_%26_Technology_OKRs#Draft_Key_Results are welcome!
Upcoming: MW 1.42 release
MW 1.42 https://mediawiki.org/wiki/MediaWiki_1.42 release is coming! The tentative target was set as May 2024 https://phabricator.wikimedia.org/T359833 and in the next few weeks, it will be the time to polish and prepare for the release. Stay tuned for updates and use Phabricator https://phabricator.wikimedia.org/project/view/6601/ to engage with us and raise potential blockers if you haven’t done yet.
Thanks all for reading,
Birgit