Hi All,
Welcome to the monthly MediaWiki Insights email!
First of all, many thanks and congratulations to the 12 new volunteer contributors: Alex (AllUsernamesArePicked), andrybak (Andrei Rybak), AutumnFeather, Claire (BlankEclair), Daniel Riedmueller, Jbol, Mahmoud Al-Qudsi (mqudsi), Newton Kamau, Nunya Klah, Rummskartoffel, XXBlackburnXx, and أنون (Lolekek). Each of them got their first patch merged in MediaWiki core, extensions or services deployed in Wikimedia production during the month of August! (or at the very end of July)
Some of the developers joining us this month are longstanding members of the Wikimedia project communities; others use MediaWiki outside of Wikimedia and submitted patches upstream for the first time; and others we don't know much about, other than that they also wanted to help with the project. Glad to see you all around!
Project Snapshots: Progress on annual plan initiatives
In July we kicked off work under the Foundation’s new annual plan https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2024-2025/Product_%26_Technology_OKRs (July 2024 - June 2025).
Two months into the new plan, we’ve made good progress on many of the initiatives to evolve the MediaWiki platform (WE5) and improve the services and tools that developers need to support the Wikimedia projects (WE6).
*WE 5.2* https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Artifacts/KR_5.2:_Simplify_feature_development is focusing on exploring ways to simplify feature development by enabling more modularity and improving the maintainability of the software. This Key Result (“KR”) kicked off with research (first quarter) that will inform the initial experiments (second quarter). We started this work by looking at the ecosystem of hooks that are used by extensions to integrate with MediaWiki https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Artifacts/KR_5.2:_Simplify_feature_development/Details_and_updates. So far, we analyzed a sample of 120 of the 1409 hook handlers we use on MediaWiki sites (implementing 60 of 460 distinct hooks). We evaluated about 20 aspects of each handler in order to identify patterns and opportunities for improvement. For instance, about half of the 60 hooks could be isolated from the database to improve maintainability and security. And it looks like 9 of the hooks could even be replaced by declarative registration, which would reduce complexity. Next, we will summarize our findings and propose experiments and explorations that could lead us towards a more expressive and sustainable interface between MediaWiki core and extensions. You can find more information on the project page https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Artifacts/KR_5.2:_Simplify_feature_development/Details_and_updates .
*WE5.3* is focusing on performance experiments by treating a page as a composition of structured fragments. The initial work consists of two pieces: Instrumentation and data gathering to inform future performance and templating improvements (5.3.1 https://phabricator.wikimedia.org/T363421) and Prototype selective html updates in Parsoid (5.3.2 https://phabricator.wikimedia.org/T371713).
We have implemented a very basic selective html update technique in Parsoid to update pages when a dependent template is edited. There are still correctness and functionality gaps, but the implementation is complete enough to let us gather preliminary performance numbers. We notice up to 5x+ speedup updating a page when a dependent template is edited - the benefit depends on the complexity of the template and the page using the template. The larger the page and simpler the template, the larger the expected benefit. There is more work to be done to ensure correctness, but we don’t expect this to impact expected performance benefits significantly. This code in Parsoid assumes that the previous version of the HTML is available in cache and the template and page meet the criteria for selective HTML updates.
The selective update technique crucially relies on a set of conditions being satisfied: previous version of page available in parser cache and template generating well-formed DOM fragments. So far, we find that previous HTML is present during parses that account for 50% of time spent in production and we expect to understand how often this is true with the collected data.
Once complete, 5.3.1 and 5.3.2 can let us estimate the benefit of fully implementing and deploying these in production. The early results from 5.3.1 and 5.3.2 look promising.
Separately, we have also been discussing and planning the work required to support the embedding of Wikifunctions on a Parsoid-enabled wiki. This work, when complete, will let Parsoid support Wikifunctions without needing to build a lot of new infrastructure for it.
A previous edition of MediaWiki Insights https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Reports/June_2024 mentioned work the MediaWiki Interfaces Team has been doing on improving the experience of REST API endpoint creators and callers. Another phase of that work was finished in August with the completion of Rest: Deprecate BodyValidato https://phabricator.wikimedia.org/T358560r. This required compatibility changes to several extensions, including: Checklists https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Checklists/+/1043823?usp=search, MenuEditor https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MenuEditor/+/1047154?usp=search, TuleapWikiFarm https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TuleapWikiFarm/+/1046727?usp=search, ContentStabilization https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ContentStabilization/+/1046739?usp=search, Workflows https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Workflows/+/1047144?usp=search, BlueSpiceExtendedSearch https://gerrit.wikimedia.org/r/c/mediawiki/extensions/BlueSpiceExtendedSearch/+/1047153?usp=search and others. Deprecating the old system directs endpoint creators down an implementation path that allows them to take advantage of recent improvements, such as convenience methods to generate response schemas https://phabricator.wikimedia.org/T368131, an initial usage of which can be seen in UpdateHandler https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1055181/6/includes/Rest/Handler/UpdateHandler.php#79. This work is part of *KR 5.1*: Interventions to increase sustainability of the MediaWiki platform.
A key goal of WE6 (Developer Services) this year is to evolve and improve our software testing capabilities. In the last Insights email https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Reports/July_2024#Project_snapshots:_Parsoid_deployed_on_first_production_wikis_and_a_new_way_of_provisioning_on-demand_test_environments_in_the_works we shared about the work that is underway to facilitate the provisioning of testing environments on-demand for automatic and exploratory testing. In the meantime, we have released the new backend for PatchDemo https://patchdemo.wmcloud.org/ that is backed by Kubernetes. You can try it out by enabling a flag on the PatchDemo UI. Next we will provide full feature completeness to the current offering and enable not just the provisioning of MediaWiki for testing, but also the testing of custom services & extensions. If you don't see your previously created environment, they are still available at patchdemo-legacy.wmcloud.org.
You can read through the full list of initiatives planned under the Knowledge Platform objectives WE5 (MediaWiki Platform) and WE6 (Developer Services) on this page https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2024-2025/Product_%26_Technology_OKRs#Hypotheses .
With this, we’re wrapping up this edition! It has been great to see some of you at Wikimania https://wikimania.wikimedia.org/wiki/2024:Wikimania in August, at the Future of MediaWiki session https://www.youtube.com/live/iRw444uS1-E?t=5698s, the hallway track and in the dev corner https://wikimania.wikimedia.org/wiki/2024:Program/Hackathon#Developer_Corner_(in_the_Hackathon_space_-_Voore_room) .
Amir Sarabadani just shared with me the link to this ticket https://phabricator.wikimedia.org/T183490, which is an effort primarily driven by Zabe: It started at Wikimania and is about to complete: MCR schema migration for External Store URLs. Many thanks to Zabe & Amir for kicking this off, and for continuing the work after the event!
Thanks all for reading,
Birgit
P.S.: For those of you who have missed the announcement on other channels: The voting period for the 2024 Board of Trustees election https://meta.wikimedia.org/wiki/Special:MyLanguage/Wikimedia_Foundation_elections/2024 is still open through September 17, 2024. Eligibility criteria for developers (and for editors and translators) can be found on this page https://meta.wikimedia.org/wiki/Wikimedia_Foundation_elections/2024/Voter_eligibility_guidelines.