Hi All, welcome to the monthly MediaWiki Insights email!
Like last time https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Reports/January_2024, we’re starting this email by celebrating volunteer contributors who got their first patch merged in MW core, WMF deployed extensions or services over the past month:
A big thanks to GergesShamon and Agent Isai for their contributions! Welcome :-)
Many thanks also to the reviewers - volunteers and staff - of patches by new contributors. Your support is invaluable for helping people contribute to MediaWiki effectively and make it a fun first experience!
The focus of this edition of the monthly MW Insights email lies on our multi-year initiatives: Migration of MediaWiki from bare metal to Kubernetes https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes in Wikimedia Production, Parser Unification https://www.mediawiki.org/wiki/Parsoid/Parser_Unification and RESTBase deprecation https://phabricator.wikimedia.org/project/profile/6289/. These initiatives involve many people and projects, require orchestrated efforts and coordination, and will help us in many different ways. Another thing that these initiatives have in common is that we’re getting closer to being able to benefit from these efforts :-).
Project Snapshots: Milestones reached on 3 multi-year projects
MediaWiki on Kubernetes migration is 75% complete, completing migration of all internal traffic, all scheduled jobs and reaches 50% global traffic milestone!
Often shortened as mw-on-k8s, MediaWiki on Kubernetes is a multi-year effort to move all of the MediaWiki deployments running on WMF production infrastructure to the new WikiKube platform.
The migration is underway and very recently a new milestone was hit where the WikiKube platform now serves 50% of end user requests https://phabricator.wikimedia.org/T290536. At half mark, having more percentage of traffic on WikiKube, Service Operations is working with Release Engineering to make changes to monitoring tools to surface any issues during deployments to continue with further ramps. Also, almost the entirety of what is called internal traffic https://phabricator.wikimedia.org/T333120, that is traffic that is generated by applications running in the infrastructure and reaching out to MediaWiki for various purposes is migrated to WikiKube. A special mention should go to MediaWiki Jobs fully migrated just a week ago https://phabricator.wikimedia.org/T349796. Feel free to follow the higher umbrella task https://phabricator.wikimedia.org/T290536 and/or MediaWiki_On_Kubernetes on Wikitech https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes.
This migration will unleash the ability to deploy multiple versions of code simultaneously. Also, this will help enhance platform capabilities to build dockerized isolated environments for coding, testing and even production debugging.
MediaWiki on Kubernetes will allow us to deprecate and eventually remove a lot of our in-house developed code. Another benefit is that we will be able to react to sudden traffic spikes, like newsworthy events better, as the flexing up and down is a matter of configuration change. This enables efficient placement of workloads, packing workloads in a more environmentally friendly way, increasing hardware utilization.
It takes a village to get this far: A big thanks to the Service Operations team (Clément Goubert, Giuseppe Lavagetto, Alexandros Kosiaris, Kamila Součková, Hugh Nowlan, Effie Mouzeli, Reuven Lazarus, Jannis Mayboom and Kavitha Appakayala) for their leadership on this project, the Release Engineering team (specifically: Dan Duvall, Jeena Huneidi, Tyler Cipriani and Ahmon Dancy) for their work on Blubber https://wikitech.wikimedia.org/wiki/Blubber, the Deployment Pipeline https://wikitech.wikimedia.org/wiki/Deployment_pipeline and Scap https://wikitech.wikimedia.org/wiki/Scap, Dom Walden from Quality and Test Engineering for crafting and executing a plan to test the first deployment of MediaWiki on Kubernetes, and everyone else who has contributed in the one way or the other! <3
More milestones:
We’re seeing the first lights at the end of the tunnel of another multi-year initiative: The *parser unification* https://www.mediawiki.org/wiki/Parsoid/Parser_Unification. One week ago, Parsoid Read Views was rolled out to first wikis: Parsoid is now the default read views renderer on the Foundations’ Office Wiki and Wikitech DiscussionTools. This early experimentation allows us to find issues in a limited space, which will help us evaluate readiness of the feature and increase our confidence for future rollouts http://mediawiki.org/wiki/Parsoid/Parser_Unification/Confidence_Framework. A huge thanks to Subbu Sastry, Mateus Santos, CScott, Isabelle Hubert-Pallatin, Arlo Breault, Shannon Bailey, Yiannis Giannelos and Sérgio Lopes for making this work! Many thanks also to Daniel Kinzler: His work on the RESTBase deprecation directly helped us get to this milestone :-)
*RESTBase deprecation*: We have been continuously working on decoupling services from RESTBase, aiming for the modernisation and sustainability of Wikimedia products in our services platform. The MediaWiki Interfaces team finished up reimplementation of Reading Lists endpoints in MW REST API https://phabricator.wikimedia.org/T348491 and are now confirming with affected callers https://phabricator.wikimedia.org/T357478 that the new endpoints meet their needs before rerouting calls https://phabricator.wikimedia.org/T348493 and retiring old code https://phabricator.wikimedia.org/T348494. The overall effort https://phabricator.wikimedia.org/T336693 will not only move us forward on RESTBase retirement, but also reduce the total amount of code we have to maintain. Many thanks to Bill Pirkle, Atieno Njira, Wendy Quarshie, and Daniel Kinzler for making this work! We also fully turned off Parsoid cache storage in RESTBase - clients will get outputs direct from MediaWiki and the cache will be handled by ParserCache. Next, we will re-route clients directly to MediaWiki and fully remove Parsoid from RESTBase (T344944 https://phabricator.wikimedia.org/T344944). The Page Content Service (PCS) will also handle its own cache https://phabricator.wikimedia.org/T348995 and we are ready to test the new capabilities in staging soon. Many thanks to Yiannis Giannelos and the Content Transform team and to Daniel Kinzler for his efforts and support on decoupling Parsoid from RESTBase!
All of these multi-year initiatives help us increase sustainability and maintainability of the platform, streamline engineering and developer workflows, and unlock the path for new and improved platform capabilities and product opportunities.
Outlook: Knowledge Platform in the annual plan 2024/2025
The Wikimedia Foundation has recently published the draft objectives by the Product & Technology department https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2024-2025/Goals/Infrastructure for the next annual plan on Meta, alongside an introduction by Selena Deckelmann and a few questions that we’re exploring https://meta.wikimedia.org/wiki/Talk:Wikimedia_Foundation_Annual_Plan/2024-2025/Goals/Infrastructure#Questions. Your input on these questions is very welcome!
The draft objectives include “Knowledge Platform I” - centered around MediaWiki platform evolution and “Knowledge Platform II” - centered around developer/engineering services and workflows. The objectives show only the high-level direction for next year. The draft “key results” (currently work in progress) will give a better idea of what areas of work we’re thinking about. We’ll be publishing these in March and share the link + invitation for feedback with this list again.
Thanks all for reading,
Birgit
I'm very happy that so many fundamental pieces are finally nearing completion this year, after years and years of work and advocacy by individuals. Things that had been initiated in discussions 4, 5 or sometimes even 15 years ago. From the ones listed in Birgit's mail, to the cleanup of the database structure, php name spacing and the jsdoc conversion, to Vue/Codex being usable etc etc.
While most of these aren't very visible to outsiders (other than an occasional unintentional disruption), they are critical if we want to keep MediaWiki working for us in the future and for all of us to more efficiently work with MediaWiki as a platform.
DJ
On Thu, Feb 29, 2024 at 5:26 PM Birgit Müller bmueller@wikimedia.org wrote:
Hi All, welcome to the monthly MediaWiki Insights email!
Like last time https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Reports/January_2024, we’re starting this email by celebrating volunteer contributors who got their first patch merged in MW core, WMF deployed extensions or services over the past month:
A big thanks to GergesShamon and Agent Isai for their contributions! Welcome :-)
Many thanks also to the reviewers - volunteers and staff - of patches by new contributors. Your support is invaluable for helping people contribute to MediaWiki effectively and make it a fun first experience!
The focus of this edition of the monthly MW Insights email lies on our multi-year initiatives: Migration of MediaWiki from bare metal to Kubernetes https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes in Wikimedia Production, Parser Unification https://www.mediawiki.org/wiki/Parsoid/Parser_Unification and RESTBase deprecation https://phabricator.wikimedia.org/project/profile/6289/. These initiatives involve many people and projects, require orchestrated efforts and coordination, and will help us in many different ways. Another thing that these initiatives have in common is that we’re getting closer to being able to benefit from these efforts :-).
Project Snapshots: Milestones reached on 3 multi-year projects
MediaWiki on Kubernetes migration is 75% complete, completing migration of all internal traffic, all scheduled jobs and reaches 50% global traffic milestone!
Often shortened as mw-on-k8s, MediaWiki on Kubernetes is a multi-year effort to move all of the MediaWiki deployments running on WMF production infrastructure to the new WikiKube platform.
The migration is underway and very recently a new milestone was hit where the WikiKube platform now serves 50% of end user requests https://phabricator.wikimedia.org/T290536. At half mark, having more percentage of traffic on WikiKube, Service Operations is working with Release Engineering to make changes to monitoring tools to surface any issues during deployments to continue with further ramps. Also, almost the entirety of what is called internal traffic https://phabricator.wikimedia.org/T333120, that is traffic that is generated by applications running in the infrastructure and reaching out to MediaWiki for various purposes is migrated to WikiKube. A special mention should go to MediaWiki Jobs fully migrated just a week ago https://phabricator.wikimedia.org/T349796. Feel free to follow the higher umbrella task https://phabricator.wikimedia.org/T290536 and/or MediaWiki_On_Kubernetes on Wikitech https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes.
This migration will unleash the ability to deploy multiple versions of code simultaneously. Also, this will help enhance platform capabilities to build dockerized isolated environments for coding, testing and even production debugging.
MediaWiki on Kubernetes will allow us to deprecate and eventually remove a lot of our in-house developed code. Another benefit is that we will be able to react to sudden traffic spikes, like newsworthy events better, as the flexing up and down is a matter of configuration change. This enables efficient placement of workloads, packing workloads in a more environmentally friendly way, increasing hardware utilization.
It takes a village to get this far: A big thanks to the Service Operations team (Clément Goubert, Giuseppe Lavagetto, Alexandros Kosiaris, Kamila Součková, Hugh Nowlan, Effie Mouzeli, Reuven Lazarus, Jannis Mayboom and Kavitha Appakayala) for their leadership on this project, the Release Engineering team (specifically: Dan Duvall, Jeena Huneidi, Tyler Cipriani and Ahmon Dancy) for their work on Blubber https://wikitech.wikimedia.org/wiki/Blubber, the Deployment Pipeline https://wikitech.wikimedia.org/wiki/Deployment_pipeline and Scap https://wikitech.wikimedia.org/wiki/Scap, Dom Walden from Quality and Test Engineering for crafting and executing a plan to test the first deployment of MediaWiki on Kubernetes, and everyone else who has contributed in the one way or the other! <3
More milestones:
We’re seeing the first lights at the end of the tunnel of another multi-year initiative: The *parser unification* https://www.mediawiki.org/wiki/Parsoid/Parser_Unification. One week ago, Parsoid Read Views was rolled out to first wikis: Parsoid is now the default read views renderer on the Foundations’ Office Wiki and Wikitech DiscussionTools. This early experimentation allows us to find issues in a limited space, which will help us evaluate readiness of the feature and increase our confidence for future rollouts http://mediawiki.org/wiki/Parsoid/Parser_Unification/Confidence_Framework. A huge thanks to Subbu Sastry, Mateus Santos, CScott, Isabelle Hubert-Pallatin, Arlo Breault, Shannon Bailey, Yiannis Giannelos and Sérgio Lopes for making this work! Many thanks also to Daniel Kinzler: His work on the RESTBase deprecation directly helped us get to this milestone :-)
*RESTBase deprecation*: We have been continuously working on decoupling services from RESTBase, aiming for the modernisation and sustainability of Wikimedia products in our services platform. The MediaWiki Interfaces team finished up reimplementation of Reading Lists endpoints in MW REST API https://phabricator.wikimedia.org/T348491 and are now confirming with affected callers https://phabricator.wikimedia.org/T357478 that the new endpoints meet their needs before rerouting calls https://phabricator.wikimedia.org/T348493 and retiring old code https://phabricator.wikimedia.org/T348494. The overall effort https://phabricator.wikimedia.org/T336693 will not only move us forward on RESTBase retirement, but also reduce the total amount of code we have to maintain. Many thanks to Bill Pirkle, Atieno Njira, Wendy Quarshie, and Daniel Kinzler for making this work! We also fully turned off Parsoid cache storage in RESTBase - clients will get outputs direct from MediaWiki and the cache will be handled by ParserCache. Next, we will re-route clients directly to MediaWiki and fully remove Parsoid from RESTBase (T344944 https://phabricator.wikimedia.org/T344944). The Page Content Service (PCS) will also handle its own cache https://phabricator.wikimedia.org/T348995 and we are ready to test the new capabilities in staging soon. Many thanks to Yiannis Giannelos and the Content Transform team and to Daniel Kinzler for his efforts and support on decoupling Parsoid from RESTBase!
All of these multi-year initiatives help us increase sustainability and maintainability of the platform, streamline engineering and developer workflows, and unlock the path for new and improved platform capabilities and product opportunities.
Outlook: Knowledge Platform in the annual plan 2024/2025
The Wikimedia Foundation has recently published the draft objectives by the Product & Technology department https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2024-2025/Goals/Infrastructure for the next annual plan on Meta, alongside an introduction by Selena Deckelmann and a few questions that we’re exploring https://meta.wikimedia.org/wiki/Talk:Wikimedia_Foundation_Annual_Plan/2024-2025/Goals/Infrastructure#Questions. Your input on these questions is very welcome!
The draft objectives include “Knowledge Platform I” - centered around MediaWiki platform evolution and “Knowledge Platform II” - centered around developer/engineering services and workflows. The objectives show only the high-level direction for next year. The draft “key results” (currently work in progress) will give a better idea of what areas of work we’re thinking about. We’ll be publishing these in March and share the link + invitation for feedback with this list again.
Thanks all for reading,
Birgit
-- Birgit Müller (she/her) Director of Product, MediaWiki and Developer Experiences
Wikimedia Foundation https://wikimediafoundation.org/ _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
wikitech-l@lists.wikimedia.org