Wikitech-l August 2023

wikitech-l@lists.wikimedia.org

33 participants
37 discussions

ORES To Lift Wing Migration
by Chris Albon 27 Sep '23

27 Sep '23

Hi everybody, TL;DR We would like users of ORES models to migrate to our new open source ML infrastructure, Lift Wing, within the next five months. We are available to help you do that, from advice to making code commits. It is important to note: All ML models currently accessible on ORES are also currently accessible on Lift Wing. As part of the Machine Learning Modernization Project ( https://www.mediawiki.org/wiki/Machine_Learning/Modernization), the Machine Learning team has deployed a Wikimedia’s new machine learning inference infrastructure, called Lift Wing ( https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing). Lift Wing brings a lot of new features such as support for GPU-based models, open source LLM hosting, auto-scaling, stability, and ability to host a larger number of models. With the creation of Lift Wing, the team is turning its attention to deprecating the current machine learning infrastructure, ORES. ORES served us really well over the years, it was a successful project but it came before radical changes in technology like Docker, Kubernetes and more recently MLOps. The servers that run ORES are at the end of their planned lifespan and so to save cost we are going to shut them down in early 2024. We have outlined a deprecation path on Wikitech ( https://wikitech.wikimedia.org/wiki/ORES), please read the page if you are a maintainer of a tool or code that uses the ORES endpoint https://ores.wikimedia.org/). If you have any doubt or if you need assistance in migrating to Lift Wing, feel free to contact the ML team via: - Email: ml(a)wikimedia.org - Phabricator: #Machine-Learning-Team tag - IRC (Libera): #wikimedia-ml The Machine Learning team is available to help projects migrate, from offering advice to making code commits. We want to make this as easy as possible for folks. High Level timeline: **By September 30th 2023: *Infrastructure powering the ORES API endpoint will be migrated from ORES to Lift Wing. For users, the API endpoint will remain the same, and most users won’t notice any change. Rather just the backend services powering the endpoint will change. Details: We'd like to add a DNS CNAME that points ores.wikimedia.org to ores-legacy.wikimedia.org, a new endpoint that offers a almost complete replacement of the ORES API calling Lift Wing behind the scenes. In an ideal world we'd migrate all tools to Lift Wing before decommissioning the infrastructure behind ores.wikimedia.org, but it turned out to be really challenging so to avoid disrupting users we chose to implement a transition layer/API. To summarize, if you don't have time to migrate before September to Lift Wing, your code/tool should work just fine on ores-legacy.wikimedia.org and you'll not have to change a line in your code thanks to the DNS CNAME. The ores-legacy endpoint is not a 100% replacement for ores, we removed some very old and not used features, so we highly recommend at least test the new endpoint for your use case to avoid surprises when we'll make the switch. In case you find anything weird, please report it to us using the aforementioned channels. **September to January: *We will be reaching out to every user of ORES we can identify and working with them to make the migration process as easy as possible. **By January 2024: *If all goes well, we would like zero traffic on the ORES API endpoint so we can turn off the ores-legacy API. If you want more information about Lift Wing, please check https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing Thanks in advance for the patience and the help! Regards, The Machine Learning Team

11 31

[Wikitech-I] Call for Projects and Mentors for Outreachy Round 27!
by Onyinyechi Onifade 20 Sep '23

20 Sep '23

Hello everyone, TLDR; Wikimedia is participating in the Outreachy Round 27 internship program <https://www.mediawiki.org/wiki/Outreachy/Round_27 > [1]. Outreachy's goal is to support people from groups underrepresented in the technology industry. Interns will work remotely with mentors from our community. We are seeking mentors to propose projects that Outreachy interns can work on during their internship. If you have some ideas for coding or non-coding (design, documentation, translation, outreach, research) projects, share them by Sept. 29, 2023 at 4 pm UTC here as a subtask of this parent task: <https://phabricator.wikimedia.org/T343871 > [2] Program Timeline As a mentor, you engage potential candidates in the application period between October–November (winter round) and help them make small contributions to your project. You work more closely with the accepted candidates during the internship period between December–March (winter round). Important dates are: - Aug. 22, 2023 at 4pm UTC - Live Q&A for Outreachy mentors <https://www.youtube.com/@outreachyinternships> - September 29, 2023 at 4pm UTC - Project submission deadline <https://www.outreachy.org/communities/cfp/wikimedia/> Guidelines for Crafting Project Proposals * Follow this task description template when you propose a project in Phabricator: < https://phabricator.wikimedia.org/tag/outreach-programs-projects> [3]. You can also use this workboard to pick an idea if you don't have one already. Add #Outreachy (Round 27) tag. * Project should require an experienced developer ~15 days and a newcomer ~3 months to complete. * Each project should have at least two mentors, including one with a technical background. * Ideally, the project has no tight deadlines, a moderate learning curve, and fewer dependencies on Wikimedia's core infrastructure. Projects addressing the needs of a language community are most welcome. Learn more about the roles and responsibilities of mentors on MediaWiki.org: <https://www.mediawiki.org/wiki/Outreachy/Mentors> [4][5] Cheers, Onyinye & Sheila (Wikimedia Org Admins for Outreachy Round 27) [1] https://www.mediawiki.org/wiki/Outreachy/Round_27 [2] https://phabricator.wikimedia.org/T343871 [3] https://phabricator.wikimedia.org/tag/outreach-programs-projects/ [4] https://www.mediawiki.org/wiki/Outreachy/Mentors [ 5] https://www.outreachy.org/mentor/mentor-faq

3 4

Attempt to deprecate the mediawiki.revision-score stream
by Luca Toscano 18 Sep '23

18 Sep '23

Hi everybody, In https://phabricator.wikimedia.org/T342116 the Machine Learning team announces its intention to deprecate the mediawiki.revision-score stream. For external users, the stream is consumable via the https://stream.wikimedia.org API and it currently has very few users. Our idea is to create smaller streams, one for each model type, instead of having a big aggregator. For example, revision 123456 for enwiki ends up with several scores from various models in the current revision-score stream, that is convenient but very hard to manage and maintain for us (since it is not clear if users are interested in all the data or only a subset of it). The revision-score stream is also very tightly coupled with the ORES' architecture, which we are trying to deprecate. In the future we plan to have smaller streams, in which every revision will get associated with a single score, from a specific model server: mediawiki.revision-score-goodfaith mediawiki.revision-score-damaging ... ... [ and also new models that will be deployed. ] To avoid creating unnecessary streams, we'll create the ones that WMF teams and the community will need and ask during the next months. If you have any requirement, please follow up with us: - Email: ml(a)wikimedia.org - Phabricator: #Machine-Learning-Team tag - IRC (Libera): #wikimedia-ml If you are a user of the Mediawiki revision-score stream please follow up on the task above explaining your use case, we'll try to do our best to find a good solution for you! Thanks in advance, Regards, Luca

1 2

Frontend stable policy: Making it official
by Jon Robson 16 Sep '23

16 Sep '23

Dear fellow developers! If you don't work on gadgets or Wikimedia code, feel free to ignore this email! For some time we've had the Stable interface policy which has been super helpful for backend-development. I would love us to have an equivalent for frontend code. For the past 3 years we have been building one with feedback and suggestions from gadget developers, WMF staff and Wikimedia volunteers. The current draft can be found at: https://www.mediawiki.org/wiki/User:Jdlrobson/Stable_interface_policy/front… I would like to make this policy official so that we can get the benefits of having a document and continue to evolve it in a more official capacity. If anyone wants to veto this, I'd like to hear from you on the talk page or by a reply to this email (either privately or publicly). When making a veto please make that explicit and include the text you find problematic and details about why. If there is no active veto after one month, this policy will be made official and moved to https://www.mediawiki.org/wiki/Stable_interface_policy/frontend. Thanks in advance for all your help with this important matter! Jon Robson PS. This note has also been sent to tech news.

1 1

ORES extension: Switching backend to Lift Wing for fiwiki and itwiki
by Ilias Sarantopoulos 12 Sep '23

12 Sep '23

Hi everybody! A couple of weeks ago we started switching the backend used by the ORES extension which is used by the Recent Changes filters. After fixing the issue that was caused in some wikis ( https://phabricator.wikimedia.org/T343308) the Machine Learning team started rolling out the change again. We have already deployed the changes to *fiwiki* and *itwiki* and will continue with the rest of the wikis starting from next week. This change is using Lift Wing ( https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing) to get revision scores instead of ORES, and is a necessary step in the process of deprecating ORES (for more info please refer to the "ORES to Lift Wing Migration <https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/…>" email). The purpose of this change is to use *exactly* the same models which are also deployed on Lift Wing so there shouldn't be anything different for our users. If however you see anything out of the ordinary, feel free to contact the Machine Learning team: IRC libera: #wikimedia-ml Phabricator: Machine-Learning-team tag Thank you! Ilias (on behalf of the Machine Learning team)

2 2

Datacenter Switchover process change
by Alexandros Kosiaris 31 Aug '23

31 Aug '23

Hello, I 'd like to inform everyone of what we consider a big change (for the better) in the Datacenter Switchover process. The full rationale, planning and implementation is documented at https://wikitech.wikimedia.org/wiki/Switch_Datacenter/Recurring,_Equinox-ba… and it includes a TL;DR that I am pasting below for everyone's convenience: Site Reliability Engineering will, starting September 2023, run a data center Switchover every 6 months, in the week of the solar Equinox <https://en.wikipedia.org/wiki/Equinox>, namely the *work weeks containing March 21st and September 21st*. If you are interested to learn more about Switchovers and why we perform them, or already know what they are and want to learn more about how this proposal would impact your workflows or the Wikimedia Movement, please read on. We hope that making the Switchover dates and duration predictable will allow the teams involved and/or utilizing a Switchover, as well as the entire movement reap the benefits we anticipate and document in the doc linked above. Regards, -- Alexandros Kosiaris Principal Site Reliability Engineer Wikimedia Foundation

1 0

Moving Wikimedia Developer Account creation
by Simon Lyngshede 31 Aug '23

31 Aug '23

Hi all, We have removed the developer account signup from Wikitech and are now directing new users towards https://idm.wikimedia.org instead. This is the first step in the implementation of our new identity manager, and a small step towards making Wikitech a little less special. Part of our documentation was updated yesterday, August 30th, and the first accounts have already been successfully created. Should there be a need to re-enable the account creation process on Wikitech, this can be done by reverting: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/952209/ If you have any questions, concerns or documentation that needs updating, please reach out to me or the infrastructure foundation team. Thank you to everyone who helped us get this far. -- Simon Lyngshede Site Reliability Engineer / Infrastructure Foundations Wikimedia Foundation

1 1

September 2023 Datacenter Switchover
by Kamila Součková 31 Aug '23

31 Aug '23

Dear Wikitechians, On *Wednesday September 20th*, the SRE team will run a planned data center switchover, moving all wikis from our data center in Virginia to the data center in Texas. This is an important periodic test of our tools and procedures, to ensure the wikis will continue to be available even in the event of major technical issues in our primary home. It also gives all our SRE and ops teams a chance to do maintenance and upgrades on systems in Virginia that normally run 24 hours a day. The switchover process requires a *brief read-only period for all Foundation-hosted wikis*, which will start at *14:00 UTC on Wednesday **September 20th*, and will last for a few minutes while we execute the migration as efficiently as possible. All our public and private wikis will be continuously available for reading as usual, but no one will be able to save edits during the process. Users will see a notification of the upcoming maintenance, and anyone still editing will be asked to try again in a few minutes. CommRel will soon begin notifying communities of the read-only window. If you like, you can follow along on the day in the public #wikimedia-operations channel on IRC (instructions for joining here <https://meta.wikimedia.org/wiki/IRC/Instructions>). To report any issues, you can reach us in #wikimedia-sre on IRC, or file a Phabricator ticket with the *datacenter-switchover* tag (pre-filled form here <https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=Data…>); we'll be monitoring closely for reports of trouble during and after the switchover. (If you're new to Phab, there's more information at Phabricator/Help.) The switchover and its preparation will be tracked in Phabricator task T345263 <https://phabricator.wikimedia.org/T345263>. On behalf of the SRE team, please excuse the disruption, and our thanks to everyone in a number of departments who've been involved in planning this work. Feel free to reply directly to me with any questions. Thank you, Kamila Součková (they/them) Senior SRE Wikimedia Foundation

1 0

Global Lua modules and templates
by Felipe Schenone 30 Aug '23

30 Aug '23

Hi! As many of you know, a central global repository for Lua modules and templates has been a frequent request since the early days of the movement. This year, I programmed a JavaScript tool called Synchronizer https://www.mediawiki.org/wiki/Synchronizer (inspired on a previous tool called DiBabel by User:Yurik) The tool allows to synchronize (that is, automatically copy) Lua modules across Wikimedia wikis, and provides other features to help developers update and maintain global modules. I also re-wrote the documentation at https://www.mediawiki.org/wiki/Multilingual_Templates_and_Modules to account for the new tool. It basically describes how to develop a Lua module that can be copied unchanged to any wiki, by abstracting things like user-readable strings and config. Admittedly, this is a "poor man's version" of a proper solution to the problem, but one I find invaluable while developing and maintaining modules. Hopefully some of you may find it useful too!

3 4

Help needed in finding a replacement to the mobile-friendly page content Rest API for my web app
by harsh410＠hotmail.com 30 Aug '23

30 Aug '23

Help needed in finding a replacement to the mobile-friendly page content Rest API for my web app Hello All, For my website webnomads.in I have been using mobile-friendly page content Rest API for years . Now that it has been stopped I am struggling to find a new suitable API as I was displaying the content in sections . On various pages I read about making use of parsoids . While I have explored the different API offerings , I could not find any API which is delivering the text based on sections . I will be very thankful if someone can guide me or point me to any article showing the use of Parsoid or any other way to deliver page content in text form and in sections . Thanks in advance . Regards Harsh

3 2

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l August 2023