Wikitech-l October 2022

wikitech-l@lists.wikimedia.org

49 participants
42 discussions

Standardizing exception handling in MediaWiki
by Daimona 26 Oct '22

26 Oct '22

Hi everyone, Today I created a task proposing that exception handling and annotations be standardized in MediaWiki: https://phabricator.wikimedia.org/T321683. This would affect all MediaWiki code written in PHP (core, extensions, and skins), and the standardization itself will require a lot of effort. Therefore, I invite you to share any questions, thoughts, or concerns that you may have on the task. Thank you, -- https://meta.wikimedia.org/wiki/User:Daimona_Eaytoy "Daimona" is not my real name -- he/him

1 0

maybe somebody can just review and approve this commit of script conversion?
by dinar qurbanov 24 Oct '22

24 Oct '22

hello can somebody help with reviewing technical aspects of a cyrillic <-> latin converter ? it has been said that the file is too big and the commit is too big. the code is nearly 3500 lines. i think it is not worth to divide it into separate library or files. as library it would be very fit for this conversion, not usable for other things. i feel making library as "premature optimisation". and laziness, feeling as useless work. also just dividing into files, into classes. 3500 lines is not very hard for me to browse. if anybody wants to separate it into classes and files, feel free to edit, if you think it is good for mediawiki. i do not want to make such change by myself. i can help with proper names (for classes, variables). i said variables because it is possible to convert some lists of php commands into for cycles with arrays, just to make it look shorter. i think it is also like premature optimisation, i have more freedom when they are as they are, in further editing. i do not want to make such change by myself. feel free to edit. quick link to the newest version of the code: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/164049/219/includes/langu… . also it was suggested to get rid of regex lookahead and lookbehind. and strtr was like suggested. i think i probably cannot easily make those changes. and i think, is strtr really faster than preg_replace? i googled for benchmark results and i do not see, in the first page, like 10 results. the code is big, but it just modifies a string. tatar wikipedians have asked me about this code, yesterday, and now i try this approach, to write to this mailing list, before other ways to go. maybe there is just a lack of some people who can review this code and approve it, and maybe this mailing list can help. other ways to go to the problem's pages: https://phabricator.wikimedia.org/T27537 , https://gerrit.wikimedia.org/r/c/mediawiki/core/+/164049 .

2 1

Infrastructure diagrams
by Krinkle 24 Oct '22

24 Oct '22

I've done a major update to a number of diagrams on Wikitech. Usually, I don't mention an update here, but I'm highlighting it now as it's been a while since we mentioned them on-list and the community and foundation have grown a lot so some of these may be new to you. Given how much has changed in recent changes, I also included a changelog and a link to where in the docs you'd normally discover this diagram on-wiki: *== 1. File:Wikipedia_webrequest_2022.png <https://wikitech.wikimedia.org/wiki/File:Wikipedia_webrequest_2022.png> (Updated) ==* This is a highly simplified diagram, covering the general shape of our stack through the example of a typical Wikipedia webrequest. ** Previous: https://upload.wikimedia.org/wikipedia/commons/b/b3/Wikipedia_webrequest_fl… New: https://upload.wikimedia.org/wikipedia/commons/4/4d/Wikipedia_webrequest_20… Documentation: wikitech:MediaWiki_at_WMF <https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF> and wikitech:Caching_overview <https://wikitech.wikimedia.org/wiki/Caching_overview>. Notable changes: * Change edge TLS termination ("HTTPS") from ats-tls to HAProxy. I wrote a "Caching overview § History <https://wikitech.wikimedia.org/wiki/Caching_overview>" section. * Change appserver TLS from Nginx- to Envoy. * Add new MainStash DB. * Include storage ExternalStore DB, ParserCache DB, and Swift media. * Include services Shellbox, Mathoid, and Kask. *== 2. File:WMF_infrastructure_2022.png <https://wikitech.wikimedia.org/wiki/File:WMF_infrastructure_2022.png> *(Updated) ==** This is a continous attempt at an overview of tier-1/user-facing infrastructure. It will likely never be complete from all POV, but.. it is more accurate and complete than it has been. Thanks to all that contributed by entertaining my many questions over the years. **** Previous (2016 by Elukey): https://upload.wikimedia.org/wikipedia/labs/4/4d/Infrastructure_overview.png New: https://upload.wikimedia.org/wikipedia/commons/4/48/WMF_infrastructure_2022… Documentation: wikitech:Wikimedia_infrastructure <https://wikitech.wikimedia.org/wiki/Wikimedia_infrastructure> and wikitech:Purged <https://wikitech.wikimedia.org/wiki/Purged> Notable changes: * Add new Drmrs data center in Marseille, France. * Add new services: purged.go, EventStreams, Thumbor, mcrouter, Envoy, etcd. * Add new distinction for Multi-DC between primary and secondary data center. * Change sessionstore from Redis to Kask/Cassandra. * Change jobqueue from Redis to EventGate/Kafka. * Include distinct MediaWiki server roles and clusters. * Include high-level MediaWiki platform components. * Include example flow for "JobQueue job" and "CDN purge". *== 3. File:MediaWiki_infrastructure_2022.png <https://wikitech.wikimedia.org/wiki/File:MediaWiki_infrastructure_2022.png> (New) ==* Similar to WMF Infra diagram, but more abstract around DC and services, and more detailed within the platform. Including more core services, and recognising extensions as their own layer. ** New: https://upload.wikimedia.org/wikipedia/commons/e/ee/MediaWiki_infrastructur… Documentation: wikitech:MediaWiki_at_WMF <https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF> *== 4. File:Wikipedia_Memcached_flow_2022.png <https://wikitech.wikimedia.org/wiki/File:Wikipedia_Memcached_flow_2022.png> *(Updated)** **** Previous: https://upload.wikimedia.org/wikipedia/commons/d/db/Wikipedia_Memcached_flo… New: https://upload.wikimedia.org/wikipedia/commons/4/45/Wikipedia_Memcached_flo… Documentation: wikitech:Memcached_for_MediaWiki <https://wikitech.wikimedia.org/wiki/Memcached_for_MediaWiki> Notable changes: * Include the three tiers of ParserCache. * Add WANCache legend to explain different keytypes you may encounter on the network. * Add full name of the mcrouter-with-onhost-tier service for greppability. * Add new WRStats service (T310662 <https://phabricator.wikimedia.org/T310662>). This was part of Multi-DC work <https://www.mediawiki.org/wiki/Wikimedia_Performance_Team/Multi-DC_MediaWiki> to reduce primary DB writes and (not bi-di replicated) Redis use in AbuseFilter. This service also replaces the old "User ping limiter" in core and is now able to serve both use cases. * Remove "on-host: soon" labels. Adopting on-host memc for WANCache was considered not worth the added runtime complexity (T264604 <https://phabricator.wikimedia.org/T264604>). Note that SRE's work on adding 10G network links for memcached hosts, and the addition of mcrouter-managed gutter pools take care of the general usecase that we were exploring on-host for. We kept it for ParserCache however (T244340 <https://phabricator.wikimedia.org/T244340>. *== Edit link ==* As before, each diagram file page has an "Edit" link in the description that takes you directly to the open-source Diagrams.net web app (loading file read-only from Google Drive). You can fork by using "Save as" in the web app. See also: wikitech:Performance/Runbook/diagrams.net_conventions <https://wikitech.wikimedia.org/wiki/Performance/Runbook/diagrams.net_conven…> -- Timo

5 4

[Train] 1.40.0-wmf.6 status update
by Antoine Musso 18 Oct '22

18 Oct '22

Hello, The 1.40.0-wmf.6 version of MediaWiki is blocked (T320511 <https://phabricator.wikimedia.org/T320511>). Upon deployment to group0 wikis I found out CheckUser to be broken: Unknown column 'cuc_private' in 'field list' I have filed it as T321041 <https://phabricator.wikimedia.org/T321041>. I am guessing CheckUser should be taught to not insert the cuc_private field unless needed and long term the schema change should be applied to all databases. I have thus blocked the train entirely until the issue get resolved. Antoine "hashar" Musso Wikimedia Release Engineering

1 1

🚂🌈Summary of 1.40.0-wmf.5 train deployment
by Dan Duvall 17 Oct '22

17 Oct '22

This email is a summary of the Wikimedia production deployment of 1.40.0-wmf.5 - Conductor: Dan Duvall - Backup Conductor: Chad Horohoe (graciously deployed to group0; thanks!) - Blocker Task: T314194 <https://phabricator.wikimedia.org/T314194> - Current Status <https://versions.toolforge.org/> 🔢 Stats Sparklines comparing the last 5 trains. - 337 Patches ▄▁▂▇█ - 0 Rollbacks █▁▄▄▁ - 0 Days of delay █▁█▁▁ - 1 Blockers ▄▃▁█▁ 😻 Trainbow Love ✨Thanks to folks who reported or resolved blockers: - Taavi Väänänen - Kosta Harlan -- Dan Duvall Staff Software Engineer, Release Engineering Wikimedia Foundation <http://wikimediafoundation.org>

1 0

Re: TDF is looking for community representatives
by Brian Wolff 15 Oct '22

15 Oct '22

So I see that User:Grunny, User:Rtnf and Dessalegn Yehuala were appointed. Respectfully, given the criteria was someone with +2 rights, I would question the decision to appoint Dessalegn Yehuala, who not only has never contributed to MediaWiki, but also appears to have never edited any Wikimedia project, or even have an account. Brian On Thu, Oct 6, 2022 at 1:58 AM Technical Decision Forum Support < TDFsupport(a)wikimedia.org> wrote: > Hi Brian, > > We have our representatives but we are always looking for some more as we > rotate throughout the year. My apologies for the late updates to the > forum. The lists are here > <https://www.mediawiki.org/wiki/Technical_decision_making/Forum>. The > community representatives have completed one feedback to the IP Masking > Proposal and next week we should have "Modern user interfaces for all > users" proposal out to the representatives for their feedback. You can see > it on the TDF <https://phabricator.wikimedia.org/project/board/5179/> > board. > > Thank you, > TDF > > > On Wed, Oct 5, 2022, 5:46 PM Brian Wolff <bawolff(a)gmail.com> wrote: > >> Out of curiosity, what happened? Aug 12 has come and gone. More to the >> point, i was wondering if TDF is even still a thing? >> >> -- >> Brian >> >> On Tuesday, August 2, 2022, Erica Litrenta <elitrenta(a)wikimedia.org> >> wrote: >> >>> Hi everyone. >>> >>> This is a reminder that the search for community representatives for the >>> Technical Decision Forum is still OPEN. >>> >>> There are a few weeks left to apply and/or recommend someone who would >>> be a great fit for the role (learn more at >>> https://www.mediawiki.org/wiki/Technical_decision_making/Community_represen… >>> ). >>> >>> Please let us know if there are any questions we can answer before you >>> submit your application, or a friend's name! Email us at >>> TDFSupport(a)wikimedia.org by Aug 12, 2022. >>> >>> Looking forward to hearing from you soon… >>> >>> The Technical Decision Forum. >>> >>> On Tue, Jul 12, 2022 at 11:17 AM Erica Litrenta <elitrenta(a)wikimedia.org> >>> wrote: >>> >>>> *(This email may not be written in your native language, and/or may be >>>> a crosspost: our apologies.)* >>>> >>>> >>>> Greetings, everyone. >>>> >>>> The Technical Decision Forum, or TDF (1), is seeking community >>>> representatives. Please visit the TDF Community Representation (2) >>>> page for more information, and email TDFSupport(a)wikimedia.org by Aug >>>> 12, 2022 to be considered for selection. >>>> >>>> >>>> Don't worry about the requirements listed on wiki, they are not set in >>>> stone; just provide your name, name of the group you "represent" if any, >>>> and a short sentence or two explaining your interest. >>>> >>>> >>>> For any questions, please post at *MediaWiki.org (3)*, or on the >>>> related Movement Strategy Forum topic (4), or simply email >>>> TDFSupport(a)wikimedia.org. >>>> >>>> Thanks, >>>> >>>> The Technical Decision Forum. >>>> >>>> (1) https://www.mediawiki.org/wiki/Technical_decision_making/Forum >>>> (2) >>>> https://www.mediawiki.org/wiki/Technical_decision_making/Community_represen… >>>> (3) https://www.mediawiki.org/wiki/Talk:Technical_decision_making/Forum >>>> (4) >>>> https://forum.movement-strategy.org/t/technical-decision-forum-tdf-is-looki… >>>> >>>> -- >>>> >>>> ------------------------------ >>>> >>>> >>>> Erica Litrenta (she/her) >>>> >>>> Senior Manager, Community Relations Specialists (Product) >>>> >>>> Wikimedia Foundation >>>> <https://meta.wikimedia.org/wiki/User:Elitre_(WMF)> >>>> >>>> >>> >>> -- >>> >>> ------------------------------ >>> >>> >>> Erica Litrenta (she/her) >>> >>> Senior Manager, Community Relations Specialists (Product) >>> >>> Wikimedia Foundation <https://meta.wikimedia.org/wiki/User:Elitre_(WMF)> >>> >>> >>>

12 16

Re: Request to post on mailing list.
by Asif Rahaman 14 Oct '22

14 Oct '22

Hi. I recently contacted Wikimedia and subscribe for the mailing list. I would like to post on your mailing list. Regards Asif Rahaman Btech second year student (IIT Bhilai) On Fri, Oct 14, 2022, 20:27 Asif Rahaman <asifrahaman162(a)gmail.com> wrote: > Hi. > I recently contacted Wikimedia and subscribe for the mailing list. I would > like to post on your mailing list. > > Regards > Asif Rahaman > Btech second year student (IIT Bhilai) >

3 2

ParserTests updates
by Subramanya Sastry 13 Oct '22

13 Oct '22

Hello everyone, The Content Transform Team <https://www.mediawiki.org/wiki/Content_Transform_Team> finally decided to act on this old task <https://phabricator.wikimedia.org/T111604> to split up the parserTests.txt file in core into a number of smaller files by functionality. We got a first patch merged and there is one more patch in gerrit that will likely merge this week. More patches will follow in the coming weeks and we expect that by the time we are done, we'll have a few 10s of parser test files which will all be significantly smaller than the parserTests.txt file now. So, when some of you write code that needs an update to parser tests,in some cases, you may have to update tests in multiple files. While I have your attention here, a few more parserTests related updates: * Over the last 6 months, we've updated MediaWiki core's parser test runner to run parser tests against Parsoid (both via CLI and in CI via PHPUnit). We call these tests integrated mode Parsoid test runs to contrast it with standalone mode Parsoid test runs (Core's test runner runs in integrated mode with the MediaWiki core code whereas Parsoid's test runner runs in standalone mode without using MediaWiki core code). * I've also been updating the parser tests wiki page <https://www.mediawiki.org/wiki/Parser_tests> to bring the documentation up to date wrt how to write tests for running with Parsoid. We will continue to improve this page and add more details as necessary. Feel free to edit the page or leave questions / comments on the talk page with requests for clarification. * We have also started enabling test runs with Parsoid in extension repositories as well as in core. If you see a 'parsoid-compatible' clause in a test file, that test file is going to be run with Parsoid. We want to make sure as we move further along with the integration of Parsoid with the rendering pipeline, we catch any newly-introduced regressions so they can be fixed. Thanks! Subbu on behalf of the Content Transform Team.

1 0

Request for feedback: Two changes related to databases in MediaWiki
by Amir Sarabadani 12 Oct '22

12 Oct '22

Hello, Two changes is being made on MediaWiki databases: *First:* We are reworking how extensions manage their database schema. The idea is to basically deprecate LoadExtensionSchemaUpdates hook[1] and replace it with structured values in extension.json. This is needed to avoid arbitrary php code execution during database update which can cause problems and was a contributing factor in one of our major outages [2]. The other reason for this change is to be able to wire abstract schema of extensions to core so it can be tested in CI (for example for having a valid json, having prefixes set correctly, etc.) You can see a POC of that in [3] Given that once the change is merged to core, breaking changs to extension registry schema would require a lot of work (updating the maniphest version and so on), it's better to get it right before merge. I have proposed something like [4]. For any feedback on this, please go to https://phabricator.wikimedia.org/T237839, *Second:* We are working to provide a replacement for raw SQL for query conditions. When the query condition is exact equality condition (e.g. rev_id = 123), you don't need to provide raw SQL but for most other cases you need to either provide it or build it via IDatabase methods. For example: 'rev_timestamp > ' . $db->addQuotes( $db->timestamp( $since ) ). You can see an example of this in [5] (being changed to something much better via a newly introduced method). This is not optimal and not very developer friendly, to solve that, we are proposing to have something like ExpressionBuilder::greater( 'rev_timestamp', $db->timestamp( $since ) ) supported in SelectQueryBuilder::where(). You can see the POC in [6]. For any feedback on this please go to https://phabricator.wikimedia.org/T210206 [1] https://www.mediawiki.org/wiki/Manual:Hooks/LoadExtensionSchemaUpdates [2] https://wikitech.wikimedia.org/wiki/Incidents/2020-04-07_Wikidata%27s_wb_it… [3] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/816262/10/tests/phpunit/s… [4] https://phabricator.wikimedia.org/T237839#8268173 [5] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/832625/6/includes/api/Api… [6] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/841940/ Thank you -- *Amir Sarabadani (he/him)* Staff Database Architect Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

🌱 Fresh 22.09 released!
by Krinkle 11 Oct '22

11 Oct '22

This release promotes Node.js 16 to be the default runtime in fresh-node. Get started by installing, updating, or learning more, at: https://gerrit.wikimedia.org/g/fresh#fresh-environment Changelog: https://gerrit.wikimedia.org/g/fresh/+/22.09.1/CHANGELOG.md Node.js 16 is now the default environment for the fresh-node command. This addresses an issue with Vue.js development using Vite 3, which did not work well under Node.js 14. (Thanks Lucas Werkmeister, T314051 <https://phabricator.wikimedia.org/T314051>). Note that most projects still test on Node.js 14 in WMF CI. [1] If you find that some of your tests don't yet pass under Node.js 16, fret not! Node.js 14 and 12 remain available via the bundled fresh-node14 and fresh-node12 commands. If you encounter problems in Fresh with Node.js 16 or experience other issues, let us know on Phabricator at https://phabricator.wikimedia.org/tag/fresh/. This is also where you can browse previous tasks. Fresh is a fast way to create isolated environments from your terminal. These can be used to work more responsibly with 'npm' developer tools such as ESLint, QUnit, Grunt, Selenium, and more. Example guide: https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing -- Timo Tijhof [1] https://phabricator.wikimedia.org/T314470

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l October 2022