Wikitech-l May 2023

wikitech-l@lists.wikimedia.org

48 participants
40 discussions

Maps as article image?
by Strainu 26 May '23

26 May '23

Hey folks, The maps and aticle image have been good additions to the multimedia capabilities in Wikipedia in the last decade, widely used on my home wiki. For now, in Wikipedia the maps are also rendered as images and become interactive only when clicked on. This makes them potential candidates for being displayed as article images in article without a photo. I would like to find out how complicated it would technically be to make the article image extension also capable of using map images? I know this is probably nowhere on the roadmap, I'm only interested in the technical part of the idea. Thanks, Strainu

2 1

Outdated codesearch results since 2023-05-16
by Kunal Mehta 24 May '23

24 May '23

Hi, For about a week MediaWiki Codesearch was serving outdated and incomplete results (and later partially down outright) because of a full disk. This has been fixed now, you may wish to re-run searches as necessary. The full disk was accidentally caused by switching between gerrit and gerrit-replica (as previously discussed on this list)[1][2], so I deleted all the old gerrit-replica repository copies and also added ~20G more disk space to the instance. Monitoring did correctly pick up that there was some issue once the disk was full, it just wasn't properly examined because of hackathon travel / activities. [1] https://phabricator.wikimedia.org/T336710#8877525 [2] https://phabricator.wikimedia.org/T337263 -- Kunal / Legoktm

1 0

svg translation on Commons
by Igal Khitron 24 May '23

24 May '23

Hello. I tried to start an svg translation. Added just a couple of switch statements, just to see how it works before the rest job. But it doesn't. Could you tell me, please, what am I doing wrong? Thank you. Igal (User:IKhitron) https://commons.wikimedia.org/w/index.php?title=File%3AMoscow_metro_map_mul…

2 2

Gerrit replica downtime (up to 2 hours) on Thu, 25 May
by Daniel Zahn 22 May '23

22 May '23

Hello The read-only Gerrit replica[0] will be down for up to 2 hours in 2 days (Thu, 25 May 2023) between 15:00–17:00 UTC[1] due to a scheduled OS upgrade [2]. During this window, git reads from the replica will not work. To my knowledge, this affects bots which rely on the replica for git read operations. Codesearch is currently using gerrit.wikimedia.org and not gerrit-replica.wikimedia.org so should not be affected. If this affects you yoy may consider switching to gerrit.wikimedia.org instead of the replica. Apologies for any inconvenience. Daniel [0]: <https://wikitech.wikimedia.org/wiki/Gerrit/Replica> [1]: <https://zonestamp.toolforge.org/1685026803> [2]: <https://phabricator.wikimedia.org/T334521> -- Daniel Zahn <dzahn(a)wikimedia.org> Site Reliability Engineer

1 0

Change to default value of install.php --scriptpath
by Gergő Tisza 21 May '23

21 May '23

Hi! The default value of the installer's script path parameter was changed in gerrit 133222 <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/133222> to make the installation process more intuitive. If you use the installer in your scripts, you might want to make sure that script path is set explicitly. (To keep the old behavior, add "--scriptpath wiki".)

1 0

Code of Conduct Committee -- call for new members
by Martin Urbanec 20 May '23

20 May '23

Hello all, The Code of Conduct Committee is a team of five trusted individuals (plus five auxiliary members) with diverse affiliations responsible for general enforcement of the Code of conduct for Wikimedia technical spaces. Committee members are in charge of processing complaints, discussing with the parties affected, agreeing on resolutions, and following up on their enforcement. For more on their duties and roles, see https://www.mediawiki.org/wiki/Code_of_Conduct/Committee. This is a call for community members interested in volunteering for appointment to this committee. Volunteers serving in this role should be experienced Wikimedians or have had experience serving in a similar position before. The current committee is doing the selection and will research and discuss candidates. Six weeks before the beginning of the next Committee term, meaning July 16, 2023, they will publish their candidate slate (a list of candidates) on-wiki. The community can provide feedback on these candidates, via private email to the group choosing the next Committee. The feedback period will be two weeks. The current Committee will then either finalize the slate, or update the candidate slate in response to concerns raised. If the candidate slate changes, there will be another two week feedback period covering the newly proposed members. After the selections are finalized, there will be a training period, after which the new Committee is appointed. The current Committee continues to serve until the feedback, selection, and training process is complete. If you are interested in serving on this committee or like to nominate a candidate, please write an email to techconductcandidates AT wikimedia.org with details of your experience on the projects, your thoughts on the code of conduct and the committee and what you hope to bring to the role and whether you have a preference in being auxiliary or main member of the committee. The committee consists of five main members plus five auxiliary members and they will serve for a year; all applications are appreciated and will be carefully considered. The deadline for applications is *the end of day on May 31 2023*. Please feel free to pass this invitation along to any users who you think may be qualified and interested. Best, Martin Urbanec, on behalf of the Code of Conduct Committee

1 0

[Train] 1.41.0-wmf.9 status update: blocked at group1 on T336962, T336964
by Brennen Bearnes 19 May '23

19 May '23

The 1.41.0-wmf.9 version of MediaWiki is blocked[0]. The new version is currently deployed to group1, but can proceed no further until these issues are resolved: * T336962 - UnexpectedValueException: Unknown image suggestions API kind: istype-depicts - https://phabricator.wikimedia.org/T336962 * T336964 - InvalidArgumentException: Data for lt_namespace and lt_title must be non-empty - https://phabricator.wikimedia.org/T336964 Once these issues are resolved train can resume. Thank you for any help resolving these issues! -- Your disgruntled train drivers [0]. https://phabricator.wikimedia.org/T330215 [1]. https://versions.toolforge.org/

2 2

+2 for Lucas Werkmeister [volunteer] in mediawiki/
by Lucas Werkmeister 19 May '23

19 May '23

Hi all, at the ongoing Hackathon, Legoktm made me nominate myself for +2 in mediawiki/* repositories ^^ see <https://phabricator.wikimedia.org/T337014>. Cheers, Lucas

3 2

Re-introducing MediaWiki CLI profiler for maintenance scripts
by Krinkle 17 May '23

17 May '23

TLDR: Those with shell access at WMF can now run maintenance scripts on mwdebug hosts, and use the *--profiler=text* option to produce a report detailing how long the script spent in each MediaWiki component, class, and function. == *What* == The MediaWiki platform at WMF consists of broadly four different deployments: app servers, api_app servers, job runners, and maint servers (diagram <https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF#/media/File:MediaWiki_…>). Each can be thought of as its own service for a specific purpose, composed of a subset of components from the MediaWiki codebase with fast local access to each. The largest of these is the appservers cluster, which is served by 150 servers of dedicated hardware across the two application data centers <https://wikitech.wikimedia.org/wiki/Data_centers>, and is responsible for responding to index.php (i.e. page views) and load.php (CSS, JS, and localisation via ResourceLoader). Today, we focus on the smallest of these: mwmaint servers <https://wikitech.wikimedia.org/wiki/Maintenance_server>. This is backed by two heavy-duty servers, one in each data center, that autonomously run essential tasks at a predefined schedule (i.e. not in direct response to a user action). Each of these ~50 different tasks is implemented as a MediaWiki maintenance script. Important examples include: sending email notifications (Echo extension), timely pruning of sensitive PII (CheckUser extension), computing mentee and link recommendation data (GrowthExperiments), and reclaiming disk space for expired caches (core/ParserCache). == *Why ==* We have detailed debug performance profiling in production for web requests via the WikimediaDebug extension, and we have detailed profiling in local development for both web requests and maintenance scripts (Docker recipe <https://www.mediawiki.org/wiki/MediaWiki-Docker/Configuration_recipes/Profi…>). What was missing is a way to profile maintenance scripts in production. This is important as maintenance scripts tend take many minutes or hours to process vast amounts of production data. While generally easy to debug locally for functional analysis, the performance bottlenecks individual teams care about are likely specific to the size of the data and the performance of other production components. Thanks also to Ahmon Dancy (RelEng), Giuseppe Lavagetto (SRE), and Aaron Schulz (Performance Team) for making this work possible, and Niklas Laxström (LangEng) for coming up with the idea. == *What's New* == Documentation: https://wikitech.wikimedia.org/wiki/WikimediaDebug#Plaintext_CLI_profile To profile a Maintenance script, run the script from the shell with *mwscript* as you normally would, but instead of connecting your terminal to mwmaint1002.eqiad.wmnet, connect to one of the *mwdebug* hosts (such as mwdebug1001.eqiad.wmnet). Then pass the *--profiler=text* option to generate a report with the performance analysis, which will be printed after the task is finished. Like so: > $ mwscript showSiteStats.php --wiki=nlwiki --profiler=text > Number of articles: 2122688 > Number of users : 1276507 > >  == *A peak behind the curtain* == Read on if you'd like to learn what hurdles we had to overcome for this to "simply" work in production, like it did for local development. The journey started when Niklas (WMF LangEng) proposed the idea at https://phabricator.wikimedia.org/T253547. *Firstly*, the profiler engine. In 2019 (blogpost <https://techblog.wikimedia.org/2019/12/16/wikimediadebug-v2-is-here/>), after we migrated from HHVM to PHP 7, we had to look for a new profiler engine for backend performance. We adopted the open source php-tideways package, and this has powered our browser-facing profiler since. Naturally, this was already installed on the mwdebug servers for that purpose. However, the package, and the accompanying *rdtsc* setting, were only set for php-fpm (web server), it was not yet enabled for php-cli (command-line). *Secondly*, the Profiler component in MediaWiki core had gotten out of sync with the needs of the Skin and Maintenance components. Over years of refactoring and more parts of each component gaining an active owner, the parts that lacked an owner eroded and stopped working, including the integration between these components. We decided to take active ownership over the remaining parts of MediaWiki-Core-Profiler and fix the disconnect. The meta work for that included identifying and re-triaging open issues under a new #mediawiki-core-profiler <https://phabricator.wikimedia.org/tag/mediawiki-core-profiler/> tag, automating discovery to our team inbox via Phabricator Herald rule, enlisting on Maintainers <https://www.mediawiki.org/wiki/Developers/Maintainers>, and automatic discovery of changesets to our code review dashboard. <https://gerrit.wikimedia.org/r/p/wikimedia/+/dashboard/teams:performance> The "output" step of the profiler was originally the responsibility of a wfLogProfilingData function that both webpages (Skin) and CLI (Maintenance) called upon toward the end of the response. After this function became deprecated, and further refactoring in the Skin and Maintenance component took on (some of) its responsibilities directly, the "output" step got lost. Adding this back in was non-trivial because by now, the code in question had gained an unintentional dependency on the WebRequest object, which is not valid in a CLI context. The code in question was simplified and decoupled in change 725152 <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/725152/> and change 725440 <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/725440>, when then made it possible to add back the "output" step in change 838884 <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/838884/>. We also removed various options <https://gerrit.wikimedia.org/r/q/message:profiler+project:mediawiki/core+br…> (such as CMDLINE_CALLBACK <https://phabricator.wikimedia.org/T305422>) that we choose to not support and maintain going forward (undocumented, unused, broken, or without known use case after research). *Thirdly*, in parallel with our work above, another team refactored MediaWiki's SettingsLoader and MaintenanceRunner components. The Profiler dates back more than a decade and still relied on the fact that settings could be changed by CLI script options such as *--profiler=text*. The new SettingsLoader and the refactored MaintenanceRunner streamlined the order of operations during process startup, which had the side effect of initialising the profiler slightly earlier than it used to. This meant it would initialise before the --profiler=text option was applied, and thus the profiler that was initialised was unconditionally the "null" profiler. This did not produce an error since that is a valid configuration, the configuration of effectively all traffic besides ad-hoc debugging. I remedied this <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/875396/> by recognising the CLI option as its own setting (separate from the main one), and passing it down via dependency injection. Thus not relying on the subtle order of operations. The Profiler component is now significantly leaner than it was before, and its requirements are either explicitly coded through dependency injection requirements, or simplified/refactored such that they do not place requirements on other components. *Fourthly*, the wmf-config repository, where we control how and when MediaWiki Profiler can be enabled, had been changed by us the year before, to feature a new sampling profiler to produce detailed flame graphs over production traffic (blog post <https://techblog.wikimedia.org/2021/03/03/profiling-php-in-production-at-sc…>). I realised that this configuration assumed a web context. Another tweak over there <https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/874419/> made this safe to enabled in CLI/Maintenance script contexts. *Fifthly*, in order to actually enable the wmf-config/Profiler code in CLI, we needed to set the same *php.ini* configuration for php-cli as we already did years earlier for php-fpm (web server). In doing so, we ran into the problem that one server in particular (deploy1002, from where the MediaWiki train runs via Scap) had two mutually exclusive copies of the MediaWiki deployment (/srv/mediawiki and /srv/mediawiki-staging). This meant that preloading a profiler for either of them from php-cli would break the other. I identified this long-standing technical debt to Ahmon Dancy (RelEng), who then went on a whole journey of his own to radically simplify our deployment servers to not have these two separate installations (T329857 <https://phabricator.wikimedia.org/T329857>). *Finally*, with a one-line change in Puppet <https://gerrit.wikimedia.org/r/c/operations/puppet/+/910882/> config, the CLI profiler is now enabled in production (Thanks Giuseppe Lavagetto, from SRE). All in all, this made the different parts of codebase, and different parts of our platform less divergent and more unified than before. Thanks again to Aaron Schulz, Ahmon Dancy, Giuseppe Lavagetto, and Niklas Laxström for their help! -- Timo Tijhof, Principal Engineer, Performance Team, Wikimedia Foundation.

1 0

Permission model for /repos/mediawiki on GitLab
by Brennen Bearnes 16 May '23

16 May '23

Hey all, RelEng are working through some of the nitty-gritty details of migrating MediaWiki repositories to GitLab, and we could use some input on a permission model for the planned /repos/mediawiki namespace. Draft permission model policy: * https://www.mediawiki.org/wiki/GitLab/Policy#MediaWiki_permission_model Task: * T336807 - Define a permissions model for the /repos/mediawiki/ namespace on GitLab - https://phabricator.wikimedia.org/T336807 Thanks! -- Brennen Bearnes Release Engineering Wikimedia Foundation

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l May 2023