Hi,
For about a week MediaWiki Codesearch was serving outdated and
incomplete results (and later partially down outright) because of a full
disk. This has been fixed now, you may wish to re-run searches as necessary.
The full disk was accidentally caused by switching between gerrit and
gerrit-replica (as previously discussed on this list)[1][2], so I
deleted all the old gerrit-replica repository copies and also added ~20G
more disk space to the instance. Monitoring did correctly pick up that
there was some issue once the disk was full, it just wasn't properly
examined because of hackathon travel / activities.
[1] https://phabricator.wikimedia.org/T336710#8877525
[2] https://phabricator.wikimedia.org/T337263
-- Kunal / Legoktm
Hello. I tried to start an svg translation. Added just a couple of switch
statements, just to see how it works before the rest job. But it doesn't.
Could you tell me, please, what am I doing wrong? Thank you.
Igal (User:IKhitron)
https://commons.wikimedia.org/w/index.php?title=File%3AMoscow_metro_map_mul…
Hi! The default value of the installer's script path parameter was changed
in gerrit 133222 <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/133222> to
make the installation process more intuitive. If you use the installer in
your scripts, you might want to make sure that script path is set
explicitly. (To keep the old behavior, add "--scriptpath wiki".)
Hello all,
The Code of Conduct Committee is a team of five trusted individuals (plus
five auxiliary members) with diverse affiliations responsible for general
enforcement of the Code of conduct for Wikimedia technical spaces.
Committee members are in charge of processing complaints, discussing with
the parties affected, agreeing on resolutions, and following up on their
enforcement. For more on their duties and roles, see
https://www.mediawiki.org/wiki/Code_of_Conduct/Committee.
This is a call for community members interested in volunteering for
appointment to this committee. Volunteers serving in this role should be
experienced Wikimedians or have had experience serving in a similar
position before.
The current committee is doing the selection and will research and discuss
candidates. Six weeks before the beginning of the next Committee term,
meaning July 16, 2023, they will publish their candidate slate (a list of
candidates) on-wiki. The community can provide feedback on these
candidates, via private email to the group choosing the next Committee. The
feedback period will be two weeks. The current Committee will then either
finalize the slate, or update the candidate slate in response to concerns
raised. If the candidate slate changes, there will be another two week
feedback period covering the newly proposed members. After the selections
are finalized, there will be a training period, after which the new
Committee is appointed. The current Committee continues to serve until the
feedback, selection, and training process is complete.
If you are interested in serving on this committee or like to nominate a
candidate, please write an email to techconductcandidates AT wikimedia.org
with details of your experience on the projects, your thoughts on the code
of conduct and the committee and what you hope to bring to the role and
whether you have a preference in being auxiliary or main member of the
committee. The committee consists of five main members plus five auxiliary
members and they will serve for a year; all applications are appreciated
and will be carefully considered. The deadline for applications is *the end
of day on May 31 2023*.
Please feel free to pass this invitation along to any users who you think
may be qualified and interested.
Best,
Martin Urbanec, on behalf of the Code of Conduct Committee
The 1.41.0-wmf.9 version of MediaWiki is blocked[0].
The new version is currently deployed to group1, but can proceed no
further until these issues are resolved:
* T336962 - UnexpectedValueException: Unknown image suggestions API
kind: istype-depicts
- https://phabricator.wikimedia.org/T336962
* T336964 - InvalidArgumentException: Data for lt_namespace and lt_title
must be non-empty
- https://phabricator.wikimedia.org/T336964
Once these issues are resolved train can resume.
Thank you for any help resolving these issues!
-- Your disgruntled train drivers
[0]. https://phabricator.wikimedia.org/T330215
[1]. https://versions.toolforge.org/
Hi all,
at the ongoing Hackathon, Legoktm made me nominate myself for +2 in mediawiki/* repositories ^^ see <https://phabricator.wikimedia.org/T337014>.
Cheers,
Lucas
TLDR: Those with shell access at WMF can now run maintenance scripts on mwdebug hosts, and use the *--profiler=text* option to produce a report detailing how long the script spent in each MediaWiki component, class, and function.
== *What* ==
The MediaWiki platform at WMF consists of broadly four different deployments: app servers, api_app servers, job runners, and maint servers (diagram <https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF#/media/File:MediaWiki_…>). Each can be thought of as its own service for a specific purpose, composed of a subset of components from the MediaWiki codebase with fast local access to each. The largest of these is the appservers cluster, which is served by 150 servers of dedicated hardware across the two application data centers <https://wikitech.wikimedia.org/wiki/Data_centers>, and is responsible for responding to index.php (i.e. page views) and load.php (CSS, JS, and localisation via ResourceLoader).
Today, we focus on the smallest of these: mwmaint servers <https://wikitech.wikimedia.org/wiki/Maintenance_server>. This is backed by two heavy-duty servers, one in each data center, that autonomously run essential tasks at a predefined schedule (i.e. not in direct response to a user action). Each of these ~50 different tasks is implemented as a MediaWiki maintenance script. Important examples include: sending email notifications (Echo extension), timely pruning of sensitive PII (CheckUser extension), computing mentee and link recommendation data (GrowthExperiments), and reclaiming disk space for expired caches (core/ParserCache).
== *Why ==*
We have detailed debug performance profiling in production for web requests via the WikimediaDebug extension, and we have detailed profiling in local development for both web requests and maintenance scripts (Docker recipe <https://www.mediawiki.org/wiki/MediaWiki-Docker/Configuration_recipes/Profi…>).
What was missing is a way to profile maintenance scripts in production. This is important as maintenance scripts tend take many minutes or hours to process vast amounts of production data. While generally easy to debug locally for functional analysis, the performance bottlenecks individual teams care about are likely specific to the size of the data and the performance of other production components.
Thanks also to Ahmon Dancy (RelEng), Giuseppe Lavagetto (SRE), and Aaron Schulz (Performance Team) for making this work possible, and Niklas Laxström (LangEng) for coming up with the idea.
== *What's New* ==
Documentation: https://wikitech.wikimedia.org/wiki/WikimediaDebug#Plaintext_CLI_profile
To profile a Maintenance script, run the script from the shell with *mwscript* as you normally would, but instead of connecting your terminal to mwmaint1002.eqiad.wmnet, connect to one of the *mwdebug* hosts (such as mwdebug1001.eqiad.wmnet). Then pass the *--profiler=text* option to generate a report with the performance analysis, which will be printed after the task is finished. Like so:
> $ mwscript showSiteStats.php --wiki=nlwiki --profiler=text
> Number of articles: 2122688
> Number of users : 1276507
>
> <!--
> 100.00% 114.964 1 - main()
> …
> 22.42% 25.776 1 - ShowSiteStats::execute
> 16.61% 19.096 2 - Wikimedia\Rdbms\LoadBalancer::getServerConnection
> 4.80% 5.522 1 - Maintenance::shutdown
> 4.41% 5.065 1 - Wikimedia\Rdbms\Database::initConnection
> 3.07% 3.530 1 - DeferredUpdates::doUpdates
> 2.66% 3.061 1 - Wikimedia\Rdbms\Database::select
> 2.38% 2.739 1 - Wikimedia\Rdbms\Database::query
> 1.95% 2.240 1 - section.SELECT * FROM `site_stats` LIMIT N
> 1.48% 1.700 1 - Wikimedia\Rdbms\DatabaseMysqli::mysqlConnect
> …
> -->
== *A peak behind the curtain* ==
Read on if you'd like to learn what hurdles we had to overcome for this to "simply" work in production, like it did for local development. The journey started when Niklas (WMF LangEng) proposed the idea at https://phabricator.wikimedia.org/T253547.
*Firstly*, the profiler engine. In 2019 (blogpost <https://techblog.wikimedia.org/2019/12/16/wikimediadebug-v2-is-here/>), after we migrated from HHVM to PHP 7, we had to look for a new profiler engine for backend performance. We adopted the open source php-tideways package, and this has powered our browser-facing profiler since. Naturally, this was already installed on the mwdebug servers for that purpose. However, the package, and the accompanying *rdtsc* setting, were only set for php-fpm (web server), it was not yet enabled for php-cli (command-line).
*Secondly*, the Profiler component in MediaWiki core had gotten out of sync with the needs of the Skin and Maintenance components. Over years of refactoring and more parts of each component gaining an active owner, the parts that lacked an owner eroded and stopped working, including the integration between these components. We decided to take active ownership over the remaining parts of MediaWiki-Core-Profiler and fix the disconnect. The meta work for that included identifying and re-triaging open issues under a new #mediawiki-core-profiler <https://phabricator.wikimedia.org/tag/mediawiki-core-profiler/> tag, automating discovery to our team inbox via Phabricator Herald rule, enlisting on Maintainers <https://www.mediawiki.org/wiki/Developers/Maintainers>, and automatic discovery of changesets to our code review dashboard. <https://gerrit.wikimedia.org/r/p/wikimedia/+/dashboard/teams:performance>
The "output" step of the profiler was originally the responsibility of a wfLogProfilingData function that both webpages (Skin) and CLI (Maintenance) called upon toward the end of the response. After this function became deprecated, and further refactoring in the Skin and Maintenance component took on (some of) its responsibilities directly, the "output" step got lost. Adding this back in was non-trivial because by now, the code in question had gained an unintentional dependency on the WebRequest object, which is not valid in a CLI context. The code in question was simplified and decoupled in change 725152 <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/725152/> and change 725440 <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/725440>, when then made it possible to add back the "output" step in change 838884 <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/838884/>. We also removed various options <https://gerrit.wikimedia.org/r/q/message:profiler+project:mediawiki/core+br…> (such as CMDLINE_CALLBACK <https://phabricator.wikimedia.org/T305422>) that we choose to not support and maintain going forward (undocumented, unused, broken, or without known use case after research).
*Thirdly*, in parallel with our work above, another team refactored MediaWiki's SettingsLoader and MaintenanceRunner components. The Profiler dates back more than a decade and still relied on the fact that settings could be changed by CLI script options such as *--profiler=text*. The new SettingsLoader and the refactored MaintenanceRunner streamlined the order of operations during process startup, which had the side effect of initialising the profiler slightly earlier than it used to. This meant it would initialise before the --profiler=text option was applied, and thus the profiler that was initialised was unconditionally the "null" profiler. This did not produce an error since that is a valid configuration, the configuration of effectively all traffic besides ad-hoc debugging. I remedied this <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/875396/> by recognising the CLI option as its own setting (separate from the main one), and passing it down via dependency injection. Thus not relying on the subtle order of operations.
The Profiler component is now significantly leaner than it was before, and its requirements are either explicitly coded through dependency injection requirements, or simplified/refactored such that they do not place requirements on other components.
*Fourthly*, the wmf-config repository, where we control how and when MediaWiki Profiler can be enabled, had been changed by us the year before, to feature a new sampling profiler to produce detailed flame graphs over production traffic (blog post <https://techblog.wikimedia.org/2021/03/03/profiling-php-in-production-at-sc…>). I realised that this configuration assumed a web context. Another tweak over there <https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/874419/> made this safe to enabled in CLI/Maintenance script contexts.
*Fifthly*, in order to actually enable the wmf-config/Profiler code in CLI, we needed to set the same *php.ini* configuration for php-cli as we already did years earlier for php-fpm (web server). In doing so, we ran into the problem that one server in particular (deploy1002, from where the MediaWiki train runs via Scap) had two mutually exclusive copies of the MediaWiki deployment (/srv/mediawiki and /srv/mediawiki-staging). This meant that preloading a profiler for either of them from php-cli would break the other. I identified this long-standing technical debt to Ahmon Dancy (RelEng), who then went on a whole journey of his own to radically simplify our deployment servers to not have these two separate installations (T329857 <https://phabricator.wikimedia.org/T329857>).
*Finally*, with a one-line change in Puppet <https://gerrit.wikimedia.org/r/c/operations/puppet/+/910882/> config, the CLI profiler is now enabled in production (Thanks Giuseppe Lavagetto, from SRE). All in all, this made the different parts of codebase, and different parts of our platform less divergent and more unified than before.
Thanks again to Aaron Schulz, Ahmon Dancy, Giuseppe Lavagetto, and Niklas Laxström for their help!
--
Timo Tijhof,
Principal Engineer,
Performance Team,
Wikimedia Foundation.
Hey all,
RelEng are working through some of the nitty-gritty details of migrating
MediaWiki repositories to GitLab, and we could use some input on a
permission model for the planned /repos/mediawiki namespace.
Draft permission model policy:
* https://www.mediawiki.org/wiki/GitLab/Policy#MediaWiki_permission_model
Task:
* T336807 - Define a permissions model for the /repos/mediawiki/
namespace on GitLab
- https://phabricator.wikimedia.org/T336807
Thanks!
--
Brennen Bearnes
Release Engineering
Wikimedia Foundation
I'd like to send an email from a python3 process on a wmcs VPS to report
errors.
I looked at https://wikitech.wikimedia.org/wiki/Help:Email_in_Cloud_VPS but
could use some help.
sudo echo "Subject: sendmail test2" | /usr/sbin/sendmail -v <myemail> works.
When I try to send the equivalent from python smtplib I get a 221 error
message.
Thanks,
Tim
