Wikitech-l April 2021

wikitech-l@lists.wikimedia.org

34 participants
53 discussions

Summary of this week's deployment of 1.37.0-wmf.1: all good!
by Lars Wirzenius 16 Apr '21

16 Apr '21

This is a summary of this week's deployment of the 1.37.0-wmf.1 branch of MediaWiki and its extensions (also known as "the train"). The primary person in charge this week is Jeena Huneidi, with Dan Duvall as backup, both from the Release Engineering team. The summary task for this week is https://phabricator.wikimedia.org/T278345 This was the first week of the 1.37.0 series. The 1.36.0 release is being made separately, and our sites are moving on to run weekly releases of the new series. It's not a momentous change. We start a series couple of times a year. The version number changes, but otherwise the changes this week are fairly mundane and usual. (There's no train deployment next week, due to a holiday on Thursday.) The new version is running all sites. There were a total of three blocker reported and resolved: - T279008 - mw-parser-output now clears after block, which is a noticeable change for Category pages where infoboxes used to render floated alongside the automatic content https://phabricator.wikimedia.org/T279008 - T279113 - Unable to parse title at new Title in Special:MediaSearch https://phabricator.wikimedia.org/T279113 - T280082 - Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'mediawikiwiki.discussiontools_subscription' doesn't exist (10.64.16.7) https://phabricator.wikimedia.org/T280082 As usual, a whole bunch of people helped to find, triage, analyze, fix, or work around problems this. Release Engineering thanks everyone, without help we wouldn't be able to deploy MediaWiki. - Anne Tomasevich - Bartosz Dziewoński - Carly Bogen - Cormac Parle - Innovative - JJMC89 - Jon Robson - Lucas Werkmeister - Mike_Peel - Rummana Yasmeen - Tacsipacsi - Timo Tijhof - TTO - Umherirrender - Urbanecm - Zabe There may have been other people, and if so, I apologize for not including them on the list above. Have a good weekend. Be well. Be safe. For more information, please see: - https://phabricator.wikimedia.org/T278345 - https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys - https://versions.toolforge.org/ -- WMF release engineering team | he/him or they/them "Imagine a world in which every single human being can freely share in the sum of all knowledge."

1 0

Performance regression expected with Debian Buster upgrade
by Kunal Mehta 15 Apr '21

15 Apr '21

Hi, We're currently in the process of upgrading the MediaWiki servers to Debian Buster and expect a performance regression to come with it. The cause appears to be better Spectre[1] mitigations in the Buster 4.19 kernel, which we can't disable. Most of the effect is seen in code that ends up invoking syscalls like filemtime, file_get_contents, etc. I posted some numbers and charts on the Phabricator investigation ticket[2]. For normal requests it looks like ~5% worse for p50/p75 and around ~13% for p95/p99. API requests look much worse, at 10% for p50 22% for p75. What now? We're going to continue with the upgrade as planned, but we also need help to try and make some performance improvements to reduce the impact of the regression. The PHP profiling flamegraphs[3] are a great tool to use to identify potentially slow spots. We now also have flamegraphs that only contain Buster requests. I created a set of differential flamegraphs[4] that compare Stretch vs Buster so you can see what specific areas slowed down. You can also use WikimediaDebug/XHGui[5] to profile a specific request. mwdebug1001/mwdebug1002 are Stretch and mwdebug1003 is Buster. If you have questions or suggestions please ask or let us know. Thanks to everyone who helped with the investigation and those who've started working on improvements already. [1] https://en.wikipedia.org/wiki/Spectre_(security_vulnerability) [2] https://phabricator.wikimedia.org/T273312#6802330 [3] https://performance.wikimedia.org/php-profiling/ [4] https://people.wikimedia.org/~legoktm/T273312/data/clean/images/flamegraphs/ [5] https://wikitech.wikimedia.org/wiki/WikimediaDebug#Request_profiling -- Kunal

3 5

Production Excellence #30: March 2021
by Krinkle 14 Apr '21

14 Apr '21

How’d we do in our strive for operational excellence last month? Read on to find out! Read on Phabricator at https://phabricator.wikimedia.org/phame/post/view/229/ Incidents 2 documented incidents. That's average for this time of year, when we usually had <https://codepen.io/Krinkle/full/wbYMZK> 1-4 incidents. Learn about recent incidents at Incident status <https://wikitech.wikimedia.org/wiki/Incident_status> on Wikitech, or Preventive measures <https://phabricator.wikimedia.org/project/view/4758/> in Phabricator. ------- Trends In March we made significant progress on the outstanding errors of previous months. Several of the 2020 months are finally starting to empty out. But with over 30 new tasks from March itself remaining, we did not break even, and ended up slightly higher than last month. This could be reversing two positive trends, but I hope not. Firstly, there was a steep increase in the number of new production errors that were not resolved within the same month. This is counter the positive trend we started in November. The past four months typically saw 10-20 errors outlive their month of discovery, and this past month saw 34 of its 48 new errors remain unresolved. Secondly, we saw the overall number of unresolved errors increase again. This January <https://phabricator.wikimedia.org/phame/post/view/227/production_excellence…> began a downward trend for the first time in thirteen months, which continued nicely through February. But, this past month we broke even and even pushed upward by one task. I hope this is just a breather and we can continue our way downward. Figure 1, Figure 2: Unresolved error reports stacked by month. <https://phabricator.wikimedia.org/phame/post/view/229/production_excellence…> ------- Outstanding errors Take a look at the workboard <https://phabricator.wikimedia.org/tag/wikimedia-production-error/> and look for tasks that could use your help. Summary over recent months, per spreadsheet <https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony…> : Jul 2019 (0 of 18 left) ✅ Last two tasks resolved! -2 Aug 2019 (1 of 14 left) ⚠️ *Unchanged (and over one year old).* Oct 2019 (3 of 12 left) ⬇️ One task resolved. -1 Nov 2019 (0 of 5 left) ✅ Last task resolved! -1 Dec 2019 (0 of 9 left) ✅ Last task resolved! -1 Jan 2020 (2 of 7 left) ⬇️ One task resolved. -1 Feb 2020 (0 of 7 left) ✅ Last task resolved! -1 Mar 2020 (2 of 2 left) ⚠️ *Unchanged (and over one year old).* Apr 2020 (5 of 14 left) ⬇️ Four tasks resolved. -4 May 2020 (5 of 14 left) ⬇️ One task resolved. -1 Jun 2020 (6 of 14 left) ⬇️ One task resolved. -1 Jul 2020 (5 of 24 issues <https://phabricator.wikimedia.org/maniphest/query/s__D8Kd0xuQH/#R>) ⬇️ Four tasks resolved. -4 Aug 2020 (15 of 53 issues <https://phabricator.wikimedia.org/maniphest/query/hu1yhWu4sXkP/#R>) ⬇️ Five tasks resolved. -5 Sep 2020 (7 of 33 issues <https://phabricator.wikimedia.org/maniphest/query/CGFQViLShnOY/#R>) ⬇️ One task resolved. -1 Oct 2020 (22 of 69 issues <https://phabricator.wikimedia.org/maniphest/query/MYnnBybPTYpd/#R>) ⬇️ Four tasks resolved. -4 Nov 2020 (9 of 38 issues <https://phabricator.wikimedia.org/maniphest/query/CkC_VqQq5VC0/#R>) ⬇️ Two tasks resolved. -2 Dec 2020 (11 of 33 issues <https://phabricator.wikimedia.org/maniphest/query/10NQy74iKaZJ/#R>) ⬇️ One task resolved. -1 Jan 2021 (4 of 50 issues <https://phabricator.wikimedia.org/maniphest/query/WIP9W8q54HB6/#R>) ⬇️ One task resolved. -1 Feb 2021 (9 of 20 issues <https://phabricator.wikimedia.org/maniphest/query/5MzPJAb5oJgv/#R>) ⬇️ Two tasks resolved. -2 *Mar 2021* (34 of 48 issues <https://phabricator.wikimedia.org/maniphest/query/RsVPep46KRY4/#R>) 34 new tasks survived and remain unresolved. +48; -14 ------- Tally 138 issues open, as of Excellence #29 <https://phabricator.wikimedia.org/phame/post/view/228/production_excellence…> (6 Mar 2021). -33 issues closed, of the previous 138 open issues. +34 new issues that survived March 2021. 139 issues open, as of today (2 Apr 2021). -------Thanks! Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks! Until next time, – Timo Tijhof Footnotes: Incident status, Wikitech <https://wikitech.wikimedia.org/wiki/Incident_status>. Wikimedia incident stats by Krinkle, CodePen <https://codepen.io/Krinkle/full/wbYMZK>. Production Excellence: Month-over-month spreadsheet and plot <https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony…> . Report charts for Wikimedia-production-error project, Phabricator <https://phabricator.wikimedia.org/project/reports/1055/>.

1 0

Does anyone know about bulk loads on Wikibase/Wikidata?
by Henry Rosales 14 Apr '21

14 Apr '21

Hi all! I hope you are doing very well! I’m working on constructing an RDF dataset, which I’m ingesting to Wikibase and leverage its ecosystem: visualization, SPARQL endpoint, a script for dump creation, etc. For ingestion, I’m using the API (explained in https://www.wikidata.org/w/api.php), and it is going well, except for the time performance. I was exploring the Quick Statements module of Wikibase, which allows the ingestion through the UI. However, I need a way to ingest data automatically. I wonder if Wikibase/Wikidata has any script for bulk loads. Any comment or suggestion will be welcome. Best Regards, Henry Rosales

2 1

April 2021 Technical Community Newsletter
by Sarah R 14 Apr '21

14 Apr '21

Hi Everyone, We’re happy to announce the April 2021 edition of the Technical Community Newsletter <https://www.mediawiki.org/wiki/Technical_Community_Newsletter/2021/April> is now available. The newsletter is compiled by the Wikimedia Developer Advocacy Team. It aims to share highlights, news, and information of interest from and about the Wikimedia technical community. Check it out, and learn about what technical contributors have been up to this past quarter, upcoming conferences & calls for papers, and how to get involved. The Wikimedia Technical Community is large and diverse, and we know we can't capture everything perfectly. We welcome your ideas for future newsletters. Let us know what you would like to see or highlights you would like us to include. Subscribe to the Technical Community Newsletter <https://www.mediawiki.org/wiki/Newsletter:Technical_Community_Newsletter>, if you'd like to keep up with essential updates and information Kindly, Sarah R. Rodlund Senior Technical Writer, Developer Advocacy <https://www.mediawiki.org/wiki/Developer_Advocacy> srodlund(a)wikimedia.org

1 0

see stacktrace from functions within mediawiki
by Sandeep Gupta 14 Apr '21

14 Apr '21

Hello, I have recently started looking into mediawiki source code and trying to understand various codeflow. I am new to php as well. I want to see the stacktrace from database functions. I have put following debug line in the function : ob_start(); debug_print_backtrace(); $data = ob_get_clean(); fwrite(STDERR, $data . PHP_EOL); But I don't see any output in /var/log/nginx/mediawiki.error ( in current setup mediawiki is served by nginx). Any help would be great. Or any other approach to access the stacktrace would also work. Thanks Sandeep

1 0

2021-04-14 Scrum of Scrums meeting notes
by James Forrester 14 Apr '21

14 Apr '21

https://www.mediawiki.org/wiki/Scrum_of_scrums/2021-04-14 = 2021-04-14 = Grace/Deb have conflicts today. Please self-organize :) == Callouts == * MediaWiki 1.36-beta has been branched: https://lists.wikimedia.org/pipermail/wikitech-l/2021-April/094424.html == Gerrit patches or GitHub Pull Requests for reviews or feedback == * None. === No updates === Community Tech, Anti-Harassment Tools, Library, Site Reliability Engineering === '''No notes provided''' === Editing, Growth, iOS native app, Android native app, Web, Product Infrastructure, Parsing, Language, Inuka, Analytics, Cloud Services, Platform, Quality and Test Engineering, Search Platform, Security == SoS Meeting Bookkeeping == * Updates: == Product == === Structured Data === * Thank yous: ** Jon Robson for getting Vue errors into logstash and helping us identify some issues in our code * Updates: ** Working to make the new MediaSearch extension (not yet deployed on Commons) more usable on other wikis ( https://www.mediawiki.org/wiki/Extension:MediaSearch) ** Image recommendations design work === Abstract Wikipedia === * Thank yous: ** Thank you to Brennen and Jeena in RelEng for discussing options for testing MW<->service integrations in CI. * Updates: ** Working on Phase δ (delta): https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Phases ** Working on our TDMP pitch for asynchronous parser fragments. === Vue.js === * Updates: ** Working on components design inventory: https://phabricator.wikimedia.org/T277047 ** Starting to work on unifying various component implementations so we can add them to the experimental WMF component library (WVUI) *** Button: https://phabricator.wikimedia.org/T278509 *** Binary inputs: https://phabricator.wikimedia.org/T279714 == Technology == === Fundraising Tech === * Updates: ** More work on email prefs page https://phabricator.wikimedia.org/T268510 ** Planning for integration with new API of backup card processor ** Work to migrate custom CRM code off drupal 7 ** CiviCRM contact deduplication enhancements ** Audit / reconciliation file processing improvements: https://phabricator.wikimedia.org/T277244, https://phabricator.wikimedia.org/T265545 === Engineering Productivity === ==== Performance ==== * Blocking: ** WMSE with performance review of WikiSpeech (in progress, almost finished) * Thank yous: ** Geno from Abstract Wikipedia for how engaged she is with the matrixing on our team * Updates: ** Cancelling our Kobiton subscription, we will build our own device lab hosted at the office instead ==== Release Engineering ==== * Updates: ** [All] Deployments/Covid-19 https://wikitech.wikimedia.org/wiki/Deployments/Covid-19 ** MediaWiki 1.36-beta has been branched: https://lists.wikimedia.org/pipermail/wikitech-l/2021-April/094424.html ** Train Health *** Last week: 1.36.0-wmf.38 [[phab:T278343]] *** This week: 1.37.0-wmf.1 [[phab:T278344]] *** This week: 1.37.0-wmf.2 [[phab:T278345]] === WMDE Technical Wishes === * Updates: ** Line numbering was deployed. Currently only on template namespace, due to feedback by the editing team. Discussion about other namespaces is still open. == Cross-cutting == * Blocked by: ** [long term] Search Platform: PHP 8.0 work is long-term blocked on the migration to ElasticSearch 7.0 https://phabricator.wikimedia.org/T263142 (or at least 6.7). * Thank yous: * Updates: ** Nothing major. ** CI tools *** Next release of mediawiki-codesniffer likely soon. *** CI tools' upgrade status: https://libraryupgrader2.wmcloud.org/status?branch=master ** PHP 8.0: *** Our target is REL1_35 (and thus also REL1_36) as well as master. *** Upstream libraries: Elastica-related PHP code is theoretically the last one. *** Core: Some unit and integration tests still fail; thank you to everyone working on fixing them. [End] Yours, -- *James D. Forrester* (he/him <http://pronoun.is/he> or they/themself <http://pronoun.is/they/.../themself>) Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

No train the week of 2021-04-19
by Tyler Cipriani 14 Apr '21

14 Apr '21

Hi All There will be no train next week (the week of 2021-04-19) due to a wmf holiday on 2021-04-22. There is a long-term calendar of upcoming deployment disruptions available on Wikitech: https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar Thanks! – Tyler

1 0

Translation request
by Timur Vorkul 14 Apr '21

14 Apr '21

Hello, dear translators! Your help with a short outreach message is needed. It is the announcement of the new suggested values feature to be deployed on all wikis. It will make filling in templates which expect specific inputs much easier to do in the VisualEditor. https://meta.wikimedia.org/wiki/User:Timur_Vorkul_(WMDE)/Suggested_values_a… I plan to send the message to all wikis on April 27, so having many translations by April 26 would be awesome! Thanks a lot! Best, Timur -- Timur Vorkul Technische Wünsche Wikimedia Deutschland e. V. | Tempelhofer Ufer 23–24 | 10963 Berlin Tel. (030) 219 158 26-0 https://wikimedia.de Bleiben Sie auf dem neuesten Stand! Aktuelle Nachrichten und spannende Geschichten rund um Wikimedia, Wikipedia und Freies Wissen im Newsletter: Zur Anmeldung <https://www.wikimedia.de/newsletter/>. Unsere Vision ist eine Welt, in der alle Menschen am Wissen der Menschheit teilhaben, es nutzen und mehren können. Helfen Sie uns dabei! https://spenden.wikimedia.de Wikimedia Deutschland — Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Problems with the CI
by Zoran Dori 14 Apr '21

14 Apr '21

Hello, in the last few days, for some reason, CI jobs related to the coverage fails with this reason: 04:16:11 Finding coverage difference in ede7dc8b68b63373b38fee56facaccf5b7d62589 04:16:11 $ php -d zend_extension=xdebug.so tests/phpunit/phpunit.php --coverage-clover /tmp/cloverETlRWo --filter '/MediaWikiIntegrationTestCase|MediaWikiTestCaseTrait|DiffHistoryBlobTest/' 04:16:11 PHP Warning: Failed loading Zend extension 'xdebug.so' (tried: /usr/lib/php/20180731/xdebug.so (/usr/lib/php/20180731/xdebug.so: cannot open shared object file: No such file or directory), /usr/lib/php/20180731/ xdebug.so.so (/usr/lib/php/20180731/xdebug.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0 04:16:11 Using PHP 7.3.27-9+0~20210227.82+debian10~1.gbpa4a3d6 04:16:14 PHPUnit 8.5.15 by Sebastian Bergmann and contributors. 04:16:14 04:16:14 Error: No code coverage driver is available 04:16:14 04:16:14 ............................S.......SS 38 / 38 (100%) 04:16:14 04:16:14 Time: 3.21 seconds, Memory: 365.00 MB 04:16:14 04:16:14 OK, but incomplete, skipped, or risky tests! 04:16:14 Tests: 38, Assertions: 87, Skipped: 3. 04:16:14 04:16:14 04:16:14 You should really speed up these slow tests (>50ms)... 04:16:14 1. 60ms to run MediaWikiIntegrationTestCaseTest:testEditPage 04:16:14 04:16:14 In CloverXml.php line 69: 04:16:14 04:16:14 String could not be parsed as XML 04:16:14 04:16:14 04:16:14 check [--sha1 [SHA1]] [--test-dir TEST-DIR] [--html [HTML]] [--command COMMAND] Example: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-patch/2841… Patch: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/675296 I would like to know why is this happening. Best regards, Zoran

4 4

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l April 2021