Wikitech-l April 2021

wikitech-l@lists.wikimedia.org

34 participants
57 discussions

2021-04-21 Scrum of Scrums meeting notes
by Grace Gellerman 21 Apr '21

21 Apr '21

https://www.mediawiki.org/wiki/Scrum_of_scrums/2021-04-21 = 2021-04-21 = == Callouts == *** Cross-cutting: There are five filed blockers to the 1.36 RC.0 release (due tomorrow), but two are unclaimed and apparently not being worked on: https://phabricator.wikimedia.org/tag/mw-1.36-release/ * … == Gerrit patches or GitHub Pull Requests for reviews or feedback == * === No updates === CommTech, Product Infrastructure, Parsing, Language, Analytics, Cloud Services, Platform, Performance, Quality and Test Engineering, Release Engineering, Search Platform, Security == SoS Meeting Bookkeeping == * Updates: == Product == === Anti-Harassment Tools === * Blocked by: * Blocking: * Thank yous: ** Thanks to Sam Reed and James Forrester for code review and advice ** Thanks to DannyS for huge help on partial blocks review/QA * Updates: ** We're working on partial blocks against actions while we await next instructions on SecurePoll === Editing === * Blocked by: * Blocking: * Thank yous: * Updates: ** Will be working with Performance and others to reduce ParserCache expiry by ~7 days (30 -> 23) to free up some space, possibly caused by DiscsussionTools reply links. === Growth === * Blocked by: * Blocking: * Thank yous: * Updates: ** Continuing work on Add Link https://wikitech.wikimedia.org/wiki/Add_Link ** Preparing to enable on-wiki configuration next week https://phabricator.wikimedia.org/T274520 ** Migrating mentor data from user preferences to a dedicated table https://phabricator.wikimedia.org/T279853 ** Deploying Growth features as opt-in to eswiki === iOS native app === * Blocked by: N/A * Blocking: N/A * Thank yous: * Updates: Version 6.8 - finally improving language varient support - is currently rolling out. === Android native app === * Blocked by: N/A * Blocking: N/A * Thank yous: * Updates: Version w/ image recommendations suggested edits should be released 2 weeks from today. === Web === * Blocked by: * Blocking: * Thank yous: * Updates: ** Finishing up Vue.js search autocomplete widget project report: https://phabricator.wikimedia.org/T249430 ** Preparing a release for WVUI to address a handful of bugs: *** https://phabricator.wikimedia.org/T277256 *** https://phabricator.wikimedia.org/T277315 *** https://phabricator.wikimedia.org/T279088 ** Beginning work on the new personal tools treatment for Desktop Improvements: https://www.mediawiki.org/wiki/Reading/Web/Desktop_Improvements/Features#Us… === Structured Data === * Blocked by: * Blocking: * Thank yous: * Updates: ** Building a tool to manually test current implementation of the image recommendations API ** Whittled down dependencies for the new MediaSearch extension; it now only requires CirrusSearch === Abstract Wikipedia === * Blocked by: ** None. * Blocking: ** None known. * Thank yous: * Updates: ** Working on Phase δ (delta): https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Phases ** Started circulating early draft of our TDMP pitch for asynchronous parser fragments. === Inuka === * Blocked by: * Blocking: * Thank yous: * Updates: ** Wikipedia Preview *** Working on the version information https://phabricator.wikimedia.org/T278377 *** Working on the debug information https://phabricator.wikimedia.org/T278381 *** Working on the Wix integration https://phabricator.wikimedia.org/T278456 ** Wikipedia KaiOS *** Optimize the confirmed characters of the search method https://phabricator.wikimedia.org/T246786 === Library === * Blocked by: * Blocking: * Thank yous: ** Language team (Niklas and Abijeet) for helping us get partner descriptions into TranslateWiki * Updates: ** You can now translate Wikipedia Library partner descriptions! https://translatewiki.net/wiki/Special:Translate?group=wikipedia-library-pa… ** Started setting up Matomo to track page views ** Started setting up tags to be translatable via TranslateWiki ** Finishing up The Wikipedia Library extension work === Vue.js === * Blocked by: * Blocking: * Thank yous: * Updates: ** Starting work to unify button implementations in WVUI and better match what's in the design style guide (https://phabricator.wikimedia.org/T278509) ** Adding contributing guidelines to WVUI: https://gerrit.wikimedia.org/r/c/wvui/+/681178/3/contributing.md (under review) ** Planning long-term migration of projects to Vue == Technology == === Fundraising Tech === * Blocked by: * Blocking: * Thank yous: * Updates: ** More work on email prefs page https://phabricator.wikimedia.org/T268510, https://phabricator.wikimedia.org/T268511 ** Planning for integration with new API of backup card processor ** Work to migrate custom CRM code off drupal 7 ** CiviCRM contact deduplication enhancements ** Audit / reconciliation file processing improvements: https://phabricator.wikimedia.org/T277244, https://phabricator.wikimedia.org/T265545 === Site Reliability Engineering === * Blocked by: ** None * Blocking: ** None * Thank yous: * Updates: ** 100% of MW appservers now on buster, no more stretch special test hosts, mwdebug1003 removed ** Gitlab: Performance testing on WMCS done, defining backup and monitoring, documentation ** The MediaWiki chart can run a php application on kubernetes https://people.wikimedia.org/~oblivian/mw-on-k8s-shared-socket.png ** Scap logs will be transitioned to ECS 2021-05-04 === WMDE Technical Wishes === * Blocked by: * Blocking: * Thank yous: * Updates: ** Reusable edit count bucketing was accepted and is live for several extensions. == Cross-cutting == * Blocked by: ** [long term] Search Platform: PHP 8.0 work is long-term blocked on the migration to ElasticSearch 7.0 https://phabricator.wikimedia.org/T263142 (or at least 6.7). * Blocking: ** None known. * Thank yous: ** Huge thanks to the Web team, especially Jon Robson, for their work on fixing multiple blockers to the 1.36 release. * Updates: ** REL1_36 release process: *** [In calls-out] There are five filed blockers to the 1.36 RC.0 release (due tomorrow), but two are unclaimed and apparently not being worked on: https://phabricator.wikimedia.org/tag/mw-1.36-release/ ** CI tools: *** Nothing major. *** Next release of mediawiki-codesniffer likely soon. *** CI tools' upgrade status: https://libraryupgrader2.wmcloud.org/status?branch=master ** PHP 8.0: *** Nothing major. *** Our target is REL1_35 (and thus also REL1_36) as well as master. *** Upstream libraries: Elastica-related PHP code is theoretically the last one. *** Core: Some unit and integration tests still fail; thank you to everyone working on fixing them. [[Category:Scrum of scrums{{#translation:}}|*]]

1 0

Working with templates: Looking for wikis who want early access to several improvements
by Johanna Strodt 21 Apr '21

21 Apr '21

Hello! Our team, Wikimedia Germany’s Technical Wishes project [1], is developing a series of improvements to make working with templates easier. Maybe you’ve already heard of some of the projects that have recently been deployed: [2] [3] [4]. We plan to release more improvements to make working with templates easier over the course of this year: It will become easier to find templates and to insert them into a page. [5] Removing a template from a page (with Visual Editor) will become easier. [6] Several problems with the existing interface for editing templates in the VisualEditor will be tackled. [7] Template creators and maintainers will be offered more guidance when they are managing TemplateData with the template documentation editor. [8] Now we’re looking for a few wikis who want to be the first to benefit from these changes. If your wiki community is interested in this, all the improvements listed above would be deployed to your wiki in a series of releases, likely between May and July 2021. Of course, each deployment would be announced beforehand. We’re hoping for wikis of different sizes, both Wikipedias and sister projects, and wikis with different scripts and writing directions. The wikis should use TemplateData and ideally have the VisualEditor as a default feature. Is your wiki community interested to have early access to these improvements? Email me or let me know on my talk page.[9] Best, Johanna [1] Technical Wishes project: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes [2] Bracket matching: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Bracket_Matching [3] Improved color scheme for syntax highlighting: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Improved_Color_Scheme… [4] Line numbering: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Line_Numbering [5] Finding and inserting templates: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Finding_and_inserting… [6] Removing a template from a page using the VisualEditor: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Removing_a_template_f… [7] VisualEditor template dialog improvements: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/VisualEditor_template… [8] TemplateData editor improvements: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Template_data_editor_… [9] talk page: https://meta.wikimedia.org/wiki/User_talk:Johanna_Strodt_(WMDE) -- Johanna Strodt Projektmanagerin Kommunikation Communitys Technische Wunschliste Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Tel. (030) 219 158 26-0 https://wikimedia.de Unsere Vision ist eine Welt, in der alle Menschen am Wissens der Menschheit teilhaben, es nutzen und mehren können. Helfen Sie uns dabei! https://spenden.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Upstream release notifications on Phabricator
by Kunal Mehta 19 Apr '21

19 Apr '21

Hi, LibUp can now notify you about new upstream releases in Phabricator. An example: <https://phabricator.wikimedia.org/T280117>. It'll check daily to see if an upstream project has a new release, and if so, file a Phabricator task. If it already filed a task that's still open, it'll leave a new comment instead. This is separate from LibUp's library upgrading in Gerrit, so it should be able to support just about any project and leave it up to the humans to decide what to do about the new version. If you'd like to get notifications for your project, file a bug in the #LibUp Phabricator project, or send a patch to the labs/libraryupgrader/config Gerrit repository (see README for instructions). Behind the scenes LibUp is using <https://release-monitoring.org/> which supports checking a bunch of different package websites so we don't have to reimplement it all. -- Legoktm

1 0

Global Watchlist extension - grant request #2
by DannyS712 Wiki 18 Apr '21

18 Apr '21

Dear all, I wanted to let you know that I have requested[1] a project grant to fund the continued development of the GlobalWatchlist extension[2]. Last year, I received a grant from the WMF to create the extension[3]. The extension was created and deployed, and is now available on metawiki. I hope to continue development of the extension - if you support this, I hope you'll consider endorsing the request. Thanks, --DannyS712 [1] https://meta.wikimedia.org/wiki/Grants:Project/DannyS712/Continued_work_on_… [2] https://www.mediawiki.org/wiki/Special:MyLanguage/Extension:GlobalWatchlist [3] https://meta.wikimedia.org/wiki/Grants:Project/DannyS712/Create_a_global_wa…

1 0

Summary of this week's deployment of 1.37.0-wmf.1: all good!
by Lars Wirzenius 16 Apr '21

16 Apr '21

This is a summary of this week's deployment of the 1.37.0-wmf.1 branch of MediaWiki and its extensions (also known as "the train"). The primary person in charge this week is Jeena Huneidi, with Dan Duvall as backup, both from the Release Engineering team. The summary task for this week is https://phabricator.wikimedia.org/T278345 This was the first week of the 1.37.0 series. The 1.36.0 release is being made separately, and our sites are moving on to run weekly releases of the new series. It's not a momentous change. We start a series couple of times a year. The version number changes, but otherwise the changes this week are fairly mundane and usual. (There's no train deployment next week, due to a holiday on Thursday.) The new version is running all sites. There were a total of three blocker reported and resolved: - T279008 - mw-parser-output now clears after block, which is a noticeable change for Category pages where infoboxes used to render floated alongside the automatic content https://phabricator.wikimedia.org/T279008 - T279113 - Unable to parse title at new Title in Special:MediaSearch https://phabricator.wikimedia.org/T279113 - T280082 - Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'mediawikiwiki.discussiontools_subscription' doesn't exist (10.64.16.7) https://phabricator.wikimedia.org/T280082 As usual, a whole bunch of people helped to find, triage, analyze, fix, or work around problems this. Release Engineering thanks everyone, without help we wouldn't be able to deploy MediaWiki. - Anne Tomasevich - Bartosz Dziewoński - Carly Bogen - Cormac Parle - Innovative - JJMC89 - Jon Robson - Lucas Werkmeister - Mike_Peel - Rummana Yasmeen - Tacsipacsi - Timo Tijhof - TTO - Umherirrender - Urbanecm - Zabe There may have been other people, and if so, I apologize for not including them on the list above. Have a good weekend. Be well. Be safe. For more information, please see: - https://phabricator.wikimedia.org/T278345 - https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys - https://versions.toolforge.org/ -- WMF release engineering team | he/him or they/them "Imagine a world in which every single human being can freely share in the sum of all knowledge."

1 0

Performance regression expected with Debian Buster upgrade
by Kunal Mehta 15 Apr '21

15 Apr '21

Hi, We're currently in the process of upgrading the MediaWiki servers to Debian Buster and expect a performance regression to come with it. The cause appears to be better Spectre[1] mitigations in the Buster 4.19 kernel, which we can't disable. Most of the effect is seen in code that ends up invoking syscalls like filemtime, file_get_contents, etc. I posted some numbers and charts on the Phabricator investigation ticket[2]. For normal requests it looks like ~5% worse for p50/p75 and around ~13% for p95/p99. API requests look much worse, at 10% for p50 22% for p75. What now? We're going to continue with the upgrade as planned, but we also need help to try and make some performance improvements to reduce the impact of the regression. The PHP profiling flamegraphs[3] are a great tool to use to identify potentially slow spots. We now also have flamegraphs that only contain Buster requests. I created a set of differential flamegraphs[4] that compare Stretch vs Buster so you can see what specific areas slowed down. You can also use WikimediaDebug/XHGui[5] to profile a specific request. mwdebug1001/mwdebug1002 are Stretch and mwdebug1003 is Buster. If you have questions or suggestions please ask or let us know. Thanks to everyone who helped with the investigation and those who've started working on improvements already. [1] https://en.wikipedia.org/wiki/Spectre_(security_vulnerability) [2] https://phabricator.wikimedia.org/T273312#6802330 [3] https://performance.wikimedia.org/php-profiling/ [4] https://people.wikimedia.org/~legoktm/T273312/data/clean/images/flamegraphs/ [5] https://wikitech.wikimedia.org/wiki/WikimediaDebug#Request_profiling -- Kunal

3 5

Production Excellence #30: March 2021
by Krinkle 14 Apr '21

14 Apr '21

How’d we do in our strive for operational excellence last month? Read on to find out! Read on Phabricator at https://phabricator.wikimedia.org/phame/post/view/229/ Incidents 2 documented incidents. That's average for this time of year, when we usually had <https://codepen.io/Krinkle/full/wbYMZK> 1-4 incidents. Learn about recent incidents at Incident status <https://wikitech.wikimedia.org/wiki/Incident_status> on Wikitech, or Preventive measures <https://phabricator.wikimedia.org/project/view/4758/> in Phabricator. ------- Trends In March we made significant progress on the outstanding errors of previous months. Several of the 2020 months are finally starting to empty out. But with over 30 new tasks from March itself remaining, we did not break even, and ended up slightly higher than last month. This could be reversing two positive trends, but I hope not. Firstly, there was a steep increase in the number of new production errors that were not resolved within the same month. This is counter the positive trend we started in November. The past four months typically saw 10-20 errors outlive their month of discovery, and this past month saw 34 of its 48 new errors remain unresolved. Secondly, we saw the overall number of unresolved errors increase again. This January <https://phabricator.wikimedia.org/phame/post/view/227/production_excellence…> began a downward trend for the first time in thirteen months, which continued nicely through February. But, this past month we broke even and even pushed upward by one task. I hope this is just a breather and we can continue our way downward. Figure 1, Figure 2: Unresolved error reports stacked by month. <https://phabricator.wikimedia.org/phame/post/view/229/production_excellence…> ------- Outstanding errors Take a look at the workboard <https://phabricator.wikimedia.org/tag/wikimedia-production-error/> and look for tasks that could use your help. Summary over recent months, per spreadsheet <https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony…> : Jul 2019 (0 of 18 left) ✅ Last two tasks resolved! -2 Aug 2019 (1 of 14 left) ⚠️ *Unchanged (and over one year old).* Oct 2019 (3 of 12 left) ⬇️ One task resolved. -1 Nov 2019 (0 of 5 left) ✅ Last task resolved! -1 Dec 2019 (0 of 9 left) ✅ Last task resolved! -1 Jan 2020 (2 of 7 left) ⬇️ One task resolved. -1 Feb 2020 (0 of 7 left) ✅ Last task resolved! -1 Mar 2020 (2 of 2 left) ⚠️ *Unchanged (and over one year old).* Apr 2020 (5 of 14 left) ⬇️ Four tasks resolved. -4 May 2020 (5 of 14 left) ⬇️ One task resolved. -1 Jun 2020 (6 of 14 left) ⬇️ One task resolved. -1 Jul 2020 (5 of 24 issues <https://phabricator.wikimedia.org/maniphest/query/s__D8Kd0xuQH/#R>) ⬇️ Four tasks resolved. -4 Aug 2020 (15 of 53 issues <https://phabricator.wikimedia.org/maniphest/query/hu1yhWu4sXkP/#R>) ⬇️ Five tasks resolved. -5 Sep 2020 (7 of 33 issues <https://phabricator.wikimedia.org/maniphest/query/CGFQViLShnOY/#R>) ⬇️ One task resolved. -1 Oct 2020 (22 of 69 issues <https://phabricator.wikimedia.org/maniphest/query/MYnnBybPTYpd/#R>) ⬇️ Four tasks resolved. -4 Nov 2020 (9 of 38 issues <https://phabricator.wikimedia.org/maniphest/query/CkC_VqQq5VC0/#R>) ⬇️ Two tasks resolved. -2 Dec 2020 (11 of 33 issues <https://phabricator.wikimedia.org/maniphest/query/10NQy74iKaZJ/#R>) ⬇️ One task resolved. -1 Jan 2021 (4 of 50 issues <https://phabricator.wikimedia.org/maniphest/query/WIP9W8q54HB6/#R>) ⬇️ One task resolved. -1 Feb 2021 (9 of 20 issues <https://phabricator.wikimedia.org/maniphest/query/5MzPJAb5oJgv/#R>) ⬇️ Two tasks resolved. -2 *Mar 2021* (34 of 48 issues <https://phabricator.wikimedia.org/maniphest/query/RsVPep46KRY4/#R>) 34 new tasks survived and remain unresolved. +48; -14 ------- Tally 138 issues open, as of Excellence #29 <https://phabricator.wikimedia.org/phame/post/view/228/production_excellence…> (6 Mar 2021). -33 issues closed, of the previous 138 open issues. +34 new issues that survived March 2021. 139 issues open, as of today (2 Apr 2021). -------Thanks! Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks! Until next time, – Timo Tijhof Footnotes: Incident status, Wikitech <https://wikitech.wikimedia.org/wiki/Incident_status>. Wikimedia incident stats by Krinkle, CodePen <https://codepen.io/Krinkle/full/wbYMZK>. Production Excellence: Month-over-month spreadsheet and plot <https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony…> . Report charts for Wikimedia-production-error project, Phabricator <https://phabricator.wikimedia.org/project/reports/1055/>.

1 0

Does anyone know about bulk loads on Wikibase/Wikidata?
by Henry Rosales 14 Apr '21

14 Apr '21

Hi all! I hope you are doing very well! I’m working on constructing an RDF dataset, which I’m ingesting to Wikibase and leverage its ecosystem: visualization, SPARQL endpoint, a script for dump creation, etc. For ingestion, I’m using the API (explained in https://www.wikidata.org/w/api.php), and it is going well, except for the time performance. I was exploring the Quick Statements module of Wikibase, which allows the ingestion through the UI. However, I need a way to ingest data automatically. I wonder if Wikibase/Wikidata has any script for bulk loads. Any comment or suggestion will be welcome. Best Regards, Henry Rosales

2 1

April 2021 Technical Community Newsletter
by Sarah R 14 Apr '21

14 Apr '21

Hi Everyone, We’re happy to announce the April 2021 edition of the Technical Community Newsletter <https://www.mediawiki.org/wiki/Technical_Community_Newsletter/2021/April> is now available. The newsletter is compiled by the Wikimedia Developer Advocacy Team. It aims to share highlights, news, and information of interest from and about the Wikimedia technical community. Check it out, and learn about what technical contributors have been up to this past quarter, upcoming conferences & calls for papers, and how to get involved. The Wikimedia Technical Community is large and diverse, and we know we can't capture everything perfectly. We welcome your ideas for future newsletters. Let us know what you would like to see or highlights you would like us to include. Subscribe to the Technical Community Newsletter <https://www.mediawiki.org/wiki/Newsletter:Technical_Community_Newsletter>, if you'd like to keep up with essential updates and information Kindly, Sarah R. Rodlund Senior Technical Writer, Developer Advocacy <https://www.mediawiki.org/wiki/Developer_Advocacy> srodlund(a)wikimedia.org

1 0

see stacktrace from functions within mediawiki
by Sandeep Gupta 14 Apr '21

14 Apr '21

Hello, I have recently started looking into mediawiki source code and trying to understand various codeflow. I am new to php as well. I want to see the stacktrace from database functions. I have put following debug line in the function : ob_start(); debug_print_backtrace(); $data = ob_get_clean(); fwrite(STDERR, $data . PHP_EOL); But I don't see any output in /var/log/nginx/mediawiki.error ( in current setup mediawiki is served by nginx). Any help would be great. Or any other approach to access the stacktrace would also work. Thanks Sandeep

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l April 2021