Wikitech-l April 2021

wikitech-l@lists.wikimedia.org

34 participants
50 discussions

by Tyler Cipriani

Hey all, This is a quick note to highlight that the REL1_36 branch has now been created for MediaWiki core and each of the extensions and skins in Wikimedia git[0]. This is the first step in the release process for MediaWiki 1.36, which should be out in late May 2021, approximately nine months after MediaWiki 1.35. The branches reflect (or are at least very close to) the code as of the last 'alpha' branch for the release, 1.36.0-wmf.38, which was deployed to Wikimedia wikis last week for MediaWiki itself and those extensions and skins available there. >From now on patches that land in the main development branch of MediaWiki and its bundled extensions and skins will be slated for the MediaWiki 1.37 release unless specifically backported[1]. If you are working on a critical bug fix that will affect the code in the release, once the patch has been merged into the development branch, you should propose it for backporting by cherry-picking to the REL1_36 branch. If you are working on a new feature, that should now not be backported. If you have an urgent case where the work should block release for everyone else, please file a task against the `mw 1.36-release` project on Phabricator.[2] If you have tickets that are tagged for `mw-1.36-release`, please finish them, untag them, or reach out to get them resolved in the next few days. We hope to issue the first release candidate, 1.36.0-rc.0, in two weeks' time, and if all goes well, to release MediaWiki 1.36.0 a few weeks after that. Thanks! – Tyler [0]: <https://www.mediawiki.org/wiki/Bundled_extensions_and_skins> [1]: <https://www.mediawiki.org/wiki/Backporting_fixes> [2]: <https://phabricator.wikimedia.org/tag/mw-1.36-release/>

2 years, 11 months

MediaWiki Extensions and Skins Security Release Supplement (1.31.13/1.35.2)

by Scott Bassett

Greetings- With the security/maintenance release of MediaWiki 1.31.13/1.35.2 [0], we would also like to provide this supplementary announcement of MediaWiki extensions and skins with now-public Phabricator tasks, security patches and backports [1]: == CommentBox == + (T270767, CVE-2021-31550) - Wg variables aren't validated by CommentBox - possible raw html insertion risk <https://gerrit.wikimedia.org/r/651934> == AbuseFilter == + (T272333, CVE-2021-31548) - Disallow the edit if blocking the user didn't succeed <https://gerrit.wikimedia.org/r/657092> == WikiLove == + (T270142, CVE-2021-31557) - mw.config.get( 'wikilove-anon' ) leaks the existence of hidden users <https://gerrit.wikimedia.org/r/q/Ibcd87abe01719222beadcfc0de13038c3021adef> == PageForms == + (T259433, CVE-2021-31551) - XSS issue in Extension:PageForms <https://gerrit.wikimedia.org/r/q/I5e0abbc2f80e6bda255b3b32a4df39a7fe7d3793> <https://gerrit.wikimedia.org/r/q/Ibe68b070ee791cd0c8e7f50eb04ac4e066b1512c> <https://gerrit.wikimedia.org/r/q/I20b63bd38779d2ccbe2d86f9879df85ca3b685f6> == AbuseFilter == + (T71617, CVE-2021-31546) - AbuseFilter logs suppression deletions <https://gerrit.wikimedia.org/r/q/I38a0a24fa32ca7a052b6940864a32b3856e84553> == AbuseFilter == + (T223654, CVE-2021-31547) - AbuseFilterCheckMatch API reveals suppressed edits and usernames <https://gerrit.wikimedia.org/r/q/I4900b1be73323599d74e3164447f81eded094d75> <https://gerrit.wikimedia.org/r/q/I3f7dbd8b873d411e37c8c3aac2339bf5ec36907d> == AbuseFilter == + (T71367, CVE-2021-31545) - page_recent_contributors leaks revdeleted user names <https://gerrit.wikimedia.org/r/q/I8d5ed9ca84282ee50832035af86123633fc88293> == AbuseFilter == + (T274152, CVE-2021-31549) - Special:AbuseFilter/examine reveals suppressed usernames <https://gerrit.wikimedia.org/r/q/I6063c02fa261c4cc0e6dbbb2db4e111eb85912c2> <https://gerrit.wikimedia.org/r/q/I71a6d521bd12931ce60eec4d2dc35af19146000f> == CheckUser == + (T275669, CVE-2021-31553) - Checkuser stores users to cu_log with trailing spaces, allowing all CUs to turn off Special:CheckuserLog at will <https://gerrit.wikimedia.org/r/666963> <https://gerrit.wikimedia.org/r/666964> == AbuseFilter == + (T152394, CVE-2021-31552) - AbuseFilter privacy concerns on action == 'createaccount' and 'accountname' <https://gerrit.wikimedia.org/r/q/I8bae477ad7e4d0190335363ac2decf28e4313da1> == AbuseFilter == + (T272244, CVE-2021-31554) - AbuseFilter blocks not working for account autocreations <https://gerrit.wikimedia.org/r/q/Ie1f4333d5b1c9d17fb2236fe38a31de427a4cc48> == OAuth == + (T277380, CVE-2021-31556) - OAuth doesn't validate length of oarc_rsa_key <https://gerrit.wikimedia.org/r/q/I13ff0350a9a0a3cd5ab3e1f82dd0d8d9c13cf9e9> == OAuth == + (T277388, CVE-2021-31555) - OAuth doesn't validate length of oarc_version <https://gerrit.wikimedia.org/r/q/I222c053b4b14ac1ad0f5b3a51565b1b9cd4c139d> The Wikimedia Security Team recommends updating these extensions and/or skins to the current master branch or relevant, supported release branch [2] as soon as possible. Some of the referenced Phabricator tasks above _may_ still be private. Unfortunately, when security issues are reported, sometimes sensitive information is exposed and since Phabricator is historical, we cannot make these tasks public without exposing this sensitive information. If you have any additional questions or concerns regarding this update, please feel free to contact security(a)wikimedia.org or file a security task within Phabricator [3]. [0] https://lists.wikimedia.org/pipermail/mediawiki-announce/2021-April/000272.… [1] https://phabricator.wikimedia.org/T270466 [2] https://www.mediawiki.org/wiki/Version_lifecycle [3] https://www.mediawiki.org/wiki/Reporting_security_bugs -- Scott Bassett sbassett(a)wikimedia.org

2 years, 11 months

Suggested values feature coming soon

by Timur Vorkul

// sorry for cross-posting Hi all, another feature will soon make usage of templates easier: Suggested values for parameters can be added to TemplateData and will be shown as a drop-down list in VisualEditor. This allows VisualEditor users to quickly select an appropriate value. Thus, it reduces potential errors and the effort needed to fill the template with values. It will still be possible to fill in values other than the suggested ones. More information can be found on the help page [1] and on the project page. [2] The WMDE Technical Wishes team plans to deploy this on all wikis on April 29. If you have any feedback, please let us know on the project’s talk page. [3] Timur for the Technical Wishes team [1] https://www.mediawiki.org/wiki/Help:TemplateData#suggestedvalues [2] https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Suggested_values_for_… [3] https://meta.wikimedia.org/wiki/Talk:WMDE_Technical_Wishes/Suggested_values… -- Timur Vorkul Technische Wünsche Wikimedia Deutschland e. V. | Tempelhofer Ufer 23–24 | 10963 Berlin Tel. (030) 219 158 26-0 https://wikimedia.de Bleiben Sie auf dem neuesten Stand! Aktuelle Nachrichten und spannende Geschichten rund um Wikimedia, Wikipedia und Freies Wissen im Newsletter: Zur Anmeldung <https://www.wikimedia.de/newsletter/>. Unsere Vision ist eine Welt, in der alle Menschen am Wissen der Menschheit teilhaben, es nutzen und mehren können. Helfen Sie uns dabei! https://spenden.wikimedia.de Wikimedia Deutschland — Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

2 years, 12 months

2021-04-21 Scrum of Scrums meeting notes

by Grace Gellerman

https://www.mediawiki.org/wiki/Scrum_of_scrums/2021-04-21 = 2021-04-21 = == Callouts == *** Cross-cutting: There are five filed blockers to the 1.36 RC.0 release (due tomorrow), but two are unclaimed and apparently not being worked on: https://phabricator.wikimedia.org/tag/mw-1.36-release/ * … == Gerrit patches or GitHub Pull Requests for reviews or feedback == * === No updates === CommTech, Product Infrastructure, Parsing, Language, Analytics, Cloud Services, Platform, Performance, Quality and Test Engineering, Release Engineering, Search Platform, Security == SoS Meeting Bookkeeping == * Updates: == Product == === Anti-Harassment Tools === * Blocked by: * Blocking: * Thank yous: ** Thanks to Sam Reed and James Forrester for code review and advice ** Thanks to DannyS for huge help on partial blocks review/QA * Updates: ** We're working on partial blocks against actions while we await next instructions on SecurePoll === Editing === * Blocked by: * Blocking: * Thank yous: * Updates: ** Will be working with Performance and others to reduce ParserCache expiry by ~7 days (30 -> 23) to free up some space, possibly caused by DiscsussionTools reply links. === Growth === * Blocked by: * Blocking: * Thank yous: * Updates: ** Continuing work on Add Link https://wikitech.wikimedia.org/wiki/Add_Link ** Preparing to enable on-wiki configuration next week https://phabricator.wikimedia.org/T274520 ** Migrating mentor data from user preferences to a dedicated table https://phabricator.wikimedia.org/T279853 ** Deploying Growth features as opt-in to eswiki === iOS native app === * Blocked by: N/A * Blocking: N/A * Thank yous: * Updates: Version 6.8 - finally improving language varient support - is currently rolling out. === Android native app === * Blocked by: N/A * Blocking: N/A * Thank yous: * Updates: Version w/ image recommendations suggested edits should be released 2 weeks from today. === Web === * Blocked by: * Blocking: * Thank yous: * Updates: ** Finishing up Vue.js search autocomplete widget project report: https://phabricator.wikimedia.org/T249430 ** Preparing a release for WVUI to address a handful of bugs: *** https://phabricator.wikimedia.org/T277256 *** https://phabricator.wikimedia.org/T277315 *** https://phabricator.wikimedia.org/T279088 ** Beginning work on the new personal tools treatment for Desktop Improvements: https://www.mediawiki.org/wiki/Reading/Web/Desktop_Improvements/Features#Us… === Structured Data === * Blocked by: * Blocking: * Thank yous: * Updates: ** Building a tool to manually test current implementation of the image recommendations API ** Whittled down dependencies for the new MediaSearch extension; it now only requires CirrusSearch === Abstract Wikipedia === * Blocked by: ** None. * Blocking: ** None known. * Thank yous: * Updates: ** Working on Phase δ (delta): https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Phases ** Started circulating early draft of our TDMP pitch for asynchronous parser fragments. === Inuka === * Blocked by: * Blocking: * Thank yous: * Updates: ** Wikipedia Preview *** Working on the version information https://phabricator.wikimedia.org/T278377 *** Working on the debug information https://phabricator.wikimedia.org/T278381 *** Working on the Wix integration https://phabricator.wikimedia.org/T278456 ** Wikipedia KaiOS *** Optimize the confirmed characters of the search method https://phabricator.wikimedia.org/T246786 === Library === * Blocked by: * Blocking: * Thank yous: ** Language team (Niklas and Abijeet) for helping us get partner descriptions into TranslateWiki * Updates: ** You can now translate Wikipedia Library partner descriptions! https://translatewiki.net/wiki/Special:Translate?group=wikipedia-library-pa… ** Started setting up Matomo to track page views ** Started setting up tags to be translatable via TranslateWiki ** Finishing up The Wikipedia Library extension work === Vue.js === * Blocked by: * Blocking: * Thank yous: * Updates: ** Starting work to unify button implementations in WVUI and better match what's in the design style guide (https://phabricator.wikimedia.org/T278509) ** Adding contributing guidelines to WVUI: https://gerrit.wikimedia.org/r/c/wvui/+/681178/3/contributing.md (under review) ** Planning long-term migration of projects to Vue == Technology == === Fundraising Tech === * Blocked by: * Blocking: * Thank yous: * Updates: ** More work on email prefs page https://phabricator.wikimedia.org/T268510, https://phabricator.wikimedia.org/T268511 ** Planning for integration with new API of backup card processor ** Work to migrate custom CRM code off drupal 7 ** CiviCRM contact deduplication enhancements ** Audit / reconciliation file processing improvements: https://phabricator.wikimedia.org/T277244, https://phabricator.wikimedia.org/T265545 === Site Reliability Engineering === * Blocked by: ** None * Blocking: ** None * Thank yous: * Updates: ** 100% of MW appservers now on buster, no more stretch special test hosts, mwdebug1003 removed ** Gitlab: Performance testing on WMCS done, defining backup and monitoring, documentation ** The MediaWiki chart can run a php application on kubernetes https://people.wikimedia.org/~oblivian/mw-on-k8s-shared-socket.png ** Scap logs will be transitioned to ECS 2021-05-04 === WMDE Technical Wishes === * Blocked by: * Blocking: * Thank yous: * Updates: ** Reusable edit count bucketing was accepted and is live for several extensions. == Cross-cutting == * Blocked by: ** [long term] Search Platform: PHP 8.0 work is long-term blocked on the migration to ElasticSearch 7.0 https://phabricator.wikimedia.org/T263142 (or at least 6.7). * Blocking: ** None known. * Thank yous: ** Huge thanks to the Web team, especially Jon Robson, for their work on fixing multiple blockers to the 1.36 release. * Updates: ** REL1_36 release process: *** [In calls-out] There are five filed blockers to the 1.36 RC.0 release (due tomorrow), but two are unclaimed and apparently not being worked on: https://phabricator.wikimedia.org/tag/mw-1.36-release/ ** CI tools: *** Nothing major. *** Next release of mediawiki-codesniffer likely soon. *** CI tools' upgrade status: https://libraryupgrader2.wmcloud.org/status?branch=master ** PHP 8.0: *** Nothing major. *** Our target is REL1_35 (and thus also REL1_36) as well as master. *** Upstream libraries: Elastica-related PHP code is theoretically the last one. *** Core: Some unit and integration tests still fail; thank you to everyone working on fixing them. [[Category:Scrum of scrums{{#translation:}}|*]]

2 years, 12 months

Working with templates: Looking for wikis who want early access to several improvements

by Johanna Strodt

Hello! Our team, Wikimedia Germany’s Technical Wishes project [1], is developing a series of improvements to make working with templates easier. Maybe you’ve already heard of some of the projects that have recently been deployed: [2] [3] [4]. We plan to release more improvements to make working with templates easier over the course of this year: It will become easier to find templates and to insert them into a page. [5] Removing a template from a page (with Visual Editor) will become easier. [6] Several problems with the existing interface for editing templates in the VisualEditor will be tackled. [7] Template creators and maintainers will be offered more guidance when they are managing TemplateData with the template documentation editor. [8] Now we’re looking for a few wikis who want to be the first to benefit from these changes. If your wiki community is interested in this, all the improvements listed above would be deployed to your wiki in a series of releases, likely between May and July 2021. Of course, each deployment would be announced beforehand. We’re hoping for wikis of different sizes, both Wikipedias and sister projects, and wikis with different scripts and writing directions. The wikis should use TemplateData and ideally have the VisualEditor as a default feature. Is your wiki community interested to have early access to these improvements? Email me or let me know on my talk page.[9] Best, Johanna [1] Technical Wishes project: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes [2] Bracket matching: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Bracket_Matching [3] Improved color scheme for syntax highlighting: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Improved_Color_Scheme… [4] Line numbering: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Line_Numbering [5] Finding and inserting templates: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Finding_and_inserting… [6] Removing a template from a page using the VisualEditor: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Removing_a_template_f… [7] VisualEditor template dialog improvements: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/VisualEditor_template… [8] TemplateData editor improvements: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Template_data_editor_… [9] talk page: https://meta.wikimedia.org/wiki/User_talk:Johanna_Strodt_(WMDE) -- Johanna Strodt Projektmanagerin Kommunikation Communitys Technische Wunschliste Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Tel. (030) 219 158 26-0 https://wikimedia.de Unsere Vision ist eine Welt, in der alle Menschen am Wissens der Menschheit teilhaben, es nutzen und mehren können. Helfen Sie uns dabei! https://spenden.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

2 years, 12 months

Upstream release notifications on Phabricator

by Kunal Mehta

Hi, LibUp can now notify you about new upstream releases in Phabricator. An example: <https://phabricator.wikimedia.org/T280117>. It'll check daily to see if an upstream project has a new release, and if so, file a Phabricator task. If it already filed a task that's still open, it'll leave a new comment instead. This is separate from LibUp's library upgrading in Gerrit, so it should be able to support just about any project and leave it up to the humans to decide what to do about the new version. If you'd like to get notifications for your project, file a bug in the #LibUp Phabricator project, or send a patch to the labs/libraryupgrader/config Gerrit repository (see README for instructions). Behind the scenes LibUp is using <https://release-monitoring.org/> which supports checking a bunch of different package websites so we don't have to reimplement it all. -- Legoktm

2 years, 12 months

Global Watchlist extension - grant request #2

by DannyS712 Wiki

Dear all, I wanted to let you know that I have requested[1] a project grant to fund the continued development of the GlobalWatchlist extension[2]. Last year, I received a grant from the WMF to create the extension[3]. The extension was created and deployed, and is now available on metawiki. I hope to continue development of the extension - if you support this, I hope you'll consider endorsing the request. Thanks, --DannyS712 [1] https://meta.wikimedia.org/wiki/Grants:Project/DannyS712/Continued_work_on_… [2] https://www.mediawiki.org/wiki/Special:MyLanguage/Extension:GlobalWatchlist [3] https://meta.wikimedia.org/wiki/Grants:Project/DannyS712/Create_a_global_wa…

2 years, 12 months

Summary of this week's deployment of 1.37.0-wmf.1: all good!

by Lars Wirzenius

This is a summary of this week's deployment of the 1.37.0-wmf.1 branch of MediaWiki and its extensions (also known as "the train"). The primary person in charge this week is Jeena Huneidi, with Dan Duvall as backup, both from the Release Engineering team. The summary task for this week is https://phabricator.wikimedia.org/T278345 This was the first week of the 1.37.0 series. The 1.36.0 release is being made separately, and our sites are moving on to run weekly releases of the new series. It's not a momentous change. We start a series couple of times a year. The version number changes, but otherwise the changes this week are fairly mundane and usual. (There's no train deployment next week, due to a holiday on Thursday.) The new version is running all sites. There were a total of three blocker reported and resolved: - T279008 - mw-parser-output now clears after block, which is a noticeable change for Category pages where infoboxes used to render floated alongside the automatic content https://phabricator.wikimedia.org/T279008 - T279113 - Unable to parse title at new Title in Special:MediaSearch https://phabricator.wikimedia.org/T279113 - T280082 - Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'mediawikiwiki.discussiontools_subscription' doesn't exist (10.64.16.7) https://phabricator.wikimedia.org/T280082 As usual, a whole bunch of people helped to find, triage, analyze, fix, or work around problems this. Release Engineering thanks everyone, without help we wouldn't be able to deploy MediaWiki. - Anne Tomasevich - Bartosz Dziewoński - Carly Bogen - Cormac Parle - Innovative - JJMC89 - Jon Robson - Lucas Werkmeister - Mike_Peel - Rummana Yasmeen - Tacsipacsi - Timo Tijhof - TTO - Umherirrender - Urbanecm - Zabe There may have been other people, and if so, I apologize for not including them on the list above. Have a good weekend. Be well. Be safe. For more information, please see: - https://phabricator.wikimedia.org/T278345 - https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys - https://versions.toolforge.org/ -- WMF release engineering team | he/him or they/them "Imagine a world in which every single human being can freely share in the sum of all knowledge."

3 years

Performance regression expected with Debian Buster upgrade

by Kunal Mehta

Hi, We're currently in the process of upgrading the MediaWiki servers to Debian Buster and expect a performance regression to come with it. The cause appears to be better Spectre[1] mitigations in the Buster 4.19 kernel, which we can't disable. Most of the effect is seen in code that ends up invoking syscalls like filemtime, file_get_contents, etc. I posted some numbers and charts on the Phabricator investigation ticket[2]. For normal requests it looks like ~5% worse for p50/p75 and around ~13% for p95/p99. API requests look much worse, at 10% for p50 22% for p75. What now? We're going to continue with the upgrade as planned, but we also need help to try and make some performance improvements to reduce the impact of the regression. The PHP profiling flamegraphs[3] are a great tool to use to identify potentially slow spots. We now also have flamegraphs that only contain Buster requests. I created a set of differential flamegraphs[4] that compare Stretch vs Buster so you can see what specific areas slowed down. You can also use WikimediaDebug/XHGui[5] to profile a specific request. mwdebug1001/mwdebug1002 are Stretch and mwdebug1003 is Buster. If you have questions or suggestions please ask or let us know. Thanks to everyone who helped with the investigation and those who've started working on improvements already. [1] https://en.wikipedia.org/wiki/Spectre_(security_vulnerability) [2] https://phabricator.wikimedia.org/T273312#6802330 [3] https://performance.wikimedia.org/php-profiling/ [4] https://people.wikimedia.org/~legoktm/T273312/data/clean/images/flamegraphs/ [5] https://wikitech.wikimedia.org/wiki/WikimediaDebug#Request_profiling -- Kunal

3 years

Production Excellence #30: March 2021

by Krinkle

How’d we do in our strive for operational excellence last month? Read on to find out! Read on Phabricator at https://phabricator.wikimedia.org/phame/post/view/229/ Incidents 2 documented incidents. That's average for this time of year, when we usually had <https://codepen.io/Krinkle/full/wbYMZK> 1-4 incidents. Learn about recent incidents at Incident status <https://wikitech.wikimedia.org/wiki/Incident_status> on Wikitech, or Preventive measures <https://phabricator.wikimedia.org/project/view/4758/> in Phabricator. ------- Trends In March we made significant progress on the outstanding errors of previous months. Several of the 2020 months are finally starting to empty out. But with over 30 new tasks from March itself remaining, we did not break even, and ended up slightly higher than last month. This could be reversing two positive trends, but I hope not. Firstly, there was a steep increase in the number of new production errors that were not resolved within the same month. This is counter the positive trend we started in November. The past four months typically saw 10-20 errors outlive their month of discovery, and this past month saw 34 of its 48 new errors remain unresolved. Secondly, we saw the overall number of unresolved errors increase again. This January <https://phabricator.wikimedia.org/phame/post/view/227/production_excellence…> began a downward trend for the first time in thirteen months, which continued nicely through February. But, this past month we broke even and even pushed upward by one task. I hope this is just a breather and we can continue our way downward. Figure 1, Figure 2: Unresolved error reports stacked by month. <https://phabricator.wikimedia.org/phame/post/view/229/production_excellence…> ------- Outstanding errors Take a look at the workboard <https://phabricator.wikimedia.org/tag/wikimedia-production-error/> and look for tasks that could use your help. Summary over recent months, per spreadsheet <https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony…> : Jul 2019 (0 of 18 left) ✅ Last two tasks resolved! -2 Aug 2019 (1 of 14 left) ⚠️ *Unchanged (and over one year old).* Oct 2019 (3 of 12 left) ⬇️ One task resolved. -1 Nov 2019 (0 of 5 left) ✅ Last task resolved! -1 Dec 2019 (0 of 9 left) ✅ Last task resolved! -1 Jan 2020 (2 of 7 left) ⬇️ One task resolved. -1 Feb 2020 (0 of 7 left) ✅ Last task resolved! -1 Mar 2020 (2 of 2 left) ⚠️ *Unchanged (and over one year old).* Apr 2020 (5 of 14 left) ⬇️ Four tasks resolved. -4 May 2020 (5 of 14 left) ⬇️ One task resolved. -1 Jun 2020 (6 of 14 left) ⬇️ One task resolved. -1 Jul 2020 (5 of 24 issues <https://phabricator.wikimedia.org/maniphest/query/s__D8Kd0xuQH/#R>) ⬇️ Four tasks resolved. -4 Aug 2020 (15 of 53 issues <https://phabricator.wikimedia.org/maniphest/query/hu1yhWu4sXkP/#R>) ⬇️ Five tasks resolved. -5 Sep 2020 (7 of 33 issues <https://phabricator.wikimedia.org/maniphest/query/CGFQViLShnOY/#R>) ⬇️ One task resolved. -1 Oct 2020 (22 of 69 issues <https://phabricator.wikimedia.org/maniphest/query/MYnnBybPTYpd/#R>) ⬇️ Four tasks resolved. -4 Nov 2020 (9 of 38 issues <https://phabricator.wikimedia.org/maniphest/query/CkC_VqQq5VC0/#R>) ⬇️ Two tasks resolved. -2 Dec 2020 (11 of 33 issues <https://phabricator.wikimedia.org/maniphest/query/10NQy74iKaZJ/#R>) ⬇️ One task resolved. -1 Jan 2021 (4 of 50 issues <https://phabricator.wikimedia.org/maniphest/query/WIP9W8q54HB6/#R>) ⬇️ One task resolved. -1 Feb 2021 (9 of 20 issues <https://phabricator.wikimedia.org/maniphest/query/5MzPJAb5oJgv/#R>) ⬇️ Two tasks resolved. -2 *Mar 2021* (34 of 48 issues <https://phabricator.wikimedia.org/maniphest/query/RsVPep46KRY4/#R>) 34 new tasks survived and remain unresolved. +48; -14 ------- Tally 138 issues open, as of Excellence #29 <https://phabricator.wikimedia.org/phame/post/view/228/production_excellence…> (6 Mar 2021). -33 issues closed, of the previous 138 open issues. +34 new issues that survived March 2021. 139 issues open, as of today (2 Apr 2021). -------Thanks! Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks! Until next time, – Timo Tijhof Footnotes: Incident status, Wikitech <https://wikitech.wikimedia.org/wiki/Incident_status>. Wikimedia incident stats by Krinkle, CodePen <https://codepen.io/Krinkle/full/wbYMZK>. Production Excellence: Month-over-month spreadsheet and plot <https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony…> . Report charts for Wikimedia-production-error project, Phabricator <https://phabricator.wikimedia.org/project/reports/1055/>.

3 years

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l April 2021