[Wikitech-l] Wikimedia production excellence (March 2019)

📘 Read on Phabricator at https://phabricator.wikimedia.org/phame/live/1/post/150/ ------- How’d we do in our strive for operational excellence last month? Read on to find out! ## 📊 Month in numbers * 8 documented incidents. [1] * 31 new Wikimedia-prod-error issues reported. [2] * 28 Wikimedia-prod-error issues closed. [3] The number of incidents this month was slightly above average compared to earlier this year (7 in February, 4 in January), and this time last year (4 in March 2018, 7 in February 2018). To read more about these incidents, their investigations, and conclusions; check https://wikitech.wikimedia.org/wiki/Incident_documentation#2019. There are currently 177 open Wikimedia-prod-error issues, similar to last month. [4]\ ## 📉 Current problems Take a look at the workboard and look for tasks that might need your help. The workboard lists known issues, grouped by the week in which they were first observed. → https://phabricator.wikimedia.org/tag/wikimedia-production-error/ Or help someone that’s already started with their patch, via "Tasks #Patch-For-Review". [5] Breakdown of recent months (past two weeks not included): * September: ✔️ Done! The last two issues were resolved. * October: ✔️ Done! The last issue was resolved. * November: 2 issues left (from 1.33-wmf.2). 1 issue was fixed. * December: 4 issues left (from 1.33-wmf.9). 1 issue was fixed. * January: 2 issues left (1.33-wmf.13 – 14). 1 issue was fixed. * February: 5 issues (1.33-wmf.16 – 19). * March: 10 new issues (1.33-wmf.20 – 23). By steward and software component, for issues remaining from February and March: * CPT / JobQueue: https://phabricator.wikimedia.org/T218692 * CPT / MCR: https://phabricator.wikimedia.org/T217329 * CPT / Parser: https://phabricator.wikimedia.org/T216664 * CPT / Revision-backend: https://phabricator.wikimedia.org/T220353, https://phabricator.wikimedia.org/T220257 * Growth / Page deletion: https://phabricator.wikimedia.org/T219816 * Growth / Watchlist: https://phabricator.wikimedia.org/T220245 * Language / Translate: https://phabricator.wikimedia.org/T217380, https://phabricator.wikimedia.org/T219736 * Multimedia / WikibaseMediaInfo: https://phabricator.wikimedia.org/T217285 * Operations / PHP-7.2: https://phabricator.wikimedia.org/T221347 * Search Platform / Elastica: https://phabricator.wikimedia.org/T219234 💡Ideas: To suggest an investigation to highlight in a future edition, feel free contact me by e-mail, or private message on IRC. ## 🎉 Thanks! Thanks to Aaron Schulz, Antoine Musso, Arlo Breault, Bartosz Dziewoński, Brad Jorsch, Daimona, James Forrester, Kosta Harlan, Leszek Manicki, Max Semenik, Niklas Laxström, Petar Petković, Santhosh Thottingal, Stephen Niedzielski, Subbu Sastry, Umherirrender, and Željko Filipin, and everyone else who helped last month by reporting, investigating, or patching errors found in production! Until next time, – Timo Tijhof 🦅 *“This isn’t flying. This is falling… with style!”* ------- Footnotes: [1] Incidents. – https://wikitech.wikimedia.org/wiki/Special:PrefixIndex?prefix=Incident+doc… [2] Tasks created. – https://phabricator.wikimedia.org/maniphest/query/As7RRbh3r2Bq/#R [3] Tasks closed. – https://phabricator.wikimedia.org/maniphest/query/9N6mnWSsJTsu/#R [4] Open tasks. – https://phabricator.wikimedia.org/maniphest/query/47MGY8BUDvRD/#R [5] Open tasks having #Patch-For-Review. – https://phabricator.wikimedia.org/maniphest/query/pzVPXPeMfRIz/#R