Wikitech-l November 2019

wikitech-l@lists.wikimedia.org

53 participants
71 discussions

WM Technical Conference 2019 update - day 2
by Deb Tankersley 14 Nov '19

14 Nov '19

Hello, We wanted to give a quick update on what happened during day 2 of the Wikimedia Technical Conference <https://www.mediawiki.org/wiki/Wikimedia_Technical_Conference/2019>, in Atlanta, Georgia. *Wednesday <https://www.mediawiki.org/wiki/Wikimedia_Technical_Conference/2019/Program#…>, November 13, 2019:* Morning sessions were large groups talking about Developer Productivity & onwiki tooling - lots of really good conversations about templates, gadgets, scripts, and modules (ticket <https://phabricator.wikimedia.org/T234661>). The afternoon sessions covered on: - Platform Stewardship (ticket <https://phabricator.wikimedia.org/T234657>) - On-wiki documentation & documentation as code (ticket <https://phabricator.wikimedia.org/T234634>) - Standardizing QA best practices (ticket <https://phabricator.wikimedia.org/T234653>) - Best practices and useful methods for remote teams - volunteers and staff alike (ticket <https://phabricator.wikimedia.org/T234658>) - Code Health/Code Review (ticket <https://phabricator.wikimedia.org/T234660>) - Quo Vadis Beta Cluster? Towards production-like testing and staging environments (ticket <https://phabricator.wikimedia.org/T234643>) - Learning how WMDE improved their organizational structure (ticket <https://phabricator.wikimedia.org/T234652>) - Release "strategies" (containers) for MediaWiki and other elements of Wikimedia platform (ticket <https://phabricator.wikimedia.org/T234644>) We ended the afternoon with cleaning up documentation from the day's sessions and moving them into the phabricator tickets (to keep all the data in one place), taking pictures of the session sticky notes / diagrams, and then adding them also into the phabricator tickets. Photos and documentation that have been uploaded can be viewed using this category <https://commons.wikimedia.org/wiki/Category:Wikimedia_Technical_Conference_…> (look for day 2 <https://commons.wikimedia.org/wiki/Category:Wikimedia_Technical_Conference_…>) on Commons: Category:Wikimedia Technical Conference 2019 (sample media: thank you goat <https://commons.wikimedia.org/wiki/File:WMTC19_thank_you_goat.jpg> and Amir and Global Templates <https://commons.wikimedia.org/wiki/File:Wikimedia_Technical_Conference_2019…> ). Cheers, The Wikimedia Technical Conference Program Committee -- deb tankersley (she/her) program manager, engineering Wikimedia Foundation

1 0

Scrum of scrums/2019-11-13
by Željko Filipin 13 Nov '19

13 Nov '19

Hi, for HTML version see https://www.mediawiki.org/wiki/Scrum_of_scrums/2019-11-13 Željko -- = 2019-11-13 = == Callouts == * Release Engineering: no train this and the next week because of Wikimedia Technical Conference and team offsite == Product == === Editing === * Updates: ** Talk pages: developing high fidelity prototype for replying to comments === Growth === * Updates: ** Finalizing v1.0 of newcomer tasks recommendation module for release next week ** Working with Scoring, Research and Search teams in the coming weeks to implement topic-matching in the task recommendation module using ORES modeling === iOS native app === * Blocked by: **Core Platform ***Section headers in compare endpoint ([[phab:T236964]]) ***Intermediate counts endpoint ([[phab:T234940]]) * Updates: ** moving 6.4.1 hotfix to 6.5 ***3D touch fix in article view ** 6.5 wrapping up development - [[phab:project/view/4245]] ***History/diff endpoints are live 🎉 thanks all! === Android native app === * Updates: **In the final stages of work towards a major release of SuggestedEdits feature **Re-started work on integrating with mobile-html endpoints for page rendering ([[phab:project/view/4318]]) === Product Infrastructure === * Updates: ** MachineVision extension is enabled for testing Computer-Aided Tagging this week === Structured Data === * Blocking: ** Search Platform: Data dumps for SDC: [[phab:T221917]] === Parsing === * Blocking: ** We are partially blocking VE testing of Parsoid/PHP and maybe completely blocking CX testing of Parsoid/PHP because of [[phab:T238161]] * Updates: ** Thanks to Parsoid client teams (VE, CX, Flow, MCS) for starting to test against Parsoid/PHP and reporting issues / filing bugs. ** We are considering enabling Parsoid/PHP on testwiki tomorrow / Monday to enable additional testing in a production-like context. ** We are close to storing Parsoid/PHP content in RESTBase storage. We have a couple issues to resolve before that. == Technology == === Analytics === * Blocking: ** Wikidata: [[gerrit:549859]] === Fundraising Tech === * Updates: ** CiviCRM *** More improvements to contact deduplication *** Creating button to send end of year summary receipt on demand *** Fixing more records of bulk mailing activity imported from Acoustic ** Payments-wiki *** Getting a bunch of variants ready for testing during end of year campaign ** CentralNotice *** Rebasing and revising sub-national geotargeting *** Refinement of processing for new impressions data pipeline === Core Platform === * Blocking: ** iOS native app *** Section headers in compare endpoint ([[phab:T236964]]) ** Intermediate counts endpoint ([[phab:T234940]]) * Updates: ** MW REST API live ** continue work on next version of API, OAuth 2.0, sessions ** planning for new === Engineering Productivity === ==== Release Engineering ==== * Updates: ** half of the team at Wikimedia Technical Conference ** Train Health *** Last week: 1.35.0-wmf.5 - [[phab:T233853]] *** This week: no train because of Wikimedia Technical Conference *** Next week: no train because of team offsite === Search Platform === * Blocked by: ** Structured Data: Data dumps for SDC: [[phab:T221917]] * Updates: ** WDQS is having high update lag for the last week - [[phab:T238229]] ** Commons search seems to have stopped indexing statements on 30 October 2019 - [[phab:T237849]] ** Search engine fails in Special:IndexPage - [[phab:T232710]] ** Make it possible to bypass automatic redirection to exact matches in commons - [[phab:T235263]] * Stuff no one else cares about: ** Warning: Deprecated field [inline] used, expected [source] instead - [[phab:T237000]] === Security === * Updates: ** Next up: Security review for MachineVision libraries - [[phab:T237588]] ** In Progress: Review the FileImporter extension's load of config from wiki pages for security concerns - [[phab:T202071]] ** In Progress: Security concept review for newcomer tasks on Special:Homepage - [[phab:T235720]] ** Waiting: Security readiness review for the MachineVision extension - [[phab:T227346]] == Wikimedia DE == === Wikidata === * Blocked by: ** Analytics: [[gerrit:549859]]

1 0

WM Technical Conference 2019 update - day 1
by Deb Tankersley 12 Nov '19

12 Nov '19

Hi all, We wanted to give a quick update on what happened during day 1 of the Wikimedia Technical Conference <https://www.mediawiki.org/wiki/Wikimedia_Technical_Conference/2019>, in Atlanta, Georgia. *Tuesday <https://www.mediawiki.org/wiki/Wikimedia_Technical_Conference/2019/Program#…>, November 12, 2019:* The first day of the event started out with the morning sessions concentrating on welcoming the participants, reviewing what has been done since last year's event, and what we want to accomplish at this year's event (notes <https://etherpad.wikimedia.org/p/WMTC19-intro> / slides <https://commons.wikimedia.org/wiki/File:Wikimedia_Tech_Conf_2019_Opening_sl…> ). The afternoon sessions concentrated on: - Our local development environment - Core (notes <https://etherpad.wikimedia.org/p/WMTC19-T234632>, ticket <https://phabricator.wikimedia.org/T234632>) - Front-end modernization and standardization (notes <https://etherpad.wikimedia.org/p/WMTC19-T234649>, ticket <https://phabricator.wikimedia.org/T234649>) - Componentization and sharing with the open source community (notes <https://etherpad.wikimedia.org/p/WMTC19-T234654>, ticket <https://phabricator.wikimedia.org/T234654>) - System level testing: patterns and anti-patterns with Selenium (notes <https://etherpad.wikimedia.org/p/WMTC19-T234635>, ticket <https://phabricator.wikimedia.org/T234635>) - Continuous Delivery/Deployment in Wikimedia: The Future of the Deployment Pipeline (notes <https://etherpad.wikimedia.org/p/WMTC19-T234641>, ticket <https://phabricator.wikimedia.org/T234641>) - Integrating contributions from other teams or volunteers (notes <https://etherpad.wikimedia.org/p/WMTC19-T234662>, ticket <https://phabricator.wikimedia.org/T234662>) We ended the afternoon with finishing up documentation from the day's sessions, taking pictures of the session notes, and then uploading into phabricator (goat sticker sample <https://commons.wikimedia.org/wiki/File:WMTC19_Goat_stickers_day_1.jpg> and badge <https://commons.wikimedia.org/wiki/File:WMTC19_badge_and_candy.jpg>). Photos and documentation that have been uploaded can be viewed using this category <https://commons.wikimedia.org/wiki/Category:Wikimedia_Technical_Conference_…> : Category:Wikimedia Technical Conference 2019. Cheers, The Wikimedia Technical Conference Program Committee -- deb tankersley (she/her) program manager, engineering Wikimedia Foundation

1 0

Pre-MCR database fields are no longer being written on group 0 (and soon on the rest of the Wikimedia wikis too)
by Brad Jorsch (Anomie) 12 Nov '19

12 Nov '19

We've reached the point in deployment of the Multi-Content Revisions changes where we're turning off the writing of data to deprecated fields. In particular, rev_text_id and ar_text_id will be 0 in all revisions, and rev_content_model, rev_content_format, and corresponding fields in the archive table will always be null. This has been the case on testwiki since October 29, and on the remaining group 0 wikis since a few minutes ago. Assuming no issues turn up, we plan to roll it out to group 1 on Thursday and group 2 early next week. If you find any MediaWiki bugs that seem as if they might be related to this change, please file them in Phabricator and tag #Core-Platform-Team. Thanks! -- Brad Jorsch (Anomie) Senior Software Engineer Wikimedia Foundation

1 0

GCI Mentor
by Anant Jain 11 Nov '19

11 Nov '19

Hello community, Myself Anant Jain, and I'm interested to be a mentor in Google Code-In 2019 Please let me know how to get started. Thanks & Regards, Anant Jain https://anantjain.me 8800624328, 7011028147

2 1

Clean up unused configs
by Amir Sarabadani 11 Nov '19

11 Nov '19

Hey, It happens quite often that people add a temporary config variable to the code and then mediawiki-config repo, they remove it from code and forget to remove the config in mediawiki-config. My quick check found these config variables to be unused: $wgSpamBlacklistEventLogging $wgUseContributionTracking $wgMFLazyLoadReferences $wgMFExperiments $wgDepictsQualifierProperties $wgMediaInfoEnableOtherStatements $wgGEHelpPanelSearchEnabled $wgDisableRollbackConfirmationFeature $wgCirrusSearchEnableSearchLogging Unused config variables are bad because they make our overly large config file even bigger (IS.php is already over 20k lines) and it can be confusing (like https://phabricator.wikimedia.org/T225183). I will add more unused config variables to the list by improving the search but please check and remove them if you introduced any of the aforementioned variables. Thanks Best -- Amir (he/him)

1 0

none
by joelfokou99＠gmail.com 10 Nov '19

10 Nov '19

Hello everyone. I am Joel and I am new in this mailing list. Please I wish to participate to the GsoC this year and I don’t now how to process. I am studying computer engineering in University of Buea (Cameroon). I have Experiences in HTML/CSS, JavaScript and PHP. Thank you beforehand. Sent from Mail for Windows 10

2 1

GitHub's "Automated Security Fixes" have been disabled on the Wikimedia Org
by Sam Reed 09 Nov '19

09 Nov '19

Just a heads up that "Automated Security Fixes" have been disabled on the Wikimedia GitHub org. See [1] The reason for this is that it generates pull requests on non canonical repositories (ie where Gerrit is the default development location) that require developers to close them. If this is a feature you want on your repo generally, because you canonically develop on GitHub, you can re-enable these on your repo by clicking the "Security" tab, and then selecting "Automated Security Fixes" in the top right corner. See [2] for more info. If you do develop canonically in GitHub, please let us know at [3]. Note, this doesn't affect the security alerts related to outdated packages etc in a repo. Thanks! Sam [1] https://phabricator.wikimedia.org/T237337 [2] https://help.github.com/en/github/managing-security-vulnerabilities/configu… [3] https://phabricator.wikimedia.org/T237470 [4] https://help.github.com/en/github/managing-security-vulnerabilities/viewing…

1 0

Production Excellence (October 2019)
by Krinkle 08 Nov '19

08 Nov '19

📘 Read on Phabricator at https://phabricator.wikimedia.org/phame/post/view/178/ ------- How’d we do in our strive for operational excellence last month? Read on to find out! ## 📊 Month in numbers * 3 documented incidents. [1] * 33 new Wikimedia-prod-error reports. [2] * 30 Wikimedia-prod-error reports closed. [3] * 207 currently open Wikimedia-prod-error reports in total. [4] There were three recorded incidents last month, which is slightly below our median of the past two years. Explore this data: < https://codepen.io/Krinkle/full/wbYMZK>. To read more about these incidents, their investigations, and pending actionables; check https://wikitech.wikimedia.org/wiki/Incident_documentation#2019 ------- ## *️⃣ To Log or not To Log MediaWiki uses the PSR-3 compliant Monolog library to send messages to Logstash (via rsyslog and Kafka [5]). These messages are used to automatically detect (by quantity) when the production cluster is in an unstable state. For example, due to an increase in application errors when deploying code, or if a backend system is failing. Two distinct issues hampered the storing of these messages this month, and both affected us simultaneously. *Elasticsearch mapping limit* The Elasticsearch storage behind Logstash optimises responses to Logstash queries with an index. This index has an upper limit to how many distinct fields (or columns) it can have. When reached, messages with fields not yet in the index are discarded. Our Logstash indexes are sharded by date and source (one for “mediawiki”, one for “syslog”, and one for everthing else). This meant that error messages were only stored if they only contained fields used before, by other errors stored that day. Which in turn would only succeed if that day’s columns weren’t already fully taken. A seemingly random subset of error messages was then rejected for a full day. Each day it got a new chance at reserving its columns, so long as the specific kind of error is triggered early enough. To unblock deployment automation and monitoring of MediaWiki, an interim solution was devised. The subset of messages from “mediawiki” that deal with application errors now have their own index shard. These error reports follow a consistent structure, and contain no free-form context fields. As such, this index (hopefully) can’t reach its mapping limit or suffer message loss. The general index mapping limit was also raised from 1000 to 2000. For now that means we’re not dropping any non-critical/debug messages. More information about the incident at https://phabricator.wikimedia.org/T234564. The general issue with accommodating debug messages in Logstash long-term, is tracked at https://phabricator.wikimedia.org/T180051. Thanks Bartosz, and Keith Herron. *Crash handling* Wikimedia’s PHP configuration has a “crash handler” that kicks in if everything else fails. For example, when the memory limit or execution timeout is reached, or if some crucial part of MediaWiki fails very early on. In that case our crash handler renders a Wikimedia-branded system error page (separate from MediaWiki and its skins). It also increments a counter metric for monitoring purposes, and sends a detailed report to Logstash. In migrating the crash handler from HHVM to PHP7, one part of the puzzle was forgotten. Namely the Logstash configuration that forwards these reports from php-fpm’s syslog channel to the one for mediawiki. As such, our deployment automation and several Logstash dashboards were blind to a subset of potential fatal errors for a few days. Regressions during that week were instead found by manually digging through the raw feed of the php-fpm channel instead. As a temporary measure, Scap was updated to consider the php-fpm’s channel as well in its automation that decides whether a deployment is “green”. We’ve created new Logstash configurations that forward PHP7 crashes in a similar way as we did for HHVM in the past. Bookmarked MW dashboards/queries you have for Logstash now provide a complete picture once again. Thanks Effie, and Cole White! – https://phabricator.wikimedia.org/T234283 ------- ## 📉 Outstanding reports Take a look at the workboard and look for tasks that might need your help. The workboard lists error reports, grouped by the month in which they were first observed. → https://phabricator.wikimedia.org/tag/wikimedia-production-error/ Or help someone that’s already started with their patch: → https://phabricator.wikimedia.org/maniphest/query/CFzrDj3vFbE_/#R Breakdown of recent months (past two weeks not included): * March: 1 report fixed. (3 of 10 reports left). * April: 8 of 14 reports left (unchanged). ⚠️ * May: (All clear!) * June: 9 of 11 reports left (unchanged). ⚠️ * July: 13 of 18 reports left (unchanged). * August: 2 reports were fixed! (6 of 14 reports left). * September: 2 reports were fixed! (10 of 12 new reports left). * October: 12 new reports survived the month of October. ------- ##### 🎉 Thanks! Thank you, to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks! Until next time, – Timo Tijhof 🌴“Gotta love crab. In time, too. I couldn't take much more of those coconuts. Coconut milk is a natural laxative. That's something Gilligan never told us.” ------- Footnotes: [1] Incidents. – https://wikitech.wikimedia.org/wiki/Special:PrefixIndex?prefix=Incident+doc… [2] Tasks created. – https://phabricator.wikimedia.org/maniphest/query/sJI9Af6LqvKL/#R [3] Tasks closed. – https://phabricator.wikimedia.org/maniphest/query/scW28HMEJemU/#R [4] Open tasks. – https://phabricator.wikimedia.org/maniphest/query/47MGY8BUDvRD/#R [5] https://wikitech.wikimedia.org/wiki/Logstash/Interface

1 0

[Data Release] Active Editors by country
by Dan Andreescu 07 Nov '19

07 Nov '19

Today we are releasing a new dataset meant to help us understand the impact of grants and programs on editing. This data was requested several years ago, and we took a long time to bring in the privacy and security experts whose help we needed to release it. With that work done, you can download the data here: https://dumps.wikimedia.org/other/geoeditors/ and read about it here: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Geoeditors/Pu… You can send questions or comments on this thread or on the discussion page.

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l November 2019