Wikitech-l November 2019

wikitech-l@lists.wikimedia.org

53 participants
66 discussions

Clean up unused configs
by Amir Sarabadani 11 Nov '19

11 Nov '19

Hey, It happens quite often that people add a temporary config variable to the code and then mediawiki-config repo, they remove it from code and forget to remove the config in mediawiki-config. My quick check found these config variables to be unused: $wgSpamBlacklistEventLogging $wgUseContributionTracking $wgMFLazyLoadReferences $wgMFExperiments $wgDepictsQualifierProperties $wgMediaInfoEnableOtherStatements $wgGEHelpPanelSearchEnabled $wgDisableRollbackConfirmationFeature $wgCirrusSearchEnableSearchLogging Unused config variables are bad because they make our overly large config file even bigger (IS.php is already over 20k lines) and it can be confusing (like https://phabricator.wikimedia.org/T225183). I will add more unused config variables to the list by improving the search but please check and remove them if you introduced any of the aforementioned variables. Thanks Best -- Amir (he/him)

1 0

none
by joelfokou99＠gmail.com 10 Nov '19

10 Nov '19

Hello everyone. I am Joel and I am new in this mailing list. Please I wish to participate to the GsoC this year and I don’t now how to process. I am studying computer engineering in University of Buea (Cameroon). I have Experiences in HTML/CSS, JavaScript and PHP. Thank you beforehand. Sent from Mail for Windows 10

2 1

GitHub's "Automated Security Fixes" have been disabled on the Wikimedia Org
by Sam Reed 08 Nov '19

08 Nov '19

Just a heads up that "Automated Security Fixes" have been disabled on the Wikimedia GitHub org. See [1] The reason for this is that it generates pull requests on non canonical repositories (ie where Gerrit is the default development location) that require developers to close them. If this is a feature you want on your repo generally, because you canonically develop on GitHub, you can re-enable these on your repo by clicking the "Security" tab, and then selecting "Automated Security Fixes" in the top right corner. See [2] for more info. If you do develop canonically in GitHub, please let us know at [3]. Note, this doesn't affect the security alerts related to outdated packages etc in a repo. Thanks! Sam [1] https://phabricator.wikimedia.org/T237337 [2] https://help.github.com/en/github/managing-security-vulnerabilities/configu… [3] https://phabricator.wikimedia.org/T237470 [4] https://help.github.com/en/github/managing-security-vulnerabilities/viewing…

1 0

Production Excellence (October 2019)
by Krinkle 08 Nov '19

08 Nov '19

📘 Read on Phabricator at https://phabricator.wikimedia.org/phame/post/view/178/ ------- How’d we do in our strive for operational excellence last month? Read on to find out! ## 📊 Month in numbers * 3 documented incidents. [1] * 33 new Wikimedia-prod-error reports. [2] * 30 Wikimedia-prod-error reports closed. [3] * 207 currently open Wikimedia-prod-error reports in total. [4] There were three recorded incidents last month, which is slightly below our median of the past two years. Explore this data: < https://codepen.io/Krinkle/full/wbYMZK>. To read more about these incidents, their investigations, and pending actionables; check https://wikitech.wikimedia.org/wiki/Incident_documentation#2019 ------- ## *️⃣ To Log or not To Log MediaWiki uses the PSR-3 compliant Monolog library to send messages to Logstash (via rsyslog and Kafka [5]). These messages are used to automatically detect (by quantity) when the production cluster is in an unstable state. For example, due to an increase in application errors when deploying code, or if a backend system is failing. Two distinct issues hampered the storing of these messages this month, and both affected us simultaneously. *Elasticsearch mapping limit* The Elasticsearch storage behind Logstash optimises responses to Logstash queries with an index. This index has an upper limit to how many distinct fields (or columns) it can have. When reached, messages with fields not yet in the index are discarded. Our Logstash indexes are sharded by date and source (one for “mediawiki”, one for “syslog”, and one for everthing else). This meant that error messages were only stored if they only contained fields used before, by other errors stored that day. Which in turn would only succeed if that day’s columns weren’t already fully taken. A seemingly random subset of error messages was then rejected for a full day. Each day it got a new chance at reserving its columns, so long as the specific kind of error is triggered early enough. To unblock deployment automation and monitoring of MediaWiki, an interim solution was devised. The subset of messages from “mediawiki” that deal with application errors now have their own index shard. These error reports follow a consistent structure, and contain no free-form context fields. As such, this index (hopefully) can’t reach its mapping limit or suffer message loss. The general index mapping limit was also raised from 1000 to 2000. For now that means we’re not dropping any non-critical/debug messages. More information about the incident at https://phabricator.wikimedia.org/T234564. The general issue with accommodating debug messages in Logstash long-term, is tracked at https://phabricator.wikimedia.org/T180051. Thanks Bartosz, and Keith Herron. *Crash handling* Wikimedia’s PHP configuration has a “crash handler” that kicks in if everything else fails. For example, when the memory limit or execution timeout is reached, or if some crucial part of MediaWiki fails very early on. In that case our crash handler renders a Wikimedia-branded system error page (separate from MediaWiki and its skins). It also increments a counter metric for monitoring purposes, and sends a detailed report to Logstash. In migrating the crash handler from HHVM to PHP7, one part of the puzzle was forgotten. Namely the Logstash configuration that forwards these reports from php-fpm’s syslog channel to the one for mediawiki. As such, our deployment automation and several Logstash dashboards were blind to a subset of potential fatal errors for a few days. Regressions during that week were instead found by manually digging through the raw feed of the php-fpm channel instead. As a temporary measure, Scap was updated to consider the php-fpm’s channel as well in its automation that decides whether a deployment is “green”. We’ve created new Logstash configurations that forward PHP7 crashes in a similar way as we did for HHVM in the past. Bookmarked MW dashboards/queries you have for Logstash now provide a complete picture once again. Thanks Effie, and Cole White! – https://phabricator.wikimedia.org/T234283 ------- ## 📉 Outstanding reports Take a look at the workboard and look for tasks that might need your help. The workboard lists error reports, grouped by the month in which they were first observed. → https://phabricator.wikimedia.org/tag/wikimedia-production-error/ Or help someone that’s already started with their patch: → https://phabricator.wikimedia.org/maniphest/query/CFzrDj3vFbE_/#R Breakdown of recent months (past two weeks not included): * March: 1 report fixed. (3 of 10 reports left). * April: 8 of 14 reports left (unchanged). ⚠️ * May: (All clear!) * June: 9 of 11 reports left (unchanged). ⚠️ * July: 13 of 18 reports left (unchanged). * August: 2 reports were fixed! (6 of 14 reports left). * September: 2 reports were fixed! (10 of 12 new reports left). * October: 12 new reports survived the month of October. ------- ##### 🎉 Thanks! Thank you, to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks! Until next time, – Timo Tijhof 🌴“Gotta love crab. In time, too. I couldn't take much more of those coconuts. Coconut milk is a natural laxative. That's something Gilligan never told us.” ------- Footnotes: [1] Incidents. – https://wikitech.wikimedia.org/wiki/Special:PrefixIndex?prefix=Incident+doc… [2] Tasks created. – https://phabricator.wikimedia.org/maniphest/query/sJI9Af6LqvKL/#R [3] Tasks closed. – https://phabricator.wikimedia.org/maniphest/query/scW28HMEJemU/#R [4] Open tasks. – https://phabricator.wikimedia.org/maniphest/query/47MGY8BUDvRD/#R [5] https://wikitech.wikimedia.org/wiki/Logstash/Interface

1 0

[Data Release] Active Editors by country
by Dan Andreescu 07 Nov '19

07 Nov '19

Today we are releasing a new dataset meant to help us understand the impact of grants and programs on editing. This data was requested several years ago, and we took a long time to bring in the privacy and security experts whose help we needed to release it. With that work done, you can download the data here: https://dumps.wikimedia.org/other/geoeditors/ and read about it here: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Geoeditors/Pu… You can send questions or comments on this thread or on the discussion page.

1 0

Wikibase: security fix available for the Query Service GUI
by Léa Lacroix 07 Nov '19

07 Nov '19

Hello all, -and sorry for cross-posting- This message is important to everyone running an instance of Wikibase including the Query Service GUI. We just released a new version of the Wikidata Query Service GUI. This release is primarily to fix a security issue described in T233213 <https://phabricator.wikimedia.org/T233213> (hidden task, will be made public soon). The fix has been successfully deployed for the Wikidata Query Service. In order to keep your instance safe, please make sure to update your Query Service GUI! Git repositories, releases and currently active version docker images also include the latest fixed code (see links below). If you have a local test setup using the docker-compose example then see: https://gist.github.com/addshore/36f8d6fe2331d28ca8f70df5abda20fd Gerrit repositories: - wikidata/query/gui after commit d9f964b88c01748e278ca8c4b8929a8ef0ef0267 - wikidata/query/gui-deploy after commit 7445472ab0ec61890b42e4d524416fbc6a18aa8a - wikidata/query/deploy after 094d9cda98f3fb706cf9c25aefa3eb33f9f6999a Docker images: - wdqs:0.3.6 (all versions of this tag) - wdqs:latest from digest sha256:04237b42d0b904a2c49ecb7059c82ace8265ba0b7f690ee2d4b3004ad39517ee - wdqs-frontend:latest from digest sha256:1308a7d6622b1e141783336fb52cd6993973321077f58359fbf907b77e105ca3 - wdqs-frontend:legacy from digest sha256:f830abd53fe5e79299011211a2aab7ad947181e56785c06eed6e9bd6b430d4ce Downloadable releases: - https://archiva.wikimedia.org/repository/snapshots/org/wikidata/query/rdf/s… If you have any questions or issues updating your code, please let us know (you can write me an email, or ask in the Wikibase Telegram group <https://t.me/joinchat/HGjGexZ9NE7BwpXzMsoDLA>) Thanks for your understanding, Cheers, -- Léa Lacroix Project Manager Community Communication for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

TechCom Radar 2019-11-06
by Alexandra Paskulin 06 Nov '19

06 Nov '19

Hi all, Here are the minutes from this week's TechCom meeting: * Discussed: RFC: Where to implement Desktop Improvements project <https://phabricator.wikimedia.org/T234907>. This RFC discusses the approach for implementing broad UI changes to the desktop interface of Wikimedia projects. * Discussed: Schema change procedure documentation blanked, no alternative given for 8 months <https://phabricator.wikimedia.org/T237229>. This task proposes re-adding the procedure for schema changes to the Wikimedia development policy. * Last call: RFC: ParallelMaintenance helper class for multi-process maintenance scripts <https://phabricator.wikimedia.org/T201970>. This RFC was placed on last call on 2019-10-23 with the intent to approve on 2019-11-13. Comments in support of or in opposition to this RFC are welcome during the last call period. * TechCom will not meet next week due to the 2019 Wikimedia Technical Conference. * No IRC meeting next week or on 2019-11-20. You can also find our meeting minutes at <https://www.mediawiki.org/wiki/Wikimedia_Technical_Committee/Minutes>. See also the TechCom RFC board <https://phabricator.wikimedia.org/tag/mediawiki-rfcs/>. If you prefer you can subscribe to our newsletter here <https://www.mediawiki.org/wiki/Newsletter:TechCom_Radar>. Thanks, - Alex -- Alex Paskulin Technical Writer Wikimedia Foundation

1 0

Scrum of scrums/2019-11-06
by Željko Filipin 06 Nov '19

06 Nov '19

Hi, for HTML version see https://www.mediawiki.org/wiki/Scrum_of_scrums/2019-11-06 Željko -- = 2019-11-06 = == Callouts == * Parsoid-PHP is about ready to be rolled out in the coming weeks for all live traffic and will serve client requests (VE, CX, Flow, MCS for Android app) instead of the current Parsoid/JS service. See Parsing team section of this etherpad for more notes about testing requests. ( [[phab:T229015]] is the tracking task ) * Product Infrastructure ** blocked by RelEng: Ensure all PI engineers have +2 rights in operations/deployment-charts [[phab:T232794]] ** Product Infrastructure blocked by SRE: Configure Google Cloud Vision credentials in production [[phab:T236426]] ** MachineVision extension is being enabled today, please let us know if anyone sees any new errors on Commons * Release Engineering ** This week train last train (1.35.0-wmf.5 - [[phab:T233853]]) is the last train before two week break ** v3 of architecture document for new continuous integration wants review and feedback [[User:LarsWirzenius/NewCI]] * Language ** Blocked by Core: Preemptive refresh in getMultiWithSetCallback() and getMultiWithUnionSetCallback() pollutes cache: [[phab:T235188]] == Product == === Editing === * Blocking: ** Parsing: [[phab:T229074]] -- Prepare VE for Parsoid-PHP switch ( Editing team ) === Growth === * Blocking: ** Parsing: [[phab:T229078]] -- Prepare Flow for Parsoid-PHP switch ( which team? but, heads up Roan ) === iOS native app === * Updates: ** 6.4.1 in beta/testing ***3D touch fix in article view ***Seeing crashes in beta so may wait on this until 6.5 ** 6.5 in active development - [[phab:project/view/4245]] ***Wrapping up history & diffs feature - integrating live endpoints this week === Android native app === * Updates: ** Had a minor release with an important bug fix related to 2FA login. ** Wrapped up our server side changes for Suggested Edits V3 feature and all set to integrate with it for a major release **Picking back momentum on mobile-html related work:[[phab:project/view/4318]] === Product Infrastructure === * Blocked by: ** RelEng: Ensure all PI engineers have +2 rights in operations/deployment-charts [[phab:T232794]] ** SRE: Configure Google Cloud Vision credentials in production [[phab:T236426]] * Blocking: ** Parsing: [[phab:T229077]] -- Prepare MCS for Parsoid-PHP switch ( Product Infrastructure team ) * Updates: ** MachineVision extension is being enabled today, please let us know if anyone sees any new errors on Commons === Structured Data === * Blocking: ** Search Platform: Data dumps for SDC: [[phab:T221917]] * Updates: ** machine vision released on testcommonswiki ** can now access MediaInfo items via Lua === Parsing === * Blocked by: ** [[phab:T236930]] needs resolution for using VE with Parsoid/PHP on non-RESTBase wikis (ex: officewiki, wikitech). Need Core Platform Team to take a look at it ** These next 4 testing / QA tasks aren't strictly a blocker yet, but will quickly become blockers as we are aiming to roll out Parsoid/PHP for all live traffic in the coming weeks (before Thanksgiving). Some teams are already working on them, but just flagging them all here for completeness' sake - thanks to you all for addressing them. *** [[phab:T229078]] -- Prepare Flow for Parsoid-PHP switch ( which team? but, heads up Roan ) *** [[phab:T229074]] -- Prepare VE for Parsoid-PHP switch ( Editing team ) *** [[phab:T229075]] -- Prepare CX for Parsoid-PHP switch ( Language team ) *** [[phab:T229077]] -- Prepare MCS for Parsoid-PHP switch ( Product Infrastructure team ) * Updates: ** As of Nov 4, Parsoid/PHP cluster is receiving 100% of Parsoid/JS change prop reparse traffic (which makes up > 90% of all traffic that Parsoid handles). Performance, error rates are all looking good. ( [[phab:T235902]] ) ** So, we are getting ready to roll this out everywhere in the coming weeks in a phased manner (wiki-by-wiki for *all* services). Appreciate testing against Parsoid/PHP in the beta cluster. See description of [[phab:T229074]] for how to test against Parsoid/PHP. === Language === * Blocked by: ** Core: [[phab:T235188]] * Blocking: ** Parsing: [[phab:T229075]] -- Prepare CX for Parsoid-PHP switch ( Language team ) * Updates: ** Testing Parsoid-PHP switch: [[phab:T229075]] === Inuka === * Updates: ** KaiOS app Quick Facts feature: [[phab:T234619]] == Technology == === Fundraising Tech === * Updates: ** CiviCRM: *** Moving more contact deduplication features from local custom hooks to a community-installable extension *** adding new data conflict resolution rules *** Adding contact locking mechanism to avoid trying to deduplicate the same contact from two different jobs ** Investigating payment processor iframe load issues ** CentralNotice: Puzzling out differences between stats from old vs new data pipelines ** CentralNotice: reviewing sub-national targeting ** End of year summary receipt changes === Core Platform === * Blocking: ** Parsing: [[phab:T236930]] needs resolution for using VE with Parsoid/PHP on non-RESTBase wikis (ex: officewiki, wikitech). Need Core Platform Team to take a look at it ** Language: [[phab:T235188]] === Engineering Productivity === ==== Performance ==== * Updates: ** Rolled out the new synthetic testing setup ready with a new Graphite instance. WebPageReplay tests has moved and WebPageTest is the next thing. All old WebPageTest dashboard will work as before but with more metrics and data. ** Gilles presenting at Velocity Berlin today: How to make sense of real user performance metrics ==== Release Engineering ==== * Blocking: ** Product Infrastructure: Ensure all PI engineers have +2 rights in operations/deployment-charts [[phab:T232794]] * Updates: ** v3 of architecture document for new continuous integration wants review and feedback [[User:LarsWirzenius/NewCI]] ** Train Health *** Last week: 1.35.0-wmf.4 - [[phab:T233852]] *** This week: 1.35.0-wmf.5 - [[phab:T233853]] - last train before two week break *** Next week: no train because of [[Wikimedia Technical Conference/2019]] === Search Platform === * Blocked by: ** Structured Data: Data dumps for SDC: [[phab:T221917]] === Site Reliability Engineering === * Blocking: ** Product Infrastructure: Configure Google Cloud Vision credentials in production [[phab:T236426]]

1 0

[TAIM] Wednesday: Technical Advice IRC Meeting
by Leszek Manicki 06 Nov '19

06 Nov '19

Sorry for cross-posting! Reminder: Technical Advice IRC meeting this week **Wednesday 4-5 pm UTC** on #wikimedia-tech. *Note the time change due to Berlin having switched to winter time!* Questions can be asked in English. The Technical Advice IRC Meeting (TAIM) is a weekly support event for volunteer developers. This week we have a special theme, related to the upcoming Wikimedia Technical Conference focusing on Developer Productivity. In particular we're interested in hearing about your experiences with developer productivity in the context of your work with on-wiki tools (gadgets, templates, modules, etc). Your input is going to be useful for the "Developer Productivity & onwiki tooling" ( https://phabricator.wikimedia.org/T234661) session held at the Conference next week. Hope to see you there! -- Leszek Manicki Engineering Manager Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0 http://wikimedia.de Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Developer Productivity and onwiki tooling: Your input is welcome :-)
by Birgit Mueller 05 Nov '19

05 Nov '19

(sorry for cross-posting!) Hello all, We’re running a session at next week’s Wikimedia Technical Conference <https://www.mediawiki.org/wiki/Wikimedia_Technical_Conference/2019> around the topic: Developer Productivity and onwiki tooling - userscripts, gadgets, templates, modules <https://phabricator.wikimedia.org/T234661>. For this we’re looking for more input from folks who won’t be at the conference. The goal of the conference is to identify changes to tooling and processes to support Wikimedia developers in working more efficiently. One aspect of that is to explore what makes it currently difficult for technical contributors working with templates, modules, userscripts and gadgets, and to discuss what could be improved or done differently. It would be wonderful if you could share your experience and comment on the following 2 questions in the phabricator ticket for the session <https://phabricator.wikimedia.org/T234661> (or send me an email, and we’ll add it to the phabricator ticket): 1. What is your background, what do you do as a technical contributor? 2. In what way your productivity as a technical contributor is affected in the context of on-wiki tooling (what slows you down, what makes your life complicated, what helps you …)? To give you two examples (made-up): 1. I am a volunteer developer and have developed several user scripts for frwiki. 2. When I’ve developed a userscript, I don’t know how many people are copying the code to use my script. When I make changes to the script, others often still have older versions. When people report bugs, I need to first find out which version they are using which is very time-consuming. 1. I am a developer of Wikibase extension, WMDE staff member 2. When I develop a new feature in Wikibase, I am often informed AFTER the feature has been released that Wikidata gadgets had been broken. Then I need to stop my current tasks, go back to my previous work and change the feature. This makes the process of my work on new features longer. Many thanks for your support! The developer productivity and onwiki tooling session crew -- Birgit Müller (she/her) Director of Technical Engagement Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l November 2019