Wikitech-l January 2020

wikitech-l@lists.wikimedia.org

68 participants
59 discussions

January 2020 Edition of Technical Community Newsletter
by Sarah R 15 Jan '20

15 Jan '20

Hi Everyone, We’re happy to announce the January 2020 edition of the Technical Community Newsletter <https://www.mediawiki.org/wiki/Technical_Community_Newsletter/2020/January>. The newsletter is compiled by the Wikimedia Developer Advocacy Team. It aims to share highlights, news, and information of interest from and about the Wikimedia technical community. Check it out, and learn about what technical contributors have been up to this past quarter, upcoming conferences & calls for papers, and how to get involved. The Wikimedia Technical Community is large and diverse, and we know we can't capture everything perfectly. We welcome your ideas for future newsletters. Let us know what you would like to see or highlights you would like us to include. Subscribe to the Technical Community Newsletter <https://www.mediawiki.org/wiki/Newsletter:Technical_Community_Newsletter>, if you'd like to keep up with essential updates and information. Kindly, Sarah R. Rodlund Technical Writer, Developer Advocacy srodlund(a)wikimedia.org

1 0

Reason for actionthrottledtext API blocking?
by Baskauf, Steven James 14 Jan '20

14 Jan '20

I am working on a script to load and update Vanderbilt researcher records in Wikidata via the API (see https://github.com/HeardLibrary/linked-data/tree/master/publications for information and https://github.com/HeardLibrary/linked-data/blob/master/publications/proces… for the relevant script). The script is working adequately; however after about 25 writes, I get blocked with an actionthrottledtext error with the message " As an anti-abuse measure, you are limited from performing this action too many times in a short space of time, and you have exceeded this limit.\nPlease try again in a few minutes." I am able to avoid being blocked if I delay three seconds between writes, but a delay of only one second results in me still being blocked. I am observing maxlag (lines 282-313) and am sending a User-Agent header, VanderBot/0.1 (steve.baskauf(a)vanderbilt.edu). I'm making the API calls using the bot account https://www.wikidata.org/wiki/User:VanderBot, which does not have a bot flag. I've been talking to another user who is running a script from paws.wmflabs.org using Pywikibot without a bot flag and he has not encountered this error. So I am wondering what I am doing wrong that is causing this error and what I can do to stop being blocked. The possible causes I can imagine are: 1. The bot account doesn't have a bot flag. https://www.wikidata.org/wiki/Wikidata:Requests_for_permissions/Bot/Danmich… implies that the lack of a bot flag might be the problem. But the script is not an autonomous bot and is still under development, so I didn't think a bot flag was necessary under those circumstances. 2. I'm not running the script from paws.wmflabs.org . Is it a problem that the API calls are coming from an unknown IP address? Does that matter? 3. I'm not using Pywikibot. But that shouldn't be necessary. The other thing that is different about what I'm doing and what is being done by the other user who is not encountering this problem is that I'm authenticating directly by establishing a session when the script starts (lines 347-349). The other user apparently somehow doesn't have to authenticate when using Pywikibot as long as he's logged in to paws.wmflabs.org . But I haven't dug in to find out how Pywikibot works, so I don't really understand how that is possible or whether that's important in establishing that the bot is legitimate and not a vandal/spam bot. Any advice you can give about how to stop being blocked would be appreciated. Eventually I will probably apply for a bot flag, but I doubt that this bot will ever be autonomous, so is that really necessary? Would it matter if I used my own account instead of a separate bot account? Steve -- Steven J. Baskauf, Ph.D. Data Science and Data Curation Specialist Jean & Alexander Heard Libraries, Vanderbilt University Nashville, TN 37235, USA Office: Eskind Biomedical Library, EMB 111 Phone: (615) 343-4582 https://my.vanderbilt.edu/baskauf/

3 2

Timelines for Parsoid/PHP to replace legacy PHP parser
by Lord_Farin 14 Jan '20

14 Jan '20

Hi Wikitech, I've been catching up on the recent achievements regarding Parsoid/PHP, well done! With WMF sites being migrated, of course non-WMF sites start to creep into the picture. As I'm involved in running one of those, I'm curious to know if and how you are going to support this upgrade? I've read about Linter and ParserMigration but I'm not clear on how they fit into the picture. I'm asking specifically because we are running some custom extensions which will probably break with the advent of Parsoid/PHP. At present we are running MW 1.33 on PHP 7.0, but we are not using VE. It would be fine if we as a maintenance team have to invest some (or even considerable) time and effort but I would like to know the size of the endeavour beforehand... Thanks, best, LF

2 2

Wikimedia Technical Talks 2020 -- Open Call
by Sarah R 14 Jan '20

14 Jan '20

2019 was a great year for Wikimedia Technical Talks <https://www.mediawiki.org/wiki/Wikimedia_Technical_Talks>! We are looking forward to a new year and a new line-up of speakers starting in February 2020! If you have an idea for a Tech Talk (or know someone who does) -- please propose a talk <https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?title=Proposa…>! We'll discuss your idea with you, and if it's a good fit, we'll get you on this year's schedule.❤📆❤ - If you are saying to yourself, *"UGH! NO WAY! I HATE PUBLIC SPEAKING!!!!"* 😐 You are not alone! Many people have a fear of public speaking <https://en.wikipedia.org/wiki/Public_speaking#Glossophobia>. One of the best ways to get past it is through practice! - If you are thinking,* "An hour is way too long to talk to a faceless audience!!!" 😬 *You are probably right! An hour is a long time to listen, too! Let's come up with a length and format that will work for you! - If you are thinking, *"I have an idea for a talk, but I don't have the time!!!"* ⏰ No problem! We can schedule your talk for a few months from now! - If you are thinking, *"Ideas? I have no ideas!"* 💡Take a look at some past tech talks <https://www.mediawiki.org/wiki/Wikimedia_Technical_Talks#2019_Technical_Tal…> for inspiration! Tech talks are a great way to share your work, refine your ideas for other professional venues (conferences, papers, etc), and be more visible in the Wikimedia technical community! 👓 Hope you'll join us this coming year as a speaker or a viewer! Please feel free to forward this email to appropriate lists. Kindly, Sarah R. Rodlund Technical Writer, Developer Advocacy <https://meta.wikimedia.org/wiki/Developer_Advocacy> srodlund(a)wikimedia.org

1 0

TechCom Radar 2020-01-08
by Kate Chapman 13 Jan '20

13 Jan '20

Hi All, Here are the minutes from this week's TechCom meeting: * Has been resolved (was not a RFC, but brought to TechCom’s attention): Service ownership and maintenance <https://phabricator.wikimedia.org/T122825> * expected to go to last call after the next TechCom meeting: Adopt a modern js framework <https://phabricator.wikimedia.org/T241180> * Reaching out to those that filed the ticket to discuss further: Standalone regex evaluation service. <https://phabricator.wikimedia.org/T240884> * Moved to Last Call ending 22nd January: Hook container with strong types and DI <https://phabricator.wikimedia.org/T240307> * Discussion on Stable Interface Policy Continues: <https://www.mediawiki.org/wiki/Stable_interface_policy> You can also find our meeting minutes at <https://www.mediawiki.org/wiki/Wikimedia_Technical_Committee/Minutes> See also the TechCom RFC board <https://phabricator.wikimedia.org/tag/mediawiki-rfcs/>. If you prefer you can subscribe to our newsletter here <https://www.mediawiki.org/wiki/Newsletter:TechCom_Radar> Thanks, Kate -- Kate Chapman Senior Program Manager, Core Platform Wikimedia Foundation kchapman(a)wikimedia.org

1 0

Timing of the MediaWiki branch cut
by Mukunda Modell 11 Jan '20

11 Jan '20

tl;dr if the timing of the weekly MediaWiki branch cut affects you, please join the discussion on T242446[1] Soon we will switch on full automation of the mediawiki branch cut! 🎉 In so doing we will switch from using the make-wmf-branch script in release/make-wmf-branch. Instead we will use release/make-release/branch.py and the associated config will now live at release/make-release/settings.yaml. The documentation on wiki has been updated to reflect the change with instructions about how to set up and use branch.py. There is more to this change than just using a different script, however. In order to turn on full automation we need to decide on when to run the branch cut. The current practice is to do it at approximately noon pacific on Tuesdays. We could move this to an earlier time but I think we should have a larger discussion about the timing of the branch cut and how that affects various teams and workflows. If the timing of the branch cut affects your team, your feedback is requested on the task at [1] [1] https://phabricator.wikimedia.org/T242446

1 0

Production Excellence (December 2019)
by Krinkle 10 Jan '20

10 Jan '20

Read on Phabricator at https://phabricator.wikimedia.org/phame/post/view/179 ------- How’d we do in our strive for operational excellence in November and December? Read on to find out! ## 📊 Month in numbers * 0 documented incidents in November, 5 incidents in December. [1] * 17 new Wikimedia-prod-error reports. [2] * 23 Wikimedia-prod-error reports closed. [3] * 190 currently open Wikimedia-prod-error reports in total. [4] November had zero reported incidents. Prior to this, the last month with no documented incidents was December 2017. To read about past incidents and unresolved actionables; check < https://wikitech.wikimedia.org/wiki/Incident_documentation#2019>. Explore Wikimedia incident graphs (interactive) at < https://codepen.io/Krinkle/full/wbYMZK>. ------- ## *️⃣ Many dots, do not a query make! David Causse investigated a flood of exceptions from SpecialSearch […]. Read the story at <https://phabricator.wikimedia.org/phame/post/view/179>. ------- ## 📉 Outstanding reports Take a look at the workboard and look for tasks that might need your help. The workboard lists error reports, grouped by the month in which they were first observed. → https://phabricator.wikimedia.org/tag/wikimedia-production-error/ Breakdown of recent months (past two weeks not included): * March: 3 of 10 reports left. (unchanged). ⚠️ * April: Three reports closed, 6 of 14 left. * May: (All clear!) * June: Three reports closed. 6 of 11 left (unchanged). ⚠️ * July: One report closed, 12 of 18 left. * August: Two reports closed, 4 of 14 left. * September: One report closed, with 9 of 12 left. * October: Four reports closed, 8 of 12 left. * November: 5 new reports survived the month of November. * December: 9 new reports survived the month of December. ------- # 🎉 Thanks! Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Until next time, – Timo Tijhof ------- Footnotes: [1] Incidents. – https://wikitech.wikimedia.org/wiki/Incident_documentation#2019 [2] Tasks created. – https://phabricator.wikimedia.org/maniphest/query/AFDaPqjd5PTe/#R [3] Tasks closed. – https://phabricator.wikimedia.org/maniphest/query/YkIxmhRvEZ8R/#R [4] Open tasks. – https://phabricator.wikimedia.org/maniphest/query/47MGY8BUDvRD/#R

1 0

Sunsetting DeepCat & Catgraph
by Michael Schönitzer 09 Jan '20

09 Jan '20

Hello everyone, In 2015 Wikimedia Germany published the Service Catgraph[1], a graph database of the categories of Wikipedias, as well as Deepcat[2] a gadget for Wikipedia users to search for articles in a category, including recursive subcategories, in the search page of Mediawiki. Since 2018 this functionality is built in the search engine of Wikipedia as well as into the Advanced Search GUI. Wikimedia will therefore sunset Catgraph as well as Deepcat. We plan to shut down the service end of January. Catgraph also powered other tools such as Catcycle[3] and Article List Generator[4], these will also stop working. If your wiki has deepcat enabled as a gadget or in your globar.js, or you have it in your common.js you should remove it and use the newer, built in 'deepcategory'. Documentation for the new search-keyword can be found here: https://www.mediawiki.org/wiki/Help:CirrusSearch/de?uselang=de#Deepcategory Documentation for the new Graph-Database is available here: https://www.mediawiki.org/wiki/Wikidata_Query_Service/Categories Cheers, Michael for the project Technical Wishes at Wikimedia Germany [1]: https://wikitech.wikimedia.org/wiki/Nova_Resource:Catgraph [2]: https://de.wikipedia.org/wiki/Hilfe:Suche/Deepcat [3]: http://tools.wmflabs.org/render-tests/catcycle-dev/catcycle.py [4]: https://tools.wmflabs.org/render/stools/alg?uilang=en -- *M. F. Schönitzer* Community Communication Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Tel. (030) 219 158 26-0 https://wikimedia.de Unsere Vision ist eine Welt, in der alle Menschen am Wissen der Menschheit teilhaben, es nutzen und mehren können. Helfen Sie uns dabei! https://spenden.wikimedia.de Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Scrum of scrums/2020-01-08
by Željko Filipin 09 Jan '20

09 Jan '20

Hi, for HTML version see https://www.mediawiki.org/wiki/Scrum_of_scrums/2020-01-08 Željko -- = 2020-01-08 = == Callouts == * Release Engineering ** January 27th-31st - No deploys at all (including train), All-Hands ** Need some help (from CPT? Maybe?) looking at jobqueue on beta cluster which is evidently running slowly: [[phab:T241448]] * SRE: ** The eqiad kubernetes cluster will be reinitialized on Tuesday Jan 14th 2020. The duration is expected to be up to 4 hours during EU morning. All requests will have been routed beforehand to codfw to avoid causing issues to users. Please refrain from trying to deploy to that cluster during that time period. ** OKRs in final drafts, reach out if your team requires SRE involvement == Product == === Editing === * Updates: ** Preparing to release v1.0 of DiscussionTools to target wikis to discuss === Growth === * Updates: ** Main focus is on releasing the first version of topic matching for suggested edits module. Initial release utilizing morelike targeted for before All Hands. === Android native app === * Updates: **Preparing for a minor release with some bug fixes and improved language variant support. The release candidate is currently in Beta : https://play.google.com/store/apps/details?id=org.wikipedia.beta **Mobile-html integration is 70% complete. The Android O board with the rest of the work will be the next major release: [[phab:project/view/4318]] === Web === * Updates: ** Summary: digging into desktop improvements (DIP). ** [[Reading/Web/Desktop_Improvements|Desktop Improvements Project (Vector / DIP)]]: *** [[phab:T232140|<nowiki>Separate out logo handling into square image logos and long text/wordmark banner logos</nowiki>]] *** [[phab:T240062|<nowiki>Extract components from VectorTemplate to Mustache files</nowiki>]] *** [[phab:T239248|<nowiki>[EPIC] Divide VectorTemplate into components</nowiki>]] *** [[phab:T117279|<nowiki>[EPIC] Core should provide inline diffs as well as side by side (Move InlineDifferenceEngine into core / remove MobileDiff)</nowiki>]] *** [[phab:T237561|<nowiki>[SPIKE 10hrs] Investigate how skin suboptions can be presented in Special:Preferences</nowiki>]] ** Mobile website (MinervaNeue / MobileFrontend): *** [[Reading/Web/Advanced_mobile_contributions|Advanced mode]]: **** [[phab:T240644|<nowiki>Regression: Label is visible on Logout button for logged in users on all wikis (non-AMC) and contribution icon on user page</nowiki>]] **** [[phab:T238364|<nowiki>[Bug] Simplified Talk Page Does Not Support Sections With Non-ascii Characters/Several ascii characters aren't supported either</nowiki>]] **** [[phab:T236328|<nowiki>Username should be on its own line (Recent changes & Watchlist)</nowiki>]] *** Miscellaneous: **** [[phab:T237050|<nowiki>The `site` and `site.styles` module should be explicitly disabled on mobile rather than abuse the targets system</nowiki>]] **** [[phab:T214647|<nowiki>[EPIC] Re-define the contract for displaying drawers and overlays in MobileFrontend</nowiki>]] **** [[phab:T241179|<nowiki>Remove Special:MobileMenu</nowiki>]] **** [[phab:T240502|<nowiki>Raw HTML in MobileFrontend</nowiki>]] **** [[phab:T234570|<nowiki>Tech debt: Drop Browser.supportsAnimations method</nowiki>]] **** [[phab:T214049|<nowiki>Technical: Every Drawer has a shield for catching clicks</nowiki>]] **** [[phab:T198265|<nowiki>Special:MobileOptions is empty for third parties and when beta is disabled and this is confusing</nowiki>]] **** [[phab:T171000|<nowiki>[EPIC] Minerva works independently of MobileFrontend</nowiki>]] **** [[phab:T206069|<nowiki>Transition MobileFrontend Gruntfile.js tasks to NPM scripts</nowiki>]] **** [[phab:T140045|<nowiki>Respect $wgAllowSiteCSSOnRestrictedPages</nowiki>]] === Product Infrastructure === * Blocked by: ** SRE on buster nodejs10 images for proton. Will resume work this week === Structured Data === * Blocking: ** Search Platform: Data dumps for SDC: [[phab:T221917]] === Inuka === * Updates: ** KaiOS app: read article in another language [[phab:T234628]] ** KaiOS app: loading experience [[phab:T240888]] ** KaiOS app: navigate to previous article [[phab:T241423]] ** KaiOS app: offline mode indicator == Technology == === Fundraising Tech === * Updates: ** CiviCRM *** Updating Civi version to latest *** More improvements to contact de-duplication ** Payments-wiki *** Implementing iDEAL and recurring card payments via our backup processor === Core Platform === * Blocked by: ** RelEng on Quibble [[phab:T236680]] ** Security OAuth 2.0 review [[phab:T239940]] * Blocking: ** Search Platform: MW Job consumers sometimes pause for several minutes [[phab:T224425]] ** RelEng: [[phab:T241448]] * Updates: ** New session storage to roll out to production -> Redis cluster going away === Engineering Productivity === ==== Release Engineering ==== * Blocked by: ** SRE ServiceOps: Rolling out PHP 7.2.26 to CI is blocked on creating a PHP 7.2.2.6 package [[phab:T241224]] ** CP: [[phab:T241448]] * Blocking: ** Core Platform: Quibble [[phab:T236680]] * Updates: ** Train Health *** week of Dec 16th: 1.35.0-wmf.11 - [[phab:T233859]] *** This week: 1.35.0-wmf.14 - [[phab:T233862]]; now on group0. New logspam is quiet so far *** Next week: 1.35.0-wmf.15 - [[phab:T233863]] *** January 27th-31st - No deploys at all (including train), All-Hands === Research === * Blocked by: ** SRE on recommendation API migration. Will reach out soon. === Scoring Platform === * Updates: *** New draft topic model coming (New taxonomy --> Better labels --> Better fitness) (FYI: Growth) *** New revscoring version 2.6.3 -- better info formatting. Expanded support for vectors. *** Jade API on Beta https://en.wikipedia.beta.wmflabs.org/w/api.php?action=help&modules=jadepro… **** + updates to Jade API docs [[Extension:Jade/API]] === Search Platform === * Blocked by: ** Structured Data: Data dumps for SDC: [[phab:T221917]] ** Core: MW Job consumers sometimes pause for several minutes [[phab:T224425]] * Updates: ** Re-import OSM data at eqiad and codfw to temporarily fix current OSM replication issues (including reducing replication interval, which might fix the issue long term) - [[phab:T239728]] ** Migrate and redeploy discovery platform analytics jobs with kerberos integration - [[phab:T241011]] === Security === * Blocking: ** Core Platform: OAuth 2.0 review [[phab:T239940]] === Site Reliability Engineering === * Blocking: ** Research on recommdendation-api migration to kubernetes. Working on it ** Product infrastructure on creating a buster nodejs image. Working on it ** Release Engineering - ServiceOps: Rolling out PHP 7.2.26 to CI is blocked on creating a PHP 7.2.2.6 package [[phab:T241224]] * Updates: ** The eqiad kubernetes cluster will be reinitialized on Tuesday Jan 14th 2020. The duration is expected to be up to 4 hours during EU morning. All requests will have been routed beforehand to codfw to avoid causing issues to users. Please refrain from trying to deploy to that cluster during that time period. ** OKRs in final drafts, reach out if your team requires SRE involvement

4 4

Quibble test runner version 0.0.40 released
by James Forrester 09 Jan '20

09 Jan '20

All, I've released a new version of Quibble just today, version 0.0.40. This has a few changes, most notable further improvements to support the work on testing Core Platform team's new RESTful API. The full change log is available at https://doc.wikimedia.org/quibble/changelog.html Thank you to Clara Andrew-Wani, Andrew Wight, and Timo Tijhof for their patches. I will be updating CI jobs to use this progressively over the next few days. Yours, -- *James D. Forrester* (he/him <http://pronoun.is/he> or they/themself <http://pronoun.is/they/.../themself>) Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l January 2020