Wikitech-l November 2017

wikitech-l@lists.wikimedia.org

60 participants
56 discussions

2017-11-08 Scrum of Scrums meeting notes
by Grace Gellerman 09 Nov '17

09 Nov '17

https://www.mediawiki.org/wiki/Scrum_of_scrums/2017-11-08 = 2017-11-08 = == Callouts == * No MW Train the week of the 20th due to Thanksgiving, SWATs will be open on Mon and Tuesday (Wednesday is “Friday” that week). * RelEng: Please port your browser tests to the nodejs framework. 7 repositories have not started yet (still in ruby, which is no longer maintained). [https://phabricator.wikimedia.org/T139740 task] * Tidy replacement: if you are an editor on some wiki, please get the word out to your wikis to fix the high-priority linter issues. We want to get Tidy removed from the cluster sooner than later.A bunch of wikis (including mediawiki) no longer have Tidy, and a few 100 small wikis can get Tidy replaced based on stats about linter issues ( https://www.mediawiki.org/wiki/Parsing/Replacing_Tidy/Linter/Stats/2017-11-… ). But would be good to get some largish wikis moved over so we can identify any lingering problems sooner than later. == Audiences == === Readers === ==== iOS native app ==== * Blocked by: * Blocking: * Updates: ** 5.7.1 in the app store today (minor bug fixes) https://phabricator.wikimedia.org/project/view/3047/ ** Continuing work on 5.8 - https://phabricator.wikimedia.org/project/view/2913/ (faster article loads, synced reading lists) ==== Android native app ==== * Blocked by: * Blocking: * Updates: ** Released version 2.6.206 to production. ** Continuing to test / integrate with Reading List service. ==== Reading Web ==== * Blocked by: ** Ops&Release engineering - How to use Chromium in Puppeteer [ https://phabricator.wikimedia.org/T178570]+ https://phabricator.wikimedia.org/T178189 *** Currently investigating on path forward in https://phabricator.wikimedia.org/T180037. * Blocking: * Updates: ** Working on the Chromium-based PDF renderer service. *** Starting performance testing within two weeks. ** Researching behaviour around the "Download to PDF" button [ https://phabricator.wikimedia.org/T177969, https://phabricator.wikimedia.org/T177969#3742056] ** Working product and technical documentation for Marvin ==== Reading Infrastructure ==== * Blocked by: * Blocking: * Updates: ** Continuing work on MCS summary, media (gallery) endpoints. ** Continuing with Reading Lists ==== Multimedia ==== * Blocked by: * Blocking: * Updates: ==== Discovery ==== * Blocked by: * Blocking: * Updates: ** continuing work on portal automation. going well. ===== Maps ===== * Blocked by: N/A * Blocking: N/A * Updates: None === Contributors === ==== Editing ==== * Blocked by: * Blocking: * Updates: ==== Parsing ==== * section + figure-inline + html5-ids changes aren't going out this week after all -- reviews not yet finalized + we need thorough testing in beta cluster. We are still hoping to get it out next week. * Tidy replacement: if you are an editor on some wiki, please get the word out to your wikis to fix the high-priority linter issues. We want to get Tidy removed from the cluster sooner than later. A bunch of wikis (including mediawiki) no longer have Tidy, and a few 100 small wikis can get Tidy replaced based on stats about linter issues ==== Global Collaboration ==== * Blocked by: nobody * Blocking: ops maybe? There's the Flow dumps issue but not sure how much of a blocker that is * Updates: ** Going to roll out RCFilters to remaining (FlaggedRevs) wikis on Nov 15 ** Early work on using wikitext instead of HTML for storing Flow content, need something like pcache except 1) it's Parsoid HTML not PHP HTML 2) Flow comments aren't pages ==== UI Standardization ==== ** OOUI v0.24.1 & v0.24.2 released (missed last week's Scrum of Scrums), https://phabricator.wikimedia.org/diffusion/GOJU/browse/master/History.md;v…; among changes: *** UI design **** WikimediaUI theme with 3 changes: SVG markup (see below), MenuSectionOptionWidget clearer on long lists and ComboBoxInputWidget providing stronger user feedback on open * Ongoing: ** OOUI & relying products: *** icons: Work on icon set to be more harmonious and align to WikimediaUI Style Guide's guidelines https://phabricator.wikimedia.org/T177432 ** Unify SVG markup across Foundation products https://phabricator.wikimedia.org/T178867 === Community Tech === * Not blocking * Not blocked * Our annual survey is underway: https://meta.wikimedia.org/wiki/2017_Community_Wishlist_Survey * Fixing issues uncovered by Unicode sections rollout to Russian projects == Technology == === Analytics === * New kafka Jumbo cluster is up, for now mirror-ing topics from analytics cluster. * New DB hosts for eventlogging back in place, purging data out of compliance at a faster rate due to better hardware. * We have spark2 in cluster! * Still troubleshooting metric difference wikistats 1.0 on wikistats 2.0. Take a look edit metrics available now (some of them have a 5% difference, so not final yet): https://stats.wikimedia.org/v2/#/am.wikipedia.org * Work on refining eventlogging data in hadoop continues so hadoop can be a easy to query backend for eventlogging data === Cloud Services === * Blocked by: * Blocking: * Updates: === Fundraising Tech === * Blocked by: * Blocking: * Updates: ** Deploying new English thank you letter ** Payments form validation improvements ( https://phabricator.wikimedia.org/T176501) ** Better API error handling for backup card processor ( https://phabricator.wikimedia.org/T177893) ** More stats reporting to prometheus ** CiviCRM UI improvements (https://phabricator.wikimedia.org/T142549, https://phabricator.wikimedia.org/T178003, https://phabricator.wikimedia.org/T177873) ** CentralNotice: allow delaying impression reporting ( https://phabricator.wikimedia.org/T176334) === MediaWiki Platform === * Blocked by: * Blocking: * Updates: === Performance === * Blocked by: * Blocking: * Updates: === Release Engineering === * Blocking ** ??? * Blocked ** Please port your browser tests to the nodejs framework. 7 repositories have not started yet (still in ruby, which is no longer maintained). *** See: https://phabricator.wikimedia.org/T139740 *** Notably: Global Collaboration Team, Fundraising Tech, Wikibase, and Mulitmedia * Updates ** No MW Train the week of the 20th due to Thanksgiving, SWATs will be open on Mon and Tuesday (Wednesday is “Friday” that week). ** [TechDebt program] First pass of the service levels for component ownership was shared with the Code Health Group last week; feedback on-going. ** [TechDebt program] The next blog post should be posted Real Soon Now™ (done on our side). ** [SSD Program] Working on getting the mathoid tests running on submit. ** [SSD Program] A new release of Blubber is on the horizon. *** [SSD Program] Aside: what it is: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Deploymen… *** [SSD Program] Bubber: https://phabricator.wikimedia.org/source/blubber/ ** [scap tech debt] Working to support both pre git-2.11 and post git 2.11 hosts (namely Trusty and Jessie (with backports)/Stretch, respectively) to let us use newer functionality (notably `--jobs`). ** [CI] Most tox jobs are moved to Docker containers ** [CI] Investigating why many docker containers are left behind after a SIGTERM === Research === * Blocked by: * Blocking: * Updates: === Scoring Platform === * Blocked by: * Blocking: * Updates: **frwiki -- ongoing issue. https://phabricator.wikimedia.org/T179711 (should be fixed by SOS) **Draft quality data is coming to English Wikipedia (ping Commtech and Global Collab) ***You can use this to help patrollers sort new pages by (Spam, Attach, Vandalism/Hoax, or OK). OK == not immediately concerning but not necessarily good. ***[Reponse from Roan: we don't plan to use draftquality directly at this point] **Wikilabels getting some improvements. ***Links to on-wiki documentation for each campaign ("info" link) ***Statistics about labelers will have usernames **Amir is working on Icelandic language support. === Search Platform === * Blocked by: * Blocking: * Updates: ** Added hiragana<->katakana translations for English wiki (not for Japanese one) https://phabricator.wikimedia.org/T176197 ** Implemented demoting disambig pages for Wikidata prefix search https://phabricator.wikimedia.org/T148411 ** Improved reusability of Searcher code https://phabricator.wikimedia.org/T178906 ** Improving completion suggester https://phabricator.wikimedia.org/T178474 ** Working on porting Selenium tests from Ruby to JS ** Working on upgrade to Elastic 5.5 ** Working on indexing Wikidata descriptions and adding them to fulltext search === Security === * Blocked by: None * Blocking: Everyone waiting for security reviews * Updates: ** Reviews: *** BeautifulSoup (done, comments will be added shortly) *** vue.js (done, same) *** mediawiki-services-chromium-render ** Preparing for MW security release === Services === * Blocked by: none * Blocking: none * Updates: ** Parsoid storage switched to new storage model for all but wikipedias ** Definitions storage switched to new model ** Planning on switching RecordLintJob to kafka-based JobQueue === Technical Operations === * Blocked by: ** Flow isAllowed gets actual revision text before it is needed https://phabricator.wikimedia.org/T172025 * Blocking: * Updates: ** labsdb1001 storage crashed; Manuel got the dbs back up, but we are leaving in read-only - https://phabricator.wikimedia.org/T179464 ** Turning up s8 databases for wikidata https://phabricator.wikimedia.org/T177208 ** wikidiff2 1.5.1 fully rolled out, enabled for group0 wikis, additional will be enabled by Adam Shoreland over the next weeks == Wikidata == * Blocked by: * Blocking: * Updates: ** getting rid of the Wikidata build (to deploy Wikibase etc. as normal extensions): https://phabricator.wikimedia.org/T173818 *** changed how JS libraries are included (as submodules, instead of via composer) * work on Lexeme (persistence) * work on multi-content revisions * Coordinates in RDF exports are going to use xsd:double: https://phabricator.wikimedia.org/T179228 == German Technical Wishlist == * Blocked by: * Blocking: * Updates:

1 0

Simple overview image about how MW loads resources in clients
by Joaquin Oltra Hernandez 09 Nov '17

09 Nov '17

Hi, We were having a session where we talked about resource loading, code entry points for the front-end, and how things work on MediaWiki, and we came up with a small pic to explain the lifecycle for people newer to MediaWiki. Maybe it could help some people get a better grasp about where files are coming from and what why the load.php urls are as they are. Please, forgive any missing details, and if there is something very wrong I'd love to correct it, please let me know. Also to clarify, "Magic" is used as "Dynamic, runtime based, dependent on the state of your code/client cache/server state & extensions" to shorten things and in a humorous key. Links: - Phab: https://phabricator.wikimedia.org/M232 - Imgur: https://i.imgur.com/DYLqtQf.png

9 12

TechCom discussion in 90 minutes: Tweaking the RFC process
by Daniel Kinzler 08 Nov '17

08 Nov '17

Hi all! As mentioned in the TechCom Radar mail earlier this week, today's TechCom IRC meeting will be about the RFC process itself. There is no concrete proposal to discuss (yet), this is intended to be a session for gathering questions, comments and suggestions. The meeting will start at 22:00 UTC (2pm PST, 23:00 CET) on #wikimedia-office. -- Daniel Kinzler Principal Platform Engineer Wikimedia Deutschland Gesellschaft zur Förderung Freien Wissens e.V.

1 0

Try out the commit message validator
by Kunal Mehta 08 Nov '17

08 Nov '17

Hi! tl;dr <https://www.mediawiki.org/wiki/commit-message-validator> Commit messages are pretty important - they make it easy to figure out what a change does without reading the entire diff. And because of that, we have some guidelines[1] on how to format your commit message in the standard way. But sometimes people aren't familiar with the guidelines, or more likely we make a typo somewhere. Here's where the commit-message-validator[2] comes in handy! $ git commit <type in commit message and save> commit-message-validator v0.5.1 The following errors were found: Line 1: First line should be <=80 characters Line 3: Bug: value must be a single phabricator task ID Please review <https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines> and update your commit message accordingly Or if you did a good job: commit-message-validator v0.5.1 Commit message is formatted properly! Keep up the good work! Installing it as a git hook is the recommended usage, and instructions on the wiki page[2] explain how to do so, even for people who aren't familiar with Python. The tool was written by Bryan Davis, Fabian Neundorf, and myself. Bugs/feature requests can be filed in the Phabricator project. [1] https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines [2] https://www.mediawiki.org/wiki/Commit-message-validator -- Legoktm

9 8

Readers monthly update for October 2017
by Chris Koerner 07 Nov '17

07 Nov '17

Hello, Here is the monthly update for October from the Readers department at the Wikimedia Foundation. **Reminder**: You can now subscribe to receive theses updates as on-wiki notifications or opt-in email! Handy if you subscribe to the digest of his mailing list or don't visit the archive frequently. Subscribe on MediaWiki.org: https://www.mediawiki.org/wiki/Newsletter:Readers_Monthly Thanks for reading, as always, feedback and questions are welcome. === Apps=== * The iOS team released version 5.7.0. This version includes an updated share-a-fact, improved on-boarding for new users and lots of fixes for iPhone X. [0] * The iOS app was named an "Editors Choice" by Apple. This is the highest recognition Apple gives apps, and in their decision they particularly called out the improved quality of the app, and focus on accessibility. * The Android app went to beta with their latest update. This version adds app shortcuts, image dimming, theming improvements for Light and Dark mode, bug fixes for Simplified and Traditional Chinese and many other refinements. [1] * The apps teams participated in a 3 day "Offsite" in Philadelphia. Read about it on MediaWiki.org. [2] === Web === * Updated Print styles arelive [3] * Regrettably, we turned off OCG. Fortunately, this unblocked Ops's decommission of the service and servers. * We're working hard on a replacement service for the Electron-based PDF rendering service. [4] Follow our progress on Phabricator. [5] === Reading Infrastructure === * the API backend for Reading Lists is in beta [6] [7] === New Readers === * New Reader updates can be found at Meta. [8] === Multimedia === * Support for uploading and viewing 3D models is coming soon to Wikimedia Commons. The feature will support the .STL file format. [9] === Discovery === The Discovery team provides weekly updates. Here are their updates for the month. * 2017-10-02 [10] * 2017-10-09 [11] * 2017-10-16 [12] * 2017-10-23 [13] == Other Noteworthy Stuff == * Sam Smith: I'm currently investigating adding up/downstream bandwidth configuration to my micro-device-lab side project. [14] [0] https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=32471523… [1] https://play.google.com/store/apps/details?id=org.wikipedia.beta [2] https://www.mediawiki.org/wiki/Wikimedia_Apps/Philadelphia_offsite/Offsite_… [3] https://twitter.com/jdlrobson/status/918264297343422465 [4] https://github.com/wikimedia/mediawiki-services-electron-render [5] https://phabricator.wikimedia.org/T176627 [6] https://www.mediawiki.org/wiki/Reading/Reading_Lists [7] http://readinglists.wmflabs.org/wiki/VagrantRoleReadingLists [8] https://meta.wikimedia.org/wiki/New_Readers/Updates [9] https://www.mediawiki.org/wiki/Help:Extension:3d [10] https://www.mediawiki.org/wiki/Discovery/Status_updates/2017-10-02 [11] https://www.mediawiki.org/wiki/Discovery/Status_updates/2017-10-09 [12] https://www.mediawiki.org/wiki/Discovery/Status_updates/2017-10-16 [13] https://www.mediawiki.org/wiki/Discovery/Status_updates/2017-10-23 [14] https://github.com/phuedx/micro-device-lab --- The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Reading/Status_updates Yours, Chris Koerner Community Liaison Wikimedia Foundation

1 0

3D model support coming soon to Commons
by Chris Koerner 07 Nov '17

07 Nov '17

Hello! Just a quick note to let you all know that support for 3D models is coming to Commons! [0] The plan is to have the feature enabled by the end of November. There are a few technical items left to address. The feature will support the STL file format, one of the most common file formats for 3D models. [1] You can see an example of how a 3D model will appear on the test wiki. [2] (.gif) [3] This feature has been discussed in the Wikimedia community for a long time. Most recently the feature was #11 on the 2015 Community Tech Wishlist and was previously discussed on Commons in April as development progressed. [4] [5] There will be a blog post on the Wikimedia Blog once the feature is enabled. You can read more about the technical details of the feature and the help documentation on MediaWiki.org. [6] [7] [0] https://commons.wikimedia.org/wiki/Commons:Village_pump#3D_model_support_co… [1] https://wikipedia.org/wiki/Special:MyLanguage/STL_(file_format) [2] https://test.wikipedia.org/wiki/File:Programmatically_created_crystal.stl [3] https://commons.wikimedia.org/wiki/File:Demo_of_3D_model_in_MediaViewer.gif [4] https://meta.wikimedia.org/wiki/2015_Community_Wishlist_Survey/Commons#3D_m… [5] https://commons.wikimedia.org/wiki/Commons:Village_pump/Archive/2017/05#Pre… [6] https://www.mediawiki.org/wiki/Extension:3D [7] https://www.mediawiki.org/wiki/Help:Extension:3d Yours, Chris Koerner Community Liaison Wikimedia Foundation

1 0

Important news about the November dumps run!
by Ariel Glenn WMF 07 Nov '17

07 Nov '17

As was previously announced on the xmldatadumps-l list, the sql/xml dumps generated twice a month will be written to an internal server, starting with the November run. This is in part to reduce load on the web/rsync/nfs server which has been doing this work also until now. We want separation of roles for some other reasons too. Because I want to get this right, and there are a lot of moving parts, and I don't want to rsync all the prefetch data over to these boxes again next month after cancelling the move: ******** If needed, the November full run will be delayed for a few days. If the November full run takes too long, the partial run, usually starting on the 20th of the month, will not take place. ********* Additionally, as described in an earlier email on the xmldatadumps-l list: ********* files will show up on the web server/rsync server with a substantial delay. Initially this may be a day or more. This includes index.html and other status files. ********* You can keep track of developments here: https://phabricator.wikimedia.org/T178893 If you know folks not on the lists in the recipients field for this email, please forward it to them and suggest that they subscribe to this list. Thanks, Ariel

2 4

TechCom Radar, 2017-11-01
by Daniel Kinzler 07 Nov '17

07 Nov '17

Hello all! Here are the minutes from this week's meeting: * TechCom would like to thank Kevin Smith for his help with facilitating and note taking. We hope to see you around, Kevin! * Previous IRC meeting: How to modify all preferences? <https://phabricator.wikimedia.org/T178449>. Outcome: Introducing a PreferenceFactory service seems like a good idea, but there are several issues to resolve with the implementation as proposed. RFC remains under discussion. * Last Call canceled: Bump PHP requirement to 5.6 in 1.31 <https://phabricator.wikimedia.org/T178538>. The RFC goes back to “under discussion” due to an issue raised by Anomie <https://phabricator.wikimedia.org/T172165#3699606>. * Discussion on wikitech-l: Proposal regarding the handling of imported usernames <https://lists.wikimedia.org/pipermail/wikitech-l/2017-October/089047.html>. * Next IRC meeting: Gathering input on the RFC process. We want to hear from you what you like or dislike about the RFC process. If you have criticism or suggestions, please let us know. Note that we have diverged quite a bit from the process as documented at Requests_for_comment/Process. This session is part of a drive to overhaul our processes and improve documentation. As always, the discussion will take place in the IRC channel #wikimedia-office on Wednesday 21:00 UTC (2pm PDT, 23:00 CEST). * Daniel has published a rant about how unclear ownership makes it hard to address technical debt <https://www.mediawiki.org/wiki/User:Daniel_Kinzler_(WMDE)/Avoiding_the_Tech…>. This topic was brought up during the discussion about FauxRequest <https://phabricator.wikimedia.org/T169266>. You can also find our meeting minutes at <https://www.mediawiki.org/wiki/Wikimedia_Technical_Committee/Minutes> See also the TechCom RFC board <https://phabricator.wikimedia.org/tag/mediawiki-rfcs/>. -- Daniel Kinzler Principal Platform Engineer Wikimedia Deutschland Gesellschaft zur Förderung Freien Wissens e.V.

1 1

Tomorrow: Weekly Technical Advice IRC Meeting
by Michael Schönitzer 07 Nov '17

07 Nov '17

Sorry for cross-posting! Reminder: Technical Advice IRC meeting again **tomorrow 4-5 pm UTC** on #wikimedia-tech. The Technical Advice IRC meeting is open for all volunteer developers, topics and questions. This can be anything from "how to get started" over "who would be the best contact for X" to specific questions on your project. If you know already what you would like to discuss or ask, please add your topic to the next meeting: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting This meeting is an offer by WMDE’s tech team. Hosts of tomorrows meeting are: @addshore & @CFisch_WMDE. Hope to see you there! Michi (for WMDE’s tech team) -- Michael F. Schönitzer Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin Tel. (030) 219 158 26-0 http://wikimedia.de Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen Wissens frei teilhaben kann. Helfen Sie uns dabei! http://spenden.wikimedia.de/ Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

1 0

Upcoming holidays/events and their impact on the deployment schedule
by Greg Grossmeier 06 Nov '17

06 Nov '17

Hello all, It's that time of year again where the density of holidays increases and at the same time our plans for fundraising also increase. Per our usual practice we will be not doing deployments at various points in the next few months. Here's the full outline: Reminder for all on we did last year: * No MW train the week of Thanksgiving (but SWAT deploys were open for high-priority things). * No deploys (at all) the last two weeks of December. People were happy with this. * The first week of January was normal (minus Monday being Jan 2nd, our observed New Year's Day holiday) deployment wise. * The second week was weird due to Dev Summit/All Hands: No MediaWiki train, only SWATs as needed on Mon/Tues/Wed. No SWATs/deploys during All Hands. I imagine we'll do similarly. In that case (looking at the calendar....) * No MW train the week of Thanksgiving (Nov 20th), SWATs open for high priority things * No deploys weeks of Dec 18th and 25th (last two weeks) * Normal week week of Jan 1st (minus no deploys that Monday) * The Dev Summit and WMF All Hands is the week of January 22nd, so that will be a "No Train but SWATs OK on Mon/Tues/Wed" week. * The following week (week of January 29th) the Release Engineering team will be on an offsite, so a week of "No Train, but SWATs and service deploys OK". This is now on-wiki at: https://wikitech.wikimedia.org/wiki/Deployments#Upcoming Best, Greg -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | Release Team Manager A18D 1138 8E47 FAC8 1C7D |

1 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l November 2017