Wikitech-l March 2019

wikitech-l@lists.wikimedia.org

94 participants
71 discussions

Discovery Weekly Update for the week starting 2019-03-04
by Chris Koerner 13 Mar '19

13 Mar '19

Good to see you, This is the weekly update from the Search Platform team for the week starting 2019-03-04. As always, feedback and questions are welcome. == Discussions == === Search === * We've updated Textcat with the new Russian wrong-keyboard and wrong-encoding models and now need CirrusSearch to be up to date as well [0] * In order to ease migration to ES6 we should migrate to 5.6.14 first (epic task is now completed) [1] ** As part of the ES6 update, we needed to upgrade logstash and the logstash elasticsearch cluster to 5.6.14 [2] ** We also upgraded logstash plugin to 5.6.14 as part of the prep for elasticsearch upgrade [3] * We've deployed & tested WikibaseCirrusSearch on beta cluster [4] * We've also deployed & tested WikibaseCirrusSearch on testwikidata and found things to be good! [5] * We found that incigna monitoring for Elasticsearch doesn't seem to notice when an out-of-memory error has happened on a node — this will be fully fixed with the next cluster restart [6] * There was some logspam due to deduplicating I18N messages between Wikibase and WikibaseCirrusSearch that we resolved [7] * A discovery was made with a WMFTimeoutException in Special:Search that was resolved by decreasing timeouts and removing our timeout hack for regexes [8] * An umbrella task to track patches of merge commits to activate elastic6 support code [9] * A ticket created way back in 2016 asked: 'the new substring search is great on mw.o. Can we have it on office wiki too?' and yes, now it is [10] === Wikidata Query Service === * We've created CI testing environment for Blazegraph on our Jenkins CI [11] * Working on hiring a contractor to help us with Blazegraph tasks [12] * We noticed that metrics from wdqs updater JMX should be prefixed while we were investigating something else, metrics exposed by jmx_exporter running on wdqs-updater should be prefixed with wdqs_updater_ to make it more clear what they are referring to (done!) [13] == Other Noteworthy Stuff == * Read Trey's new blog post about 'the anatomy of search: a place for my stuff' [14] [0] https://phabricator.wikimedia.org/T216083 [1] https://phabricator.wikimedia.org/T215931 [2] https://phabricator.wikimedia.org/T216052 [3] https://phabricator.wikimedia.org/T216993 [4] https://phabricator.wikimedia.org/T215684 [5] https://phabricator.wikimedia.org/T217276 [6] https://phabricator.wikimedia.org/T76090 [7] https://phabricator.wikimedia.org/T216839 [8] https://phabricator.wikimedia.org/T216860 [9] https://phabricator.wikimedia.org/T217043 [10] https://phabricator.wikimedia.org/T150153 [11] https://phabricator.wikimedia.org/T216855 [12] https://phabricator.wikimedia.org/T200558 [13] https://phabricator.wikimedia.org/T208215 [14] https://wikimediafoundation.org/2019/03/12/the-anatomy-of-search-a-place-fo… ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner (he/him) Community Relations Specialist Wikimedia Foundation

1 0

Old tiles.wfmlabs.org maps services need maintainers to keep them alive after december 18th
by Derk-Jan Hartman 12 Mar '19

12 Mar '19

This is just another small reminder that, because the servers which host tiles.wmflabs.org and wma.wmflabs.org (wikiminiatlas) and overpass-wiki run on a version of the OS (Ubuntu Trusty) that is no longer supported (and hasn't been available for new instances since november 2017). These services need maintainers and support by community members in order to keep them alive after dec 18th (after which wmflabs will phase out those versions) and before the EOL of early 2019 of the OS. Unfortunately it seems no one is stepping up so far to convert these machines. This issue is tracked at https://phabricator.wikimedia.org/T204506 As I was curious, I looked around on the tile server a bit and used what I could find to update https://wikitech.wikimedia.org/wiki/OSM_Tileserver#Technology_stack This is all the information that I could gather, but i'm FAR from sure if that is complete information and if I would break anything with a rebuild basing myself on that info, so any information on missing elements etc. would be appreciated. I've not gotten around to looking at wikiminiatlas. If the services are not rebuild then likely they will just disappear at some point for all layer variants. This includes the mapnik, black and white, hill shading, hike bike layers. As I have no idea how many users of these services there are, it is hard to say what the effect of that would be. DJ

7 17

Improving reading experience on mobile
by Volker E. 12 Mar '19

12 Mar '19

Hi all, in my part of the Design team at Wikimedia Foundation, I'd like to share an upcoming change in typography, that might be of interest for you: Improving reading experience on mobile [0] – As many of our projects are putting textual content first, we are consistently aiming at best possible reading experience for our users, regardless of the device, software, or language of our readers. Typography, and specifically font choices, build the base for readability. Therefore we have been proposing to rely on so-called system fonts as our default mobile font choice in the mobile skin MinervaNeue. Both major platforms, iOS and Android, but also operating systems like macOS and Windows come out-of-box with better suited system fonts than the general fallback `sans-serif`. Those specific fonts - deliver a better native experience for readers, - improve cross-platform and - improve cross-language readability. Please see the project page on mediawiki.org [0] for further technical details of the changes and an overview of our wide-range testing. Our current plan is to rollout the change to Beta-Cluster next week. We welcome your feedback! Best regards, Volker [0] – https://www.mediawiki.org/wiki/Design/Projects/Improve_mobile_reading_exper… Senior UX Engineer, UI Standardization Lead (he/him) Wikimedia Foundation volker.e(a)wikimedia.org | @Volker_E

1 0

ManualLogEntries::publish do not store tags when published to `udp`
by Piotr Miazga 12 Mar '19

12 Mar '19

Hi All, I noticed that ManualLogEntry items could have tags only when those entries are published to `rc` or `rcandudp`. Then the extensions can attach tags via RecentChange_save hook and everything works perfectly. The problem I encountered happens when the log entry is published to `udp` only[1], then tags are ignored. The only way to apply tags to the LogEntry is to call `ChangeTags::addTags()` after log entry creation. In the AMC[2] project we would like to tag user actions with `advanced mobile edit`[3]. Some of those actions, like "Thanks" are not published as RecentChange, and we want to keep it as it is right now[4]. What is the reason for this? Can I safely add support for tagging log entries that are published to udp only inside the ManualLogEntry class? Otherwise, I'll need to update Thanks extension code and - create a new hook that passes the LogEntry object, for example, "ThanksLogEntryCreation" - in the MobileFrontend listen to that hook and call the `ChangeTags::addTags( [ 'advanced mobile edit' ], null, null, $logId );` Doesn't sound that bad, but then I'll have to apply the same code to other places where we create log entries that are not published as RecentChange. It sounds like a technical debt to me. Also, the current implementation feels broken, as code quietly ignores tags when logs are published to `udp` only. It looks like the ChangeTags already supports adding tags to LogEntries only - both rc_id and rev_id are nullable, and the only question is how to tag logs published to the RecentChange (do we add tags to both LogEntry and RecentChange objects?). Additionally, I'd like to introduce a Taggable interface[6], that provides a one way to tag objects (right now RecentChange exposes addTags() method but the ManualLogEntry exposes setTags() method). [1] https://github.com/wikimedia/mediawiki/blob/278ac40e9609b0b226a85e020f5e574… [2] https://www.mediawiki.org/wiki/Reading/Web/Advanced_mobile_contributions [3] https://phabricator.wikimedia.org/T212959 [4] https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Thanks/+/493740/ [5] https://github.com/wikimedia/mediawiki-extensions-Thanks/blob/ac1c3906efabd… [6] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/493464/ -- *Piotr Miazga* (he/him) Fullstack Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

2 1

White paper Technical Wishes 2018: Building tools for diverse users
by Johanna Strodt 11 Mar '19

11 Mar '19

Hi, Over the past years, Wikimedia Germany’s Technical Wishes team has been working on new and improved tools for the diverse communities and users of the Wikimedia projects. Apart from developing software, we also developed methods and approaches for building software in a collaborative environment. We learned lots and wrote the most important parts of it down. So, if you’re interested in software development that involves a diverse community, please have a look at our white paper. It is version 1.1 of the paper we published in December 2018. You can find it on Meta wiki: https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Approach/Lessons_lear… and a PDF version on Wikimedia Commons: https://commons.wikimedia.org/wiki/File:White_Paper_Technical_Wishes_2018.p… We hope you’ll find our shared learnings helpful. If you have any questions, please reach out to us on this talk page: https://meta.wikimedia.org/wiki/Talk:WMDE_Technical_Wishes/Approach/Lessons… For the Technical Wishes team of Wikimedia Germany Birgit, Michi, Lea and Johanna -- Johanna Strodt Project manager community communication for the Technical Wishlist Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Tel. (030) 219 158 26-0 https://wikimedia.de Unsere Vision ist eine Welt, in der alle Menschen am Wissens der Menschheit teilhaben, es nutzen und mehren können. Helfen Sie uns dabei! https://spenden.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Wednesday: Technical Advice IRC Meeting
by Michael Schönitzer 11 Mar '19

11 Mar '19

Reminder: Technical Advice IRC meeting this week **(Wednesday) 4-5 pm UTC** on #wikimedia-tech. Question can be asked in English, German & Spanish! The Technical Advice IRC Meeting is a weekly support event for volunteer developers. Every Wednesday, two full-time developers are available to help you with all your questions about Mediawiki, gadgets, tools and more! This can be anything from "how to get started" over "who would be the best contact for X" to specific questions on your project. If you know already what you would like to discuss or ask, please add your topic to the next meeting: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting Hope to see you there! Michi (for the Technical Advice IRC Meeting crew) -- Michael F. Schönitzer Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Tel. (030) 219 158 26-0 https://wikimedia.de Unsere Vision ist eine Welt, in der alle Menschen am Wissens der Menschheit teilhaben, es nutzen und mehren können. Helfen Sie uns dabei! https://spenden.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

2 1

BagOStuff::modifySimpleRelayEvent removed from MediaWiki 1.33
by Aaron Schulz 08 Mar '19

08 Mar '19

The modifySimpleRelayEvent() method was narrowly intended (and only usable) for use with a WANObjectCache that uses EventRelayer. The later dependency has since been removed from WANObjectCache. It was part of an experimental approach for relaying object cache purges accross WMF datacenters, which was abandoned in favor of mcrouter/dynomite. -- -Aaron

1 0

2019-03-06 Scrum of Scrums meeting notes
by Grace Gellerman 07 Mar '19

07 Mar '19

https://www.mediawiki.org/wiki/Scrum_of_scrums/2019-03-06 =*2019-03-06*= == Callouts == * Language blocked on RelEng: CI issues/Merge Blockers: https://phabricator.wikimedia.org/T217654 and https://phabricator.wikimedia.org/T216689 * SRE informs that we have a Network maintenance (Tuesday 19th 14:00UTC) https://phabricator.wikimedia.org/T187960 ** Will require s2 to be read only (T217441 - we will use that window to upgrade MySQL on the master) ** Will also impact x1 master (which will be read only for a few seconds and cannot be disabled on mediawiki layer) * Search Platform -> RelEng: CI configuration for Blazegraph: https://gerrit.wikimedia.org/r/c/integration/config/+/493340 * Release Engineering: Train blocked: ** Growth - Flow\Exception\WikitextException: ParseEntityRef: no name - https://phabricator.wikimedia.org/T217766 ** Cloud Services - labtestweb2001: Fatal error: unknown class AuthPlugin in /srv/mediawiki/php-1.33.0-wmf.20/extensions/LdapAuthentication/LdapAuthenticationPlugin.php on line 21 - https://phabricator.wikimedia.org/T217692 == Audiences == === Contributors === ==== Community Tech ==== * Blocked by: * Blocking: * Updates: ** ==== Anti-Harassment Tools ==== * Blocked by: * Blocking: * Updates: ** ==== Editing ==== * Blocked by: * Blocking: ** Updates: ** ==== Growth ==== * Blocked by: * Blocking: ** Performance: CR for avoiding Watchlist DB writes on HTTP GET. https://gerrit.wikimedia.org/r/416198 ** Release Engineering - Flow\Exception\WikitextException: ParseEntityRef: no name - https://phabricator.wikimedia.org/T217766 * Updates: ** ==== Language ==== * Blocked by: * Blocking: * Updates: ** === Readers === ==== iOS native app ==== * Blocked by: * Blocking: * Updates: **Working on 6.2.1 ( https://phabricator.wikimedia.org/tag/ios-app-v6.2.1-beluga-on-stilts/) ***bug fixes ***editing enhancements ==== Android native app ==== * Blocked by: * Blocking: * Updates: ** ==== Readers Web ==== * Blocked by: * Blocking: * Updates: ** Summary: adding some features to QuickSurveys, medium term planning, and continuing the Advanced Mobile Contributions and MobileFrontend Architecture investment projects. ** Responsive website (MinervaNeue / MobileFrontend): *** Advanced mobile contributions https://www.mediawiki.org/wiki/Reading/Web/Advanced_mobile_contributions **** Provide mechanism to allow dynamically tag log entries T215675 **** Add X-Analytics tag for AMC webrequests T212961 **** Cannot access user contributions when following red link to user page on mobile T201339 *** Invest in the MobileFrontend & MinervaNeue frontend architecture https://www.mediawiki.org/wiki/Reading/Web/Projects/Invest_in_the_MobileFro… **** Refactor TalkSectionAddOverlay T217102 **** Refactor ImageOverlay T216198 **** Post-inheritance cleanup T217220 **** Limit mobile.startup's mw.config variables T216848 **** Refactor TalkOverlay T215370 **** Refactor LanguageOverlay T215657 *** Update list styles T150377 *** Miscellaneous bug fixes and maintenance T216628 *** QuickSurveys **** Consultation with Research **** Support sampling by country T213847 **** Support sampling by edits T139317 **** Support sampling by page T213696 **** QuickSurveys are hidden on pages without infoboxes and with table of contents T210646 ** Desktop website (Vector, Popups) *** Vector **** Vector should replace rendering from PHP with Mustache T217172 *** Popups https://www.mediawiki.org/wiki/Page_Previews **** WMDE reference previews review and support T67114 https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/ReferencePreviews ** Medium-term planning. ==== Readers Infrastructure ==== * Blocked by: Security on WikimediaEditorTasks review (they know, it was assigned yesterday) * Blocking: * Updates: ** WikimediaEditorTasks extension security review in progress ** Reminder: summary 1.4.0 endpoint change: titles in the content_urls properties are encoded now (T216739) ==== Multimedia ==== * Updates ** ==== Parsing ==== * Blocked by: * Blocking: * Updates: ==== UI Standardization ==== * Blocked by: * Blocking: * Updates: ** == Technology == === Analytics === * Blocked by: * Blocking: * Updates: ** === Cloud Services === * Blocked by: * Blocking: Release Engineering - labtestweb2001: Fatal error: unknown class AuthPlugin in /srv/mediawiki/php-1.33.0-wmf.20/extensions/LdapAuthentication/LdapAuthenticationPlugin.php on line 21 - https://phabricator.wikimedia.org/T217692 * Updates: ** === Fundraising Tech === * Blocked by: * Blocking: * Updates: ** Reviewing audit code for main card processor to make sure we're catching everything https://phabricator.wikimedia.org/T217582 ** Fixing CentralNotice sanitization over-strictness https://phabricator.wikimedia.org/T216150 ** Better fraud attempt queries for CiviCRM https://phabricator.wikimedia.org/T199268 ** Tweaks to Thank You mail https://phabricator.wikimedia.org/T207674 ** Building notification stream for changes to active CentralNotice banners and included bits https://phabricator.wikimedia.org/T208511 ** Working on recording opt-in choices even when payment attempt fails https://phabricator.wikimedia.org/T216293 ** Still trying to coordinate timing of payments-wiki upgrade to PHP7 & MW 1.31 https://phabricator.wikimedia.org/T184460 === Core Platform === * Blocked by: * Blocking: * Updates: ** Documentation Portal product plan draft ** Tech writer hiring ** PEG rule parameters ** Kask to K8S (logging, readiness) ** Engineering manager hiring last mile ** === Performance === * Blocked by: ** Parsing Team: CR for Parser save timing optimisation. https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/294774/ ** Growth Team: CR for avoiding Watchlist DB writes on HTTP GET. https://gerrit.wikimedia.org/r/416198 * Blocking: ** * Updates: ** Aaron made database messages to Logstash nicer and less verbose. ** New Fresnel release (per-patch MW perf tester), scenarios now have names. ** Working on sampling profiler instrastructure for PHP 7, to match HHVM/Xenon flame graphs. === Release Engineering === * Blocked by: Train blocked: ** Growth - Flow\Exception\WikitextException: ParseEntityRef: no name - https://phabricator.wikimedia.org/T217766 ** Cloud Services - labtestweb2001: Fatal error: unknown class AuthPlugin in /srv/mediawiki/php-1.33.0-wmf.20/extensions/LdapAuthentication/LdapAuthenticationPlugin.php on line 21 - https://phabricator.wikimedia.org/T217692 * Blocking: ** Language: CI issues/Merge Blockers: https://phabricator.wikimedia.org/T217654 and https://phabricator.wikimedia.org/T216689 ** Search Platform: CI configuration for Blazegraph: https://gerrit.wikimedia.org/r/c/integration/config/+/493340 * Updates: ** Choosing tools for continuous integration https://phabricator.wikimedia.org/phame/post/view/148/choosing_tools_for_co… ** Projects, Forms and Subtypes oh my! https://phabricator.wikimedia.org/phame/post/view/147/projects_forms_and_su… ** Train Health: *** Last week: 1.33.0-wmf.19 - https://phabricator.wikimedia.org/T206673 **** Still one blocker left - TranslateRenderJob: Cannot render translation page for Wikimania/pa-guru! https://phabricator.wikimedia.org/T217380 *** This week: 1.33.0-wmf.20 - https://phabricator.wikimedia.org/T206674 **** We have two blockers: ***** Flow\Exception\WikitextException: ParseEntityRef: no name - https://phabricator.wikimedia.org/T217766 ***** labtestweb2001: Fatal error: unknown class AuthPlugin in /srv/mediawiki/php-1.33.0-wmf.20/extensions/LdapAuthentication/LdapAuthenticationPlugin.php on line 21 - https://phabricator.wikimedia.org/T217692 *** Next week: 1.33.0-wmf.21 - https://phabricator.wikimedia.org/T206675 === Research === * Blocked by: None * Blocking: None * Updates: ** Ran the reader demographics survey: https://phabricator.wikimedia.org/T217080 ** Google summer of code preparation: https://phabricator.wikimedia.org/T217655 ** Collected new set of data for the CitationUsage schema for quality assurance: https://phabricator.wikimedia.org/T213969 ** Improving article-recommender scripts: https://phabricator.wikimedia.org/T211981 === Scoring Platform === * Blocked by: * Blocking: * Updates: === Search Platform === * Blocked by: RelEng: CI configuration for Blazegraph: https://gerrit.wikimedia.org/r/c/integration/config/+/493340 * Blocking: * Updates: ** WikibaseCirrusSearch deployed tested on Beta cluster, works fine, testwikidata next: https://phabricator.wikimedia.org/T215684 ** Working on ES 6 upgrade: https://phabricator.wikimedia.org/T183282 ** Working on language-specific lowercasing in search indexing: https://phabricator.wikimedia.org/T217602 ** Updated extensions to use new phan (thanks Legoktm, bugs found!) ** Working on query parsing refactoring: https://phabricator.wikimedia.org/T185108 ** Working on moving CirrusSearch code out of Wikibase to a separate extension: https://phabricator.wikimedia.org/T190022 === Security === * Blocked by:None * Blocking:None * Updates: **https://phabricator.wikimedia.org/T217289: ass*igned 03/05 **https://phabricator.wikimedia.org/T215048: assigned 03/05 **https://phabricator.wikimedia.org/T216692: TBD **https://phabricator.wikimedia.org/T163827: complete by end of week **https://phabricator.wikimedia.org/T216419: will start this week **https://phabricator.wikimedia.org/T211489: complete by end of week **https://phabricator.wikimedia.org/T201492: TBD **https://phabricator.wikimedia.org/T103011: TBD **https://phabricator.wikimedia.org/T207990: will start this week === Services === * Blocked by: * Blocking: * Updates: ** === Site Reliability Engineering === * Blocked by: ** None * Blocking: ** None * Updates: ** Network maintenance (Tuesday 19th 14:00UTC) T187960 *** Will require s2 to be read only (T217441 - we will use that window to upgrade MySQL on the master) *** ill also impact x1 master (which will be read only for a few seconds and cannot be disabled on mediawiki layer) == Wikidata == * Blocked by: * Blocking: * Updates: ** == German Technical Wishlist == * Blocked by: * Blocking: * Updates: ** == SoS Meeting Bookkeeping == * Updates: **

2 1

Unstable search results - possible causes
by Hogan (US), Michael C 07 Mar '19

07 Mar '19

I'm using the built-in Mediawiki search engine. We just updated from 1.30.0 to 1.31.0. Since the update, search results are unstable. The same search term gives different results in different web browsers. We also see different results across browser sessions. Any advice on how I can troubleshoot this? Thank you! -Michael

4 4

Revised Security Reviews Process
by Charlotte Portero 07 Mar '19

07 Mar '19

The Security Team has revised the Standard Operating Procedure (SOP). The revised SOP can be found at: https://www.mediawiki.org/wiki/Security/SOP/Security_Readiness_Reviews For any questions or comments, please go the Talk Page at https://www.mediawiki.org/wiki/Talk:Security/SOP/Security_Readiness_Reviews. Thank you, Charlotte Portero Project Manager, Security

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l March 2019