Wikitech-l August 2018

wikitech-l@lists.wikimedia.org

101 participants
97 discussions

Readers Monthly update for July 2018
by Chris Koerner 06 Aug '18

06 Aug '18

Hello all, This is the monthly update from the Readers team at the Wikimedia Foundation for July 2018. Feedback and questions are encouraged. ==Discussions== === Apps=== * Wikipedia for iOS released a first beta of the next major version, 6.0.0. This update includes a fully user controlled Explore feed, similar to the recent changes on Android. It also includes major design and navigation improvements. === Web === * Page Previews now aborts in-flight requests if the user has shifted their focus, resulting in less data being sent to the user. [0] Indeed, Page Previews is now sending ~25% of the requests that it used to at peak. [1] *We are continuing work on advanced mobile contributions. [2] We conducted more than 20 interviews with editors at Wikimania and collected information on the most requested pages and workflows for editing and contributing on mobile. === Reading Infrastructure === * Doing definition work for T199096ː Add support for wikidata summaries in the /page/summary/ endpoint consulting with Wikidata and Readers web team [3] * Page Content Service: Improving mobile-html (fka. content-html) endpoint output. [4] Added several transformations from the #wikimedia-page-library. [5] * T198622ː Migrating maps servers to stretch with the current style and subtasks (Mapnik library upgrades for the stretch migration) [6] === New Readers === * New Reader updates can be found at Meta. [7] === Multimedia === * Video downloads will start being produced in VP9 format soon. [8] [0] https://phabricator.wikimedia.org/T197700 [1] https://grafana.wikimedia.org/dashboard/db/reading-web-page-previews?orgId=… [2] https://www.mediawiki.org/wiki/Reading/Web/Advanced_mobile_contributions [3] https://phabricator.wikimedia.org/T199096 [4] https://www.mediawiki.org/wiki/Page_Content_Service#Mobile_HTML [5] https://github.com/wikimedia/wikimedia-page-library [6] https://phabricator.wikimedia.org/T198622 [7] https://meta.wikimedia.org/wiki/New_Readers/Updates [8] https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition --- Subscribe to receive theses updates as on-wiki notifications or opt-in email. https://www.mediawiki.org/wiki/Newsletter:Readers_Monthly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Reading/Status_updates Yours, Chris Koerner Community Relations Specialist Wikimedia Foundation

1 0

Discovery Weekly Update for the week starting 2018-07-30
by Chris Koerner 06 Aug '18

06 Aug '18

Greetings, This is the weekly update from the Search Platform team for the week starting 2018-07-30. As always, feedback and questions welcome. == Discussions == === Search === * David did a lot of work surrounding the upgrade to ElasticSearch 6.3: ** updated an extra plugin[0] ** updated the LTR plugin [1] ** worked on an older issue that we were waiting to fix for an experimental plugin that was throwing an exception on some requests [2] ** upgraded the experimental highlighter [3] ** upgraded the extra-analysis plugin [4] * Trey fixed an address stemming issue in Polish analyzer for search [5] and it'll be fully in production when a re-index occurs [6] * Trey also worked on a review of the Esperanto Morphological Libraries when a volunteer wanted to help in this effort (and knew Esperanto) [7] * David worked on a few cirrus integration tests are were randomly and consistently failing (V => Venom > V:N) [8] * Trey took on the task of reviewing the work of applying Indonesian Analysis Chain for Malay (writeup and review) [9] [10] * David took on the fairly massive task of changing how SpecialSearch/SearchEngine handles the 'prefix' URI param set by the InputBox extension with lots of help from others [11] * David did quite a bit of work on deprecating the SearchEngine::replacePrefixes with several related patches [12] * Trey worked on exploring the potential applications for NLP to be applied in Search (review and write-up [13] [14] * Stas implemented fulltext search for Lexemes when Lexeme namespace is requested in the search [15] * Stas added collecting click data from Wikidata prefix search into event logs so that we know what the users are selecting when using Wikidata prefix search [16] and it'll be in production late this week with the train * Erik and David worked on the completion suggester code which didn't gracefully handle shard failures during the fetch phase, which caused the response received by cirrus to not contain the necessary information [17] and it'll be in production late this week with the train * After a lot of discussion, Erik fixed an issue where intitle search didn't match the stop words [18] and it'll be in production late this week with the train * Gehel and Erik worked on ensuring discovery.query_clicks_* data is purged per privacy policy, with adding support for hourly or daily partition dropping [19] === Wikidata Query Service === * Wikibase Constraints violations are now loaded into WDQS database and are queryable. [20] [0] https://phabricator.wikimedia.org/T198067 [1] https://phabricator.wikimedia.org/T198066 [2] https://phabricator.wikimedia.org/T178425 [3] https://phabricator.wikimedia.org/T198069 [4] https://phabricator.wikimedia.org/T198068 [5] https://phabricator.wikimedia.org/T186046 [6] https://phabricator.wikimedia.org/T200037 [7] https://phabricator.wikimedia.org/T197240 [8] https://phabricator.wikimedia.org/T199445 [9] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Analysis_of_Applying… [10] https://phabricator.wikimedia.org/T196780 [11] https://phabricator.wikimedia.org/T198318 [12] https://phabricator.wikimedia.org/T198860 [13] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Potential_Applicatio… [14] https://phabricator.wikimedia.org/T193070 [15] https://phabricator.wikimedia.org/T196188 [16] https://phabricator.wikimedia.org/T196186 [17] https://phabricator.wikimedia.org/T180807 [18] https://phabricator.wikimedia.org/T68969 [19] https://phabricator.wikimedia.org/T189845 [20] https://phabricator.wikimedia.org/T172380 --- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Relations Specialist Wikimedia Foundation

1 0

Wikilabels now enforces SSL
by Amir Sarabadani 06 Aug '18

06 Aug '18

Hey, Some people reported that they have trouble labelling edits in wikilabels [1] (specially for Wikidata and that's why I'm cross-posting to wikidata-l too). That was due to the fact that wikilabels is behind a proxy on cloud VPS which confuses the application and sometimes it thinks the request is made over HTTP and changes the links/redirects to http instead of https and basically jumping over to SSL and not-SSL continuously causing OAuth redirect loops (because cookies can't jump around like requests). There were several options but the most secure and sane one was to enforce SSL and tell the application to consider everything as SSL request. This would solve lots of bugs and issues in wikilabels. We apologize for any inconvenience caused by it. It was difficult to understand the underlying problem because it was happening for some people and not all and was hard to reproduce the bug. Also, If you encounter any (new) issue, Please let me know immediately. More info: https://phabricator.wikimedia.org/T184437 [1]: https://labels.wmflabs.org/ Best -- Amir Sarabadani Software Engineer Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Tel. (030) 219 158 26-0 http://wikimedia.de Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen Wissens frei teilhaben kann. Helfen Sie uns dabei! http://spenden.wikimedia.de/ Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Re: [Wikitech-l] How to make oauth authentication with wikipedia?
by rupert THURNER 06 Aug '18

06 Aug '18

On Sun, Jul 29, 2018 at 4:30 PM, Bryan Davis <bd808(a)wikimedia.org> wrote: > On Sun, Jul 29, 2018 at 12:37 AM rupert THURNER > <rupert.thurner(a)gmail.com> wrote: > > > > if one takes an example, lke https://tools.wmflabs.org/video2commons/, > is > > this implemented like it should? is there any difference from "any" > > application or applications on the tools server? am looking at the code > > here currently: > > https://github.com/toolforge/video2commons/blob/master/ > video2commons/frontend/app.py > > the "dologin" method. > > Yes, there is a major difference between a web application like the > video2commons tool and a device native application like an Android > app. That difference is that in a web application secret data can be > kept on the web server side that is not visible to the end user. This > allows the OAuth application secret to be used in signing requests to > the Wikimedia servers without exposing that secret to anyone who is > looking at the source code of the web application. This separation is > not possible when the application is running on end-user controlled > devices as a phone or desktop application does. > > interesting, never thought about it. i found an entry on stackexchange confirming what you said. additionally it states that oauth is not for authenticaiton. oauth's purpose is to access users resources from some resource provider, while openid_connect should be used to authenticate. does openid_connect work with wikipedia and is it the best option currently? [0] https://security.stackexchange.com/questions/133065/why-is-it-a-bad-idea-to… [1] https://connect2id.com/learn/openid-connect [2] https://www.mediawiki.org/wiki/Extension:OpenID_Connect rupert

2 1

TechCom Radar 2018-08-01
by Kate Chapman 06 Aug '18

06 Aug '18

Hi All, Here are the minutes from this week's TechCom meeting: * IRC Meeting hosted for: RFC: Modern Event Platform - Choose Schema Tech <https://phabricator.wikimedia.org/T198256> consensus in meeting was to move forward with JSONSchema with strict policies about what is allowed (e.g. no additionalProperties) and how to do schema evolution. * Discussion on RFC: Block users by page/namespace discussion continues <https://phabricator.wikimedia.org/T199917> * Triaged: Pages do not have stable identifiers <https://phabricator.wikimedia.org/T196950> and was decided to postpone until after Unify various deletion systems <https://phabricator.wikimedia.org/T20493> which may solve the issue * Triaged: Editor should flag mobile editing warnings <https://phabricator.wikimedia.org/T200880> seeking clarification and then will review again. * Triaged and shortly scheduling IRC meeting for: RfC: Release notes automation <https://phabricator.wikimedia.org/T200392> You can also find our meeting minutes at <https://www.mediawiki.org/wiki/Wikimedia_Technical_Committee/Minutes> See also the TechCom RFC board <https://phabricator.wikimedia.org/tag/mediawiki-rfcs/>. If you prefer you can subscribe to our newsletter here: <https://www.mediawiki.org/wiki/Newsletter:TechCom_Radar> Thanks, Kate -- Kate Chapman TechCom Facilitator (Contractor)

1 0

Making PolyGerrit the default ui for gerrit
by Paladox 06 Aug '18

06 Aug '18

Hi, i have created this task [1] with i have uploaded this patch [2] to make polygerrit the default ui. The reason why is upstream are preparing to remove the gwtui very soon. In matter of fact upstream have disabled the gwtui on *.googlesource.com. Upstream already have this change [3] to remove the ui. Making PolyGerrit the default ui will get new users use to the new ui. GWTUI will still be available with ui switcher in the footer or you can append the url like https://gerrit.wikimedia.org/r/?polygerrit=0 PolyGerrit is stable, secure and also fast. It also has features that you cannot see in gwtui like user status, naming your patchiest (description), cc feature and also being able to tell who added you as a reviewer. This email is advanced notice before we change the default ui. any bugs todo with polygerrit / gerrit can be filled at https://phabricator.wikimedia.org/project/view/330/ and we can forward it upstream. [1] https://phabricator.wikimedia.org/T196812 [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/439444 [3] https://gerrit-review.googlesource.com/c/gerrit/+/116790

2 6

MediaWiki action api - query categories property of allimages list?
by Hogan (US), Michael C 04 Aug '18

04 Aug '18

Is there a way to use the allimages API to get categories applied to each File: page? If not, would it make sense to suggest adding a categories property using Phabricator or by some other communication channel? https://www.mediawiki.org/wiki/API:Allimages Thank you, Michael

2 1

2018-08-01 Scrum of Scrums meeting notes
by Grace Gellerman 03 Aug '18

03 Aug '18

https://www.mediawiki.org/wiki/Scrum_of_scrums/2018-08-01 = 2018-08-01= == Callouts == * SREs say FYI: ** Switchover tentative timeline announced. Please take a look and voice concerns if any Switchover: Services: Tuesday, September 11th 2018 14:30 UTC Media storage/Swift: Tuesday, September 11th 2018 15:00 UTC Traffic: Tuesday, September 11th 2018 19:00 UTC Mediawiki: Wednesday, September 12th 2018: 14:00 UTC Switchback: Traffic: Tuesday, October 9th 2018 19:00 UTC (and maybe some prep work on Monday) Mediawiki: Wednesday, October 10th 2018: 14:00 UTC Services: Thursday, October 11th 2018 14:30 UTC Media storage/Swift: Thursday, October 11th 2018 15:00 UTC * MCR Schema migrations are happening ** Beta is set to write both / read new with no problems identified ** Production set to write both / read old this week ** Starting with migration scripts to copy data from old schema to new one == Audiences == === Readers === ==== iOS native app ==== * Blocked by: - * Blocking: - * Updates: **6.0 in beta ( https://phabricator.wikimedia.org/tag/ios-app-v6.0-walrus-on-a-unicycle/), expected to push another beta this week ==== Android native app ==== * Blocked by: * Blocking: * Updates: ** Maintenance update released to Beta, will go to production tomorrow (Thurs) ** User testing of navigation design prototypes will commence shortly. ==== Readers Web ==== * Blocked by: * Blocking: * Updates: ** RFC: Modernise our JavaScript with the goal that the startup module and mediawiki object should not depend on jQuery https://phabricator.wikimedia.org/T200868 ** Mobile website (MinervaNeue / MobileFrontend): *** Page issues UI and instrumentation: T191532 T191303 T197932 T197931 T200867 *** Invest in the MobileFrontend & MinervaNeue frontend architecture: T188261 T197133 T197639 *** Other fixes and hygiene T193172 T200491 T186823 T199282 T199000 T199066 T197110 T198151 ** Product and design are collating research and interviews from Wikimania and working on advanced contributions ==== Readers Infrastructure ==== * Blocked by: * Blocking: * Updates: ** mobile-html exposed but needs fixes for CSP header and a URL to CSS. Won't run CollapseTable transform on server for the time being. ** Maps: *** New tile regeneration cron job for low-zoom vector tiles https://phabricator.wikimedia.org/T194787 *** Snapshot low DPI bug is waiting for a pull request review from upstream on third party library https://phabricator.wikimedia.org/T152196 *** Working on migrating test servers to Cloud VPS *** Working with SRE (Gehel) on migrating all servers to Debian Stretch ==== Multimedia ==== * No representative today - two team members on vacation and Mark has a conflicting meeting * Updates ** Breaking change in MediaWiki Core OOUI widget nearly caught us out - luckily something else broke in the meantime so the related fatal never occurred because of another fatal. Both fixed now. ** Search for SDC coming along slowly with various people on vacation. ** Next up: OOUI rendering for SDC first feature (captions), coming down the pipeline soon™ ** MCR rollout plan basically set, we'll need some time to sort out our integration, but should all work out well. === Contributors === ==== Community Tech ==== * Blocked by: * Blocking: * Updates: ** ==== Anti-Harassment Tools ==== * Blocked by: * Blocking: * Updates: ** ==== Editing ==== * Blocked by: None * Blocking: None ** Updates: **Unblocked languages on previous issues. ** ==== Parsing ==== * Blocked by: * Blocking: * Updates: ==== Growth ==== * Blocked by: * Blocking: * Updates: ** ==== Language ==== * Blocked by: ** CX patches still blocked from merging due to test failures occuring in Wikibase Lexeme ** Thanks to everyone who helped/are helping to solve issues (4 separate ones) that prevent merging patches to CX&Translate * Blocking: ** Providing guidance on https://phabricator.wikimedia.org/T195910 (ios app language selection related issue) * Updates: ** Grant Metrics enabled for translation in translatewiki.net ** Niklas will work on https://phabricator.wikimedia.org/T195293 (follow-ups to an issue causing an outage a while ago) === Audiences Design === ==== UI Standardization ==== * Blocked by: * Blocking: * Updates: ** Preparation of OOUI v0.27.6 == Technology == === Analytics === * Blocked by: * Blocking: ** Outage in Eventlogging * Updates: ** Added data quality checks in mediawiki history and mediawiki history reduced datasets ** Cleaning of user directories in hadoop ** Improved data sanitization in Eventlogging (salt creation/rotation) ** Improved partition cleaning in hive === Cloud Services === * Blocked by: * Blocking: * Updates: ** === Fundraising Tech === * Blocked by: * Blocking: * Updates: ** Making our exchange rates available on a WMF-visible spreadsheet for travel reimbursement ** Work continues on the ingress scripts for EventLogging data from banner and landing page impressions ** Experimenting with different ways to let donors from certain countries opt in to emails ** CiviCRM: new search filters, looking at mailing house data import bug and data erasure requests ** Reviewing CiviCRM core team code for contact view editor ** More error-handling work prompted by full-scale tests of card processor's new API === MediaWiki Core Platform === * Blocked by: * Blocking: * Updates: ** Cherry-picked API statsd improvement into production. Updated API dashboard in Grafana. *** https://grafana.wikimedia.org/dashboard/db/api-backend-summary?refresh=5m&o… ** Platform Evolution kick-offs and TechConf prep are ongoing ** Language variants for summary REST API deployed - Page Previews will now show your preferred variant *** Only serbian language is supported now, but more will be added as Parsoid supports more conversions ** Memory issues with EventStreams service in codfw affected other services, now resolved. == Multi-Content Revisions == * Blocked by: * Blocking: * Updates: ** MCR Schema migrations are happening *** Beta is set to write both / read new with no problems identified *** Production set to write both / read old this week *** Starting with migration scripts to copy data from old schema to new one ** Discussion on-going on whether a production test wiki is necessary in addition to a test wiki on beta: https://phabricator.wikimedia.org/T197616 *** Requires Wikibase change propagation ** Development and code review continues === Performance === * Blocked by: ** * Blocking: ** * Updates: ** === Release Engineering === * Blocked by: ** Related to 1.32.0-wmf.14 train https://wikitech.wikimedia.org/wiki/Incident_documentation/20180724-Train#A… *** Feedback needed from Wikidata team on how T200340 (*Wikibase\DataModel\Entity\EntityIdParsingException $serialization must not be an empty string*) could be prevented. *** Feedback needed from Readers/Performance team(s) on how T200346 (*wmf.14 failing to execute ThumbnailRender jobs "error: ThumbnailRenderJob::run: HTTP request failure"*) could be prevented. *** Feedback needed from ORES/Wikidata team(s) on how T200412 (*PageTriage requires ORES to be installed*) could be prevented. *** Feedback needed from Wikidata team on how T200420 (*Wikidata dispatching stuck (not releasing lockmanager locks)*) could be prevented. *** Feedback needed from Readers team on how T200456 (*MapCacheLRU::has called with invalid key. Must be string or integer*) could be prevented. * Blocking: * Updates: ** Moved the beta deployment server to deployment-deploy01, yell if anything seems amiss there === Research === * Blocked by: None * Blocking: None * Updates: ** We've been analyzing the Citation usage data and improving the schema. - https://phabricator.wikimedia.org/T199457 ** Working on recommending missing articles based on translation pageview predictions. ** Have been interviewing candidates for a research position. === Scoring Platform === * Blocked by: ** Waiting for a TechCom review of JADE proposal for deployment. *** Trying to get Global Collaboration's attention about this. *** "Everything is a wiki page", yet we're not allowed to deploy an extension that might increase the number of wiki pages... Who will be blocked next? * Blocking: * Updates: ** GSoC project by Saurabh Batra to identify donations fraud has resulted in a successful model! We're beginning integration work now. === Search Platform === * Blocked by: * Blocking: * Updates: ** Lexeme fulltext search merged: https://phabricator.wikimedia.org/T196188 ** Started collecting click stats for Wikidata completion search: https://phabricator.wikimedia.org/T196186 ** Quality constraints violations are now in WDQS: https://phabricator.wikimedia.org/T172380 ** Fixed intitle search not matching stopwords: https://phabricator.wikimedia.org/T68969 ** Fixed problems in completion search on shard failure: https://phabricator.wikimedia.org/T180807 ** Added task to purge query click data according to privacy policy: https://phabricator.wikimedia.org/T189845 ** Working on running multiple Elastic instances on the same hardware: https://phabricator.wikimedia.org/T193654 ** Working on ES 6.3 upgrade: https://phabricator.wikimedia.org/T197960 ** Working on Esperanto analyzer: https://phabricator.wikimedia.org/T200099 ** Working on query parsing refactoring: https://phabricator.wikimedia.org/T185108 === Security === * Blocked by: * Blocking: * Updates: ** === Site Reliability Engineering === * Blocked by: ** None * Blocking: ** None * Updates: ** eqiad row B and C upgrades went quite fine with some minor hiccups ** work on the switchover preparation ongoing. Tentative timeline posted. Take a look at calls out section == Wikidata == * Blocked by: * Blocking: ** ContentTranslation CI red because of failing Wikidata-related tests: https://phabricator.wikimedia.org/T200693 *** working on understanding and fixing failures * Updates: ** Work in Lexicographical data continues: *** adding senses *** improving the editor workflow *** https://phabricator.wikimedia.org/project/view/2292/ == German Technical Wishlist == * Blocked by: * Blocking: * Updates: ** == SoS Meeting Bookkeeping == * Updates: ** move the meeting to 2 hours earlier as soon as week of 2018-08-06- look for new invite! ** some changes to the notes template ** keep the meeting as is- a forum for teams to share updates and ask for help and keep the notes as useful as possible ** decision to handle goals dependencies outside this meeting

2 1

phone cannot play mobile site video, but standard site video
by rupert THURNER 03 Aug '18

03 Aug '18

why is it that the excellent player is available on the standard wikipedia view, and not on mobile view? is this just my configuration? as eyample: * https://en.wikipedia.org/wiki/Polar_orbit * https://en.m.wikipedia.org/wiki/Polar_orbit the effect is that my mobile can play the standard view video, but not the mobile view video. rupert

2 1

New Beta Cluster Deployment Servers
by Tyler Cipriani 03 Aug '18

03 Aug '18

Hi All! There are new deployment servers in beta cluster: deployment-deploy01 and deployment-deploy02. These servers replace deployment-tin and deployment-mira (which I just shut down today). If you had a home directory on either deployment-tin or deployment-mira I've moved it to the new machines at: deployment-deploy01:~/deployment-tin-home deployment-deploy02:~/deployment-mira-home If you see anything amiss that you feel may be a result of this move, please file a task on phab or find me on IRC. Thanks to Daniel Zahn and Alex Monk for all their work on this task! Thanks! -- Tyler

1 0

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l August 2018