https://www.mediawiki.org/wiki/Scrum_of_scrums/2018-04-11
= 2018-04-11 =
*Q4 FY1718*
== Callouts ==
* Fundraising campaigns
https://meta.wikimedia.org/wiki/CentralNotice/Calendar - Netherlands
2018-04-03 through 2018-05-01
* OpenStack upgrades on Friday 2018-04-13 -- Horizon and CI
(Zuul/Jenkins/Nodepool) will be disabled for a while beginning around 14:00
UTC
* Still need help from RelEng with WDQS GUI build automation:
https://phabricator.wikimedia.org/T189811
== Audiences ==
=== Readers ===
==== iOS native app ====
* Blocked by:
* Blocking:
* Updates:
**
==== Android native app ====
* Blocked by:
* Blocking:
* Updates:
** Reading list sync rolled out to 10% of production (~800K users)
** On track to roll out to full production next week.
==== Readers Web ====
* Blocked by:
[performance] https://phabricator.wikimedia.org/T42812 - nice
collaboration going on here, to improve mw-collapsible behaviour on desktop
- [releng] https://phabricator.wikimedia.org/T190710 - getting browser
tests ported to Node for Minerva - Thanks Željko for being so responsive!
* Blocking:
* Updates:
** Updating clients to reflect new wikidata description changes (
https://phabricator.wikimedia.org/T190953)
**Deploying Page previews to German Wikipedia (right now!!)
*Quarterly goal dependency update:
**Outcome 1, Objective 4: Continue improving the ways that users can
download articles of interest for later consumption
*** Reading Web depends on SRE, RelEng, Reading Infra
==== Readers Infrastructure ====
* Blocked by:
* Blocking:
* Updates:
** CSS endpoint going to be deployed in RESTBase after MediaWiki train
completes
** Working on very simplistic browser extension to add a single page to a
reading list
** TemplateStyles was deployed on ruwiki
*Quarterly goal dependency update:
**Outcome 1, Objective 4: Continue improving the ways that users can
download articles of interest for later consumption
*** Reading Web depends on SRE, RelEng, Reading Infra
**Increase code sharing of client apps by coalescing and moving more logic
to the server.
***Reading Infra depends on Parsing, Services
===== Maps =====
* Blocked by: None
* Blocking: None
* Updates:
** Working on internationalization and enwiki rollout
==== Multimedia ====
* Updates
** Wikibase patches for file page prototype going through
** Search work continuing
** File captions - currently working through changing the name of the field
for MediaInfo - it's a bear
*Quarterly goal dependency update:
**Objective 3.1 Prepare for launch of the first Structured Data on Commons
feature (multilingual file captions)
***SDC depends on Multimedia,SRE, WMDE, Search Platform, MediaWiki
Platform, Research
** Objective 2.1 Integrate structured file captions into search
*** SDC depends on Search Platform, Multimedia
**Segment 4, Outcome 2: Develop a better understanding of existing needs
for Structured Commons- T171252
***Research depends on Multimedia
=== Contributors ===
==== Community Tech ====
* Blocked by:
* Blocking:
* Updates:
** CodeMirror is getting ready to promotion from beta
** Test deployment of GlobalPreferences today
==== Anti-Harassment Tools ====
* Blocked by:
* Blocking:
* Updates:
Blocking Tool Improvments
==== Editing ====
* Blocked by:
* Blocking:
** Updates:
** Goal 3.1 Improve and consolidate our unified editing platform so that
it's great on all devices
***Editing depends on Parsing
==== Parsing ====
* Blocked by:
* Blocking:
* Updates:
** Tidy replaced with RemexHtml on 40 more wikis today (see
https://phabricator.wikimedia.org/T190731 ) Currently have about 650
running RemexHTML, another 250 yet to convert.
** Parsoid is exploring the use of Finite State Transducer formalism for
language variants ( see https://phabricator.wikimedia.org/T191925 && gerrit
- https://gerrit.wikimedia.org/r/#/c/423197/ )
** TTO and us had an IRC meeting last week about the Poem extension and TTO
summarized the decisions in the phabricator task ( see
https://phabricator.wikimedia.org/T54061#4110198 )
** No quarterly goal dependency updates at this time. Services, Editing,
Parsing teams will be syncing up this/next week to identify any blockers.
*Quarterly goal dependency update:
**Goal 3.6 Support work towards unifying MediaWiki's parser
implementations, in liaison with Technology's MediaWiki team
***Parsing depends on MediaWiki Platform, Services
**Increase code sharing of client apps by coalescing and moving more logic
to the server.
***Reading Infra depends on Parsing, Services
** Goal 3.1 Improve and consolidate our unified editing platform so that
it's great on all devices
***Editing depends on Parsing
Outcome 2: Objective 1: Revision storage scaling
- Services depends on SRE, Parsing
==== Collaboration ====
* Blocked by:
* Blocking:
* Updates:
** Hoping to dark-deploy maps localization this week; had some false starts
and reverted deploys
==== Language ====
* Blocked by:
* Blocking:
* Updates:
**
*Quarterly goal dependency update:
**Goal 3.1 Improve and consolidate our unified editing platform so that
it's great on all devices
***Language depends on Editing
=== Audiences Design ===
* Blocked by:
* Blocking:
* Updates:
**
==== UI Standardization ====
* Blocked by:
* Blocking:
* Updates:
** OOUI – v0.26.3 released:
https://phabricator.wikimedia.org/diffusion/GOJU/browse/master/History.md;v…
*** Apex theme now uses WikimediaUI icon, this was long overdue in regards
of maintainability cost and that there were under 10% Apex specific icons
left
*** 4 new icons for 'tableMoveColumn*' & 'tableMoveRow*'
*** CheckboxInput-/MenuOptionWidget/MenuToolGroup: Don't specify icon in
CSS, reduces CSS output (Bartosz Dziewoński)
** Work on size unification aftermath in different products (Notifications,
Flow, RevisionSlider)
** Style Guide technical and contents improvements ongoing with 5 open PRs
currently
*** PR with big impact: Using system fonts as sans-serif choice –
https://phabricator.wikimedia.org/T188799
** Transparency Report: Wrapping up accessiblity measurements (continuation
from last week)
**
== Technology ==
=== Analytics ===
* Blocked by: traffic in a bit of issues
* Blocking:
* Updates:
**
** Major change on hadoop cluster to spark2, we try to avoid non backwards
compatible changes
** but this actually non backwards compatible so it will be a
stop-the-world change for some hours TODAY:
https://phabricator.wikimedia.org/T159962
** Mystery of the week around geo-location of pageviews for opera mini,
almost there, talking to opera team this week:
https://phabricator.wikimedia.org/T187014
** Continue working on measuring popups on all sites (now feature is
enabled everywhere minus dewiki and enwiki).
** Completed first release of geowiki data (editors per country), available
now in superset internally:
https://superset.wikimedia.org/superset/dashboard/9/
** Issues with mirrormarker and newer versions of kafka producers not
solved, job topics blocked from jumbo cluster
=== Cloud Services ===
* Blocked by:
* Blocking:
* Updates:
** OpenStack upgrades on Friday -- Horizon and CI will be disabled for a
while beginning around 14:00 UTC
** Short blog post published on Q3 FY17/18 highlights for WMCS <
https://phabricator.wikimedia.org/phame/post/view/91/cloud_services_team_q3…
>
** Andrew fixed a privledge escalation security issue with Horizon's puppet
managment screens (T191433)
** Brooke is working on Wiki Replica view changes and needs some
instruction from the DBA team on how to depool servers for maintenance
there.
** Chase is working out Neutron gateway HA configuration in labtestn
** Madhu & Ariel migrated the last user-facing Dumps endpoints to the new
Cloud Services managed servers
=== Fundraising Tech ===
* CentralNotice: deploy CSP headers and violation alert on forced banner
previews
** Need to determine exact header to add
** Curious about the state of the larger CSP work
* Custom data field changes for our CiviCRM instance
* Custom reporting for CiviCRM to help find fraudsters faster
* More API update work for main CC processor, lately focused on the
recurring donation bits
=== MediaWiki Platform ===
* Blocked by:
* Blocking:
* Updates:
** MW 1.31 release planning
*** Pencils down on April 16
*** considering additional bundled extensions (T178349 and subtasks)
*** Making Remex default tidy implementation w/ James_F (T185753)
*** Running phan for extensions that are proposed to be bundled -
https://gerrit.wikimedia.org/r/#/q/status:open+topic:phan
*** Run tests with PHPUnit 6 -
https://gerrit.wikimedia.org/r/#/q/topic:phpunit-6
*** Handle extension dependencies in the installer (for new bundled
extensions, still WIP) - https://gerrit.wikimedia.org/r/#/c/424967/
*** Also planning to do license reviews for new bundled extensions - e.g.
https://gerrit.wikimedia.org/r/#/c/424726/
*** Implement PlatformSettings.php RfC -
https://gerrit.wikimedia.org/r/#/c/423577/
** MCR:
*** ar_rev_id population patch merged!
*** maintenance/populateArchiveRevId.php run on all wikis!
*** Preparation schema change for ar_text dropping is still in progress
*** Working on MCR schema migration script
*** API updates for query modules are in Gerrit:
https://gerrit.wikimedia.org/r/#/c/413223/
*** Started working on the API "templated parameters" described in
https://phabricator.wikimedia.org/T174032#3990488. Not in Gerrit yet.
*** Code review
** Actor table:
*** Further deployment is waiting on the schema change (T188299)
*** 17 extension updates still need review:
https://gerrit.wikimedia.org/r/#/q/status:open+topic:bug/T167246-actor-migr…
** Comment table:
*** Follow up schema change is still in progress
https://gerrit.wikimedia.org/r/#/c/417039/
*** Toolforge view updating patch is merged and pending updates
*** The image_comment_temp merging patches are in Gerrit
*** All except one are blocked on the schema change
** Improving API test coverage:
*** 8 changes outstanding: improved test coverage for ApiMain,
ApiStructureTest, ApiBase, ApiParse, ApiMove, and ApiUserrights; two
improvements to ApiTestCase (getting rid of manual session mangling and
doLogin); and API usage errors in one more extension.
*** This is the last week for Aryeh
** Code review and bug fixes
** Quarterly goal dependency update:
**Goal 3.6 Support work towards unifying MediaWiki's parser
implementations, in liaison with Technology's MediaWiki team
***Parsing depends on MediaWiki Platform, Services
**** MWPT working on relevant planning for Platform Evolution CDP to
prepare for next FY and will coordinate with Parsing as this progresses
**Prepare for launch of the first Structured Data on Commons feature
(multilingual file captions)
***SDC depends on Multimedia/,SRE, WMDE, Search Platform, MediaWiki
Platform, Research
**** MWPT meets weekly with SDC and WMDE
** Objective 1.1 Assist with deploying MultiContent Revisions on Commons
T174022 Implement multi-content revisions, T174023 Implement MCR storage
layer, T174045 DB schema migration for MCR, T174044 Deploy MCR storage
layer,T174043 Deploy Multi-Content Revisions
***SDC depends on MediaWiki Platform, WMDE
**** MWPT meets weekly with SDC and WMDE
=== Performance ===
* Blocked by: None
* Blocking: None
* Updates:
** Aaron/Timo are helping with train delay
**Moving our apps out of puppet, so that we don't need to bug SRE in order
to deploy changes
** Implemented WebP lossless compression support in thumbor on Vagrant for
testing, and it looks good. Gilles will be reachingout to folks on SRE and
Traffic to discuss some of the potential impacts of implementing in prod.
** bunches of code review
=== Release Engineering ===
* Blocked by:
* Blocking:
** scap and git-lfs work is blocking ores deployment. Mukunda is working on
it. Should be unblocked by the end of this week.
* Updates:
** Train is delayed, see task for updates:
https://phabricator.wikimedia.org/T183968
**Quarterly goal dependency update:
***Outcome 1, Objective 4: Continue improving the ways that users can
download articles of interest for later consumption
**** Reading Web depends on SRE, RelEng, Reading Infra
=== Research ===
* Blocked by:
* Blocking:
* Updates:
*Quarterly goal dependency update:
**Prepare for launch of the first Structured Data on Commons feature
(multilingual file captions)
***SDC depends on Multimedia/,SRE, WMDE, Search Platform, MediaWiki
Platform, Research
**Segment 4, Outcome 2: Develop a better understanding of existing needs
for Structured Commons- T171252
***Research depends on Multimedia
=== Scoring Platform ===
* Blocked by:
** Still blocked by git-lfs. This is urgent for us, Adam is happy to work
on it synchronously if this might be helpful for RelEng?
* Blocking:
* Updates:
=== Search Platform ===
* Blocked by:
* Blocking:
* Updates:
**Need help with WDQS GUI build automation:
https://phabricator.wikimedia.org/T189811
**Evaluating feature generation by `query_explorer` functionality of LTR
plugin: https://phabricator.wikimedia.org/T187148
**Working on some job queue issues:
https://phabricator.wikimedia.org/T191024
**Working on query parsing refactoring:
https://phabricator.wikimedia.org/T185108
**Working on Lexeme completion search:
https://phabricator.wikimedia.org/T190970
**Some new docs about running non-wikidata WDQS setups:
https://www.mediawiki.org/wiki/Wikidata_query_service/Implementation/Standa…
*Quarterly goal dependency update:
**Prepare for launch of the first Structured Data on Commons feature
(multilingual file captions)
***SDC depends on Multimedia/,SRE, WMDE, Search Platform, MediaWiki
Platform, Research
**Objective 2.1 Integrate structured file captions into search
*** SDC depends on Search Platform, Multimedia
=== Security ===
* Blocked by:
* Blocking:
* Updates:
**
*Quarterly goal dependency update:
**Goal 3.6: Support work towards unifying MediaWiki's parser
implementations, in liaison with Technology's MediaWiki team
***Parsing depends on MediaWiki Platform, Services
=== Services ===
* Blocked by: Search on CirrusSearchElasticaWrite job serialization T191024
* Blocking: none
* Updates:
** Switched a big bulk of jobs (9) for test wikis to the kafka queue
** restbase-dev cluster work continues, so ignore any alert for it
*Quarterly goal dependency update:
**Increase code sharing of client apps by coalescing and moving more logic
to the server
***Reading Infra/Parsing, Services
**** working with Reading Infra on new CSS endpoints rollout
**Goal 3.6 Support work towards unifying MediaWiki's parser
implementations, in liaison with Technology's MediaWiki team
***Parsing depends on MediaWiki Platform, Services
Outcome 2: Objective 1: Revision storage scaling
***Services depends on SRE, Parsing
-
=== Site Reliability Engineering ===
* Blocked by:
** None
* Blocking:
** None
* Updates:
** ICU migration current ongoing https://phabricator.wikimedia.org/T189295
** mcrouter 0.37 released and packaged already for Debian stretch. Jessie
is a bit more difficult, WIP
** Various countries routed to eqsin (Singapore PoP) once more (Japan,
India, Pakistan, Australia is being experimented with still)
** image scalers no longer received thumb.php traffic
** ping offloading to a dedicated host is happening
https://phabricator.wikimedia.org/T190090. Keep in mind that pings to LVS
IPs from now on will not be replied to by the hosts they used to be replied
from
*Quarterly goal dependency update:
**Outcome 1, Objective 4: Continue improving the ways that users can
download articles of interest for later consumption
*** Reading Web depends on SRE, RelEng, Reading Infra
*** services/SRE will schedule a meeting to discuss this
**Prepare for launch of the first Structured Data on Commons feature
(multilingual file captions)
***SDC depends on Multimedia/,SRE, WMDE, Search Platform, MediaWiki
Platform, Research
Outcome 2: Objective 1: Revision storage scaling
- Services depends on SRE, Parsing
== Wikidata ==
* Blocked by:
* Blocking:
* Updates:
**
*Quarterly goal dependency update:
**Prepare for launch of the first Structured Data on Commons feature
(multilingual file captions)
***SDC depends on Multimedia/,SRE, WMDE, Search Platform, MediaWiki
Platform, Research
** Objective 1.1 Assist with deploying MultiContent Revisions on Commons
T174022 Implement multi-content revisions, T174023 Implement MCR storage
layer, T174045 DB schema migration for MCR, T174044 Deploy MCR storage
layer,T174043 Deploy Multi-Content Revisions
***SDC depends on MediaWiki Platform, WMDE
== German Technical Wishlist ==
* Blocked by:
* Blocking:
* Updates:
** Raising FileImporter test coverage, currently at 48%:
https://phabricator.wikimedia.org/T190290
** Looking into MediaWiki CodeSniffer performance:
https://gerrit.wikimedia.org/r/#/projects/mediawiki/tools/codesniffer,dashb…
** Documenting FileImporter for performance review:
https://phabricator.wikimedia.org/T191283
** Currently deploying new Wikidiff2 to beta:
https://phabricator.wikimedia.org/T190717
** Blogging about the learnings on Wikidiff2:
https://phabricator.wikimedia.org/T190176
== SoS Meeting Bookkeeping ==
* Updates:
**please add links to Phab tickets in the goals update section
If it's gone, that's coincidence. Flagging this to look into, thanks for
the report. Please follow that ticket,
https://phabricator.wikimedia.org/T184258 for more info.
On Tue, Apr 10, 2018 at 5:35 PM, Derk-Jan Hartman <
d.j.hartman+wmf_ml(a)gmail.com> wrote:
> It seems that the pagecounts-ez sets disappeared from
> dumps.wikimedia.org starting this date. Is that a coincidence ?
> Is it https://phabricator.wikimedia.org/T189283 perhaps ?
>
> DJ
>
> On Thu, Mar 29, 2018 at 2:42 PM, Ariel Glenn WMF <ariel(a)wikimedia.org>
> wrote:
> > Here it comes:
> >
> > For the April 1st run and all following runs, the Wikidata dumps of
> > pages-meta-current.bz2 will be produced only as separate downloadable
> > files, no recombined single file will be produced.
> >
> > No other dump jobs will be impacted.
> >
> > A reminder that each of the single downloadable pieces has the siteinfo
> > header and the mediawiki footer so they may all be processed separately
> by
> > whatever tools you use to grab data out of the combined file. If your
> > workflow supports it, they may even be processed in parallel.
> >
> > I am still looking into what the best approach is for the pags-articles
> > dumps.
> >
> > Please forward wherever you deem appropriate. For further updates, don't
> > forget to check the Phab ticket! https://phabricator.wikimedia.
> org/T179059
> >
> > On Mon, Mar 19, 2018 at 2:00 PM, Ariel Glenn WMF <ariel(a)wikimedia.org>
> > wrote:
> >
> >> A reprieve! Code's not ready and I need to do some timing tests, so the
> >> March 20th run will do the standard recombining.
> >>
> >> For updates, don't forget to check the Phab ticket!
> >> https://phabricator.wikimedia.org/T179059
> >>
> >> On Mon, Mar 5, 2018 at 1:10 PM, Ariel Glenn WMF <ariel(a)wikimedia.org>
> >> wrote:
> >>
> >>> Please forward wherever you think appropriate.
> >>>
> >>> For some time we have provided multiple numbered pages-articles bz2
> file
> >>> for large wikis, as well as a single file with all of the contents
> combined
> >>> into one. This is consuming enough time for Wikidata that it is no
> longer
> >>> sustainable. For wikis where the sizes of these files to recombine is
> "too
> >>> large", we will skip this recombine step. This means that downloader
> >>> scripts relying on this file will need to check its existence, and if
> it's
> >>> not there, fall back to downloading the multiple numbered files.
> >>>
> >>> I expect to get this done and deployed by the March 20th dumps run.
> You
> >>> can follow along here: https://phabricator.wikimedia.org/T179059
> >>>
> >>> Thanks!
> >>>
> >>> Ariel
> >>>
> >>
> >>
> > _______________________________________________
> > Wikitech-l mailing list
> > Wikitech-l(a)lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
Please forward wherever you think appropriate.
For some time we have provided multiple numbered pages-articles bz2 file
for large wikis, as well as a single file with all of the contents combined
into one. This is consuming enough time for Wikidata that it is no longer
sustainable. For wikis where the sizes of these files to recombine is "too
large", we will skip this recombine step. This means that downloader
scripts relying on this file will need to check its existence, and if it's
not there, fall back to downloading the multiple numbered files.
I expect to get this done and deployed by the March 20th dumps run. You
can follow along here: https://phabricator.wikimedia.org/T179059
Thanks!
Ariel
Sorry for cross-posting!
Reminder: Technical Advice IRC meeting again **tomorrow, Wednesday 3-4 pm
UTC** on #wikimedia-tech.
The Technical Advice IRC meeting is open for all volunteer developers,
topics and questions. This can be anything from "how to get started" over
"who would be the best contact for X" to specific questions on your project.
If you know already what you would like to discuss or ask, please add your
topic to the next meeting:
https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
Hope to see you there!
Michi (for WMDE’s tech team)
--
Michael F. Schönitzer
Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de
Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/681/51985.
Hello all,
The Code of Conduct Committee has published the list of candidates for
the next 6 months term:
https://www.mediawiki.org/wiki/Code_of_Conduct/Committee/Candidates/2018-I
If nominated, these candidates will be trusted to enforce the code of
conduct for Wikimedia
technical spaces. You can read it at
https://www.mediawiki.org/wiki/Code_of_Conduct.
Any feedback or concern about a candidate can be submitted in private
to techconduct(a)wikimedia.org
for the next two weeks, until Tuesday 2018-04-24.
If there is any need to change the candidates slate following the
community feedback,
the committee will submit a new list, and a new two weeks period will
take place.
--
For the Code of Conduct Committee,
Sébastien Santoro aka Dereckson
https://www.dereckson.be/
Hey,
Scoring platform team aims to support more wikis but keeping track of how
much support they need is not easy. This is why we built a tool that
automatically gets updated and shows us an overview of the current support
and specially it shows progress of labelling campaigns in different wikis
so it's easier for us and the community to see which wiki is about to
finish or which wiki is stalled.
You can find the tool in https://tools.wmflabs.org/ores-support-checklist/
The source code is in http://github.com/wiki-ai/ores-support-checklist.
Pull requests are welcome
To report problems or request new features, feel free to file a phabricator
ticket tagged with ores-support-checklist (
https://phabricator.wikimedia.org/tag/ores-support-checklist/)
Best
--
Amir Sarabadani
Software Engineer
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de
Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/
Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
In 2005–2006 a sysadmin blocked 79 IP addresses on all wikis as being
automatically-detected open proxies, without recording them in the block
log or attributing the block to any user account. These incomplete records
are now causing errors when MediaWiki tries to access them in various
places, see https://phabricator.wikimedia.org/T189840.
Since these are all over 12 years old, it seems reasonably likely that many
of these are no longer open proxies. Rather than trying to fix the
incomplete records, I'm just going to remove them.
Any existing blocks of these IPs that are not causing errors will not be
removed. At first glance this seems relevant mainly to enwiki, where only 5
of the IPs have incomplete records. 21 are currently blocked there with
complete records (19 since 2005 or earlier), and the other 53 are not
currently blocked there.
The list of IPs is at https://phabricator.wikimedia.org/P6876 in case
anyone wants to review them for potential reblocking.
--
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation