[Wikitech-l] 2018-01-10 Scrum of Scrums meeting notes

https://www.mediawiki.org/wiki/Scrum_of_scrums/2018-01-10 == Callouts == * Search PF would like somebody from performance/MediaWiki team to look at: https://phabricator.wikimedia.org/T183101we’ve got some fails on LinksUpdate which blocked search index update which makes Wikidata items invisible in search (~50 items) ** [Perf] We've been looking at it, but we haven't identified a cause for this issue yet. ** [SP] I guess we will just watch out for it and hope it was an one-time thing. * [Perf] Trying to compile a list of everything running in Prod that *requires* Zend PHP5. Currently only aware of the XML dump job (has syntax errors in output if run under HHVM). Anything else, please reach out to Ian (email: imarlier@, irc: marlier) * Reminder: Whitelist your EventLogging schema if you want data older than 90 days to be retained, talk to the Analytics team if you're not sure what this means == Audiences == === Readers === ==== iOS native app ==== * Blocked by: none * Blocking: none * Updates: **Continuing work on 5.8 - synced reading lists ==== Android native app ==== * Blocked by: Patch to unblock DELETE endpoints in varnish: https://gerrit.wikimedia.org/r/402433 DONE! * Blocking: none * Updates: ** Completed work to integrate with the new Reading List service! Continuing to test and measure performance. ==== Reading Infrastructure ==== * Blocked by: ops on https://gerrit.wikimedia.org/r/#/c/402433/ <= this is already merged since 2018-01-09 * Blocking: none * Updates: ** working on request batching in reading list service ** continuing work on WP0 piracy ** Summary endpoint switchover from TextExtracts to PCS implementation planned for Wednesday ** wrapping up media and references endpoints ** metadata and CSS endpoints to follow ==== Web ==== * Wrapping up a rework of the mobile web beta/settings page. * Nothing new to report with regards to challenges/deliverables ==== Maps ==== * Blocked by: Services team * Blocking: None * Updates ** Everyone is back from vacation ** Several issues waiting on service team ==== Multimedia ==== * Nothing to report === Contributors === ==== Parsing ==== * Sole update: Approaching additional wikis for replacing Tidy on those wikis on Jan 31st. ==== Global Collaboration ==== * Blocked: Security on https://phabricator.wikimedia.org/T144467 * Blocking: Ops on Flow dumps. Matt's back from sabbatical but has been sick off and on over the last week === Community Tech === * Investigating new year's proposals * Will file for security/performance/DBA review for GlobalPreferences today ==== UI Standardization ==== ** OOUI v0.25.0 released https://phabricator.wikimedia.org/diffusion/GOJU/browse/master/History.md *** 6 breaking and 5 deprecating changes, and 16 UI related improvements, among those **** Drop the `constructive` flag entirely **** Remove `BookletLayout#getClosestPage` Also we significantly reduced SVG file size for our interface elements by up to 21% with fine-tuned markup optimizations. * Ongoing: ** OOUI & based products: *** icons: Unify, refine and align to WikimediaUI Style Guide https://phabricator.wikimedia.org/T177432 – first patches in: https://gerrit.wikimedia.org/r/#/c/402757/ ** More optimizations/size savings for RL embedded SVGs as data URIs == Technology == == Technical Operations == * Blocking: ** None * Blocked: ** Global Collaboration on Flow. https://phabricator.wikimedia.org/T172025 * Updates ** s5 => s8 migration for wikidata completed successfully. Feel free to reach out to DBAs for followup questions and/or details ** meltdown/spectre week, ops busy with the mitigations, do expect some slower response times ** This is a big hardware refresh quarter. Work is already decided and prioritized, but please do provide input when/if requested ASAP. Thanks === Release Engineering === * Blocking ** None? * Blocked ** ops: zuul package update (blocks gerrit upgrade) ** ops: node-tunnel-agent package update (blocks moving node testing to docker in ci) * Updates ** 2 weeks of normal MediaWiki deploys (this and next) followed by 2 weeks of no MediaWiki but SWATs as needed (DevSummit/All Hands followed by RelEng team offsite) ** Currently building nightlies of Mediawiki on the new “releases” (aka non-CI) Jenkins host. Working with Security on best way to handle security patches (which is the goal, to ensure security patches stay cleanly applicable). === Performance === * Blocking: None * Blocked: None * Updates ** As noted above, we've been investigating LinksUpdate failures, haven't been able to identify a cause at this point, haven't been able to reproduce. ** Fixes to Thumbor implementation happening as issues are reported (no major ones at this point) ** Published the first in a series of blog posts detailing how we measure performance: https://phabricator.wikimedia.org/phame/post/view/83/measuring_wikipedia_pa… ** Trying to get a list of everything running in Prod that *requires* PHP5 (Zend) -- only thing that I know of for sure is the XML dump job. === Security === * No updates * Sam and Brian are traveling this week * Darian's last day is January 15th === Fundraising Tech === * Blocking: None * Blocked: None * Updates: ** Upstreaming stuff to Civi ** Making our Amazon Pay SDK fork support TCP proxy ** Re-starting work on new API for our main credit card processor ** Stats projects: *** Andrew Green's druid banner impressions lib: https://github.com/AndrewGreen/centralnotice_analytics *** Jack Gleeson's stats collector: https://github.com/jackgleeson/stats-collector *** More consideration of how the get banner stats to payments in realtime (talking with Analytics) *** Also need to talk with analytics about preserving aggregated banner stats for December === Scoring Platform === * Blocking: no * Blocked: no * Updates ** We added Extension:ORES to in-house code coverage reporting :D *** https://doc.wikimedia.org/cover-extensions/ORES/index.html *** https://scrutinizer-ci.com/g/wikimedia/mediawiki-extensions-ORES/ === Analytics === * https://superset.wikimedia.org You can now create dashboards on top of druid data, no automatic user creation. Created users for PMs and analysts, let us know if you want a user. Perf to be seen. * Clickstream data now public. Research community has access to the paths readers follow through the site (aggreggated counts, not per user) * Eventlogging purging on master box almost done. * Working on wikistats UI bugs and small features to make project selection more friendly, once the UI tweaks are done we will leave this work on backburner until next year. * Work on new set of APIs to return pageviews per country continues. Privacy study of this data: https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews/Pageviews_by_co… * Meeting with Asaf for our next big project: geowiki (editors per country, quite sensitive data) ** Mostly quiet (holidays) === Search Platform === * Blocked by: none * Blocking: none * Updates: * Improving LTR training mechanisms ( https://phabricator.wikimedia.org/T184547) * Working on fixes for completion suggester & redirects namespaces ( https://phabricator.wikimedia.org/T115756) * Investigating ElasticSearch phonetic search ( https://phabricator.wikimedia.org/T182708) * Working on refactoring search profiles to make them more config-like https://phabricator.wikimedia.org/T183279 * Processing feedback for Wikidata fulltext & prefix search rankings * Running test for machine-learning ranking on Hebrew wiki ( https://phabricator.wikimedia.org/T182616) * Discussing publishing data from WDQS query logs ( https://phabricator.wikimedia.org/T143819) == Wikidata == * Gaining speed after the holidays. * Working on Lexeme continues, now working on diffs: https://phabricator.wikimedia.org/T182424 * Working on Quality Constraint integration continues, 1.0 rollout planned for Q1. * Preparing for Developer Summit. * Planning collaboration with Multimedia (MediaInfo) as well as Search for Q1. == MediaWiki Platform== * Blocked by: Wikibase on https://phabricator.wikimedia.org/T183019 * Blocking: N/A * Updates: * Tim: Spectre/Meltdown review * Multi-Content Revisions: ** Actor table: 380669 needs code review ** Working on https://phabricator.wikimedia.org/T36925 and https://phabricator.wikimedia.org/T182678 ** Comment table: ***Schema changes are almost done. Then we can start trying to use it. ** cleanupUsersWithNoId: *** enwiki finished *** Hopefully dewiki and wikidatawiki can be run this week * ExternalStore de-PHP-serialization: Patch done before vacation. But Jaime raised some interesting discussion on https://phabricator.wikimedia.org/T183419 that may determine how we go about updating ExternalStore * Code coverage: ** Added a bunch of @covers tags in core, we also validate them pre-merge now (see giant spike in https://tools.wmflabs.org/coverage/mediawiki/chart.png ) ** Implemented coverage reports for extensions ( https://phabricator.wikimedia.org/T71685, https://doc.wikimedia.org/cover-extensions/ ), still in a beta period, will announce as ready for usage end of this or next week once https://phabricator.wikimedia.org/T182749 is implemented and documented ** Brainstorming/working on proof of concept for pre-merge coverage reports that don't take hours to generate * Audiences Technology Working Group planning * Developer Summit planning == SoS meeting == * ggellerman has ongoing conflicts, looking for another facilitator every other week * Check in on goals here, talk about cross-team dependencies in this meeting, rather than waiting until the end of each quarter.