https://www.mediawiki.org/wiki/Scrum_of_scrums/2016-06-01
= 2016-06-01 =
== Product ==
=== Reading ===
==== Reading Infrastructure ====
* AuthManager coming soon - it's on the beta cluster now! Please indicate if you hae a serious blocker now at https://phabricator.wikimedia.org/T135504
==== Web ==== * Starting Hovercards A/B test during this sprint (with 1% in HU.wiki) * Rolling lazy loaded images / references to medium sized wikis * Blocked (Ops/Analytics) - https://phabricator.wikimedia.org/T127883
==== iOS ====
==== Android ==== * Production release 2.2.146 rollout out this Tuesday. https://lists.wikimedia.org/pipermail/mobile-l/2016-May/010243.html (New features include: reading lists, edit here, Show redirect source in search results, updates for AuthManager, No longer download and widen high-quality images when on a metered connection.) * Working on new feed UI
==== Mobile Content Service ==== * Adding first couple of new feed endpoints this week (enwiki featured article, most read articles, aggregated endpoint for previous and future feed microservices)
=== Community Tech === * We're going to be testing the PageAssessments extension on Labs this week (https://www.mediawiki.org/wiki/Extension:PageAssessments) * Continuing work on new CopyPatrol Tool ( http://tools.wmflabs.org/plagiabot) * More investigation on Category collation * No blockers
=== Editing ===
==== Language ==== * Blocked: Parallel Corpora dumps: https://phabricator.wikimedia.org/T127793 (Tech Ops)
==== Multimedia ==== * Blocked: New repository request for FileAnnotations so we can experiment and do design/code review on Mark's crappy prototype * Updates ** ImageTweaks waiting on Thumbor and a few other things, no news there AFAIK ** Gallery slideshow core work coming along, no huge updates from prtksxna ** UW fatals rate waaaay down since last year, thanks Sentry people for helping us track it
==== Parsing ==== * Tidy replacement: With latest HTML5Depurate and visual diff test run, ~93.4% of test pages render with pixel perfect accuracy. Working through remaining diff scenarios (a lot of the diffs are harmless minor pixel shifts) to identify real issues and having wikitext be fixed or fix core parser as required. ** Tim working on a HTML5 parser to fix some core parser issues around doBlockLevels. * Will attempt switching Parsoid to service-runner sometime next week (Arlo & Services) * Arlo close to getting html2html endpoint done (last remaining blocker to enable Parsoid new HTML version with separate data-mw) ** VE & CX: Please start planning how to handle this new version. * Kunal working on using the new Linker code for files/images.
==== Collaboration ==== * Blocked - None * Blocking - No change * Updates ** Working on Echo special page, for both JS and no-JS users ** Schema changes we requested are done, we now need to backfill and use the new schema
== Technology ==
=== Services === * RESTBase ** enforcing rate limits as of today *** pageview: 10 req/s *** transforms: 5 req/s * Change Propagation ** now handles summary and MCS updates as well as purges ** partially moved RestbaseUpdateJobs from job runners to it *** still missing transclusions * MathML enabled by default across all projects ** proper caching and purging still a TODO *** RESTBase side for purging proposed in https://github.com/wikimedia/restbase/pull/628 *** need Ops help for the Varnish side of things - https://phabricator.wikimedia.org/T136205#2343850
=== Technical Operations === * '''Blocking''': none * '''Blocked''': none * Updates: ** libicu update successful ** Started preliminary work on migrating appservers to jessie ** Last day of Riccardo ** Getting graphite into better shape
=== Analytics === * Loaded 3 months of pageview data into Druid and querying it is very fast * Cleaning up limn-flow-data, limn-edit-data, and limn-languge-data a little bit, deploying today * Working on processing data from the mediawiki databases and turning it into an analytics-friendly schema (one example is slowly changing dimensions such as page_title recorded as (page_id, page_title, valid_from, valid_to)) * pageview requests that miss the cache are rate-limited to 10 req/s. More req/s than that will throw 429
=== Security === * Two-factor usability surveying prep continues * Reviews: Ex:DoubleWiki, Ex:TemplateStyles, Use of wheels for deployment by Analytics
=== Discovery === * '''Blocking''': none * '''Blocked''': none * Tomasz last day at WMF was on Friday, Katie Horn leading Discovery now * ElasticSearch 2.3 upgrade in progress: https://phabricator.wikimedia.org/T133124 * TextCat A/B test for english wiki started * Updated Portal language stats.
=== Fundraising Tech === * Fixed CentralNotice tests, now enforcing ES3 * More work to get off ActiveMQ, remove SPOF * Testing new processor in Israel, Japan and Ukraine ** We quit messing with language fallback and just got the last few missing messages translated to Ukrainian * Ready to test new PayPal integration method * Enhancing fraud & dos mitigation measures
=== Release Engineering === * '''Blocking''': None * '''Blocked''': Nada * '''Update:''' ** Auth Manager rolling out *soon*
Hi Grace,
I'd like to request a clarification about RESTBase.
These notes say: "* RESTBase ** enforcing rate limits as of today *** pageview: 10 req/s *** transforms: 5 req/s"
Can you expand on what kinds of requests these rate limits are limiting?
Thanks, Pine On Jun 1, 2016 11:06, "Grace Gellerman" ggellerman@wikimedia.org wrote:
https://www.mediawiki.org/wiki/Scrum_of_scrums/2016-06-01
= 2016-06-01 =
== Product ==
=== Reading ===
==== Reading Infrastructure ====
- AuthManager coming soon - it's on the beta cluster now! Please indicate
if you hae a serious blocker now at https://phabricator.wikimedia.org/T135504
==== Web ====
- Starting Hovercards A/B test during this sprint (with 1% in HU.wiki)
- Rolling lazy loaded images / references to medium sized wikis
- Blocked (Ops/Analytics) - https://phabricator.wikimedia.org/T127883
==== iOS ====
==== Android ====
- Production release 2.2.146 rollout out this Tuesday.
https://lists.wikimedia.org/pipermail/mobile-l/2016-May/010243.html (New features include: reading lists, edit here, Show redirect source in search results, updates for AuthManager, No longer download and widen high-quality images when on a metered connection.)
- Working on new feed UI
==== Mobile Content Service ====
- Adding first couple of new feed endpoints this week (enwiki featured
article, most read articles, aggregated endpoint for previous and future feed microservices)
=== Community Tech ===
- We're going to be testing the PageAssessments extension on Labs this week
(https://www.mediawiki.org/wiki/Extension:PageAssessments)
- Continuing work on new CopyPatrol Tool (
http://tools.wmflabs.org/plagiabot)
- More investigation on Category collation
- No blockers
=== Editing ===
==== Language ====
- Blocked: Parallel Corpora dumps:
https://phabricator.wikimedia.org/T127793 (Tech Ops)
==== Multimedia ====
- Blocked: New repository request for FileAnnotations so we can experiment
and do design/code review on Mark's crappy prototype
- Updates
** ImageTweaks waiting on Thumbor and a few other things, no news there AFAIK ** Gallery slideshow core work coming along, no huge updates from prtksxna ** UW fatals rate waaaay down since last year, thanks Sentry people for helping us track it
==== Parsing ====
- Tidy replacement: With latest HTML5Depurate and visual diff test run,
~93.4% of test pages render with pixel perfect accuracy. Working through remaining diff scenarios (a lot of the diffs are harmless minor pixel shifts) to identify real issues and having wikitext be fixed or fix core parser as required. ** Tim working on a HTML5 parser to fix some core parser issues around doBlockLevels.
- Will attempt switching Parsoid to service-runner sometime next week (Arlo
& Services)
- Arlo close to getting html2html endpoint done (last remaining blocker to
enable Parsoid new HTML version with separate data-mw) ** VE & CX: Please start planning how to handle this new version.
- Kunal working on using the new Linker code for files/images.
==== Collaboration ====
- Blocked - None
- Blocking - No change
- Updates
** Working on Echo special page, for both JS and no-JS users ** Schema changes we requested are done, we now need to backfill and use the new schema
== Technology ==
=== Services ===
- RESTBase
** enforcing rate limits as of today *** pageview: 10 req/s *** transforms: 5 req/s
- Change Propagation
** now handles summary and MCS updates as well as purges ** partially moved RestbaseUpdateJobs from job runners to it *** still missing transclusions
- MathML enabled by default across all projects
** proper caching and purging still a TODO *** RESTBase side for purging proposed in https://github.com/wikimedia/restbase/pull/628 *** need Ops help for the Varnish side of things - https://phabricator.wikimedia.org/T136205#2343850
=== Technical Operations ===
- '''Blocking''': none
- '''Blocked''': none
- Updates:
** libicu update successful ** Started preliminary work on migrating appservers to jessie ** Last day of Riccardo ** Getting graphite into better shape
=== Analytics ===
- Loaded 3 months of pageview data into Druid and querying it is very fast
- Cleaning up limn-flow-data, limn-edit-data, and limn-languge-data a
little bit, deploying today
- Working on processing data from the mediawiki databases and turning it
into an analytics-friendly schema (one example is slowly changing dimensions such as page_title recorded as (page_id, page_title, valid_from, valid_to))
- pageview requests that miss the cache are rate-limited to 10 req/s. More
req/s than that will throw 429
=== Security ===
- Two-factor usability surveying prep continues
- Reviews: Ex:DoubleWiki, Ex:TemplateStyles, Use of wheels for deployment
by Analytics
=== Discovery ===
- '''Blocking''': none
- '''Blocked''': none
- Tomasz last day at WMF was on Friday, Katie Horn leading Discovery now
- ElasticSearch 2.3 upgrade in progress:
https://phabricator.wikimedia.org/T133124
- TextCat A/B test for english wiki started
- Updated Portal language stats.
=== Fundraising Tech ===
- Fixed CentralNotice tests, now enforcing ES3
- More work to get off ActiveMQ, remove SPOF
- Testing new processor in Israel, Japan and Ukraine
** We quit messing with language fallback and just got the last few missing messages translated to Ukrainian
- Ready to test new PayPal integration method
- Enhancing fraud & dos mitigation measures
=== Release Engineering ===
- '''Blocking''': None
- '''Blocked''': Nada
- '''Update:'''
** Auth Manager rolling out *soon* _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 3 June 2016 at 00:14, Pine W wiki.pine@gmail.com wrote:
I'd like to request a clarification about RESTBase.
These notes say: "* RESTBase ** enforcing rate limits as of today *** pageview: 10 req/s *** transforms: 5 req/s"
Can you expand on what kinds of requests these rate limits are limiting?
The first one is to the page content requests, documented at https://en.wikipedia.org/api/rest_v1/?doc#/Page_content.
The second one, similarly, is to content transformation requests, shown at https://en.wikipedia.org/api/rest_v1/?doc#/Transforms.
As always, our first mission is to keep the sites up and running well; as with other API access points, this means dealing with abusive requests where we see them, and (as in this case) limiting access to a reasonable level to avoid overloading the system. Unless you're running a tool against this API at a very high speed, ignoring the Terms of Use, it won't affect you.
J.
On Fri, Jun 3, 2016 at 10:21 AM, James Forrester jforrester@wikimedia.org wrote:
On 3 June 2016 at 00:14, Pine W wiki.pine@gmail.com wrote:
I'd like to request a clarification about RESTBase.
These notes say: "* RESTBase ** enforcing rate limits as of today *** pageview: 10 req/s *** transforms: 5 req/s"
Can you expand on what kinds of requests these rate limits are limiting?
Small correction: The "pageview" reference here is to the page view API end points:
- https://wikimedia.org/api/rest_v1/?doc#!/Pageviews_data/get_metrics_pageview... - https://wikimedia.org/api/rest_v1/?doc#!/Pageviews_data/get_metrics_pageview... - https://wikimedia.org/api/rest_v1/?doc#!/Pageviews_data/get_metrics_pageview...
See https://phabricator.wikimedia.org/T135240 for background on these limits on the pageview API. tl;dr: The backend for pageview data has limited traffic capacity right now, but the analytics team is working on a hardware upgrade. In the meantime, we need to make sure that the limited capacity is used fairly, by enforcing conservative per-client limits.
Unless you're running a tool against this API at a very high speed, ignoring the Terms of Use, it won't affect you.
Indeed. For the (relatively expensive) transform end points, we picked the limits so that none of the current users are actually affected by them.
Gabriel
wikitech-l@lists.wikimedia.org