Indeed, in December we've rolled out a change that removed the bogus code that previously prevented redirection to canonical URLs. However, search JS uses raw URL-encoding which results in people visiting a lot of non-canonical URLs. I attempted a fix in https://gerrit.wikimedia.org/r/#/c/42766/ - hope to squeeze it in today's deployment. On Tue, Jan 8, 2013 at 4:47 AM, Asher Feldman <afeldman(a)wikimedia.org>wrote;wrote:

On Jan 8, 2013, at 6:57 AM, Diederik van Liere <dvanliere(a)wikimedia.org> wrote: https://mediawiki.org/wiki/Extension:MobileFrontend/Deployments

On 01/08/2013 02:39 PM, Erik Zachte wrote: Wouldn't watch the page and enable email notifications accomplish the same purpose? http://www.mediawiki.org/w/index.php?title=Extension:MobileFrontend/Deploym… -- Quim Gil Technical Contributor Coordinator @ Wikimedia Foundation http://www.mediawiki.org/wiki/User:Qgil

On Tue, Jan 8, 2013 at 2:55 PM, Rob Lanphier <robla(a)wikimedia.org> wrote: I think it's reasonable for engineering teams to QA releases to ensure they don't mangle urls or do anything else that jacks up user roundtrip latency, or needlessly increases apache requests and db queries. Analytics shouldn't be the ones proactively looking for such things. The only time it makes sense for mobile, features, etc. to loop in analytics is when a deployment intentionally results in additional requests that look like pageviews. More likely though, such requests would be relative to the api or bits. Analytics should also stop counting /wiki/article requests with a 301 response as pageviews, as it's pretty much always double counting. They should still be counted in a separate requests metric though, it's still important to see when things like mobile requests double, especially if unexpected.

The example I gave (verified on my iPhone yesterday) was a link to a non-canonical URL from within article HTML mangled by MobileFrontend. Search JS is a problem, but the bigger issue appears to be server side. On Tuesday, January 8, 2013, Max Semenik wrote:

http://stats.wikimedia.org/EN/TablesPageViewsMonthlyOriginalMobile.htm For now I have assumed the problem started 14 Dec and reran page view report extrapolating from first 13 days of month. For most languages figures are now less obviously unbelievable. Of course they might err on the conservative side now, as we miss the extra traffic from Christmas presents. But there is still a +73% MoM for Russian Wikipedia. And again only for mobile. Did we launch the code earlier on ru.wikipedia.org? Erik From: analytics-bounces(a)lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Arthur Richards Sent: Tuesday, January 08, 2013 6:51 PM To: Asher Feldman Cc: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics.; Tomasz Finc; mobile-tech; Max Semenik; Brion Vibber Subject: Re: [Analytics] Suspicious increase in mobile pv's I believe there is some JS that updates links in the UI that uses encodeURIComponent(), which create a non-canonical URL when with things like spaces, colons, and slashes are present. I'm pretty sure we fixed the canonical URL problem server side like Max said, be we'll double check. On Tue, Jan 8, 2013 at 9:34 AM, Asher Feldman <afeldman(a)wikimedia.org> wrote: The example I gave (verified on my iPhone yesterday) was a link to a non-canonical URL from within article HTML mangled by MobileFrontend. Search JS is a problem, but the bigger issue appears to be server side. On Tuesday, January 8, 2013, Max Semenik wrote: Indeed, in December we've rolled out a change that removed the bogus code that previously prevented redirection to canonical URLs. However, search JS uses raw URL-encoding which results in people visiting a lot of non-canonical URLs. I attempted a fix in https://gerrit.wikimedia.org/r/#/c/42766/ - hope to squeeze it in today's deployment. On Tue, Jan 8, 2013 at 4:47 AM, Asher Feldman <afeldman(a)wikimedia.org> wrote: It looks like the December numbers are indeed skewed, possibly due to a link encoding change in mobilefrontend that is doubling many requests. I counted en.m.wikipedia/wiki/article requests and my total count agrees with stats.wikimedia (around 2B), but http 301 redirects went from being around 1% of responses to >18% in one random mid-december day. An example is: cp1044.wikimedia.org 214632678 2013-01-08T00:27:24 0.094004869 0.0.0.0 miss/301 20 GET http://en.m.wikipedia.org/wiki/Pawn%20(chess) - text/html; charset=utf-8 http://en.m.wikipedia.org/wiki/Rules_of_chess - Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%206_0_1%20like%20Mac%20OS%20X)%20 AppleWebKit/536.26%20(KHTML,%20like%20Gecko)%20Version/6.0%20Mobile/10A523%2 0Safari/8536.25 en-us - The correct article name for the above example is http://en.m.wikipedia.org/wiki/Pawn_(chess). The desktop site in safari provides a correct link, but the mobile site converts the underscore to an encoded space. There may be some other issues as well - the number of requests with a blank referrer have increased heavily. This is partially due to ios6 using google over ssl, but the number of blank referrers increased heavily for android as well. My rough estimate is that mobile pageviews actually increased by around 1/3 in december. On Mon, Jan 7, 2013 at 2:00 PM, Diederik van Liere <dvanliere(a)wikimedia.org> wrote: Hi all, It has come to our attention that there is quite a big jump in mobile pageviews on http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm We are looking into the exact cause of it right now, but suffice to say for now, do not use these numbers for the month of December 2012. Best, Diederik _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics -- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687

On Tue, Jan 8, 2013 at 6:48 PM, Max Semenik <msemenik(a)wikimedia.org> wrote: And it was deployed. Do we have enough data since yesterday to figure out how significant was the drop in requests?

301 redirects are occuring on < 1% of en.m pageviews now, looks like Max's fix was successful. On Wed, Jan 9, 2013 at 3:01 PM, Tomasz Finc &lt;tfinc(a)wikimedia.org&gt; wrote: