Hi all,
It has come to our attention that there is quite a big jump in mobile pageviews on http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We are looking into the exact cause of it right now, but suffice to say for now, do not use these numbers for the month of December 2012.
Best, Diederik
It looks like the December numbers are indeed skewed, possibly due to a link encoding change in mobilefrontend that is doubling many requests.
I counted en.m.wikipedia/wiki/article requests and my total count agrees with stats.wikimedia (around 2B), but http 301 redirects went from being around 1% of responses to >18% in one random mid-december day.
An example is:
cp1044.wikimedia.org 214632678 2013-01-08T00:27:24 0.094004869 0.0.0.0 miss/301 20 GET http://en.m.wikipedia.org/wiki/Pawn%20(chess) - text/html; charset=utf-8 http://en.m.wikipedia.org/wiki/Rules_of_chess - Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%206_0_1%20like%20Mac%20OS%20X)%20AppleWebKit/536.26%20(KHTML,%20like%20Gecko)%20Version/6.0%20Mobile/10A523%20Safari/8536.25 en-us -
The correct article name for the above example is http://en.m.wikipedia.org/wiki/Pawn_(chess). The desktop site in safari provides a correct link, but the mobile site converts the underscore to an encoded space.
There may be some other issues as well - the number of requests with a blank referrer have increased heavily. This is partially due to ios6 using google over ssl, but the number of blank referrers increased heavily for android as well. My rough estimate is that mobile pageviews actually increased by around 1/3 in december.
On Mon, Jan 7, 2013 at 2:00 PM, Diederik van Liere dvanliere@wikimedia.orgwrote:
Hi all,
It has come to our attention that there is quite a big jump in mobile pageviews on http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We are looking into the exact cause of it right now, but suffice to say for now, do not use these numbers for the month of December 2012.
Best, Diederik
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
CC'ing mobile team to find out what caused this
--tomasz
On Mon, Jan 7, 2013 at 4:47 PM, Asher Feldman afeldman@wikimedia.orgwrote:
It looks like the December numbers are indeed skewed, possibly due to a link encoding change in mobilefrontend that is doubling many requests.
I counted en.m.wikipedia/wiki/article requests and my total count agrees with stats.wikimedia (around 2B), but http 301 redirects went from being around 1% of responses to >18% in one random mid-december day.
An example is:
cp1044.wikimedia.org 214632678 2013-01-08T00:27:24 0.094004869 0.0.0.0 miss/301 20 GET http://en.m.wikipedia.org/wiki/Pawn%20(chess) - text/html; charset=utf-8 http://en.m.wikipedia.org/wiki/Rules_of_chess - Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%206_0_1%20like%20Mac%20OS%20X)%20AppleWebKit/536.26%20(KHTML,%20like%20Gecko)%20Version/6.0%20Mobile/10A523%20Safari/8536.25 en-us -
The correct article name for the above example is http://en.m.wikipedia.org/wiki/Pawn_(chess). The desktop site in safari provides a correct link, but the mobile site converts the underscore to an encoded space.
There may be some other issues as well - the number of requests with a blank referrer have increased heavily. This is partially due to ios6 using google over ssl, but the number of blank referrers increased heavily for android as well. My rough estimate is that mobile pageviews actually increased by around 1/3 in december.
On Mon, Jan 7, 2013 at 2:00 PM, Diederik van Liere < dvanliere@wikimedia.org> wrote:
Hi all,
It has come to our attention that there is quite a big jump in mobile pageviews on http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We are looking into the exact cause of it right now, but suffice to say for now, do not use these numbers for the month of December 2012.
Best, Diederik
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Indeed, in December we've rolled out a change that removed the bogus code that previously prevented redirection to canonical URLs. However, search JS uses raw URL-encoding which results in people visiting a lot of non-canonical URLs. I attempted a fix in https://gerrit.wikimedia.org/r/#/c/42766/ - hope to squeeze it in today's deployment.
On Tue, Jan 8, 2013 at 4:47 AM, Asher Feldman afeldman@wikimedia.orgwrote:
It looks like the December numbers are indeed skewed, possibly due to a link encoding change in mobilefrontend that is doubling many requests.
I counted en.m.wikipedia/wiki/article requests and my total count agrees with stats.wikimedia (around 2B), but http 301 redirects went from being around 1% of responses to >18% in one random mid-december day.
An example is:
cp1044.wikimedia.org 214632678 2013-01-08T00:27:24 0.094004869 0.0.0.0 miss/301 20 GET http://en.m.wikipedia.org/wiki/Pawn%20(chess) - text/html; charset=utf-8 http://en.m.wikipedia.org/wiki/Rules_of_chess - Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%206_0_1%20like%20Mac%20OS%20X)%20AppleWebKit/536.26%20(KHTML,%20like%20Gecko)%20Version/6.0%20Mobile/10A523%20Safari/8536.25 en-us -
The correct article name for the above example is http://en.m.wikipedia.org/wiki/Pawn_(chess). The desktop site in safari provides a correct link, but the mobile site converts the underscore to an encoded space.
There may be some other issues as well - the number of requests with a blank referrer have increased heavily. This is partially due to ios6 using google over ssl, but the number of blank referrers increased heavily for android as well. My rough estimate is that mobile pageviews actually increased by around 1/3 in december.
On Mon, Jan 7, 2013 at 2:00 PM, Diederik van Liere < dvanliere@wikimedia.org> wrote:
Hi all,
It has come to our attention that there is quite a big jump in mobile pageviews on http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We are looking into the exact cause of it right now, but suffice to say for now, do not use these numbers for the month of December 2012.
Best, Diederik
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
So what would be really useful is to have a canonical place where mobile deployments are logged with some notes if there are changes to how links / URL's are handled / processed. It can be as simple as a wiki, and maybe a quick email to the Analytics list to notify us as well.
D
On Tue, Jan 8, 2013 at 9:48 AM, Max Semenik msemenik@wikimedia.org wrote:
Indeed, in December we've rolled out a change that removed the bogus code that previously prevented redirection to canonical URLs. However, search JS uses raw URL-encoding which results in people visiting a lot of non-canonical URLs. I attempted a fix in https://gerrit.wikimedia.org/r/#/c/42766/ - hope to squeeze it in today's deployment.
On Tue, Jan 8, 2013 at 4:47 AM, Asher Feldman afeldman@wikimedia.orgwrote:
It looks like the December numbers are indeed skewed, possibly due to a link encoding change in mobilefrontend that is doubling many requests.
I counted en.m.wikipedia/wiki/article requests and my total count agrees with stats.wikimedia (around 2B), but http 301 redirects went from being around 1% of responses to >18% in one random mid-december day.
An example is:
cp1044.wikimedia.org 214632678 2013-01-08T00:27:24 0.094004869 0.0.0.0 miss/301 20 GET http://en.m.wikipedia.org/wiki/Pawn%20(chess) - text/html; charset=utf-8 http://en.m.wikipedia.org/wiki/Rules_of_chess - Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%206_0_1%20like%20Mac%20OS%20X)%20AppleWebKit/536.26%20(KHTML,%20like%20Gecko)%20Version/6.0%20Mobile/10A523%20Safari/8536.25 en-us -
The correct article name for the above example is http://en.m.wikipedia.org/wiki/Pawn_(chess). The desktop site in safari provides a correct link, but the mobile site converts the underscore to an encoded space.
There may be some other issues as well - the number of requests with a blank referrer have increased heavily. This is partially due to ios6 using google over ssl, but the number of blank referrers increased heavily for android as well. My rough estimate is that mobile pageviews actually increased by around 1/3 in december.
On Mon, Jan 7, 2013 at 2:00 PM, Diederik van Liere < dvanliere@wikimedia.org> wrote:
Hi all,
It has come to our attention that there is quite a big jump in mobile pageviews on http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We are looking into the exact cause of it right now, but suffice to say for now, do not use these numbers for the month of December 2012.
Best, Diederik
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On Jan 8, 2013, at 6:57 AM, Diederik van Liere dvanliere@wikimedia.org wrote:
So what would be really useful is to have a canonical place where mobile deployments are logged with some notes if there are changes to how links / URL's are handled / processed. It can be as simple as a wiki, and maybe a quick email to the Analytics list to notify us as well.
https://mediawiki.org/wiki/Extension:MobileFrontend/Deployments
D
On Tue, Jan 8, 2013 at 9:48 AM, Max Semenik msemenik@wikimedia.org wrote: Indeed, in December we've rolled out a change that removed the bogus code that previously prevented redirection to canonical URLs. However, search JS uses raw URL-encoding which results in people visiting a lot of non-canonical URLs. I attempted a fix in https://gerrit.wikimedia.org/r/#/c/42766/ - hope to squeeze it in today's deployment.
On Tue, Jan 8, 2013 at 4:47 AM, Asher Feldman afeldman@wikimedia.org wrote: It looks like the December numbers are indeed skewed, possibly due to a link encoding change in mobilefrontend that is doubling many requests.
I counted en.m.wikipedia/wiki/article requests and my total count agrees with stats.wikimedia (around 2B), but http 301 redirects went from being around 1% of responses to >18% in one random mid-december day.
An example is:
cp1044.wikimedia.org 214632678 2013-01-08T00:27:24 0.094004869 0.0.0.0 miss/301 20 GET http://en.m.wikipedia.org/wiki/Pawn%20(chess) - text/html; charset=utf-8 http://en.m.wikipedia.org/wiki/Rules_of_chess - Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%206_0_1%20like%20Mac%20OS%20X)%20AppleWebKit/536.26%20(KHTML,%20like%20Gecko)%20Version/6.0%20Mobile/10A523%20Safari/8536.25 en-us -
The correct article name for the above example is http://en.m.wikipedia.org/wiki/Pawn_(chess). The desktop site in safari provides a correct link, but the mobile site converts the underscore to an encoded space.
There may be some other issues as well - the number of requests with a blank referrer have increased heavily. This is partially due to ios6 using google over ssl, but the number of blank referrers increased heavily for android as well. My rough estimate is that mobile pageviews actually increased by around 1/3 in december.
On Mon, Jan 7, 2013 at 2:00 PM, Diederik van Liere dvanliere@wikimedia.org wrote: Hi all,
It has come to our attention that there is quite a big jump in mobile pageviews on http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We are looking into the exact cause of it right now, but suffice to say for now, do not use these numbers for the month of December 2012.
Best, Diederik
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
--
Good to know that page exists. But is it rather impractical for us to visit that page daily and scan for changes that might influence how stats are gathered.
So please if you expect stats will be affected, drop us a line. Thanks so much.
Erik
From: analytics-bounces@lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Maryana Pinchuk Sent: Tuesday, January 08, 2013 5:41 PM To: Diederik van Liere Cc: mobile-tech; Tomasz Finc; Brion Vibber; Arthur Richards; A mailing list for the Analytics Team at WMF and everybody who has aninterest in Wikipedia and analytics. Subject: Re: [Analytics] Suspicious increase in mobile pv's
On Jan 8, 2013, at 6:57 AM, Diederik van Liere dvanliere@wikimedia.org wrote:
So what would be really useful is to have a canonical place where mobile deployments are logged with some notes if there are changes to how links / URL's are handled / processed. It can be as simple as a wiki, and maybe a quick email to the Analytics list to notify us as well.
https://mediawiki.org/wiki/Extension:MobileFrontend/Deployments
D
On Tue, Jan 8, 2013 at 9:48 AM, Max Semenik msemenik@wikimedia.org wrote:
Indeed, in December we've rolled out a change that removed the bogus code that previously prevented redirection to canonical URLs. However, search JS uses raw URL-encoding which results in people visiting a lot of non-canonical URLs. I attempted a fix in https://gerrit.wikimedia.org/r/#/c/42766/ - hope to squeeze it in today's deployment.
On Tue, Jan 8, 2013 at 4:47 AM, Asher Feldman afeldman@wikimedia.org wrote:
It looks like the December numbers are indeed skewed, possibly due to a link encoding change in mobilefrontend that is doubling many requests.
I counted en.m.wikipedia/wiki/article requests and my total count agrees with stats.wikimedia (around 2B), but http 301 redirects went from being around 1% of responses to >18% in one random mid-december day.
An example is:
cp1044.wikimedia.org 214632678 2013-01-08T00:27:24 0.094004869 0.0.0.0 miss/301 20 GET http://en.m.wikipedia.org/wiki/Pawn%20(chess) - text/html; charset=utf-8 http://en.m.wikipedia.org/wiki/Rules_of_chess - Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%206_0_1%20like%20Mac%20OS%20X)%20AppleWebKit/536.26%20(KHTML,%20like%20Gecko)%20Version/6.0%20Mobile/10A523%20Safari/8536.25 en-us -
The correct article name for the above example is http://en.m.wikipedia.org/wiki/Pawn_(chess). The desktop site in safari provides a correct link, but the mobile site converts the underscore to an encoded space.
There may be some other issues as well - the number of requests with a blank referrer have increased heavily. This is partially due to ios6 using google over ssl, but the number of blank referrers increased heavily for android as well. My rough estimate is that mobile pageviews actually increased by around 1/3 in december.
On Mon, Jan 7, 2013 at 2:00 PM, Diederik van Liere dvanliere@wikimedia.org wrote:
Hi all,
It has come to our attention that there is quite a big jump in mobile pageviews on http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We are looking into the exact cause of it right now, but suffice to say for now, do not use these numbers for the month of December 2012.
Best,
Diederik
_______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
_______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On 01/08/2013 02:39 PM, Erik Zachte wrote:
Good to know that page exists. But is it rather impractical for us to visit that page daily and scan for changes that might influence how stats are gathered.
So please if you expect stats will be affected, drop us a line. Thanks so much.
Wouldn't watch the page and enable email notifications accomplish the same purpose?
http://www.mediawiki.org/w/index.php?title=Extension:MobileFrontend/Deployme...
On Tue, Jan 8, 2013 at 2:44 PM, Quim Gil qgil@wikimedia.org wrote:
On 01/08/2013 02:39 PM, Erik Zachte wrote:
So please if you expect stats will be affected, drop us a line. Thanks so much.
Wouldn't watch the page and enable email notifications accomplish the same purpose?
http://www.mediawiki.org/w/index.php?title=Extension:MobileFrontend/Deployme...
The problem is that 99% of MobileFrontend changes should have no effect on logging (either that, or you're doing it wrong). What Erik is asking for (which I believe is reasonable) is for developers to be aware of how the changes they make might affect pageview statistics, and work with him and others when that's likely to happen.
Also in the future, whenever we have some area where the numbers are off, we should collectively ask "was there something in the thing that we're monitoring that changed?" It seems in this case that there was a lot of focus on the analytics infrastructure being the sole problem, when in fact it was an interaction between our infrastructure and the MobileFrontend code.
Rob
On Tue, Jan 8, 2013 at 2:55 PM, Rob Lanphier robla@wikimedia.org wrote:
On Tue, Jan 8, 2013 at 2:44 PM, Quim Gil qgil@wikimedia.org wrote:
On 01/08/2013 02:39 PM, Erik Zachte wrote:
So please if you expect stats will be affected, drop us a line. Thanks so much.
Wouldn't watch the page and enable email notifications accomplish the
same
purpose?
http://www.mediawiki.org/w/index.php?title=Extension:MobileFrontend/Deployme...
The problem is that 99% of MobileFrontend changes should have no effect on logging (either that, or you're doing it wrong). What Erik is asking for (which I believe is reasonable) is for developers to be aware of how the changes they make might affect pageview statistics, and work with him and others when that's likely to happen.
I think it's reasonable for engineering teams to QA releases to ensure they don't mangle urls or do anything else that jacks up user roundtrip latency, or needlessly increases apache requests and db queries. Analytics shouldn't be the ones proactively looking for such things. The only time it makes sense for mobile, features, etc. to loop in analytics is when a deployment intentionally results in additional requests that look like pageviews. More likely though, such requests would be relative to the api or bits.
Analytics should also stop counting /wiki/article requests with a 301 response as pageviews, as it's pretty much always double counting. They should still be counted in a separate requests metric though, it's still important to see when things like mobile requests double, especially if unexpected.
On Wed, Jan 9, 2013 at 2:39 AM, Erik Zachte ezachte@wikimedia.org wrote:
Good to know that page exists. But is it rather impractical for us to visit that page daily and scan for changes that might influence how stats are gathered.****
So please if you expect stats will be affected, drop us a line. Thanks so much.
If we only knew it would affect stats... :)
The example I gave (verified on my iPhone yesterday) was a link to a non-canonical URL from within article HTML mangled by MobileFrontend. Search JS is a problem, but the bigger issue appears to be server side.
On Tuesday, January 8, 2013, Max Semenik wrote:
Indeed, in December we've rolled out a change that removed the bogus code that previously prevented redirection to canonical URLs. However, search JS uses raw URL-encoding which results in people visiting a lot of non-canonical URLs. I attempted a fix in https://gerrit.wikimedia.org/r/#/c/42766/ - hope to squeeze it in today's deployment.
On Tue, Jan 8, 2013 at 4:47 AM, Asher Feldman <afeldman@wikimedia.org<javascript:_e({}, 'cvml', 'afeldman@wikimedia.org');>
wrote:
It looks like the December numbers are indeed skewed, possibly due to a link encoding change in mobilefrontend that is doubling many requests.
I counted en.m.wikipedia/wiki/article requests and my total count agrees with stats.wikimedia (around 2B), but http 301 redirects went from being around 1% of responses to >18% in one random mid-december day.
An example is:
cp1044.wikimedia.org 214632678 2013-01-08T00:27:24 0.094004869 0.0.0.0 miss/301 20 GET http://en.m.wikipedia.org/wiki/Pawn%20(chess) - text/html; charset=utf-8 http://en.m.wikipedia.org/wiki/Rules_of_chess - Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%206_0_1%20like%20Mac%20OS%20X)%20AppleWebKit/536.26%20(KHTML,%20like%20Gecko)%20Version/6.0%20Mobile/10A523%20Safari/8536.25 en-us -
The correct article name for the above example is http://en.m.wikipedia.org/wiki/Pawn_(chess). The desktop site in safari provides a correct link, but the mobile site converts the underscore to an encoded space.
There may be some other issues as well - the number of requests with a blank referrer have increased heavily. This is partially due to ios6 using google over ssl, but the number of blank referrers increased heavily for android as well. My rough estimate is that mobile pageviews actually increased by around 1/3 in december.
On Mon, Jan 7, 2013 at 2:00 PM, Diederik van Liere < dvanliere@wikimedia.org <javascript:_e({}, 'cvml', 'dvanliere@wikimedia.org');>> wrote:
Hi all,
It has come to our attention that there is quite a big jump in mobile pageviews on http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We are looking into the exact cause of it right now, but suffice to say for now, do not use these numbers for the month of December 2012.
Best, Diederik
Analytics mailing list Analytics@lists.wikimedia.org <javascript:_e({}, 'cvml', 'Analytics@lists.wikimedia.org');> https://lists.wikimedia.org/mailman/listinfo/analytics
I believe there is some JS that updates links in the UI that uses encodeURIComponent(), which create a non-canonical URL when with things like spaces, colons, and slashes are present. I'm pretty sure we fixed the canonical URL problem server side like Max said, be we'll double check.
On Tue, Jan 8, 2013 at 9:34 AM, Asher Feldman afeldman@wikimedia.orgwrote:
The example I gave (verified on my iPhone yesterday) was a link to a non-canonical URL from within article HTML mangled by MobileFrontend. Search JS is a problem, but the bigger issue appears to be server side.
On Tuesday, January 8, 2013, Max Semenik wrote:
Indeed, in December we've rolled out a change that removed the bogus code that previously prevented redirection to canonical URLs. However, search JS uses raw URL-encoding which results in people visiting a lot of non-canonical URLs. I attempted a fix in https://gerrit.wikimedia.org/r/#/c/42766/ - hope to squeeze it in today's deployment.
On Tue, Jan 8, 2013 at 4:47 AM, Asher Feldman afeldman@wikimedia.orgwrote:
It looks like the December numbers are indeed skewed, possibly due to a link encoding change in mobilefrontend that is doubling many requests.
I counted en.m.wikipedia/wiki/article requests and my total count agrees with stats.wikimedia (around 2B), but http 301 redirects went from being around 1% of responses to >18% in one random mid-december day.
An example is:
cp1044.wikimedia.org 214632678 2013-01-08T00:27:24 0.094004869 0.0.0.0 miss/301 20 GET http://en.m.wikipedia.org/wiki/Pawn%20(chess) - text/html; charset=utf-8 http://en.m.wikipedia.org/wiki/Rules_of_chess- Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%206_0_1%20like%20Mac%20OS%20X)%20AppleWebKit/536.26%20(KHTML,%20like%20Gecko)%20Version/6.0%20Mobile/10A523%20Safari/8536.25 en-us -
The correct article name for the above example is http://en.m.wikipedia.org/wiki/Pawn_(chess). The desktop site in safari provides a correct link, but the mobile site converts the underscore to an encoded space.
There may be some other issues as well - the number of requests with a blank referrer have increased heavily. This is partially due to ios6 using google over ssl, but the number of blank referrers increased heavily for android as well. My rough estimate is that mobile pageviews actually increased by around 1/3 in december.
On Mon, Jan 7, 2013 at 2:00 PM, Diederik van Liere < dvanliere@wikimedia.org> wrote:
Hi all,
It has come to our attention that there is quite a big jump in mobile pageviews on http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We are looking into the exact cause of it right now, but suffice to say for now, do not use these numbers for the month of December 2012.
Best, Diederik
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
http://stats.wikimedia.org/EN/TablesPageViewsMonthlyOriginalMobile.htm
For now I have assumed the problem started 14 Dec and reran page view report extrapolating from first 13 days of month.
For most languages figures are now less obviously unbelievable. Of course they might err on the conservative side now, as we miss the extra traffic from Christmas presents.
But there is still a +73% MoM for Russian Wikipedia. And again only for mobile. Did we launch the code earlier on ru.wikipedia.org?
Erik
From: analytics-bounces@lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Arthur Richards Sent: Tuesday, January 08, 2013 6:51 PM To: Asher Feldman Cc: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics.; Tomasz Finc; mobile-tech; Max Semenik; Brion Vibber Subject: Re: [Analytics] Suspicious increase in mobile pv's
I believe there is some JS that updates links in the UI that uses encodeURIComponent(), which create a non-canonical URL when with things like spaces, colons, and slashes are present. I'm pretty sure we fixed the canonical URL problem server side like Max said, be we'll double check.
On Tue, Jan 8, 2013 at 9:34 AM, Asher Feldman afeldman@wikimedia.org wrote:
The example I gave (verified on my iPhone yesterday) was a link to a non-canonical URL from within article HTML mangled by MobileFrontend. Search JS is a problem, but the bigger issue appears to be server side.
On Tuesday, January 8, 2013, Max Semenik wrote:
Indeed, in December we've rolled out a change that removed the bogus code that previously prevented redirection to canonical URLs. However, search JS uses raw URL-encoding which results in people visiting a lot of non-canonical URLs. I attempted a fix in https://gerrit.wikimedia.org/r/#/c/42766/ - hope to squeeze it in today's deployment.
On Tue, Jan 8, 2013 at 4:47 AM, Asher Feldman afeldman@wikimedia.org wrote:
It looks like the December numbers are indeed skewed, possibly due to a link encoding change in mobilefrontend that is doubling many requests.
I counted en.m.wikipedia/wiki/article requests and my total count agrees with stats.wikimedia (around 2B), but http 301 redirects went from being around 1% of responses to >18% in one random mid-december day.
An example is:
cp1044.wikimedia.org 214632678 2013-01-08T00:27:24 0.094004869 0.0.0.0 miss/301 20 GET http://en.m.wikipedia.org/wiki/Pawn%20(chess) - text/html; charset=utf-8 http://en.m.wikipedia.org/wiki/Rules_of_chess - Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%206_0_1%20like%20Mac%20OS%20X)%20 AppleWebKit/536.26%20(KHTML,%20like%20Gecko)%20Version/6.0%20Mobile/10A523%2 0Safari/8536.25 en-us -
The correct article name for the above example is http://en.m.wikipedia.org/wiki/Pawn_(chess). The desktop site in safari provides a correct link, but the mobile site converts the underscore to an encoded space.
There may be some other issues as well - the number of requests with a blank referrer have increased heavily. This is partially due to ios6 using google over ssl, but the number of blank referrers increased heavily for android as well. My rough estimate is that mobile pageviews actually increased by around 1/3 in december.
On Mon, Jan 7, 2013 at 2:00 PM, Diederik van Liere dvanliere@wikimedia.org wrote:
Hi all,
It has come to our attention that there is quite a big jump in mobile pageviews on http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We are looking into the exact cause of it right now, but suffice to say for now, do not use these numbers for the month of December 2012.
Best,
Diederik
_______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On what particular day did the rollout occur in December? The chart suggests 14 Dec ?
Also please let me know when the issue has been fixed.
I am asking because I hope to present better stats by extrapolation,
after discarding input from last x days of December and first y days of January.
http://stats.wikimedia.org/EN/TablesPageViewsMonthlyOriginalMobile.htm
We cannot fix the original data, those were transient, udp messages updated counter, and got discarded.
For traffic reports based on 1:1000 sampled squid/varnish log, we can filter unwanted traffic, if we know exactly what to filter out, and rerun.
Erik
cid:image001.png@01CDE938.42042260
From: analytics-bounces@lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Max Semenik Sent: Tuesday, January 08, 2013 3:48 PM To: Asher Feldman Cc: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics.; Tomasz Finc; mobile-tech; Brion Vibber; Arthur Richards Subject: Re: [Analytics] Suspicious increase in mobile pv's
Indeed, in December we've rolled out a change that removed the bogus code that previously prevented redirection to canonical URLs. However, search JS uses raw URL-encoding which results in people visiting a lot of non-canonical URLs. I attempted a fix in https://gerrit.wikimedia.org/r/#/c/42766/ - hope to squeeze it in today's deployment.
On Tue, Jan 8, 2013 at 4:47 AM, Asher Feldman afeldman@wikimedia.org wrote:
It looks like the December numbers are indeed skewed, possibly due to a link encoding change in mobilefrontend that is doubling many requests.
I counted en.m.wikipedia/wiki/article requests and my total count agrees with stats.wikimedia (around 2B), but http 301 redirects went from being around 1% of responses to >18% in one random mid-december day.
An example is:
cp1044.wikimedia.org 214632678 2013-01-08T00:27:24 0.094004869 0.0.0.0 miss/301 20 GET http://en.m.wikipedia.org/wiki/Pawn%20(chess) - text/html; charset=utf-8 http://en.m.wikipedia.org/wiki/Rules_of_chess - Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%206_0_1%20like%20Mac%20OS%20X)%20 AppleWebKit/536.26%20(KHTML,%20like%20Gecko)%20Version/6.0%20Mobile/10A523%2 0Safari/8536.25 en-us -
The correct article name for the above example is http://en.m.wikipedia.org/wiki/Pawn_(chess). The desktop site in safari provides a correct link, but the mobile site converts the underscore to an encoded space.
There may be some other issues as well - the number of requests with a blank referrer have increased heavily. This is partially due to ios6 using google over ssl, but the number of blank referrers increased heavily for android as well. My rough estimate is that mobile pageviews actually increased by around 1/3 in december.
On Mon, Jan 7, 2013 at 2:00 PM, Diederik van Liere dvanliere@wikimedia.org wrote:
Hi all,
It has come to our attention that there is quite a big jump in mobile pageviews on http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We are looking into the exact cause of it right now, but suffice to say for now, do not use these numbers for the month of December 2012.
Best,
Diederik
_______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On Tue, Jan 8, 2013 at 6:48 PM, Max Semenik msemenik@wikimedia.org wrote:
Indeed, in December we've rolled out a change that removed the bogus code that previously prevented redirection to canonical URLs. However, search JS uses raw URL-encoding which results in people visiting a lot of non-canonical URLs. I attempted a fix in https://gerrit.wikimedia.org/r/#/c/42766/ - hope to squeeze it in today's deployment.
And it was deployed. Do we have enough data since yesterday to figure out how significant was the drop in requests?
On Wed, Jan 9, 2013 at 12:55 PM, Max Semenik msemenik@wikimedia.org wrote:
And it was deployed. Do we have enough data since yesterday to figure out how significant was the drop in requests?
I talked with Diederik today and he should be able to run these jobs latest by Monday to see if were back on track.
--tomasz
301 redirects are occuring on < 1% of en.m pageviews now, looks like Max's fix was successful.
On Wed, Jan 9, 2013 at 3:01 PM, Tomasz Finc tfinc@wikimedia.org wrote:
On Wed, Jan 9, 2013 at 12:55 PM, Max Semenik msemenik@wikimedia.orgwrote:
And it was deployed. Do we have enough data since yesterday to figure out how significant was the drop in requests?
I talked with Diederik today and he should be able to run these jobs latest by Monday to see if were back on track.
--tomasz
