Page view data with Wikipedia app?

List overview All Threads
Download

newer

older

Context: massive FB decline with...

FastCCI (Catgraph for Commons...

Denny Vrandečić

13 Jan 2014 13 Jan '14

10:34 p.m.

According to the stats, the last 12 months have seen a decline in overall page views by about 9-10%. As far as I can tell these numbers do not include access to Wikipedia and sister projects through the API. And if I understand correctly, this means that e.g. access through the Wikipedia app is ignored. Considering that mobile access through the browser is about 15-20% of the overall access (if I read the graphs correctly), is it possible that the current page view numbers are underreported? If a reasonable portion of mobile users use the app instead of the browser, then these numbers should somehow be included in the page view statistics, or wrong conclusions might be drawn. Any ideas? Denny

Attachments:

attachment.htm (text/html — 845 bytes)

Show replies by date

Erik Zachte

13 Jan 13 Jan

10:47 p.m.

Hi Denny, Thanks for notice, We are busy vetting the page view results and trying to find out if the decline is real or an artefact. I am running queries on our raw sampled squid log data, to see if the pattern from the patched webstatscollector data can be found there as well. API calls are indeed something we need to look into further. Do you (or anyone else) happen to know which specific API parameters I should search for in the squid log? Thanks, Erik From: analytics-bounces(a)lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Denny Vrandecic Sent: Monday, January 13, 2014 21:35 To: Wikimedia Analytics Subject: [Analytics] Page view data with Wikipedia app? According to the stats, the last 12 months have seen a decline in overall page views by about 9-10%. As far as I can tell these numbers do not include access to Wikipedia and sister projects through the API. And if I understand correctly, this means that e.g. access through the Wikipedia app is ignored. Considering that mobile access through the browser is about 15-20% of the overall access (if I read the graphs correctly), is it possible that the current page view numbers are underreported? If a reasonable portion of mobile users use the app instead of the browser, then these numbers should somehow be included in the page view statistics, or wrong conclusions might be drawn. Any ideas? Denny

Denny Vrandečić

10:56 p.m.

Unfortunately I do not know that. I guess the folks n the Mobile Apps team [1] should be able to help here. [1] https://www.mediawiki.org/wiki/Mobile_Apps/Team On Mon Jan 13 2014 at 12:47:48 PM, Erik Zachte <ezachte(a)wikimedia.org> wrote:

...

Hi Denny, Thanks for notice, We are busy vetting the page view results and trying to find out if the decline is real or an artefact. I am running queries on our raw sampled squid log data, to see if the pattern from the patched webstatscollector data can be found there as well. API calls are indeed something we need to look into further. Do you (or anyone else) happen to know which specific API parameters I should search for in the squid log? Thanks, Erik *From:* analytics-bounces(a)lists.wikimedia.org [mailto: analytics-bounces(a)lists.wikimedia.org] *On Behalf Of *Denny Vrandecic *Sent:* Monday, January 13, 2014 21:35 *To:* Wikimedia Analytics *Subject:* [Analytics] Page view data with Wikipedia app? According to the stats, the last 12 months have seen a decline in overall page views by about 9-10%. As far as I can tell these numbers do not include access to Wikipedia and sister projects through the API. And if I understand correctly, this means that e.g. access through the Wikipedia app is ignored. Considering that mobile access through the browser is about 15-20% of the overall access (if I read the graphs correctly), is it possible that the current page view numbers are underreported? If a reasonable portion of mobile users use the app instead of the browser, then these numbers should somehow be included in the page view statistics, or wrong conclusions might be drawn. Any ideas? Denny

Christian Aistleitner

11:34 p.m.

Hi, On Mon, Jan 13, 2014 at 09:47:34PM +0100, Erik Zachte wrote:

...

API calls are indeed something we need to look into further. Do you (or anyone else) happen to know which specific API parameters I should search for in the squid log?

As webstatscollector limits itself to /wiki/ requests, it does not see api requests. Some documentation of the api actions can be found on https://en.wikipedia.org/w/api.php The most relevant actions are parse and mobileview (search for "action=parse", or "action=mobileview" on the page). But they (in sum) currently amount to ~10% of api requests. When I last checked API requests were ~8% of all requests. So impact might be ~0.8% of requests. However, when looking for potential page views, /w/index.php might be a better candidate. Webstatscollector never cared about it, but last time I checked, it was about 6% of requests, and it breaks down >90% into actions raw, view, and render. Documentation for the actions is available at https://www.mediawiki.org/wiki/Manual:Parameters_to_index.php So impact might be ~5% of requests. Have fun, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian(a)quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ --------------------------------------------------------------- OpenPGP key transition from 0xEF78CCDE to 0x13C1072F: http://quelltextlich.at/openpgp-transition-0xEF78CCDE-to-0x13C1072F.txt

Denny Vrandečić

11:54 p.m.

Thanks Christian, that is indeed helpful. When was the last time you checked, roughly? When you got the 8%. On Mon Jan 13 2014 at 1:34:23 PM, Christian Aistleitner < christian(a)quelltextlich.at> wrote:

...

The most relevant actions are parse and mobileview (search for "action=parse", or "action=mobileview" on the page). But they (in sum) currently amount to ~10% of api requests. When I last checked API requests were ~8% of all requests.

Christian Aistleitner

14 Jan 14 Jan

12:53 a.m.

Hi, On Mon, Jan 13, 2014 at 09:54:08PM +0000, Denny Vrandečić wrote:

...

Thanks Christian, that is indeed helpful. When was the last time you checked, roughly? When you got the 8%.

Around end of November. I quickly rechecked for the most recent logfile. It had: ~25% /wiki/ ~10% /w/index.php ~4% /w/api.php But that's just a single day. It might move a bit for longer periods of time. And also note that those numbers are requests not pageviews. Wikistats for example recruits pageviews from webstatscollector, which itself is limited to the /wiki/ requests. Best regards, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian(a)quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ --------------------------------------------------------------- OpenPGP key transition from 0xEF78CCDE to 0x13C1072F: http://quelltextlich.at/openpgp-transition-0xEF78CCDE-to-0x13C1072F.txt

Denny Vrandečić

1:29 a.m.

So actually /w/api.php makes about 20% of the requests compared to /wiki/? (a bit confused, sorry) On Mon Jan 13 2014 at 2:53:47 PM, Christian Aistleitner < christian(a)quelltextlich.at> wrote:

...

Hi, On Mon, Jan 13, 2014 at 09:54:08PM +0000, Denny Vrandečić wrote:

Thanks Christian, that is indeed helpful. When was the last time you checked, roughly? When you got the 8%.

Daniel Schwen

1:36 a.m.

...

So actually /w/api.php makes about 20% of the requests compared to /wiki/? (a bit confused, sorry)

That does not seem too surprising, as it includes autocomplete as you type requests in the search widget, which must make a large contribution by sheer request number. Daniel

Erik Zachte

4:05 a.m.

Hi all, I added two pages to Google Doc " Analysis of pageviews trend after correction for overreporting" scroll to bottom https://docs.google.com/a/wikimedia.org/document/d/1kpJrfataS5KAxGXFoygQVhM… One updated chart, plus two new charts,and some raw data. Quick first observations: (may be refined in coming days) The page view reports http://stats.wikimedia.org/EN/TablesPageViewsSitemap.htm are based on webstatscollector data, which are totally based on GET /wiki/ (as was known). They miss out on GET /w/index.php (a considerable amount of page views, of all kind, including search) and GET /w/api.php (not so much). ‘Get other’ is not plotted in the chart. It is partly portal pages, e.g. http://wikipedia./org , partly invalid requests. (I may need to break this up further) So with above limitations squid based counts 'GET /wiki/' are roughly same order of magnitude as webstatscollector data, but curves do not exactly match. More importantly data from squid log show no consistent decline over several months. We also see no decline in share of traffic from Google. All in all it seems we need to look for internal data collecting/reporting bug as explanation. Or review the patch made in December (for filtering bogus traffic) once again, but it was thoroughly and independently vetted by two people already, not a likely cause. Any further comments tomorrow Erik -----Original Message----- From: analytics-bounces(a)lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Daniel Schwen Sent: Tuesday, January 14, 2014 0:36 To: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics. Subject: Re: [Analytics] Page view data with Wikipedia app?

...

So actually /w/api.php makes about 20% of the requests compared to /wiki/? (a bit confused, sorry)

That does not seem too surprising, as it includes autocomplete as you type requests in the search widget, which must make a large contribution by sheer request number. Daniel _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Erik Moeller

4:15 a.m.

Thanks, Erik. We've known for a while that the /wiki/ limitation is a pretty significant one. A further quantitative breakdown of the other common groupings (search, edits, etc.) would definitely be helpful. -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation

Erik Zachte

4:18 a.m.

Sure, will look into it. -----Original Message----- From: analytics-bounces(a)lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Erik Moeller Sent: Tuesday, January 14, 2014 3:15 To: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics. Subject: Re: [Analytics] Page view data with Wikipedia app? Thanks, Erik. We've known for a while that the /wiki/ limitation is a pretty significant one. A further quantitative breakdown of the other common groupings (search, edits, etc.) would definitely be helpful. -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Dario Taraborelli

4:28 a.m.

Erik – thanks so much for this and for staying up late to crunch and publish this data. I agree that we’re not in a position to ascribe changes to external factors until we’ve nailed down what share of API requests or otherwise non-standard requests excluded by WSC should be counted or discounted as PVs. I am worried that an increasing number of requests may have moved to the API not just due to apps gaining larger shares of mobile traffic (as per Denny), but also other crawlers/scrapers hitting /w/* instead of /wiki/*. I look forward to catching up on these threads tomorrow. Dario On Jan 13, 2014, at 6:18 PM, Erik Zachte <ezachte(a)wikimedia.org> wrote:

...

Matthew Flaschen

15 Jan 15 Jan

4:17 a.m.

On 01/13/2014 09:15 PM, Erik Moeller wrote:

...

Yes. Regular page views can also use index.php (https://en.wikipedia.org/w/index.php?title=Earth). For example, navigation popups constructs the main article link like this. index.php with only a title URL parameter should be treated identically to /wiki/ requests without URL parameters. Matt Flaschen

Christian Aistleitner

14 Jan 14 Jan

3:05 p.m.

Hi, On Tue, Jan 14, 2014 at 03:05:03AM +0100, Erik Zachte wrote:

...

[...] webstatscollector data, which are totally based on GET /wiki/ (as was known).

it should hardly have impact, but just to avoid misconceptions as GET was coined several times through the email ... I do not know of a filter to GET in webstatscollector. Looking for it again, I could not find it. Where does that GET filter live? Best regards, Christian P.S.: Just to avoid confusion, from my point of view, webstatscollector does limit to /wiki/, but not to GET /wiki/. -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian(a)quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ --------------------------------------------------------------- OpenPGP key transition from 0xEF78CCDE to 0x13C1072F: http://quelltextlich.at/openpgp-transition-0xEF78CCDE-to-0x13C1072F.txt

Erik Zachte

5:02 p.m.

The point of yesterdays execrcise was to crosscheck webstatscollector, so the filters in webstatscollector served as inspiration but not as guiding principle. I broke down traffic patterns from squids log using ad hoc criteria that seemed most descriptive. Nearly all html traffic is sent as response on GET requests. This may be obvious for web experts, so it hardly merits mentioning, but it wasn't obvious to me. (for example I was curious about ratio GET vs HEAD) I am also not sure yet webstatscollector should use the range of ip addresses it does right now to filter out internal traffic, Some are very busy ranges, others show no activity at all (could be tuned out for better webstatscollector performance). And is the configuration complete at all, or could we be missing some ranges? I'll follow up on that off-list. Erik -----Original Message----- From: analytics-bounces(a)lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Christian Aistleitner Sent: Tuesday, January 14, 2014 14:06 To: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics. Subject: Re: [Analytics] Page view data with Wikipedia app? Hi, On Tue, Jan 14, 2014 at 03:05:03AM +0100, Erik Zachte wrote:

...

[...] webstatscollector data, which are totally based on GET /wiki/ (as was known).

Denny Vrandečić

8:50 p.m.

Thank you Erik for this amazing work. I think I understood most of it :) The overreporting patch leads to a significant difference, as discussed in the first two pages, starting end of 2013. I wonder for the curve before the divergence, why does the patch have no effect? I.e. why is there only a divergence since mid-2013? Can the numbers be retroactively recalculated after the patch, or is the original data not available to do that? I wonder if there was systemic overreporting before mid-2013 as well. Again, thank you very very much for this great work! On Tue Jan 14 2014 at 7:02:15 AM, Erik Zachte <ezachte(a)wikimedia.org> wrote:

...

[...] webstatscollector data, which are totally based on GET /wiki/ (as was known).

Federico Leva (Nemo)

11:47 p.m.

Denny Vrandečić, 14/01/2014 19:50:

...

The overreporting patch leads to a significant difference, as discussed in the first two pages, starting end of 2013. I wonder for the curve before the divergence, why does the patch have no effect? I.e. why is there only a divergence since mid-2013?

Denny Vrandečić

11:50 p.m.

Ah, OK, understood. And we are sure that the patch did have no other consequences than remove these requests, i.e. a recalculation would lead to the same numbers, I assume. On Tue Jan 14 2014 at 1:47:41 PM, Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote:

...

Denny Vrandečić, 14/01/2014 19:50:

The overreporting was caused by some /wiki/Special:* requests added by SUL2 <https://www.mediawiki.org/wiki/SUL2> of which you can see the deployment timeline... nowhere, apparently, gotta dig https://wikitech.wikimedia.org/wiki/Deployments/Archive/2013 Nemo _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Christian Aistleitner

15 Jan 15 Jan

1:02 a.m.

Hi Denny, On Tue, Jan 14, 2014 at 09:50:24PM +0000, Denny Vrandečić wrote:

...

And we are sure that the patch did have no other consequences than remove these requests,

I am pretty convinced that the webstatscollector patch does only what it is supposed to do. Andrew and Erik looked over the patch, but please have a look at the patch yourself: https://gerrit.wikimedia.org/r/#/c/98505/1 Just to be sure it that the patch does the right thing, I also added a short test harness and added test cases to make sure the change does not misbehave. See the bottom part of: https://gerrit.wikimedia.org/r/#/c/98524/2/tests/test.sh The patch has the known short-coming of handling only non-localized versions of the Special:CentralAutoLogin requests. But the localized versions were very, very few in number, and discussion was already ongoing to no longer localize them, as localization was causing problems for Ops as well. In the meanwhile, localizing Special:CentralAutoLogin has been removed: https://gerrit.wikimedia.org/r/#/c/98461/ https://bugzilla.wikimedia.org/show_bug.cgi?id=54195

...

[...] i.e. a recalculation would lead to the same numbers, I assume.

Erik Z did the patching of the files back then. I did recalculate the data and it looked good back then. But since we were also concerned that ... maybe we overlooked something ... I recomputed the trends that wikistats exposed once again directly from the cache logs (using the wikistats pageview definition). And it matches the picture wikistats shows for the last months. So I also arrived at the different patterns of ruwiki, ptwiki, eswiki, kowiki, ... There are many other possible sources of problems (and general room for improvement), but to me, employing the wikistats pageview definition to either the stored cache logs or the stored webstatscollector data, we really arrive changes that wikistats is depicting. Best regards, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian(a)quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ --------------------------------------------------------------- OpenPGP key transition from 0xEF78CCDE to 0x13C1072F: http://quelltextlich.at/openpgp-transition-0xEF78CCDE-to-0x13C1072F.txt

Denny Vrandečić

1:55 a.m.

Thank you very much! You really did a great job here. On Tue Jan 14 2014 at 3:02:18 PM, Christian Aistleitner < christian(a)quelltextlich.at> wrote:

...

Hi Denny, On Tue, Jan 14, 2014 at 09:50:24PM +0000, Denny Vrandečić wrote:

And we are sure that the patch did have no other consequences than remove these requests,

[...] i.e. a recalculation would lead to the same numbers, I assume.

Erik Moeller

2:21 a.m.

On Tue, Jan 14, 2014 at 3:02 PM, Christian Aistleitner <christian(a)quelltextlich.at> wrote:

...

There are many other possible sources of problems (and general room for improvement), but to me, employing the wikistats pageview definition to either the stored cache logs or the stored webstatscollector data, we really arrive changes that wikistats is depicting.

For the time being, in both cases we're counting crawler traffic, correct? If so, have we attempted to quantify the degree of variation in crawler traffic over time? My main concern is that major changes in crawler behavior could distort the overall numbers. Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation

Federico Leva (Nemo)

16 Jan 16 Jan

1:27 a.m.

Erik Moeller, 15/01/2014 01:21:

...

What sort of crawlers/bots are more likely to have such an impact? Are things like /wiki/Special:Export* or /wiki/*?action=render and so on filtered out? No idea how many of those (and all sorts of parameters to index.php explicit or implicit) there could be. Matthew Flaschen, 15/01/2014 03:17:

...

Yes. Regular page views can also use index.php (https://en.wikipedia.org/w/index.php?title=Earth). For example, navigation popups constructs the main article link like this.

Navigation popups are really a bad example to mention in this case. :)

...

index.php with only a title URL parameter should be treated identically to /wiki/ requests without URL parameters.

How many actual/"legit" page views can we expect from there, as opposed to bots of all sorts, stuff loaded in background etc.? I think the subject "Page view data with Wikipedia app" went in the right direction, it's probably better to hunt for biggest sources of views legit but missing or fake but counted. Nemo

Matthew Flaschen

2:08 a.m.

On 01/15/2014 06:27 PM, Federico Leva (Nemo) wrote:

...

Matthew Flaschen, 15/01/2014 03:17:

Yes. Regular page views can also use index.php (https://en.wikipedia.org/w/index.php?title=Earth). For example, navigation popups constructs the main article link like this.

Navigation popups are really a bad example to mention in this case. :)

Why? The magnitude is not overwhelming, but they are legit views, not something loaded in the background. Basically, if you mouseover a link, then click the main link in the popup, that's an index.php URL with just title parameter, but it's a regular view. Popups does load things in the background, but I think they all use api.php.

...

index.php with only a title URL parameter should be treated identically to /wiki/ requests without URL parameters.

How many actual/"legit" page views can we expect from there, as opposed to bots of all sorts, stuff loaded in background etc.?

Any index.php with only a title parameter is equivalent to a regular /wiki/ view. If we decide to exclude bots or crawlers (obviously an important discussion), they should probably be excluded based on User-Agent or IP, not simply because they should a less common URL that's equivalent to the common one.

...

it's probably better to hunt for biggest sources of views legit but missing or fake but counted.

I agree this is a good way to look at it, though different people will have different definitions of "fake". Matt Flaschen

Christian Aistleitner

14 Jan 14 Jan

2:40 a.m.

Hi, [ rearranged due to top-posting ] On Mon, Jan 13, 2014 at 11:29:12PM +0000, Denny Vrandečić wrote:

...

On Mon Jan 13 2014 at 2:53:47 PM, Christian Aistleitner < christian(a)quelltextlich.at> wrote:

I quickly rechecked for the most recent logfile. It had: ~25% /wiki/ ~10% /w/index.php ~4% /w/api.php

So actually /w/api.php makes about 20% of the requests compared to /wiki/?

Yes. As Daniel replied it is mostly “opensearch” or “query” actions. Best regards, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian(a)quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ --------------------------------------------------------------- OpenPGP key transition from 0xEF78CCDE to 0x13C1072F: http://quelltextlich.at/openpgp-transition-0xEF78CCDE-to-0x13C1072F.txt

Federico Leva (Nemo)

13 Jan 13 Jan

11:37 p.m.

Erik Zachte, 13/01/2014 21:47:

...

Do you (or anyone else) happen to know which specific API parameters I should search for in the squid log?

There's also the API log which contains each and all API requests by the way, no idea if it's more informative. Nemo

Denny Vrandečić

11:54 p.m.

Uh, that sounds interesting. I didn't know about the API log. Is it also on for Wikidata? On Mon Jan 13 2014 at 1:37:48 PM, Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote:

...

Erik Zachte, 13/01/2014 21:47:

Do you (or anyone else) happen to know which specific API parameters I should search for in the squid log?

There's also the API log which contains each and all API requests by the way, no idea if it's more informative. Nemo _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Federico Leva (Nemo)

14 Jan 14 Jan

4:06 p.m.

Denny Vrandečić, 13/01/2014 22:54:

...

Uh, that sounds interesting. I didn't know about the API log. Is it also on for Wikidata?

Sure, for all projects. I don't know much, but for the last few months I've been adding all I discover to https://wikitech.wikimedia.org/wiki/Logs Nemo

3746

days inactive

3749

days old

analytics@lists.wikimedia.org

Manage subscription

26 comments

8 participants

tags (0)

participants (8)

Christian Aistleitner
Daniel Schwen
Dario Taraborelli
Denny Vrandečić
Erik Moeller
Erik Zachte
Federico Leva (Nemo)
Matthew Flaschen