Unfortunately I do not know that. I guess the folks n the Mobile Apps team [1] should be able to help here.

[1] https://www.mediawiki.org/wiki/Mobile_Apps/Team

On Mon Jan 13 2014 at 12:47:48 PM, Erik Zachte ezachte@wikimedia.org wrote:

Hi Denny,

Thanks for notice,

We are busy vetting the page view results and trying to find out if the decline is real or an artefact.

I am running queries on our raw sampled squid log data, to see if the pattern from the patched webstatscollector data can be found there as well.

API calls are indeed something we need to look into further.

Do you (or anyone else) happen to know which specific API parameters I should search for in the squid log?

Thanks,

Erik

*From:* analytics-bounces@lists.wikimedia.org [mailto: analytics-bounces@lists.wikimedia.org] *On Behalf Of *Denny Vrandecic *Sent:* Monday, January 13, 2014 21:35 *To:* Wikimedia Analytics *Subject:* [Analytics] Page view data with Wikipedia app?

According to the stats, the last 12 months have seen a decline in overall page views by about 9-10%.

As far as I can tell these numbers do not include access to Wikipedia and sister projects through the API. And if I understand correctly, this means that e.g. access through the Wikipedia app is ignored.

Considering that mobile access through the browser is about 15-20% of the overall access (if I read the graphs correctly), is it possible that the current page view numbers are underreported?

If a reasonable portion of mobile users use the app instead of the browser, then these numbers should somehow be included in the page view statistics, or wrong conclusions might be drawn.

Any ideas?

Denny

Thanks Christian, that is indeed helpful. When was the last time you checked, roughly? When you got the 8%.

On Mon Jan 13 2014 at 1:34:23 PM, Christian Aistleitner < christian@quelltextlich.at> wrote:

The most relevant actions are parse and mobileview (search for "action=parse", or "action=mobileview" on the page). But they (in sum) currently amount to ~10% of api requests. When I last checked API requests were ~8% of all requests.

So actually /w/api.php makes about 20% of the requests compared to /wiki/? (a bit confused, sorry)

On Mon Jan 13 2014 at 2:53:47 PM, Christian Aistleitner < christian@quelltextlich.at> wrote:

Hi,

On Mon, Jan 13, 2014 at 09:54:08PM +0000, Denny Vrandečić wrote:

...

Thanks Christian, that is indeed helpful. When was the last time you checked, roughly? When you got the 8%.

Around end of November. I quickly rechecked for the most recent logfile.

It had: ~25% /wiki/ ~10% /w/index.php ~4% /w/api.php

But that's just a single day. It might move a bit for longer periods of time.

And also note that those numbers are requests not pageviews. Wikistats for example recruits pageviews from webstatscollector, which itself is limited to the /wiki/ requests.

Best regards, Christian

-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian@quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/

---

OpenPGP key transition from 0xEF78CCDE to 0x13C1072F: http://quelltextlich.at/openpgp-transition-0xEF78CCDE-to-0x13C1072F.txt _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Hi all,

I added two pages to Google Doc " Analysis of pageviews trend after correction for overreporting" scroll to bottom https://docs.google.com/a/wikimedia.org/document/d/1kpJrfataS5KAxGXFoygQVhMl...

One updated chart, plus two new charts,and some raw data.

Quick first observations: (may be refined in coming days) The page view reports http://stats.wikimedia.org/EN/TablesPageViewsSitemap.htm are based on webstatscollector data, which are totally based on GET /wiki/ (as was known). They miss out on GET /w/index.php (a considerable amount of page views, of all kind, including search) and GET /w/api.php (not so much). ‘Get other’ is not plotted in the chart. It is partly portal pages, e.g. http://wikipedia./org , partly invalid requests. (I may need to break this up further)

So with above limitations squid based counts 'GET /wiki/' are roughly same order of magnitude as webstatscollector data, but curves do not exactly match.

More importantly data from squid log show no consistent decline over several months. We also see no decline in share of traffic from Google.

All in all it seems we need to look for internal data collecting/reporting bug as explanation. Or review the patch made in December (for filtering bogus traffic) once again, but it was thoroughly and independently vetted by two people already, not a likely cause.

Any further comments tomorrow

Erik

-----Original Message----- From: analytics-bounces@lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Daniel Schwen Sent: Tuesday, January 14, 2014 0:36 To: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics. Subject: Re: [Analytics] Page view data with Wikipedia app?

So actually /w/api.php makes about 20% of the requests compared to /wiki/? (a bit confused, sorry)

That does not seem too surprising, as it includes autocomplete as you type requests in the search widget, which must make a large contribution by sheer request number. Daniel

_______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Sure, will look into it.

-----Original Message----- From: analytics-bounces@lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Erik Moeller Sent: Tuesday, January 14, 2014 3:15 To: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics. Subject: Re: [Analytics] Page view data with Wikipedia app?

Thanks, Erik. We've known for a while that the /wiki/ limitation is a pretty significant one. A further quantitative breakdown of the other common groupings (search, edits, etc.) would definitely be helpful.

-- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation

_______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

On 01/13/2014 09:15 PM, Erik Moeller wrote:

Thanks, Erik. We've known for a while that the /wiki/ limitation is a pretty significant one. A further quantitative breakdown of the other common groupings (search, edits, etc.) would definitely be helpful.

Yes. Regular page views can also use index.php (https://en.wikipedia.org/w/index.php?title=Earth). For example, navigation popups constructs the main article link like this.

index.php with only a title URL parameter should be treated identically to /wiki/ requests without URL parameters.

Matt Flaschen

The point of yesterdays execrcise was to crosscheck webstatscollector, so the filters in webstatscollector served as inspiration but not as guiding principle.

I broke down traffic patterns from squids log using ad hoc criteria that seemed most descriptive. Nearly all html traffic is sent as response on GET requests. This may be obvious for web experts, so it hardly merits mentioning, but it wasn't obvious to me. (for example I was curious about ratio GET vs HEAD)

I am also not sure yet webstatscollector should use the range of ip addresses it does right now to filter out internal traffic, Some are very busy ranges, others show no activity at all (could be tuned out for better webstatscollector performance). And is the configuration complete at all, or could we be missing some ranges? I'll follow up on that off-list.

Erik

-----Original Message----- From: analytics-bounces@lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Christian Aistleitner Sent: Tuesday, January 14, 2014 14:06 To: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics. Subject: Re: [Analytics] Page view data with Wikipedia app?

Hi,

On Tue, Jan 14, 2014 at 03:05:03AM +0100, Erik Zachte wrote:

[...] webstatscollector data, which are totally based on GET /wiki/ (as was known).

it should hardly have impact, but just to avoid misconceptions as GET was coined several times through the email ... I do not know of a filter to GET in webstatscollector. Looking for it again, I could not find it. Where does that GET filter live?

Best regards, Christian

P.S.: Just to avoid confusion, from my point of view, webstatscollector does limit to /wiki/, but not to GET /wiki/.

-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian@quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ --------------------------------------------------------------- OpenPGP key transition from 0xEF78CCDE to 0x13C1072F: http://quelltextlich.at/openpgp-transition-0xEF78CCDE-to-0x13C1072F.txt

Denny Vrandečić, 14/01/2014 19:50:

The overreporting patch leads to a significant difference, as discussed in the first two pages, starting end of 2013. I wonder for the curve before the divergence, why does the patch have no effect? I.e. why is there only a divergence since mid-2013?

The overreporting was caused by some /wiki/Special:* requests added by SUL2 https://www.mediawiki.org/wiki/SUL2 of which you can see the deployment timeline... nowhere, apparently, gotta dig https://wikitech.wikimedia.org/wiki/Deployments/Archive/2013

Nemo

Hi Denny,

On Tue, Jan 14, 2014 at 09:50:24PM +0000, Denny Vrandečić wrote:

And we are sure that the patch did have no other consequences than remove these requests,

I am pretty convinced that the webstatscollector patch does only what it is supposed to do. Andrew and Erik looked over the patch, but please have a look at the patch yourself: https://gerrit.wikimedia.org/r/#/c/98505/1

Just to be sure it that the patch does the right thing, I also added a short test harness and added test cases to make sure the change does not misbehave. See the bottom part of: https://gerrit.wikimedia.org/r/#/c/98524/2/tests/test.sh

The patch has the known short-coming of handling only non-localized versions of the Special:CentralAutoLogin requests. But the localized versions were very, very few in number, and discussion was already ongoing to no longer localize them, as localization was causing problems for Ops as well. In the meanwhile, localizing Special:CentralAutoLogin has been removed: https://gerrit.wikimedia.org/r/#/c/98461/ https://bugzilla.wikimedia.org/show_bug.cgi?id=54195

[...] i.e. a recalculation would lead to the same numbers, I assume.

Erik Z did the patching of the files back then. I did recalculate the data and it looked good back then.

But since we were also concerned that ... maybe we overlooked something ... I recomputed the trends that wikistats exposed once again directly from the cache logs (using the wikistats pageview definition). And it matches the picture wikistats shows for the last months. So I also arrived at the different patterns of ruwiki, ptwiki, eswiki, kowiki, ...

There are many other possible sources of problems (and general room for improvement), but to me, employing the wikistats pageview definition to either the stored cache logs or the stored webstatscollector data, we really arrive changes that wikistats is depicting.

Best regards, Christian

On Tue, Jan 14, 2014 at 3:02 PM, Christian Aistleitner christian@quelltextlich.at wrote:

There are many other possible sources of problems (and general room for improvement), but to me, employing the wikistats pageview definition to either the stored cache logs or the stored webstatscollector data, we really arrive changes that wikistats is depicting.

For the time being, in both cases we're counting crawler traffic, correct? If so, have we attempted to quantify the degree of variation in crawler traffic over time? My main concern is that major changes in crawler behavior could distort the overall numbers.

Erik

On 01/15/2014 06:27 PM, Federico Leva (Nemo) wrote:

Matthew Flaschen, 15/01/2014 03:17:

...

Yes. Regular page views can also use index.php (https://en.wikipedia.org/w/index.php?title=Earth). For example, navigation popups constructs the main article link like this.

Navigation popups are really a bad example to mention in this case. :)

Why? The magnitude is not overwhelming, but they are legit views, not something loaded in the background. Basically, if you mouseover a link, then click the main link in the popup, that's an index.php URL with just title parameter, but it's a regular view.

Popups does load things in the background, but I think they all use api.php.

...

index.php with only a title URL parameter should be treated identically to /wiki/ requests without URL parameters.

How many actual/"legit" page views can we expect from there, as opposed to bots of all sorts, stuff loaded in background etc.?

Any index.php with only a title parameter is equivalent to a regular /wiki/ view.

If we decide to exclude bots or crawlers (obviously an important discussion), they should probably be excluded based on User-Agent or IP, not simply because they should a less common URL that's equivalent to the common one.

it's probably better to hunt for biggest sources of views legit but missing or fake but counted.

I agree this is a good way to look at it, though different people will have different definitions of "fake".

Matt Flaschen

Erik Zachte, 13/01/2014 21:47:

Do you (or anyone else) happen to know which specific API parameters I should search for in the squid log?

There's also the API log which contains each and all API requests by the way, no idea if it's more informative.

Nemo

Denny Vrandečić, 13/01/2014 22:54:

Uh, that sounds interesting. I didn't know about the API log. Is it also on for Wikidata?

Sure, for all projects. I don't know much, but for the last few months I've been adding all I discover to https://wikitech.wikimedia.org/wiki/Logs

Nemo