Hi Folks,
TLDR: Directional data suggests that the project-wide drop we see in pageviews is, in part, caused by shorter sessions on mobile web compared to desktop (and a migration from desktop to mobile web)
*Context:* Danny and I took some time last week to try and understand the dramatic drop in pageviews that we saw globally and in the global south just over the last quarter. The numbers we quoted in the Q1 quarterly review https://commons.wikimedia.org/wiki/File:WMF_Reading_Quarterly_Review_Q1_2015-16.pdf last week were (pageviews across all projects, platforms and geographies):
- -12.4% Quarter over quarter - -7% Year over year*
*The YoY data is sampled and there may be anomalies that force us to consider an inaccuracy of up to +-5% difference possible (i.e. a YoY change from -2% to -12%) (source: Tilman, Hive).
Here is the answer to one question we had: Is this drop due to fewer people or fewer pages per user? I will preface this by saying that I erred on the side of getting this out sooner than in making it incredibly replicable, share-able. If you are interested in the specific queries or access to the raw data, I will prep and send it out. Otherwise, curious to hear your questions/concerns/suggestions.
*Details:*
Is this drop due to fewer people or fewer pages per user?
The answer here, is interesting and the impact is more significant than I would have expected. On Desktop the pageviews per visit (as internal referers/external+ unknown) is relatively constant. On Mobile web, however, the pageviews per visit is much lower and appears to be dropping. The following graphs explain:
Daily *desktop* pageviews, by referer 4/13--present (all wikipedias, all geos). There is strong correlation between pageviews that come from the outside v. the inside: [image: Inline image 3]
[image: Inline image 4]
Daily *Mobile web* pageviews, by referer 4/13--present (all wikipedias, all geos). [image: Inline image 2]
Compared the 60% we have on desktop, you can see that the ratio is 40% (33% smaller) on mobile and that this gap has widened (though not in the last 2 years): [image: Inline image 5]
I don't know if we can explain all of our traffic decreases to the drop in session length, but it is certainly a big factor. Basically 60% of our pageviews (internal) shrink by 33% on mobile. So all else being equal, if we transfer all our traffic to mobile we lose 33% of our pageviews. Right now we're at 50%. This assumes that there is no change in numbers of sessions...on which we have no data right now.
*Next Steps:* Unless I hear otherwise, I think the next steps are to start thinking through what the implications are.
- Do we try to identify reasons users might be skipping out earlier on mobile and fixing those? - Do we try to make it easier for people to find content on mobile? - Maybe sessions across the internet are just shorter on mobile and we should focus our efforts on helping people find us?
Regardless, I find this a bit comforting, because having the same number of users who spend less time would be much better than reaching fewer people: controlling the experience once they found us is relatively easier than altering the channels by which people find us in the first place.
Again questions/concerns/suggestions encouraged.
-J
mobile, which is a 15% drop in pageviews (again, assuming a 1:1 traffic switch).
Hi Jon --
Great analysis, thanks for doing this and making it available publicly. As far as I know, this is the first time we've looked at the data this way.
Some questions/thoughts -- I think it is reassuring that an explanation for this page view drop we see is that we are still reaching the same number of people but they just browse less. But for me, it's still an issue that they are seeing less of our content.
1. There's a strange artifact in the desktop page views where they drop by about 75mm at the end of 2013 but don't recover. Do we understand what happened here. 2. Does this mean that mobile users are less likely to fall into the rabbit hole? I'd be interested in getting some more targeted research around how mobile and desktop use differ. Maybe the apps can help. 3. It feels like we should keep things simple and see what we can do about engagement in the mobile web. We have a few ideas in the pipeline for this quarter; I wonder if there are more things we can do.
More analysis and questions are welcome.
-Toby
On Tue, Oct 13, 2015 at 10:14 PM, Jon Katz jkatz@wikimedia.org wrote:
Hi Folks,
TLDR: Directional data suggests that the project-wide drop we see in pageviews is, in part, caused by shorter sessions on mobile web compared to desktop (and a migration from desktop to mobile web)
*Context:* Danny and I took some time last week to try and understand the dramatic drop in pageviews that we saw globally and in the global south just over the last quarter. The numbers we quoted in the Q1 quarterly review https://commons.wikimedia.org/wiki/File:WMF_Reading_Quarterly_Review_Q1_2015-16.pdf last week were (pageviews across all projects, platforms and geographies):
- -12.4% Quarter over quarter
- -7% Year over year*
*The YoY data is sampled and there may be anomalies that force us to consider an inaccuracy of up to +-5% difference possible (i.e. a YoY change from -2% to -12%) (source: Tilman, Hive).
Here is the answer to one question we had: Is this drop due to fewer people or fewer pages per user? I will preface this by saying that I erred on the side of getting this out sooner than in making it incredibly replicable, share-able. If you are interested in the specific queries or access to the raw data, I will prep and send it out. Otherwise, curious to hear your questions/concerns/suggestions.
*Details:*
Is this drop due to fewer people or fewer pages per user?
The answer here, is interesting and the impact is more significant than I would have expected. On Desktop the pageviews per visit (as internal referers/external+ unknown) is relatively constant. On Mobile web, however, the pageviews per visit is much lower and appears to be dropping. The following graphs explain:
Daily *desktop* pageviews, by referer 4/13--present (all wikipedias, all geos). There is strong correlation between pageviews that come from the outside v. the inside: [image: Inline image 3]
[image: Inline image 4]
Daily *Mobile web* pageviews, by referer 4/13--present (all wikipedias, all geos). [image: Inline image 2]
Compared the 60% we have on desktop, you can see that the ratio is 40% (33% smaller) on mobile and that this gap has widened (though not in the last 2 years): [image: Inline image 5]
I don't know if we can explain all of our traffic decreases to the drop in session length, but it is certainly a big factor. Basically 60% of our pageviews (internal) shrink by 33% on mobile. So all else being equal, if we transfer all our traffic to mobile we lose 33% of our pageviews. Right now we're at 50%. This assumes that there is no change in numbers of sessions...on which we have no data right now.
*Next Steps:* Unless I hear otherwise, I think the next steps are to start thinking through what the implications are.
- Do we try to identify reasons users might be skipping out earlier on
mobile and fixing those?
- Do we try to make it easier for people to find content on mobile?
- Maybe sessions across the internet are just shorter on mobile and we
should focus our efforts on helping people find us?
Regardless, I find this a bit comforting, because having the same number of users who spend less time would be much better than reaching fewer people: controlling the experience once they found us is relatively easier than altering the channels by which people find us in the first place.
Again questions/concerns/suggestions encouraged.
-J
mobile, which is a 15% drop in pageviews (again, assuming a 1:1 traffic switch).
reading-wmf mailing list reading-wmf@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/reading-wmf
On Wed, Oct 14, 2015 at 1:43 AM, Toby Negrin tnegrin@wikimedia.org wrote:
- Does this mean that mobile users are less likely to fall into the
rabbit hole? I'd be interested in getting some more targeted research around how mobile and desktop use differ. Maybe the apps can help
I think this is pretty much a given - session length on mobile is known to be shorter than desktop. The upside here is that we should be able to get more sessions from a user over a given time period because a users phone is with them at all times. So we should be able to compensate for length of session with number of sessions.
Upping number of sessions is tied pretty tightly to notifications - Facebook and Twitter get people to come back to their apps by pushing notifications when something interesting happens. Pushing notifications for items in a users feed, or changes to watched/saved pages, etc could be a way to accomplish this.
Agreed, although Twitter greatly exceeds my personal tolerance level for spammy emails even with supposedly limited traffic settings in my preferences.
Pine On Oct 14, 2015 9:04 AM, "Corey Floyd" cfloyd@wikimedia.org wrote:
On Wed, Oct 14, 2015 at 1:43 AM, Toby Negrin tnegrin@wikimedia.org wrote:
- Does this mean that mobile users are less likely to fall into the
rabbit hole? I'd be interested in getting some more targeted research around how mobile and desktop use differ. Maybe the apps can help
I think this is pretty much a given - session length on mobile is known to be shorter than desktop. The upside here is that we should be able to get more sessions from a user over a given time period because a users phone is with them at all times. So we should be able to compensate for length of session with number of sessions.
Upping number of sessions is tied pretty tightly to notifications - Facebook and Twitter get people to come back to their apps by pushing notifications when something interesting happens. Pushing notifications for items in a users feed, or changes to watched/saved pages, etc could be a way to accomplish this.
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
TL;DR: never underestimate https://xkcd.com/214/ ? :) We must be careful NOT to provide users [only] what they are looking for.
Toby Negrin, 14/10/2015 07:43:
There's a strange artifact in the desktop page views where they drop by about 75mm at the end of 2013 but don't recover. Do we understand what happened here.
Irrecoverable effects of One Direction album? ;) https://lists.wikimedia.org/pipermail/mobile-l/2013-December/006342.html
In that period there was a lot of work around redirects and forcing users to use the mobile site. One major forceful redirect was in June 2014 (tablets) and you clearly see the impact on Jon's graphs.
SAL and https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments are not particularly useful for historical digging, it would be great to have a timeline of major mobile configuration changes (or at least of redirects). http://bugs.wmflabs.org/buglist.cgi?chfield=resolution&chfieldfrom=2013-... finds e.g. http://bugs.wmflabs.org/show_bug.cgi?id=57127 and http://bugs.wmflabs.org/show_bug.cgi?id=49653
Things worth testing (or data worth recovering from the past if possible) for their effect on session length: * collapsing sections by default, providing easy "expand all" features etc.; * collapsing/hiding other content (and maybe interface elements too); * forceful redirects by device (e.g. did the forceful redirect for tablets increase session length on tablets?); * other major alterations AKA hacks compared to desktop.
Then the specific subcases can be switched to the experience which proved most immersive.
Nemo
On Tue, Oct 13, 2015 at 10:14 PM, Jon Katz jkatz@wikimedia.org wrote:
Do we try to identify reasons users might be skipping out earlier on mobile and fixing those?
Somewhat related: is this consistent across geos? Seems like that would help us say if this is an artifact of slow load speed on mobile in much of the world.
Luis
Great question, Luis, and something we should dig into. I'll add it to the list of inquiries we should be making.
Regarding sessions and 'reasons to come back' like notifications, I agree with what has been said, and also think it will take a lot of work for us to get right (and not annoy nice people)! Looking forward to exploring this soon.
-J
On Wed, Oct 14, 2015 at 10:36 AM, Luis Villa lvilla@wikimedia.org wrote:
On Tue, Oct 13, 2015 at 10:14 PM, Jon Katz jkatz@wikimedia.org wrote:
Do we try to identify reasons users might be skipping out earlier on mobile and fixing those?
Somewhat related: is this consistent across geos? Seems like that would help us say if this is an artifact of slow load speed on mobile in much of the world.
Luis
-- Luis Villa Sr. Director of Community Engagement Wikimedia Foundation *Working towards a world in which every single human being can freely share in the sum of all knowledge.*
By the way, what about unique visitors (compared to number of sessions and number of pageviews)? And can we track unique visitors across devices?
Pine On Oct 14, 2015 11:04 AM, "Jon Katz" jkatz@wikimedia.org wrote:
Great question, Luis, and something we should dig into. I'll add it to the list of inquiries we should be making.
Regarding sessions and 'reasons to come back' like notifications, I agree with what has been said, and also think it will take a lot of work for us to get right (and not annoy nice people)! Looking forward to exploring this soon.
-J
On Wed, Oct 14, 2015 at 10:36 AM, Luis Villa lvilla@wikimedia.org wrote:
On Tue, Oct 13, 2015 at 10:14 PM, Jon Katz jkatz@wikimedia.org wrote:
Do we try to identify reasons users might be skipping out earlier on mobile and fixing those?
Somewhat related: is this consistent across geos? Seems like that would help us say if this is an artifact of slow load speed on mobile in much of the world.
Luis
-- Luis Villa Sr. Director of Community Engagement Wikimedia Foundation *Working towards a world in which every single human being can freely share in the sum of all knowledge.*
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Yay, data! Thanks for putting this together, Jon.
On Tue, Oct 13, 2015 at 10:14 PM, Jon Katz jkatz@wikimedia.org wrote:
I don't know if we can explain all of our traffic decreases to the drop in session length, but it is certainly a big factor. Basically 60% of our pageviews (internal) shrink by 33% on mobile. So all else being equal, if we transfer all our traffic to mobile we lose 33% of our pageviews. Right now we're at 50%. This assumes that there is no change in numbers of sessions...on which we have no data right now.
We would only lose 20% (33% of 60%). But that is a completely unjustified assumption. (Also, there is another unjustified assumption that every mobile user is an ex-desktop user, ie. that this really is a shift in the habit of an existing userbase, not Wikipedia becoming less attractive to the old userbase and attractive to a new one. While it is not strictly relevant to this analysis whether the user disappearing on desktop and the one appearing on mobile is the same one, it is very relevant to how we interpret it.)
We don't have to guess the session numbers, though - your analysis assumes that non-internal pageviews start a session and internal ones continue it, in which case the number of sessions is simply the number of non-internal pageviews. That is, we get desktop: 510M pageviews / 335M sessions -> 275M pageviews / 175M sessions mobile: 150M pageviews / 100M sessions -> 230M pageviews / 175M sessions total: 660M pageviews / 435M sessions -> 505M pageviews / 350M sessions between the two ends of the graph (numbers are vague; I just looked at the image and guessed averages). That's a 20% decrease in sessions vs. the 25% decrease in page views, so it shouldn't cause much change in how worried we are about the drop.
Actually I did a study of this a while back using session reconstruction analysis, which doesn't depend on the referer field.
(Jon, if your team isn't familiar with it, I'm happy to talk it through with you (it works and is peer-reviewed!))
While I can't comment on the specific numbers here, the broad pattern we saw was:
1. Mobile sessions produce fewer pageviews and last less time; 2. Mobile users have more sessions, but; 3. Not enough to offset the fewer pageviews.
So, the pattern of mobile dropping pageviews is absolutely expected. I'd suggest performing some of this sort of analysis too just so we have a banded box rather than a single datapoint; "approach A produces X outcome, approach B produces Y outcome, so the answer is probably somewhere between the two" is always preferable. Happy to provide assistance with thinking through the analysis here and providing the tools you'd need to do it.
On 14 October 2015 at 19:29, Gergo Tisza gtisza@wikimedia.org wrote:
Yay, data! Thanks for putting this together, Jon.
On Tue, Oct 13, 2015 at 10:14 PM, Jon Katz jkatz@wikimedia.org wrote:
I don't know if we can explain all of our traffic decreases to the drop in session length, but it is certainly a big factor. Basically 60% of our pageviews (internal) shrink by 33% on mobile. So all else being equal, if we transfer all our traffic to mobile we lose 33% of our pageviews. Right now we're at 50%. This assumes that there is no change in numbers of sessions...on which we have no data right now.
We would only lose 20% (33% of 60%). But that is a completely unjustified assumption. (Also, there is another unjustified assumption that every mobile user is an ex-desktop user, ie. that this really is a shift in the habit of an existing userbase, not Wikipedia becoming less attractive to the old userbase and attractive to a new one. While it is not strictly relevant to this analysis whether the user disappearing on desktop and the one appearing on mobile is the same one, it is very relevant to how we interpret it.)
We don't have to guess the session numbers, though - your analysis assumes that non-internal pageviews start a session and internal ones continue it, in which case the number of sessions is simply the number of non-internal pageviews. That is, we get desktop: 510M pageviews / 335M sessions -> 275M pageviews / 175M sessions mobile: 150M pageviews / 100M sessions -> 230M pageviews / 175M sessions total: 660M pageviews / 435M sessions -> 505M pageviews / 350M sessions between the two ends of the graph (numbers are vague; I just looked at the image and guessed averages). That's a 20% decrease in sessions vs. the 25% decrease in page views, so it shouldn't cause much change in how worried we are about the drop.
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
