Tablet pageview data pre and post redirect

List overview All Threads
Download

newer

older

Re: [WikimediaMobile] More Menu...

Re: [WikimediaMobile] [zero]...

Maryana Pinchuk

20 Jun 2014 20 Jun '14

7:33 p.m.

Good news, everyone! Oliver ran some stats for us on where tablet users were landing (desktop view or mobile view) a few days before and after the tablet redirect on Tuesday: Before the redirect, *13% of pageviews from tablets were hitting the mobile site*, meaning a non-trivial chunk of users had opted into the mobile experience even before we were serving it to them as the default. After the redirect, *95% of pageviews from tablets were hitting the mobile site*, meaning a fairly small number of users are opting out of the mobile experience we're serving – smaller than the number of people who were opting out of the desktop experience before the redirect :) (Note that the total pageview numbers haven't changed significantly – the slightly downward trendline in the past few days just reflects the normal dip we observe toward the end of every week.) So, tl;dr, it looks like we're serving the needs of the majority of our users. Graph below, courtesy of our beloved British data analyst/road warrior. We'll continue to monitor these numbers, of course, but the early results look good – nice work, team! [image: Inline image 1] -- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org

Attachments:

attachment.htm (text/html — 1.6 KB)
2014-06-12_to_2014-06-18.png (image/png — 308.0 KB)

Show replies by date

Tomasz Finc

20 Jun 20 Jun

7:52 p.m.

Great to see the positive quantitate response. Maryana, what's the high level of view of how your thinking about moving forward? --tomasz On Fri, Jun 20, 2014 at 10:33 AM, Maryana Pinchuk <mpinchuk(a)wikimedia.org> wrote:

...

Maryana Pinchuk

21 Jun 21 Jun

12:29 a.m.

On Fri, Jun 20, 2014 at 10:52 AM, Tomasz Finc <tfinc(a)wikimedia.org> wrote:

...

Great to see the positive quantitate response. Maryana, what's the high level of view of how your thinking about moving forward?

I think we'll want to let the changes simmer for another week or two and then dig into a few specific things: 1) Do the pageview stats hold? e.g., are we still getting relatively few opt-outs? 2) How has this change impacted new mobile user numbers (registrations, 1st time editors, active editors, etc.)? Early results[1] show a healthy spike in new signups coming from mobile in the past few days, and we need to keep track of these users to make sure they're going through the new and active editor funnels at a reasonable rate. 3) How does the heatmap of taps look? e.g., how often are people accessing hamburger menu items, language selection, history, etc. Based on the data from the above, we could do some UX optimization – the designers are very keen to refine the UI for page actions (instead of dumping features like language selection and page history at the top/bottom of the page kind of at random). If the new and active editor conversion rates are low, we'll need to prioritize work on new contribution funnels, but realistically that's probably not something we'll be able to tackle until Q2 of next fiscal. 1. http://ee-dashboard.wmflabs.org/graphs/enwiki_daily_reg_mobile

...

--tomasz On Fri, Jun 20, 2014 at 10:33 AM, Maryana Pinchuk <mpinchuk(a)wikimedia.org> wrote:

-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org

Federico Leva (Nemo)

23 Jun 23 Jun

10:57 a.m.

Maryana Pinchuk, 21/06/2014 00:29:

...

2) How has this change impacted [...] 1st time editors, active editors, etc.)? [...] we need to keep track of these users to make sure they're going through the new and active editor funnels at a reasonable rate.

Nice! I think this also needs to be reflected (at a very high level of course) in the yearly goals: https://www.mediawiki.org/w/index.php?title=Talk%3AWikimedia_Engineering%2F… Nemo

Maryana Pinchuk

6:48 p.m.

...

On Jun 23, 2014, at 1:57 AM, "Federico Leva (Nemo)" <nemowiki(a)gmail.com> wrote: Maryana Pinchuk, 21/06/2014 00:29:

Nice! I think this also needs to be reflected (at a very high level of course) in the yearly goals: https://www.mediawiki.org/w/index.php?title=Talk%3AWikimedia_Engineering%2F…

It is :) The quarterly quantitative targets in that draft doc mention maintaining and growing "new mobile active editors" -- this metric represents new users registering on mobile and then going on to hit the active editor threshold (5+ edits per month) on either mobile or desktop. I'll make that more explicit in the doc in case it's not clear.

Tomasz Finc

7:41 p.m.

On Fri, Jun 20, 2014 at 3:29 PM, Maryana Pinchuk <mpinchuk(a)wikimedia.org> wrote:

...

I think we'll want to let the changes simmer for another week or two and then dig into a few specific things:

Yes. We really need more data to make future decisions.

...

1) Do the pageview stats hold? e.g., are we still getting relatively few opt-outs?

Do we have this in LIMN ?

...

2) How has this change impacted new mobile user numbers (registrations, 1st time editors, active editors, etc.)? Early results[1] show a healthy spike in new signups coming from mobile in the past few days, and we need to keep track of these users to make sure they're going through the new and active editor funnels at a reasonable rate.

Great. I tried to find any other metrics on the ee dashboard but I just got a white page. Do you have a better link for us? I looked on the mobile report card and found this which just can't be right http://mobile-reportcard.wmflabs.org/graphs/successful-edits-main

...

3) How does the heatmap of taps look? e.g., how often are people accessing hamburger menu items, language selection, history, etc.

Eager to see the write ups of this. Based on the data from the above, we could do some UX optimization – the

...

designers are very keen to refine the UI for page actions (instead of dumping features like language selection and page history at the top/bottom of the page kind of at random). If the new and active editor conversion rates are low, we'll need to prioritize work on new contribution funnels, but realistically that's probably not something we'll be able to tackle until Q2 of next fiscal.

That makes sense. Great work thus far. --tomasz

Oliver Keyes

7:47 p.m.

On 23 June 2014 10:41, Tomasz Finc <tfinc(a)wikimedia.org> wrote:

...

On Fri, Jun 20, 2014 at 3:29 PM, Maryana Pinchuk <mpinchuk(a)wikimedia.org> wrote:

I think we'll want to let the changes simmer for another week or two and then dig into a few specific things:

Yes. We really need more data to make future decisions.

1) Do the pageview stats hold? e.g., are we still getting relatively few opt-outs?

Do we have this in LIMN ?

Nope, just static reports at the moment. Given the speedy nature of the request (Both figuratively - there was a narrow window to produce it - and literally, because I wrote most of the code while travelling through Oregon at 85 MP/H) I'm not tremendously confident in the ability of the code to indefinitely generate data.

...

3) How does the heatmap of taps look? e.g., how often are people accessing hamburger menu items, language selection, history, etc.

Eager to see the write ups of this. Based on the data from the above, we could do some UX optimization – the

That makes sense. Great work thus far. --tomasz

-- Oliver Keyes Research Analyst Wikimedia Foundation

Tomasz Finc

7:58 p.m.

On Mon, Jun 23, 2014 at 10:47 AM, Oliver Keyes <okeyes(a)wikimedia.org> wrote:

...

Dario Taraborelli

25 Jun 25 Jun

8:57 p.m.

Tomasz, one of the analytics goals for Q1-2015 is to deliver traffic metric definitions (primarily: pageviews, unique clients) and their breakdown by target site, device or device class and geography. We will keep monitoring page requests using the interim definitions Oliver applied to the sampled logs [1], but there’s more work that needs to be done to turn these into fully vetted, production-level reports generated from the unsampled logs. We’re currently turning the mobile analytics priorities discussed with Howie, Maryana and Dan into cards and we’ll share the list once it’s completed. Mobile is a focus area for Q1 with virtually two dedicated people from Research & Data supporting the team with traffic and contribution research. We’ll be also reinforcing our traffic crunching capacity with a new dedicated research position that we’ll be opening in Q1 and realistically we should expect to have onboard in Q2. Dario [1] https://trello.com/c/DCd58xGQ/334-daily-pv-from-sampled-logs On Jun 23, 2014, at 10:58 AM, Tomasz Finc <tfinc(a)wikimedia.org> wrote:

...

On Mon, Jun 23, 2014 at 10:47 AM, Oliver Keyes <okeyes(a)wikimedia.org> wrote:

That's fine. Maryana, it would be good to keep track of this over the quarter. Where would this sit on your priority list of analytics requests that need to have complete/scalable implementations ? if there is already a backlog of these then feel free to point me to it. --tomasz _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

Oliver Keyes

27 Jun 27 Jun

10:37 p.m.

...

On Mon, Jun 23, 2014 at 10:47 AM, Oliver Keyes <okeyes(a)wikimedia.org>

wrote:

> Nope, just static reports at the moment. Given the speedy nature of the > request (Both figuratively - there was a narrow window to produce it -

and

> literally, because I wrote most of the code while travelling through

Oregon

at 85 MP/H) I'm not tremendously confident in the ability of the code to indefinitely generate data

-- Oliver Keyes Research Analyst Wikimedia Foundation

Dario Taraborelli

11:26 p.m.

On the contribution side, the switchover also had a positive impact on new user activation [1]. iPad users who were previously signing up on the desktop site saw a significantly (and substantially) higher activation rate [2] as a result of the redirection, for most days we sampled after the switchover. It’s too early to tell if this change in new user engagement will persist (we probably had a large number of testers among the first signups) and we know that by design (anonymous edit restrictions, prominent CTAs) we should expect to see a higher conversion for editors on the mobile site, but these results are very encouraging: we’re seeing a much higher rate of new users to contribute to start editing Wikipedia in an environment that is more appropriate for tablet devices. Dario [1] measured as the proportion of newly registered users who complete at least 1 edit in their first 24 hours, English Wikipedia. [2] https://trello.com/c/LKVK7RfL/351-tablet-switchover-and-editor-activation On Jun 27, 2014, at 1:37 PM, Oliver Keyes <okeyes(a)wikimedia.org> wrote:

...

So; we have the second week of results. To set the stage: ISP caches will be cleared, so we might see desktop drop. Weekend traffic will show up, which means we might see the type of people accessing wikipedia change, and bring new preferences and demographics into play. It's been longer since the switchover, so people who dislike the change have had more chances to find the opt-out. It's the data wheel of fortune, and nobody knows where your spin will take you! The answer is "a good place". The weekend made absolutely no difference; it bumped the amount of desktop traffic by a tiny amount, compared to a big increase for mobile, which suggests not just that most people have happily switched over but that the people who have switched are our most frequent and active visitors. Day-on-day, we saw no significant increase in desktop opt-ins - in fact, a slight decrease from the (already tiny) 5-ish percent. Looks like being WP:BOLD and switching our tablet users to mobile in one fell swoop was a good decision, and our readers think so too :). On 25 June 2014 14:57, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote: Tomasz, one of the analytics goals for Q1-2015 is to deliver traffic metric definitions (primarily: pageviews, unique clients) and their breakdown by target site, device or device class and geography. We will keep monitoring page requests using the interim definitions Oliver applied to the sampled logs [1], but there’s more work that needs to be done to turn these into fully vetted, production-level reports generated from the unsampled logs. We’re currently turning the mobile analytics priorities discussed with Howie, Maryana and Dan into cards and we’ll share the list once it’s completed. Mobile is a focus area for Q1 with virtually two dedicated people from Research & Data supporting the team with traffic and contribution research. We’ll be also reinforcing our traffic crunching capacity with a new dedicated research position that we’ll be opening in Q1 and realistically we should expect to have onboard in Q2. Dario [1] https://trello.com/c/DCd58xGQ/334-daily-pv-from-sampled-logs On Jun 23, 2014, at 10:58 AM, Tomasz Finc <tfinc(a)wikimedia.org> wrote:

On Mon, Jun 23, 2014 at 10:47 AM, Oliver Keyes <okeyes(a)wikimedia.org> wrote:

-- Oliver Keyes Research Analyst Wikimedia Foundation <2014-06-13_to_2014-06-25.png>

Erik Moeller

11:56 p.m.

On Fri, Jun 27, 2014 at 2:26 PM, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote:

...

Very cool data, thanks Dario! Is it possible to quantify before/after total edit volume to account for the impact of the anonymous edit restriction? Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation

Dario Taraborelli

28 Jun 28 Jun

2 a.m.

On Jun 27, 2014, at 2:56 PM, Erik Moeller <erik(a)wikimedia.org> wrote:

...

On Fri, Jun 27, 2014 at 2:26 PM, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote:

Very cool data, thanks Dario! Is it possible to quantify before/after total edit volume to account for the impact of the anonymous edit restriction?

I am looking into overall edit volume as well. We should keep in mind that this will be a noisy measure (much less controlled than new user activation rates). Other hypotheses we’re formulating on the expected impact of the tablet switchover are listed here [1], feel free to add more. Dario [1] http://etherpad.wikimedia.org/p/TabletSwitchover-UserAcquisition

Dario Taraborelli

27 Jun 27 Jun

11:36 p.m.

On the contribution side, the switchover also had a positive impact on new user activation [1]. It’s too early to tell if this change in new user engagement will persist (we probably had a large number of testers among the first signups) and we know that by design (anonymous edit restrictions, prominent CTAs) we should expect to see a higher conversion for editors on the mobile site, but these results are very encouraging: we’re seeing a much higher rate of new users to contribute to start editing Wikipedia in an environment that is more appropriate for tablet devices. Dario [1] measured as the proportion of newly registered users who complete at least 1 edit in their first 24 hours, English Wikipedia. [2] https://trello.com/c/LKVK7RfL/351-tablet-switchover-and-editor-activation

Oliver Keyes

28 Jun 28 Jun

12:04 a.m.

New subject: Fwd: Tablet pageview data pre and post redirect

Forwarding to mobile-l, because it needs approval, and commonsifying the graphic - https://commons.wikimedia.org/wiki/File:Tablet_desktop_to_mobile_switchover… ---------- Forwarded message ---------- From: Oliver Keyes <okeyes(a)wikimedia.org> Date: 27 June 2014 16:37 Subject: Re: [WikimediaMobile] Tablet pageview data pre and post redirect To: Dario Taraborelli <dtaraborelli(a)wikimedia.org> Cc: Tomasz Finc <tfinc(a)wikimedia.org>rg>, mobile-l < mobile-l(a)lists.wikimedia.org> So; we have the second week of results. To set the stage: ISP caches will be cleared, so we might see desktop drop. Weekend traffic will show up, which means we might see the type of people accessing wikipedia change, and bring new preferences and demographics into play. It's been longer since the switchover, so people who dislike the change have had more chances to find the opt-out. It's the data wheel of fortune, and nobody knows where your spin will take you! The answer is "a good place". The weekend made absolutely no difference; it bumped the amount of desktop traffic by a tiny amount, compared to a big increase for mobile, which suggests not just that most people have happily switched over but that the people who have switched are our most frequent and active visitors. Day-on-day, we saw no significant increase in desktop opt-ins - in fact, a slight decrease from the (already tiny) 5-ish percent. Looks like being WP:BOLD and switching our tablet users to mobile in one fell swoop was a good decision, and our readers think so too :). On 25 June 2014 14:57, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote:

...

On Mon, Jun 23, 2014 at 10:47 AM, Oliver Keyes <okeyes(a)wikimedia.org>

wrote:

> Nope, just static reports at the moment. Given the speedy nature of the > request (Both figuratively - there was a narrow window to produce it -

and

> literally, because I wrote most of the code while travelling through

Oregon

at 85 MP/H) I'm not tremendously confident in the ability of the code to indefinitely generate data

-- Oliver Keyes Research Analyst Wikimedia Foundation -- Oliver Keyes Research Analyst Wikimedia Foundation

Tomasz Finc

8 Jul 8 Jul

4:55 a.m.

On Wed, Jun 25, 2014 at 11:57 AM, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote:

...

We’re currently turning the mobile analytics priorities discussed with Howie, Maryana and Dan into cards and we’ll share the list once it’s completed.

Thanks Dario. Do send out when its ready. --tomasz

Max Semenik

20 Jun 20 Jun

8:05 p.m.

I wonder how much of these numbers - tablets on mobile site before the switchover and on desktop site after it - are due to discrepancies in tablet detection between analytics and varnish. Varnishes used to detect them by matching user-agent against (iPad|Android.3|(?i)tablet|PlayBook|Wii) - how does analytics do that? On Fri, Jun 20, 2014 at 10:33 AM, Maryana Pinchuk <mpinchuk(a)wikimedia.org> wrote:

...

-- Best regards, Max Semenik ([[User:MaxSem]])

Oliver Keyes

8:55 p.m.

We use tobie's ua-parser, which should indeed hit a slightly wider range of tablets and would explain some of that 5%. We should probably have a different conversation with ops about trying to rebalance the varnish tablet identification (I've seen the request logs, and that won't catch all of the tablets). Re the drop - it's actually the mid-week drop (mobile devices, tablets included, gain a chunk of our user share on weekends). The script is set to loop every Friday, so there'll be additional data this time next week which should allow us to see if the percentages later as, e.g, ISP-side caching changes. On Friday, June 20, 2014, Max Semenik <maxsem.wiki(a)gmail.com> wrote:

...

-- Best regards, Max Semenik ([[User:MaxSem]])

-- Oliver Keyes Research Analyst Wikimedia Foundation

Max Semenik

9 p.m.

On Fri, Jun 20, 2014 at 11:55 AM, Oliver Keyes <okeyes(a)wikimedia.org> wrote:

...

We should probably have a different conversation with ops about trying to rebalance the varnish tablet identification (I've seen the request logs, and that won't catch all of the tablets).

Luckily, Varnish doesn't need to detect tablets anymore - it now handles all mobile devices equally.

Oliver Keyes

9:48 p.m.

But how are 'mobile devices.' Defined? (Let's split this thread off) On Friday, June 20, 2014, Max Semenik <maxsem.wiki(a)gmail.com> wrote:

...

On Fri, Jun 20, 2014 at 11:55 AM, Oliver Keyes <okeyes(a)wikimedia.org <javascript:_e(%7B%7D,'cvml','okeyes@wikimedia.org');>> wrote:

We should probably have a different conversation with ops about trying to rebalance the varnish tablet identification (I've seen the request logs, and that won't catch all of the tablets).

Luckily, Varnish doesn't need to detect tablets anymore - it now handles all mobile devices equally.

-- Oliver Keyes Research Analyst Wikimedia Foundation

Max Semenik

9:56 p.m.

On Fri, Jun 20, 2014 at 12:48 PM, Oliver Keyes <okeyes(a)wikimedia.org> wrote:

...

But how are 'mobile devices.' Defined? (Let's split this thread off)

http://git.wikimedia.org/blob/operations%2Fpuppet.git/7cd814686241a19ba5b4f… lines 24, 25 and 27. -- Best regards, Max Semenik ([[User:MaxSem]])

Erik Moeller

21 Jun 21 Jun

2:14 a.m.

I agree these early numbers are very promising. Thanks for sharing! :) Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation

3592

days inactive

3610

days old

mobile-l@lists.wikimedia.org

Manage subscription

21 comments

7 participants

tags (0)

participants (7)

Dario Taraborelli
Erik Moeller
Federico Leva (Nemo)
Maryana Pinchuk
Max Semenik
Oliver Keyes
Tomasz Finc