Hi Lila,
Great questions. My responses inline in green. 
Best,

Ja

On Wed, Jan 21, 2015 at 12:42 PM, Lila Tretikov <lila@wikimedia.org> wrote:
Jon, 

What are you testing for? What is the success criteria?

Given that this is a pilot, I would like to quantify initial success in terms of what we learn.  Given that we are not likely to have a long period of time before April 1 with a live feature, here are the draft measurement goals for this feature in Q3:
  • Begin collecting baseline usage statistics (creating, adding, consuming) and characterizing use cases
  • Decide on further development on this feature based on initial baseline stats on:
    • number of creators (as compared to mobile web editors and desktop page creators)
    • user class of creators (are we reaching outside the power editor group?)
    • number of shares
    • pageview traffic to content viewed via shares


If by "success" you mean: whether or not we move forward, I appreciate the ask. I think that we will need further develop an understanding of what we can build this quarter and I will need to look at our baselines for comparable features (if thats possible), before we can define a reasonable baseline. You can certainly hold me to having rough expectations prior to launch. Any thoughts you have as to what your minimum expectations would be for key metrics are welcome.

Your numbers are statistically significant, so I am not convinced that total entry numbers are an issue at this point. This depends on what the test is meant to examine, but so far I think you have enough total uniques to do analysis.

The numbers of people who click on the entry point are, as you pointed out, large enough to run tests for that action.   However, given that this is a funnel with a few steps, rather than a single action, the number of users creating collections will be dramatically smaller.  We have <50k logged-in users per month who click the star and will see "collections" as an option.  If .3% finish a collection (using the same conversion metric we have for star-->account creation), then we only have <150 users across wikis finishing a collection in a month. That number would be hard to draw conclusions from, especially as we don't want to wait 30 days to analyze results.  Let me know if you disagree or want to discuss further. 
 
L

On Thu, Jan 15, 2015 at 3:38 PM, Jon Katz <jkatz@wikimedia.org> wrote:
+ product, analytics as fyi

On Thu, Jan 15, 2015 at 3:27 PM, Jon Katz <jkatz@wikimedia.org> wrote:
Hi Folks,
Just sharing some data gleaned from the logs that I pulled in relation to the collections project.  I looked at clicks on the watchlist star (see image), because we are considering that to be the first entry point for creating a collection from an article on mobile web.   Nothing earth-shattering, but the data suggests that the pilot will not generate enough data without:
  • a tutorial element/signage
  • improved account creation funnel
  • additional entry points (such as search results)
-J


Inline image 1Inline image 4

Summary:  
  • Data suggests additional entry points might be necessary in order to derive meaningful results from the pilot.
    • Current expected user population reaching collections entry point without further intervention 2.4M per month.  Of those, 28.6k are logged in and able to access the feature funnel. 
    • Another 8.5k users end up signing up as a result of the prompt (.3% conversion rate). 
  • Baseline established to measure interventions against. 
  • More baselines to come.

Results
  • in a 30 day period in April, ~28.6k logged in users clicked on watchlist star (2.4M anonymous clicked, prompted to create account)
  • in a 30 day period in June*, ~8.5k of the previously logged-out users successfully created accounts after clicking on watchlist star (.3% conversion rate). I do not know if these should be counted in the 28.6k logged in "watch" events.
  • Logins that resulted from CTA not tracked, but assumed to be low and might trigger the measured events we did track.
  • Chart view, clicks on watchlist star (see methods>assumptions, below, for anonymous user calc):
Inline image 2

How this informs Collections:
  • If nothing changes, ~2.4M people have potential to see watchlist creation prompt, of whom 28.6k are immediately eligible to create (don't need login/account?). I think this latter number is a little low for a pilot.  You?  Should we explore other entry points for pilot, like search results?
  • We have a baseline of ~81.5k watchlist clicks per month by logged in users and 2.8M from loggedout. How much can we increase these numbers?  I don't think this number needs to grow significantly for success, because 1 collection could drive infinite views, but it would definitely help
    • Any overlay or product introduction will need to increase this number to be successful
    • Changing the icon from a star to a heart or some thing else more collection-y would also have to increase this number to be succesful
  • Account creation rate has room to grow
    • The CTA has a .3% conversion rate, which is very very low.  If we can boost this, we might be able to have a large impact on the number of people who are eligible to create.  
    •  Can we drive this number up by offering collections as an incentive?  
Next Steps
  • We need to explore alternate entry points - such as search, as we continue to build out the MVP.
  • I think we should measure the following baselines to establish what we should measure against and set realistic expectations:
    • Book creations over time (to compare to collection creations)
    • Edits on mobile (to compare to collection creations/edits)
    • Watchlist menu clicks (similar path to editing your collections)
    • Referrals via social (to establish baseline for social/share traffic)
    • Any others?
Methods:
  • Queries used:
    • Queried MobileBetaWatchlist_5281061 for 30 days--tracks clicks on star (April 2014).  Link to query.
    • Queried ServerSideAccountCreation_5487345 for 30 days in (June 2014) Link to query.
  • Assumptions
    • Assume unique [IP+UserAgent] = unique anonymous user.  Definitely undercounts users this way.
    • Assume number of users who login as a result of clicking on watchlist is small (not currently measured) and/or succesful logins from this prompt sucesfully add article to watchlist and trigger event that we did measure. So the above count might be underestimating total events.
    • Assuming addition is appropriate between the logged in watchlist clicks and the account creations. I ran a join to see if a succesful account creation then triggered a successful version of the original watchlist event (i.e. anon click to watch article-->account signup prompt-->create account-->article watch click triggered), but it kept stalling out. This is the query, in case you're curious.
  • Data alert
    • I had a rough time with some queries because the event logs had events from Jan- Dec, but there were obvious dropoffs and spikes due to changes that were only caught when I ran a daily report for events in each table.  Without this the data would have been very very skewed (my original count for anon watchlist clicks was 57k per month...it is actually 2.4M).  Here is a link to a helpful daily event query.

*necessary to use different time periods because the logs did not cleanly overlap

 


_______________________________________________
Wmfproduct mailing list
Wmfproduct@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wmfproduct