+ product, analytics as fyi

On Thu, Jan 15, 2015 at 3:27 PM, Jon Katz <jkatz@wikimedia.org> wrote:

Hi Folks,

Just sharing some data gleaned from the logs that I pulled in relation to the collections project.  I looked at clicks on the watchlist star (see image), because we are considering that to be the first entry point for creating a collection from an article on mobile web.   Nothing earth-shattering, but the data suggests that the pilot will not generate enough data without:

• a tutorial element/signage
• improved account creation funnel
• additional entry points (such as search results)

-J

Summary:  

• Data suggests additional entry points might be necessary in order to derive meaningful results from the pilot.
• Current expected user population reaching collections entry point without further intervention 2.4M per month.  Of those, 28.6k are logged in and able to access the feature funnel. 
• Another 8.5k users end up signing up as a result of the prompt (.3% conversion rate). 
• Baseline established to measure interventions against. 
• More baselines to come.

Results

• in a 30 day period in April, ~28.6k logged in users clicked on watchlist star (2.4M anonymous clicked, prompted to create account)
• in a 30 day period in June*, ~8.5k of the previously logged-out users successfully created accounts after clicking on watchlist star (.3% conversion rate). I do not know if these should be counted in the 28.6k logged in "watch" events.
• Logins that resulted from CTA not tracked, but assumed to be low and might trigger the measured events we did track.
• Chart view, clicks on watchlist star (see methods>assumptions, below, for anonymous user calc):

How this informs Collections:

• If nothing changes, ~2.4M people have potential to see watchlist creation prompt, of whom 28.6k are immediately eligible to create (don't need login/account?). I think this latter number is a little low for a pilot.  You?  Should we explore other entry points for pilot, like search results?
• We have a baseline of ~81.5k watchlist clicks per month by logged in users and 2.8M from loggedout. How much can we increase these numbers?  I don't think this number needs to grow significantly for success, because 1 collection could drive infinite views, but it would definitely help
• Any overlay or product introduction will need to increase this number to be successful
• Changing the icon from a star to a heart or some thing else more collection-y would also have to increase this number to be succesful
• Account creation rate has room to grow
• The CTA has a .3% conversion rate, which is very very low.  If we can boost this, we might be able to have a large impact on the number of people who are eligible to create.  
•  Can we drive this number up by offering collections as an incentive?  

Next Steps

• We need to explore alternate entry points - such as search, as we continue to build out the MVP.
• I think we should measure the following baselines to establish what we should measure against and set realistic expectations:
• Book creations over time (to compare to collection creations)
• Edits on mobile (to compare to collection creations/edits)
• Watchlist menu clicks (similar path to editing your collections)
• Referrals via social (to establish baseline for social/share traffic)
• Any others?

Methods:

• Queries used:
• Queried MobileBetaWatchlist_5281061 for 30 days--tracks clicks on star (April 2014).  Link to query.
• Queried ServerSideAccountCreation_5487345 for 30 days in (June 2014) Link to query.
• Assumptions
• Assume unique [IP+UserAgent] = unique anonymous user.  Definitely undercounts users this way.
• Assume number of users who login as a result of clicking on watchlist is small (not currently measured) and/or succesful logins from this prompt sucesfully add article to watchlist and trigger event that we did measure. So the above count might be underestimating total events.
  
• Assuming addition is appropriate between the logged in watchlist clicks and the account creations. I ran a join to see if a succesful account creation then triggered a successful version of the original watchlist event (i.e. anon click to watch article-->account signup prompt-->create account-->article watch click triggered), but it kept stalling out. This is the query, in case you're curious.
• Data alert
• I had a rough time with some queries because the event logs had events from Jan- Dec, but there were obvious dropoffs and spikes due to changes that were only caught when I ran a daily report for events in each table.  Without this the data would have been very very skewed (my original count for anon watchlist clicks was 57k per month...it is actually 2.4M).  Here is a link to a helpful daily event query.

*necessary to use different time periods because the logs did not cleanly overlap

 

_______________________________________________
Wmfproduct mailing list
Wmfproduct@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wmfproduct