+ product, analytics as fyi
On Thu, Jan 15, 2015 at 3:27 PM, Jon Katz <jkatz(a)wikimedia.org> wrote:
Hi Folks,
Just sharing some data gleaned from the logs that I pulled in relation to
the collections project. I looked at clicks on the watchlist star (see
image), because we are considering that to be the first entry point for
creating a collection from an article on mobile web. Nothing
earth-shattering, but the data suggests that the pilot will not generate
enough data without:
- a tutorial element/signage
- improved account creation funnel
- additional entry points (such as search results)
-J
[image: Inline image 1][image: Inline image 4]
*Summary: *
- Data suggests additional entry points might be necessary in order to
derive meaningful results from the pilot.
- Current expected user population reaching collections entry point
without further intervention 2.4M per month. Of those, 28.6k are logged in
and able to access the feature funnel.
- Another 8.5k users end up signing up as a result of the prompt
(.3% conversion rate).
- Baseline established to measure interventions against.
- More baselines to come.
*Results*
- in a 30 day period in April, ~28.6k logged in users clicked on
watchlist star (2.4M anonymous clicked, prompted to create account)
- in a 30 day period in June*, ~8.5k of the previously logged-out
users successfully created accounts after clicking on watchlist star (.3%
conversion rate). I do not know if these should be counted in the 28.6k
logged in "watch" events.
- Logins that resulted from CTA not tracked, but assumed to be low and
might trigger the measured events we did track.
- Chart view, clicks on watchlist star (see methods>assumptions,
below, for anonymous user calc):
[image: Inline image 2]
*How this informs Collections:*
- If nothing changes, ~2.4M people have potential to see watchlist
creation prompt, of whom 28.6k are immediately eligible to create (don't
need login/account?). I think this latter number is a little low for a
pilot. You? Should we explore other entry points for pilot, like search
results?
- We have a baseline of ~81.5k watchlist clicks per month by logged in
users and 2.8M from loggedout. How much can we increase these numbers? I
don't think this number *needs* to grow significantly for success,
because 1 collection could drive infinite views, but it would definitely
help
- Any overlay or product introduction will need to increase this
number to be successful
- Changing the icon from a star to a heart or some thing else more
collection-y would also have to increase this number to be succesful
- Account creation rate has room to grow
- The CTA has a .3% conversion rate, which is very very low. If we
can boost this, we might be able to have a large impact on the number of
people who are eligible to create.
- Can we drive this number up by offering collections as an
incentive?
*Next Steps*
- We need to explore alternate entry points - such as search, as we
continue to build out the MVP.
- I think we should measure the following baselines to establish what
we should measure against and set realistic expectations:
- Book creations over time (to compare to collection creations)
- Edits on mobile (to compare to collection creations/edits)
- Watchlist menu clicks (similar path to editing your collections)
- Referrals via social (to establish baseline for social/share
traffic)
- Any others?
*Methods:*
- Queries used:
- Queried MobileBetaWatchlist_5281061 for 30 days--tracks clicks on
star (April 2014). Link to query
<https://docs.google.com/a/wikimedia.org/document/d/1-bfW9H2kApvOOOTKL40TjYKH0eO7dUeErpg0dNW554I/edit#heading=h.eqhhm1kvdn6e>
.
- Queried ServerSideAccountCreation_5487345 for 30 days in (June
2014) Link to query
<https://docs.google.com/a/wikimedia.org/document/d/1-bfW9H2kApvOOOTKL40TjYKH0eO7dUeErpg0dNW554I/edit#heading=h.d11qmgmu7pqb>
.
- Assumptions
- Assume unique [IP+UserAgent] = unique anonymous user. Definitely
undercounts users this way.
- Assume number of users who login as a result of clicking on
watchlist is small (not currently measured) and/or succesful logins from
this prompt sucesfully add article to watchlist and trigger event that we
*did* measure. So the above count might be underestimating total
events.
- Assuming addition is appropriate between the logged in watchlist
clicks and the account creations. I ran a join to see if a succesful
account creation then triggered a successful version of the original
watchlist event (i.e. anon click to watch article-->account signup
prompt-->create account-->article watch click triggered), but it kept
stalling out. This is the query, in case you're curious
<https://docs.google.com/a/wikimedia.org/document/d/1-bfW9H2kApvOOOTKL40TjYKH0eO7dUeErpg0dNW554I/edit#heading=h.eg0uxzfv84qe>
.
- Data alert
- I had a rough time with some queries because the event logs had
events from Jan- Dec, but there were obvious dropoffs and spikes due to
changes that were only caught when I ran a daily report for events in each
table. Without this the data would have been very very skewed (my original
count for anon watchlist clicks was 57k per month...it is actually 2.4M). Here
is a link
<https://docs.google.com/a/wikimedia.org/document/d/1-bfW9H2kApvOOOTKL40TjYKH0eO7dUeErpg0dNW554I/edit#heading=h.d0ywz6oufh2m>
to a helpful daily event query.
*necessary to use different time periods because the logs did not cleanly
overlap