Hi People Interested in Gather,
Given the reorg and the traffic being driven to beta, we need to revisit
Gather's Q4 goals:
TLDR: Continue working on gather to finish MVP features (end of the month,
max), not pushing to stable unless we see 2x the number of logged-in edits
on beta (or >10x current state).
Before the serious stuff:
Top 5 best collections since Monday:
https://en.m.wikipedia.org/wiki/Special:Gather/by/Johnrenfro
https://en.m.wikipedia.org/wiki/Special:Gather/id/3454/Philadelphia_watch
https://en.m.wikipedia.org/wiki/Special:Gather/id/3401/engineering
https://en.m.wikipedia.org/wiki/Special:Gather/id/3532/Mental_Health
https://en.m.wikipedia.org/wiki/Special:Gather/id/3132/Libertarianismo
Okay, now the goals. Please feel free to comment in email or in this google
doc
<https://docs.google.com/a/wikimedia.org/document/d/1DK1pa3PEpIbiON0Bj6BrhGAnePqZcKSmqTW1z6sTKvU/edit?usp=sharing>
Thanks,
Jon
Given the reorg and an improvement made to beta, we need to revisit
Gather's goals
Pushing to Stable
Given that we can now test Gather numbers in beta, the original goal of
launching on stable to test adoption is no longer valid. It is expensive
in terms of future maintenance and commitments to launch features to
stable, so we only want to do so after we have proven success, if possible.
Target numbers
Originally, the benchmarks for success on stable that were agreed on were
low (10k creators a month on stable, and 1k shares). Given the current
beta numbers, it looks like we will blow the first number out of the
water. Share has been deprioritized given current usage patterns.
However, given that we now have 4 engineers in charge of the entire web,
the standards for what we work on have to be more rigorous. We cannot
allocate multiple engineers to an experimental product unless it shows
promise of impacting greater numbers
1. Goal
1.
New Goal:
1.
Round out Gather hypothesis (criteria below)
2.
By end of June know whether or not we want to push Gather to stable
1. Next Eng+PM Steps (to round out hypothesis)
1.
Improve onboarding (a few tasks)
2.
Surface collections publicly (this is a big missing feature)
3.
Qualitative and quantitative research
2. Criteria for passing to stable:
We don’t have a great way to measure success of Gather based on usage by a
proportion of users or logged in users. However, we can compare to
something similar like edits. In terms of pure value to WP, we can
consider a collection to be like a low-value edit.
-
There are 2x ‘good’ collections made as there are total logged-in
edits/month
-
in May there were 2,180 edits by logged in users (2,694 logged out).
-
At current rates this suggests ~4,500 collections per month is our
target.
-
‘good’ here, means >1 collection--it’s not a perfect definition, but
it’s a strong proxy.
-
Our current rate of ‘good’ collections is roughly 250 a month, so we
will need to increase the number by almost 20x.
-
It might be worth exploring % of those 2,180 edits that are reverted
and adjusting down accordingly
OR
-
Views of collections or where collection is the referrer > .5% of total
pageviews
-
Currently, the views of collections are minimal.
-
If very few people create collections, but they drive a significant
boost in page views, say .5% of total PVs (not an end goal, but also not
bad for an MVP introducing a new use case), then the feature is a success.
1. What if we don’t pass to stable:
Lets burn that bridge when we get to it :) . Seriously though, until we get
some qualitative data back from our readers, we will not be able to make
important calls on the feature as is. There are a few great alternatives I
can think of right off the bat:
-
Keep code in beta and work on Gather opportunistically or as qualitative
data dictates
-
One example might be to make collections private by default and
launch as bookmarker for readers (primary current use-case)
-
Promote as beta feature on desktop
-
Use codebase as start of multiple watchlists (some good work started
here by JRobson)
-
Codebase is fairly generic list table that has some basic and
interesting features built in that could be used for a number of other ends
1. Validation and why we choose this criteria:
Qualitative questions to answer:
Why aren't more people using Gather? (correctly)
Why aren't people returning to use Gather more
Success metric questions to answer (that we can’t already):
What % of logged in users use Gather
What % of users who visit > 1 page use Gather
What is our denominator
Status
Measure logins and signup funnel directed from Gather, what is % success
rate?
This is not instrumented
Measure % of sessions with more than 1 pageview
Working on this
Measure % of sessions with logged in users
Cannot get this
What is our baseline? Logged in edits is probably the best thing to
measure against.