On Thu, Mar 25, 2010 at 9:01 PM, Gregory Maxwell <gmaxwell(a)gmail.com> wrote:
On Thu, Mar 25, 2010 at 11:48 PM, Naoko Komura
<nkomura(a)wikimedia.org>
wrote:
Hello.
The usability beta has been tried out by over 570,000 users across all
Wikimedia projects and roughly 80% of those beta users continue using it
[1].
Hello Naoko. I believe the 80% number is failing to account for users
which have responded to the beta by logging out and simply not logging
back in.
While I do not intend to suggest that there is anything wrong with the
beta— since non-logged-in users were solicited and most people have no
cause to log into the wikimedia projects, I expect any metric
generated by counting the preference would be incredibly inflated and
80% may, in fact be a very low number indicating a serious problem.
To the extent that this is worth measuring, it's probably worth
measuring correctly.
Hello, Gregory.
I agree that our data is not perfect. We had a similar concern back in
October, so we asked Erik Zachte to provide us the stats on newly created
accounts. We could not find any significant activities in account creation,
so we did not dig deeper.
This is the chart of newly created accounts for English Wikipedia last
year. The beta started on August 6. Please ignore the October column, as
it is incomplete due to the cut-off date, which happened before the end of
the month.
http://usability.wikimedia.org/wiki/File:Newly_created_accounts_from_Jan_to…
Although there was an increase in account creation for roughly 20,000
accounts in the month of August and September, this increase does not look
significant compare to the account creation increase earlier in 2009.
I also would like to supplement that average retention rate is
cumulative.[1]
http://en.wikipedia.org/w/index.php?title=Special:PrefStats/skin&inc=672
[1] If you look at the retention rate in daily or weekly span, the opt-out
rate is a lot higher. Opting out happens significantly within the first four
weeks even within the day. However, the monthly retention rate does not
show significant variance from culumative retention rate.
http://biturl.cc/Jpr (Second sheet)
While there is no concrete evidence that users who created new accounts in
order to try out the beta actually opt-out or just log out, the number who
opted out from the beta is close to 40,000 users in the last four weeks (the
number here is for English Wikipedia only). Even If we make an extreme
assumption that the account creation increase (20,000 observed in August and
September in 2009) was due to the beta trial, there is another 20,000 beta
users who actually opt-out even under this extreme scenario.
So although the most recent account creation stats and Commons specific
statistics for the account creation are needed, I am inclined to say that
explict opting out from the beta is significant enough that beta users who
created accounts just to try out the beta and stopped using it by logging
out instead of opting out, does not require significant attention. I agree
we can make the data more precise if we tracked the IP or http sesion data.
But as this survey is meant for the short term, I hope the rough analysis is
acceptable.
Thank you for the great question. My brain is twisted and exhausted now.
:-)
- Naoko
--
Support Free Knowledge:
http://wikimediafoundation.org/wiki/Donate
