On Thu, Mar 25, 2010 at 9:01 PM, Gregory Maxwell <gmaxwell@gmail.com> wrote:

On Thu, Mar 25, 2010 at 11:48 PM, Naoko Komura <nkomura@wikimedia.org> wrote:
> Hello.
>
> The usability beta has been tried out by over 570,000 users across all
> Wikimedia projects and roughly 80% of those beta users continue using it
> [1].

Hello Naoko. I believe the 80% number is failing to account for users
which have responded to the beta by logging out and simply not logging
back in.

While I do not intend to suggest that there is anything wrong with the
beta— since non-logged-in users were solicited and most people have no
cause to log into the wikimedia projects, I expect any metric
generated by counting the preference would be incredibly inflated and
80% may, in fact be a very low number indicating a serious problem.
To the extent that this is worth measuring, it's probably worth
measuring correctly.

Hello, Gregory.

I agree that our data is not perfect. We had a similar concern back in October, so we asked Erik Zachte to provide us the stats on newly created accounts. We could not find any significant activities in account creation, so we did not dig deeper.

This is the chart of newly created accounts for English Wikipedia last year. The beta started on August 6. Please ignore the October column, as it is incomplete due to the cut-off date, which happened before the end of the month.

http://usability.wikimedia.org/wiki/File:Newly_created_accounts_from_Jan_to_Oct_2009.png

Although there was an increase in account creation for roughly 20,000 accounts in the month of August and September, this increase does not look significant compare to the account creation increase earlier in 2009.

I also would like to supplement that average retention rate is cumulative.[1]
http://en.wikipedia.org/w/index.php?title=Special:PrefStats/skin&inc=672

[1] If you look at the retention rate in daily or weekly span, the opt-out rate is a lot higher. Opting out happens significantly within the first four weeks even within the day. However, the monthly retention rate does not show significant variance from culumative retention rate. http://biturl.cc/Jpr (Second sheet)

While there is no concrete evidence that users who created new accounts in order to try out the beta actually opt-out or just log out, the number who opted out from the beta is close to 40,000 users in the last four weeks (the number here is for English Wikipedia only). Even If we make an extreme assumption that the account creation increase (20,000 observed in August and September in 2009) was due to the beta trial, there is another 20,000 beta users who actually opt-out even under this extreme scenario.

So although the most recent account creation stats and Commons specific statistics for the account creation are needed, I am inclined to say that explict opting out from the beta is significant enough that beta users who created accounts just to try out the beta and stopped using it by logging out instead of opting out, does not require significant attention. I agree we can make the data more precise if we tracked the IP or http sesion data. But as this survey is meant for the short term, I hope the rough analysis is acceptable.

Thank you for the great question. My brain is twisted and exhausted now. :-)

- Naoko