Yeah, I think there are a lot of problems with the opt-in beta model. I
much prefer releasing new features to a small % of users, logging
events/usage, and if we suspect something has the potential to be
disruptive/offputting, letting them know the feature they're seeing is beta
and letting them turn it off.
That said, beta is still useful for sandboxing new features and in-person
user testing, so I don't think we should kill it altogether. I just think we
need to supplement it with a graduated release model – which we're already
doing with stuff like WikiGrok :)
On Tue, Jan 27, 2015 at 11:10 AM, Sam Smith <samsmith(a)wikimedia.org>
wrote:
Have we thought about automatically opting people into beta mode e.g.
a sample of our users in a certain geographic region / certain zero
enabled area/ all users in a certain bucket based on their user id ?
I like this idea. In fact, I'm for it, provided that we make it clear to
the user that they've been entered into an experiment and they're seeing
non-standard UI.
How many users could beta actually handle?
Not sure. But, interestingly, we can find out by bucketing users and
slowly assigning them the beta variant.
Is this technically possible?
Yes. If we're generating and storing tokens on the client, which we do
for anonymous users in other experiments, then we can enter anonymous users
into the experiment at the cost of a little control over how tokens are
stored.
If someone was bucketed into beta would they be
able to opt out into
stable again under any of the above situations?
See my first inline response. We must make it clear to the user that
they're seeing a variant of an experiment… and we must make it simple to opt
out of the experiment.
Also, all instrumentation for beta features will need to be augmented
with a is_beta_opt_in flag.
–Sam
On Tue, Jan 27, 2015 at 10:18 AM, Jon Robson <jrobson(a)wikimedia.org>
wrote:
>
> One of the frustrations I have heard so far is that the audience there
> is too small to get meaningful data around various experiments.
> Currently people have to opt in by going to
>
https://en.m.wikipedia.org/wiki/Special:MobileOptions which is hidden
> away in the mobile interface. They can do this whilst anonymous or
> logged in.
Have we thought about automatically opting people into beta mode e.g.
a sample of our users in a certain geographic region / certain zero
enabled area/ all users in a certain bucket based on their user id ?
>
How many users could beta actually handle?
Is this technically possible?
If someone was bucketed into beta would they be able
to opt out into
stable again under any of the above situations?
>
> Jon
>
> _______________________________________________
> Mobile-l mailing list
> Mobile-l(a)lists.wikimedia.org
>
https://lists.wikimedia.org/mailman/listinfo/mobile-l
_______________________________________________
Mobile-l mailing list
Mobile-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l
_______________________________________________
Mobile-l mailing list
Mobile-l(a)lists.wikimedia.org