One of the frustrations I have heard so far is that the audience there is too small to get meaningful data around various experiments. Currently people have to opt in by going to https://en.m.wikipedia.org/wiki/Special:MobileOptions which is hidden away in the mobile interface. They can do this whilst anonymous or logged in.
Have we thought about automatically opting people into beta mode e.g. a sample of our users in a certain geographic region / certain zero enabled area/ all users in a certain bucket based on their user id ?
How many users could beta actually handle? Is this technically possible? If someone was bucketed into beta would they be able to opt out into stable again under any of the above situations?
Jon
Have we thought about automatically opting people into beta mode e.g. a sample of our users in a certain geographic region / certain zero enabled area/ all users in a certain bucket based on their user id ?
I like this idea. In fact, I'm for it, provided that we make it clear to the user that they've been entered into an experiment and they're seeing non-standard UI.
How many users could beta actually handle?
Not sure. But, interestingly, we can find out by bucketing users and slowly assigning them the beta variant.
Is this technically possible?
Yes. If we're generating and storing tokens on the client, which we do for anonymous users in other experiments, then we can enter anonymous users into the experiment at the cost of a little control over how tokens are stored.
If someone was bucketed into beta would they be able to opt out into
stable again under any of the above situations?
See my first inline response. We must make it clear to the user that they're seeing a variant of an experiment… and we must make it simple to opt out of the experiment.
Also, all instrumentation for beta features will need to be augmented with a is_beta_opt_in flag.
–Sam
On Tue, Jan 27, 2015 at 10:18 AM, Jon Robson jrobson@wikimedia.org wrote:
One of the frustrations I have heard so far is that the audience there is too small to get meaningful data around various experiments. Currently people have to opt in by going to https://en.m.wikipedia.org/wiki/Special:MobileOptions which is hidden away in the mobile interface. They can do this whilst anonymous or logged in.
Have we thought about automatically opting people into beta mode e.g. a sample of our users in a certain geographic region / certain zero enabled area/ all users in a certain bucket based on their user id ?
How many users could beta actually handle? Is this technically possible? If someone was bucketed into beta would they be able to opt out into stable again under any of the above situations?
Jon
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Yeah, I think there are a lot of problems with the opt-in beta model. I much prefer releasing new features to a small % of users, logging events/usage, and if we suspect something has the potential to be disruptive/offputting, letting them know the feature they're seeing is beta and letting them turn it off.
That said, beta is still useful for sandboxing new features and in-person user testing, so I don't think we should kill it altogether. I just think we need to supplement it with a graduated release model – which we're already doing with stuff like WikiGrok :)
On Tue, Jan 27, 2015 at 11:10 AM, Sam Smith samsmith@wikimedia.org wrote:
Have we thought about automatically opting people into beta mode e.g.
a sample of our users in a certain geographic region / certain zero enabled area/ all users in a certain bucket based on their user id ?
I like this idea. In fact, I'm for it, provided that we make it clear to the user that they've been entered into an experiment and they're seeing non-standard UI.
How many users could beta actually handle?
Not sure. But, interestingly, we can find out by bucketing users and slowly assigning them the beta variant.
Is this technically possible?
Yes. If we're generating and storing tokens on the client, which we do for anonymous users in other experiments, then we can enter anonymous users into the experiment at the cost of a little control over how tokens are stored.
If someone was bucketed into beta would they be able to opt out into
stable again under any of the above situations?
See my first inline response. We must make it clear to the user that they're seeing a variant of an experiment… and we must make it simple to opt out of the experiment.
Also, all instrumentation for beta features will need to be augmented with a is_beta_opt_in flag.
–Sam
On Tue, Jan 27, 2015 at 10:18 AM, Jon Robson jrobson@wikimedia.org wrote:
One of the frustrations I have heard so far is that the audience there is too small to get meaningful data around various experiments. Currently people have to opt in by going to https://en.m.wikipedia.org/wiki/Special:MobileOptions which is hidden away in the mobile interface. They can do this whilst anonymous or logged in.
Have we thought about automatically opting people into beta mode e.g. a sample of our users in a certain geographic region / certain zero enabled area/ all users in a certain bucket based on their user id ?
How many users could beta actually handle? Is this technically possible? If someone was bucketed into beta would they be able to opt out into stable again under any of the above situations?
Jon
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Jon, Are there any technical reasons not to explore the % rollout model for collections instead of beta? -J
On Tue, Jan 27, 2015 at 11:20 AM, Maryana Pinchuk mpinchuk@wikimedia.org wrote:
Yeah, I think there are a lot of problems with the opt-in beta model. I much prefer releasing new features to a small % of users, logging events/usage, and if we suspect something has the potential to be disruptive/offputting, letting them know the feature they're seeing is beta and letting them turn it off.
That said, beta is still useful for sandboxing new features and in-person user testing, so I don't think we should kill it altogether. I just think we need to supplement it with a graduated release model – which we're already doing with stuff like WikiGrok :)
On Tue, Jan 27, 2015 at 11:10 AM, Sam Smith samsmith@wikimedia.org wrote:
Have we thought about automatically opting people into beta mode e.g.
a sample of our users in a certain geographic region / certain zero enabled area/ all users in a certain bucket based on their user id ?
I like this idea. In fact, I'm for it, provided that we make it clear to the user that they've been entered into an experiment and they're seeing non-standard UI.
How many users could beta actually handle?
Not sure. But, interestingly, we can find out by bucketing users and slowly assigning them the beta variant.
Is this technically possible?
Yes. If we're generating and storing tokens on the client, which we do for anonymous users in other experiments, then we can enter anonymous users into the experiment at the cost of a little control over how tokens are stored.
If someone was bucketed into beta would they be able to opt out into
stable again under any of the above situations?
See my first inline response. We must make it clear to the user that they're seeing a variant of an experiment… and we must make it simple to opt out of the experiment.
Also, all instrumentation for beta features will need to be augmented with a is_beta_opt_in flag.
–Sam
On Tue, Jan 27, 2015 at 10:18 AM, Jon Robson jrobson@wikimedia.org wrote:
One of the frustrations I have heard so far is that the audience there is too small to get meaningful data around various experiments. Currently people have to opt in by going to https://en.m.wikipedia.org/wiki/Special:MobileOptions which is hidden away in the mobile interface. They can do this whilst anonymous or logged in.
Have we thought about automatically opting people into beta mode e.g. a sample of our users in a certain geographic region / certain zero enabled area/ all users in a certain bucket based on their user id ?
How many users could beta actually handle? Is this technically possible? If someone was bucketed into beta would they be able to opt out into stable again under any of the above situations?
Jon
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
CC'ing Greg to help us think how to approach percentage based releases and what would be in our way to do them.
--tomasz
On Tue, Jan 27, 2015 at 1:18 PM, Jon Katz jkatz@wikimedia.org wrote:
Jon, Are there any technical reasons not to explore the % rollout model for collections instead of beta? -J
On Tue, Jan 27, 2015 at 11:20 AM, Maryana Pinchuk mpinchuk@wikimedia.org wrote:
Yeah, I think there are a lot of problems with the opt-in beta model. I much prefer releasing new features to a small % of users, logging events/usage, and if we suspect something has the potential to be disruptive/offputting, letting them know the feature they're seeing is beta and letting them turn it off.
That said, beta is still useful for sandboxing new features and in-person user testing, so I don't think we should kill it altogether. I just think we need to supplement it with a graduated release model – which we're already doing with stuff like WikiGrok :)
On Tue, Jan 27, 2015 at 11:10 AM, Sam Smith samsmith@wikimedia.org wrote:
Have we thought about automatically opting people into beta mode e.g. a sample of our users in a certain geographic region / certain zero enabled area/ all users in a certain bucket based on their user id ?
I like this idea. In fact, I'm for it, provided that we make it clear to the user that they've been entered into an experiment and they're seeing non-standard UI.
How many users could beta actually handle?
Not sure. But, interestingly, we can find out by bucketing users and slowly assigning them the beta variant.
Is this technically possible?
Yes. If we're generating and storing tokens on the client, which we do for anonymous users in other experiments, then we can enter anonymous users into the experiment at the cost of a little control over how tokens are stored.
If someone was bucketed into beta would they be able to opt out into stable again under any of the above situations?
See my first inline response. We must make it clear to the user that they're seeing a variant of an experiment… and we must make it simple to opt out of the experiment.
Also, all instrumentation for beta features will need to be augmented with a is_beta_opt_in flag.
–Sam
On Tue, Jan 27, 2015 at 10:18 AM, Jon Robson jrobson@wikimedia.org wrote:
One of the frustrations I have heard so far is that the audience there is too small to get meaningful data around various experiments. Currently people have to opt in by going to https://en.m.wikipedia.org/wiki/Special:MobileOptions which is hidden away in the mobile interface. They can do this whilst anonymous or logged in.
Have we thought about automatically opting people into beta mode e.g. a sample of our users in a certain geographic region / certain zero enabled area/ all users in a certain bucket based on their user id ?
How many users could beta actually handle? Is this technically possible? If someone was bucketed into beta would they be able to opt out into stable again under any of the above situations?
Jon
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
<quote name="Tomasz Finc" date="2015-02-06" time="14:04:55 -0800">
CC'ing Greg to help us think how to approach percentage based releases and what would be in our way to do them.
Hi there!
On Tue, Jan 27, 2015 at 1:18 PM, Jon Katz jkatz@wikimedia.org wrote:
Jon, Are there any technical reasons not to explore the % rollout model for collections instead of beta?
I'd love to see what you come up with for this that takes into account all of the oddities of our deployment system as it is (ie: versions of mediawiki and extensions are different depending on which wiki you visit).
Jon or whoever (Sam?) can you include me and my team in your explorations here? I want to figure out something that we can do, even if for only a limited sub-set of things, to make percentage based rollouts possible.
The "real" answer is something along the lines of what Facebook does: * No artificial distinction in code version depending on site * All servers serving the same code for all users (except during deployments, which are also percentage based, but that's handled at the LVS laye) * Staff are transparently sent to the staging build * Features can be turned on by region/percentage/etc with the toggle of a button (enabled by Gatekeeper, which is basically fancy feature flag toggling. It gives a PM or whoever a UI to turn the dials on who sees what when. See: https://www.quora.com/How-does-Facebooks-Gatekeeper-service-work?share=1
We aren't there yet :) So anything you can think of that works reliably might be useful here.
Can someone start a conversation with my team/ops as needed? Feel free to bring strawmen proposals!
Thanks!