Have we thought about automatically opting people into beta mode e.g.
a sample of our users in a certain geographic region / certain zero
enabled area/ all users in a certain bucket based on their user id ?
I like this idea. In fact, I'm for it, provided that we make it clear to the user that they've been entered into an experiment and they're seeing non-standard UI.
How many users could beta actually handle?
Not sure. But, interestingly, we can find out by bucketing users and slowly assigning them the beta variant.
Is this technically possible?
Yes. If we're generating and storing tokens on the client, which we do for anonymous users in other experiments, then we can enter anonymous users into the experiment at the cost of a little control over how tokens are stored.
If someone was bucketed into beta would they be able to opt out into
stable again under any of the above situations?
See my first inline response. We must make it clear to the user that they're seeing a variant of an experiment… and we must make it simple to opt out of the experiment.
Also, all instrumentation for beta features will need to be augmented with a is_beta_opt_in flag.
–Sam