Thanks, Baha.

I think the following reinforces your point, but just to clarify, the only reason I compared beta to stable was to identify if any of the anomalies we saw in beta (decrease in beta pvs, huge spike in 'random' clicks) were artifacts of beta adoption or were attributable to normal traffic trends.

On Thu, Aug 13, 2015 at 10:42 AM, Bahodir Mansurov <bmansurov@wikimedia.org> wrote:

On Thu, 2015-08-13 at 10:07 -0700, Jon Katz wrote:
> We need to either increase beta users, a/b test, or test in stable
> more (which would also mean a/b testing on a small % of the
> population)

Thanks, Jon for the write up. I strongly believe this is the way
forward for our future experiments. Comparing beta results to stable
doesn't make much sense.

Baha