Thanks, Baha.
I think the following reinforces your point, but just to clarify, the only reason I compared beta to stable was to identify if any of the anomalies we saw in beta (decrease in beta pvs, huge spike in 'random' clicks) were artifacts of beta adoption or were attributable to normal traffic trends.
On Thu, Aug 13, 2015 at 10:42 AM, Bahodir Mansurov bmansurov@wikimedia.org wrote:
On Thu, 2015-08-13 at 10:07 -0700, Jon Katz wrote:
We need to either increase beta users, a/b test, or test in stable more (which would also mean a/b testing on a small % of the population)
Thanks, Jon for the write up. I strongly believe this is the way forward for our future experiments. Comparing beta results to stable doesn't make much sense.
Baha