[Wikimedia-l] Multivariate Fundraising Tests (Re: compromise?)

James, On Fri, Dec 28, 2012 at 2:11 PM, James Salsman <jsalsman(a)gmail.com> wrote: I obviously cannot speak for what Zack will end up doing but let's talk shop for a moment on how this would be implemented. The tests you indicated play banner, landing page impressions, and donation amount against each other. It appears that everyone saw a collection of random banners (ie: the test was not bucketed.) Are these the same variables you want to test? Regardless of the answer to the above; how do you propose we normalize our tests across time of day, day of week, and day of month factors - we've seen evidence that these all play a role. I don't know how many banner variations we actually have to test but it's likely we won't be able to test them all at the same time (In fact with the current weighting setup we can only test 30 banners at a time). Do we just take each group as it stands -- find the best performers in the group and then test the winners against each other? An additional considering is that we have four buckets to play with; buckets are independent so we could potentially test 120 banners at a time to four different groups. Presumably if we did this we would want a couple of control banners in each to normalize with? An additional something to consider is how long do we have to run these tests to gain statistical significance? At least a day I'm guessing. Are we going to account for banner fatigue at all? IE: show banners during only the first 10 visits like we just did with this most recent campaign? -- ~Matt Walker