[Wikimedia-l] Fundraising updates?

[Thomas Dalton]

Have you considered doing some longer tests? Lasting a week, say. It would
enable you to do proper multivariate testing, including dependencies
between variables (which I don't think you have done any real tests of
yet). It would also let you test time dependence. Eg., does a particular
message work better in the morning than in the afternoon? (Different types
of people browse at different times, so it wouldn't surprise me) You could
also model banner fatigue properly, which could be very useful.
On Dec 17, 2012 4:28 PM, "Zack Exley" <zexley at wikimedia.org> wrote:

> On Sat, Dec 15, 2012 at 11:01 AM, James Salsman <jsalsman at gmail.com>
> wrote:
>
> > Hi Zack,
> >
> > Thanks very much for your updates:
> >
> > > What saved us was taking text from the personal appeals and putting it
> > into
> > > the banner itself. These banners did very well. These new
> message-driven
> > > banners are what made us split the campaign in two -- because we knew
> we
> > > were going to develop a lot of new messages and not have time to
> > translate
> > > them well....
> >
> > As you know I've been saying for years that the variance among the
> > volunteer-supplied messages, originally submitted in 2009 and hundreds
> > of which have not yet been tested (as far as I know), was large enough
> > to suggest that some messages would certainly outperform the
> > traditional banners and appeals. While it's refreshing to be
> > validated, as you might imagine I feel like Cassandra much of the time
> > for reasons that have nothing to do with the underlying mathematical
> > reasoning involved.
> >
> > The last time I heard from you, you said that you intended to test the
> > untried messaging from 2009 with multivariate analysis. However,
> > http://meta.wikimedia.org/wiki/Fundraising_2012/We_Need_A_Breakthrough
> > shows only three very small-N multivariate tests, the last of which
> > was in October, and no recent testing.
> >
> > Do you still intend to test the untried volunteer-submitted messages
> > with multivariate analysis? If so, when? Thank you.
> >
> >
> James -
>
> We can only do big multivariate tests for banner click rates. But banner
> click rates have very little to do with donations in our present context.
>
> For example, the new banners have about 30% the click rate of the old ones,
> but they make about 3 or 4 times as much money.
>
> To determine how well a banner message does for donations, we usually need
> a sample size between 500 and 5,000 donations per banner, depending on the
> difference in performance between the banners. That takes from 30 minutes
> to several hours to collect -- if we're only testing two banners at a time.
>
> Regarding the banners suggested in past years: I've explained this before,
> and will repeat: We tested tons of those banners. I think that we tested
> virtually every different (serious) theme that was suggested. They all had
> BOTH far lower click rates and even lower donation rates -- usually by
> orders of magnitude. This was also true for the new short slogans that we
> came up with ourselves on the fundraising team.
>
> Now we're pretty clear on why: A short slogan isn't enough to get people
> over all their questions about why they should support Wikipedia. More text
> was needed. In our marketing-slogan-obsessed culture, the idea that we'd
> have to present people with a long paragraph was very counterintuitive. We
> didn't think of it on the fundraising team and none of the volunteers who
> submitted suggestions thought of it either. Several marketing professionals
> who contacted us with advice even told us to get rid of the appeal on then
> landing page altogether because "people don't read!"
>
> As it turns out, Wikipedia users DO like to read -- and want all the facts
> before they donate.
>
> Where we're at today, just to emphasize my previous point, is that with the
> new banners, changes in messages effect donations totally independently of
> click rate. And we typically need an hour or two -- or five -- to detect
> even a 10%-%15 percent difference in message performance. That's why we're
> not running big multivariate tests with tons of difference banners.
>
> You'll be happy to know, though, that we are running multivariate tests
> when we're able. For example, if we have a tweak to the landing pages that
> we think is fairly independent of the banner effect, then we sometimes run
> a multivariate test. Or if we have a design tweak (like color) that we're
> confident will always effect click rate in the same direction as donations,
> then we can combine that with message testing.
>
>
> > Sincerely,
> > James Salsman
> >
>
>
>
> --
> Zack Exley
> Chief Revenue Officer
> Wikimedia Foundation
> 415 506 9225
> _______________________________________________
> Wikimedia-l mailing list
> Wikimedia-l at lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
>