[Wikimedia-l] Fundraising updates?

[Zack Exley]

On Sat, Dec 15, 2012 at 11:01 AM, James Salsman <jsalsman at gmail.com> wrote:

> Hi Zack,
>
> Thanks very much for your updates:
>
> > What saved us was taking text from the personal appeals and putting it
> into
> > the banner itself. These banners did very well. These new message-driven
> > banners are what made us split the campaign in two -- because we knew we
> > were going to develop a lot of new messages and not have time to
> translate
> > them well....
>
> As you know I've been saying for years that the variance among the
> volunteer-supplied messages, originally submitted in 2009 and hundreds
> of which have not yet been tested (as far as I know), was large enough
> to suggest that some messages would certainly outperform the
> traditional banners and appeals. While it's refreshing to be
> validated, as you might imagine I feel like Cassandra much of the time
> for reasons that have nothing to do with the underlying mathematical
> reasoning involved.
>
> The last time I heard from you, you said that you intended to test the
> untried messaging from 2009 with multivariate analysis. However,
> http://meta.wikimedia.org/wiki/Fundraising_2012/We_Need_A_Breakthrough
> shows only three very small-N multivariate tests, the last of which
> was in October, and no recent testing.
>
> Do you still intend to test the untried volunteer-submitted messages
> with multivariate analysis? If so, when? Thank you.
>
>
James -

We can only do big multivariate tests for banner click rates. But banner
click rates have very little to do with donations in our present context.

For example, the new banners have about 30% the click rate of the old ones,
but they make about 3 or 4 times as much money.

To determine how well a banner message does for donations, we usually need
a sample size between 500 and 5,000 donations per banner, depending on the
difference in performance between the banners. That takes from 30 minutes
to several hours to collect -- if we're only testing two banners at a time.

Regarding the banners suggested in past years: I've explained this before,
and will repeat: We tested tons of those banners. I think that we tested
virtually every different (serious) theme that was suggested. They all had
BOTH far lower click rates and even lower donation rates -- usually by
orders of magnitude. This was also true for the new short slogans that we
came up with ourselves on the fundraising team.

Now we're pretty clear on why: A short slogan isn't enough to get people
over all their questions about why they should support Wikipedia. More text
was needed. In our marketing-slogan-obsessed culture, the idea that we'd
have to present people with a long paragraph was very counterintuitive. We
didn't think of it on the fundraising team and none of the volunteers who
submitted suggestions thought of it either. Several marketing professionals
who contacted us with advice even told us to get rid of the appeal on then
landing page altogether because "people don't read!"

As it turns out, Wikipedia users DO like to read -- and want all the facts
before they donate.

Where we're at today, just to emphasize my previous point, is that with the
new banners, changes in messages effect donations totally independently of
click rate. And we typically need an hour or two -- or five -- to detect
even a 10%-%15 percent difference in message performance. That's why we're
not running big multivariate tests with tons of difference banners.

You'll be happy to know, though, that we are running multivariate tests
when we're able. For example, if we have a tweak to the landing pages that
we think is fairly independent of the banner effect, then we sometimes run
a multivariate test. Or if we have a design tweak (like color) that we're
confident will always effect click rate in the same direction as donations,
then we can combine that with message testing.


> Sincerely,
> James Salsman
>



-- 
Zack Exley
Chief Revenue Officer
Wikimedia Foundation
415 506 9225