Wikimedians--
As many of you know, last month we began work on exploring the visibility of the donate button on all Wikimedia projects. After a long comment period, we received many comments and many new ideas. Some of these ideas we have incorporated into a new set of test buttons. Thank you to everyone who took the time to evaluate Round 1 buttons. You can see those discussions here: http://meta.wikimedia.org/wiki/Fundraising_2009/Donation_buttons_upgrade/Rou...
We have 4 designs that we will be testing on the Wikipedia:EN main skin during August and the first part of September. We are going to evaluate each button for one full week. This process will unfold over the next two months.
You can see the designs and timeline at this link: http://meta.wikimedia.org/wiki/Fundraising_2009/Donation_buttons_upgrade
-Rand
On Mon, Jul 20, 2009 at 4:50 PM, Rand Montoyarmontoya@wikimedia.org wrote:
Wikimedians--
As many of you know, last month we began work on exploring the visibility of the donate button on all Wikimedia projects. After a long comment period, we received many comments and many new ideas. Some of these ideas we have incorporated into a new set of test buttons. Thank you to everyone who took the time to evaluate Round 1 buttons. You can see those discussions here: http://meta.wikimedia.org/wiki/Fundraising_2009/Donation_buttons_upgrade/Rou...
We have 4 designs that we will be testing on the Wikipedia:EN main skin during August and the first part of September. We are going to evaluate each button for one full week. This process will unfold over the next two months.
You can see the designs and timeline at this link: http://meta.wikimedia.org/wiki/Fundraising_2009/Donation_buttons_upgrade
Testing should be done in parallel, not in sequence. History has demonstrated that donors have a tendency to respond disproportionately to "the new thing". Which means that whatever button you test first will have an advantage over whichever one you test last. Probably the easiest way to get a reasonable distribution is to vary which button people see based on their IP.
-Robert Rohde
On Mon, Jul 20, 2009 at 6:04 PM, Robert Rohde rarohde@gmail.com wrote:
On Mon, Jul 20, 2009 at 4:50 PM, Rand Montoyarmontoya@wikimedia.org wrote:
Wikimedians--
As many of you know, last month we began work on exploring the visibility of the donate button on all Wikimedia projects. After a long comment period, we received many comments and many new ideas. Some of these ideas we have incorporated into a new set of test buttons. Thank you to everyone who took the time to evaluate Round 1 buttons. You can see those discussions here:
http://meta.wikimedia.org/wiki/Fundraising_2009/Donation_buttons_upgrade/Rou...
We have 4 designs that we will be testing on the Wikipedia:EN main skin during August and the first part of September. We are going to evaluate each button for one full week. This process will unfold over the next two months.
You can see the designs and timeline at this link: http://meta.wikimedia.org/wiki/Fundraising_2009/Donation_buttons_upgrade
Testing should be done in parallel, not in sequence. History has demonstrated that donors have a tendency to respond disproportionately to "the new thing". Which means that whatever button you test first will have an advantage over whichever one you test last. Probably the easiest way to get a reasonable distribution is to vary which button people see based on their IP.
-Robert Rohde
It's also necessary to control for seasonal traffic (and thus donation) variations. I note that the first three button tests are at the end of summer while the fourth coincides with the beginning of the school year. It could be the case that there is no variation, or that the variation is highly significant. Since nobody has looked there is no way to tell if the test results are valid.
2009/7/21 Robert Rohde rarohde@gmail.com:
Testing should be done in parallel, not in sequence. History has demonstrated that donors have a tendency to respond disproportionately to "the new thing". Which means that whatever button you test first will have an advantage over whichever one you test last. Probably the easiest way to get a reasonable distribution is to vary which button people see based on their IP.
Or simply to randomise it entirely.
If either of those aren't possible for technical reasons, it might be practical to rotate them - run each button for x many hours at a stretch, rotating them so as to ensure they don't regularly go up at the same time (of the day or of the week) and so that they get roughly equal coverage.
On Tue, Jul 21, 2009 at 11:02 AM, Andrew Gray andrew.gray@dunelm.org.ukwrote:
2009/7/21 Robert Rohde rarohde@gmail.com:
Testing should be done in parallel, not in sequence. History has demonstrated that donors have a tendency to respond disproportionately to "the new thing". Which means that whatever button you test first will have an advantage over whichever one you test last. Probably the easiest way to get a reasonable distribution is to vary which button people see based on their IP.
Or simply to randomise it entirely.
If either of those aren't possible for technical reasons, it might be practical to rotate them - run each button for x many hours at a stretch, rotating them so as to ensure they don't regularly go up at the same time (of the day or of the week) and so that they get roughly equal coverage.
Rotating them would seem like a more viable solution than randomised - We don't want the situation where every new page in WP someone reads there is a new/different coloured donation button where last week there was none at all - to go from nothing to that would be almost as bad as a flashing "donate here, now!" banner. I suspect that if there are any complaints about this new button system they will be about whether it makes us look like we're giving the 'hard sell' to people. Finding the middle ground between making the donation button clear/prominent and making it subtle/non-distracting will be difficult but also very important. We want people to know that we depend on donations but we don't want them to be annoyed by being constantly distracted by a big colourful push for their wallet (especially if it keeps changing).
It's a tricky line to tread - but I'm sure we can do it!
-Liam [[witty lama]]
wittylama.com/blog Peace, love & metadata Sent from Sydney, Nsw, Australia
--
- Andrew Gray
andrew.gray@dunelm.org.uk
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
2009/7/20 Liam Wyatt liamwyatt@gmail.com:
Rotating them would seem like a more viable solution than randomised - We don't want the situation where every new page in WP someone reads there is a new/different coloured donation button where last week there was none at all
- to go from nothing to that would be almost as bad as a flashing "donate
here, now!" banner.
Indeed, that's the reasoning behind the proposed approach. We don't want it to typically be changing constantly for an individual user. Yes, a sequential run does introduce various problematic biases.
An IP-address based hack could work, but would need to take into account dynamic IP addresses and such, without introducing strange new biases of its own. We'll discuss a bit further - good ideas / algorithms welcome. :-)
On Mon, Jul 20, 2009 at 9:19 PM, Erik Moellererik@wikimedia.org wrote:
Indeed, that's the reasoning behind the proposed approach. We don't want it to typically be changing constantly for an individual user. Yes, a sequential run does introduce various problematic biases.
An IP-address based hack could work, but would need to take into account dynamic IP addresses and such, without introducing strange new biases of its own. We'll discuss a bit further - good ideas / algorithms welcome. :-)
For this the normal procedure is to give users a session cookie of some kind (either one handed out by the server or one just generated on the client) and base the selection on that.
For caching reasons I suppose you'd just want to do this all client side. Should work fine.
Alternatively, someone rigs up the front end caches to do this substitution based on IP at serving time. This would be non-trivial with squid. It would be much easier with varnish, alas.
In any case, I strongly agree with the argument against running them sequentially. Not only do you get the uncertainty from changing habits over time but later buttons will suffer from the influence of prior ones. Whatever can be done to avoid sequential testing should be done.
On Tue, Jul 21, 2009 at 12:30 PM, Gregory Maxwell gmaxwell@gmail.comwrote:
On Mon, Jul 20, 2009 at 9:19 PM, Erik Moellererik@wikimedia.org wrote:
Indeed, that's the reasoning behind the proposed approach. We don't want it to typically be changing constantly for an individual user. Yes, a sequential run does introduce various problematic biases.
An IP-address based hack could work, but would need to take into account dynamic IP addresses and such, without introducing strange new biases of its own. We'll discuss a bit further - good ideas / algorithms welcome. :-)
For this the normal procedure is to give users a session cookie of some kind (either one handed out by the server or one just generated on the client) and base the selection on that.
For caching reasons I suppose you'd just want to do this all client side. Should work fine.
Alternatively, someone rigs up the front end caches to do this substitution based on IP at serving time. This would be non-trivial with squid. It would be much easier with varnish, alas.
In any case, I strongly agree with the argument against running them sequentially. Not only do you get the uncertainty from changing habits over time but later buttons will suffer from the influence of prior ones. Whatever can be done to avoid sequential testing should be done.
Didn't we do this kind of trial during the last fundraiser - with the messages at the top? They were rotated each new day weren't they? In any case, whatever we worked out for last time can't we use that method again?
wittylama.com/blog Peace, love & metadata Sent from Sydney, Nsw, Australia
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
What about changing it every hour? Then you have sufficient randomness over time, and no flashy buttons.
-- eia
2009/7/21 Erik Moeller erik@wikimedia.org
2009/7/20 Liam Wyatt liamwyatt@gmail.com:
Rotating them would seem like a more viable solution than randomised - We don't want the situation where every new page in WP someone reads there
is a
new/different coloured donation button where last week there was none at
all
- to go from nothing to that would be almost as bad as a flashing "donate
here, now!" banner.
Indeed, that's the reasoning behind the proposed approach. We don't want it to typically be changing constantly for an individual user. Yes, a sequential run does introduce various problematic biases.
An IP-address based hack could work, but would need to take into account dynamic IP addresses and such, without introducing strange new biases of its own. We'll discuss a bit further - good ideas / algorithms welcome. :-) -- Erik Möller Deputy Director, Wikimedia Foundation
Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
On Tue, Jul 21, 2009 at 2:15 AM, effe iets anderseffeietsanders@gmail.com wrote:
What about changing it every hour? Then you have sufficient randomness over time, and no flashy buttons.
You still have the problem that peoples response to future buttons will depend on the past buttons they've seen.
On Mon, Jul 20, 2009 at 11:28 PM, Gregory Maxwellgmaxwell@gmail.com wrote:
On Tue, Jul 21, 2009 at 2:15 AM, effe iets anderseffeietsanders@gmail.com wrote:
What about changing it every hour? Then you have sufficient randomness over time, and no flashy buttons.
You still have the problem that peoples response to future buttons will depend on the past buttons they've seen.
Yeah, I'd suggest it is better to show each person one button type and stick with it.
One can approximate that behavior either by assigning each user randomly and tracking their type via a cookie or by selecting based on higher order bytes in the IP address and assuming those are fairly stable (e.g. using BBB mod 4 where the address is AAA.BBB.CCC.DDD).
Neither approach is foolproof or totally without bias, but one can do fairly well.
-Robert Rohde
Donate Now Every donation helps us to keep free for everyone. Donate Now Keep Wikipedia free for everyone.
Is no one else concerned by the use of the word "free" in the message options being tested. I wouldn't want these ambigous messages like these on the site no matter if they beat out the no message option by 10 to 1. Why can't we test messages that are actually clear and honest? Wikipedia will still be free for everyone if not a single further donation is ever made.
Birgitte SB
on 7/21/09 10:33 AM, Birgitte SB at birgitte_sb@yahoo.com wrote:
Donate Now Every donation helps us to keep free for everyone. Donate Now Keep Wikipedia free for everyone.
Is no one else concerned by the use of the word "free" in the message options being tested. I wouldn't want these ambigous messages like these on the site no matter if they beat out the no message option by 10 to 1. Why can't we test messages that are actually clear and honest? Wikipedia will still be free for everyone if not a single further donation is ever made.
Birgitte SB
I agree with you very much, Birgitte. Both of these messages sound like threats.
Marc Riddell
I agree with this, and said so at the original discussion--where I think the consensus was not to use that phrase.
David Goodman, Ph.D, M.L.S. http://en.wikipedia.org/wiki/User_talk:DGG
On Tue, Jul 21, 2009 at 10:39 AM, Marc Riddellmichaeldavid86@comcast.net wrote:
on 7/21/09 10:33 AM, Birgitte SB at birgitte_sb@yahoo.com wrote:
Donate Now Every donation helps us to keep free for everyone. Donate Now Keep Wikipedia free for everyone.
Is no one else concerned by the use of the word "free" in the message options being tested. I wouldn't want these ambigous messages like these on the site no matter if they beat out the no message option by 10 to 1. Why can't we test messages that are actually clear and honest? Wikipedia will still be free for everyone if not a single further donation is ever made.
Birgitte SB
I agree with you very much, Birgitte. Both of these messages sound like threats.
Marc Riddell
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
wikimedia-l@lists.wikimedia.org