I’d like to give everybody on this list some information on the Berkman/Sciences Po
research project that many of you have been discussing here.
On Thursday the Wikimedia Foundation announced the launch of a banner to support a study
led by a team at the Berkman Center/Sciences Po and recruiting participants from the
English Wikipedia editor community [1]. The banner was taken down within hours of its
launch after concerns raised in various community forums (the Admin Noticeboard [2], the
Village Pump Tech [3], various IRC channels and mailing lists such as foundation-l [4] and
internal-l [5]) that the design was confusing, that it was perceived as a commercial ad
and that the community approval process and privacy terms were unclear and hardly
visible.
Here’s what happened until the launch, what went wrong after the launch and what we are
planning to do next.
==The prequel==
This proposal went through a long review process, involving community forums, the Research
Committee and various WMF departments since early 2010.
The Berkman research team first approached WMF to discuss this study in January 2010. They
suggested a protocol to recruit English Wikipedia contributors to participate in an early
version of this study by March 2010 and posted a proposal to the Administrators’
noticeboard to get community feedback [6]. The community response at that time opposed the
proposed recruitment protocol (posting individual invitation messages on user talk pages).
It was suggested instead that the recruitment should be handled through a CentralNotice
banner to be displayed to registered editors, but concerns were raised on how to minimize
the disruption.
To address these concerns, the proposal went through a full review with the Wikimedia
Research Committee, that was completed in July 2011. The RCom evaluated the methods, the
recruitment strategy, the language used in the survey and approved the proposal pending a
final solution for the recruitment taking into account the concerns expressed by the
community [7].
Based on suggestions made by community members (e.g. [8]) the research team started to
work on a technical solution to selectively display a banner to a subset of registered
editors of the English Wikipedia meeting certain eligibility conditions. WMF agreed to
invest engineering effort into a system that would allow CentralNotice to serve contents
to a specific set of editors – functionality that would benefit future campaigns run by
the community, chapters or the Foundation [9] [10].
A new CentralNotice backend was then designed to look up various editor metrics (i.e.
number of contributions, account registration date and editor privileges) – all public
information available from our database – and to perform a participant eligibility check
against these metrics. A banner would then be displayed to eligible participants, posting
the above data (user ID + editor metrics) along with a unique token to the server hosting
the survey upon clicking. On the landing page of the survey, participants would have the
possibility to read the privacy terms of the survey and decide whether to take it or not.
Throughout the review process of this recruitment protocol, the research team received
constant feedback from the Foundation’s legal team, the community department, the tech
department and the communication team before the campaign went live.
The campaign was announced in the CentralNotice calendar one month before its launch [11]
and the launch was with a post on the Foundation’s blog. The banner was enabled on
December 8 at 11:00pm UTC. 800+ participants completed the study within a few hours since
its launch. The banner was then taken down by a meta-admin a few hours after the launch
due to the concerns described above.
So what went wrong?
==A few explanations we owe you==
• Is the Foundation running ads?
No, this banner is a recruitment campaign for a research project that has been thoroughly
reviewed by the Research Committee. We have a long tradition of supporting recruitment for
research about our communities via various sitenotices. The methodology of this project is
sound and the recruitment method less invasive than thousands of individual messages
posted on user talk pages. We believe this research will help advance our understanding of
the dynamics of participation in our projects. Receiving support by the Research Committee
implies that all published output and anonymized data produced by this study will be made
available under open licenses. [12] The banner also received full Wikimedia Foundation
approval before its launch.
• Is this campaign conflicting with the fundraiser?
No, this banner is running only for a subset of logged-in editors for whom the main
fundraiser campaign has already been taken down. We carefully timed this campaign to
minimize the impact on the fundraiser and we scheduled it on the CentralNotice calendar
with a month notice for this reason.
• Is this campaign running at 100% on the English Wikipedia?
No, the banner has been designed to target a subsample of the English Wikipedia registered
editor population. Based on estimates by the research team, the eligibility criteria apply
to about 10,000 very active contributors and about 30,000 new editors of the English
Wikipedia. The target number of completed responses is 1500.
• Why does the banner include logos of organizations not affiliated with Wikimedia?
The design of the banner was based on the decision to give participants as much
information as possible about the research team running the project and to set accurate
expectations about the study.
==What we are doing now==
We realize that despite an extensive review, the launch of this project was not fully
advertised on community forums. We plan to shortly resume the campaign (for the time
needed by the researchers to complete their responses) after a full redesign of the
recruitment protocol in order to address the concerns raised by many of you over the last
24 hours. Here’s what we are doing:
• Provide you with better information about the project
We asked the research team to promptly set up a FAQ section on the project page on Meta
[13], and to be available to address any concern about the study on the discussion page of
this project. The project page on Meta will be linked from the recruitment banner
itself.
• Redesign the banner
We understand that the banner design has been interpreted by some as ad-like (even if the
goal was to make clear that this study was not being run by WMF, as it implied a
redirection to a third party website for performing the experiment). In coordination with
the research team, we will come up with a banner design that will be more in line with the
concerns expressed by the community (for instance by removing the logos from the banner).
• Make privacy terms as transparent as possible
Upon clicking on the banner, participants accept to share their username, edit count and
user privileges with the research team. The previous version didn’t make it explicit and
we are working to address this problem. To make the process totally transparent we will
make the acceptance of these terms explicit in the banner itself.
Once redirected to the landing page, participants will have to accept the terms of
participation in order to enter the study. The project is funded by the European Research
Council: the data collected in this study is subject to strict European privacy protocols.
The research team will use this data for research purposes only. The research team is not
exposed to and does not record participants’ IP addresses.
==How you can help==
We would like to hear from you on the redesign of the banner to make sure it meets the
expectations of the community and doesn’t lend itself to any kind of confusion. We will
post the new banners to Meta and try to address all pending questions before we resume the
campaign.
This is one of the first times we’re supporting a complex, important research initiative
like this one, and I apologize for the bumps in the road. We believe that supporting
research is part of our mission: it helps advance our understanding of ourselves. So
thanks again for all support you can give in making this a success.
Dario Taraborelli
Senior Research Analyst, Wikimedia Foundation
[1]
http://blog.wikimedia.org/2011/12/08/experiment-decision-making/
[2]
http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Incide…
[3]
http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Search_…
[4]
http://lists.wikimedia.org/pipermail/foundation-l/2011-December/070742.html
[5]
https://lists.wikimedia.org/mailman/private/internal-l/2011-December/018842…
[6]
http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archiv…
[7]
http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interaction…
[8]
http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065580.html
[9]
http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065558.html
[10]
http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines
[11]
http://meta.wikimedia.org/w/index.php?title=CentralNotice/Calendar&oldi…
[12]
http://meta.wikimedia.org/wiki/Research:Subject_recruitment
[13]
meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and_Behavi…