I do, however, have concerns about any research that expects to contact
40,000 editors and involve 1500 of them; that is a very significant portion
of our active editorship on the English Wikipedia project.
Commenting on this: out of those targeted 40,000 editors, 30,000 or so are
*newly registered users*, so that the sample remains somewhat
representative of the diversity we find on en:wp. The rest of it indeed are
active contributors.
Regards,
Jérôme.
2011/12/10 Risker <risker.wp(a)gmail.com>
On 9 December 2011 22:51, Dario Taraborelli
<dtaraborelli(a)wikimedia.org
wrote:
I’d like to give everybody on this list some
information on the
Berkman/Sciences Po research project that many of you have been
discussing
here.
On Thursday the Wikimedia Foundation announced the launch of a banner to
support a study led by a team at the Berkman Center/Sciences Po and
recruiting participants from the English Wikipedia editor community [1].
The banner was taken down within hours of its launch after concerns
raised
in various community forums (the Admin
Noticeboard [2], the Village Pump
Tech [3], various IRC channels and mailing lists such as foundation-l [4]
and internal-l [5]) that the design was confusing, that it was perceived
as
a commercial ad and that the community approval
process and privacy terms
were unclear and hardly visible.
Here’s what happened until the launch, what went wrong after the launch
and what we are planning to do next.
==The prequel==
This proposal went through a long review process, involving community
forums, the Research Committee and various WMF departments since early
2010.
The Berkman research team first approached WMF to discuss this study in
January 2010. They suggested a protocol to recruit English Wikipedia
contributors to participate in an early version of this study by March
2010
and posted a proposal to the Administrators’
noticeboard to get community
feedback [6]. The community response at that time opposed the proposed
recruitment protocol (posting individual invitation messages on user talk
pages). It was suggested instead that the recruitment should be handled
through a CentralNotice banner to be displayed to registered editors, but
concerns were raised on how to minimize the disruption.
To address these concerns, the proposal went through a full review with
the Wikimedia Research Committee, that was completed in July 2011. The
RCom
evaluated the methods, the recruitment strategy,
the language used in the
survey and approved the proposal pending a final solution for the
recruitment taking into account the concerns expressed by the community
[7].
Based on suggestions made by community members (e.g. [8]) the research
team started to work on a technical solution to selectively display a
banner to a subset of registered editors of the English Wikipedia meeting
certain eligibility conditions. WMF agreed to invest engineering effort
into a system that would allow CentralNotice to serve contents to a
specific set of editors – functionality that would benefit future
campaigns run by the community, chapters or the Foundation [9] [10].
A new CentralNotice backend was then designed to look up various editor
metrics (i.e. number of contributions, account registration date and
editor
privileges) – all public information available
from our database – and to
perform a participant eligibility check against these metrics. A banner
would then be displayed to eligible participants, posting the above data
(user ID + editor metrics) along with a unique token to the server
hosting
the survey upon clicking. On the landing page of
the survey, participants
would have the possibility to read the privacy terms of the survey and
decide whether to take it or not.
Throughout the review process of this recruitment protocol, the research
team received constant feedback from the Foundation’s legal team, the
community department, the tech department and the communication team
before
the campaign went live.
The campaign was announced in the CentralNotice calendar one month before
its launch [11] and the launch was with a post on the Foundation’s blog.
The banner was enabled on December 8 at 11:00pm UTC. 800+ participants
completed the study within a few hours since its launch. The banner was
then taken down by a meta-admin a few hours after the launch due to the
concerns described above.
So what went wrong?
==A few explanations we owe you==
• Is the Foundation running ads?
No, this banner is a recruitment campaign for a research project that has
been thoroughly reviewed by the Research Committee. We have a long
tradition of supporting recruitment for research about our communities
via
various sitenotices. The methodology of this
project is sound and the
recruitment method less invasive than thousands of individual messages
posted on user talk pages. We believe this research will help advance our
understanding of the dynamics of participation in our projects. Receiving
support by the Research Committee implies that all published output and
anonymized data produced by this study will be made available under open
licenses. [12] The banner also received full Wikimedia Foundation
approval
before its launch.
• Is this campaign conflicting with the fundraiser?
No, this banner is running only for a subset of logged-in editors for
whom
the main fundraiser campaign has already been
taken down. We carefully
timed this campaign to minimize the impact on the fundraiser and we
scheduled it on the CentralNotice calendar with a month notice for this
reason.
• Is this campaign running at 100% on the English Wikipedia?
No, the banner has been designed to target a subsample of the English
Wikipedia registered editor population. Based on estimates by the
research
team, the eligibility criteria apply to about
10,000 very active
contributors and about 30,000 new editors of the English Wikipedia. The
target number of completed responses is 1500.
• Why does the banner include logos of organizations not affiliated with
Wikimedia?
The design of the banner was based on the decision to give participants
as
much information as possible about the research
team running the project
and to set accurate expectations about the study.
==What we are doing now==
We realize that despite an extensive review, the launch of this project
was not fully advertised on community forums. We plan to shortly resume
the
campaign (for the time needed by the researchers
to complete their
responses) after a full redesign of the recruitment protocol in order to
address the concerns raised by many of you over the last 24 hours. Here’s
what we are doing:
• Provide you with better information about the project
We asked the research team to promptly set up a FAQ section on the
project
page on Meta [13], and to be available to address
any concern about the
study on the discussion page of this project. The project page on Meta
will
be linked from the recruitment banner itself.
• Redesign the banner
We understand that the banner design has been interpreted by some as
ad-like (even if the goal was to make clear that this study was not being
run by WMF, as it implied a redirection to a third party website for
performing the experiment). In coordination with the research team, we
will
come up with a banner design that will be more in
line with the concerns
expressed by the community (for instance by removing the logos from the
banner).
• Make privacy terms as transparent as possible
Upon clicking on the banner, participants accept to share their username,
edit count and user privileges with the research team. The previous
version
didn’t make it explicit and we are working to
address this problem. To
make
the process totally transparent we will make the
acceptance of these
terms
explicit in the banner itself.
Once redirected to the landing page, participants will have to accept the
terms of participation in order to enter the study. The project is funded
by the European Research Council: the data collected in this study is
subject to strict European privacy protocols. The research team will use
this data for research purposes only. The research team is not exposed to
and does not record participants’ IP addresses.
==How you can help==
We would like to hear from you on the redesign of the banner to make sure
it meets the expectations of the community and doesn’t lend itself to any
kind of confusion. We will post the new banners to Meta and try to
address
all pending questions before we resume the
campaign.
This is one of the first times we’re supporting a complex, important
research initiative like this one, and I apologize for the bumps in the
road. We believe that supporting research is part of our mission: it
helps
advance our understanding of ourselves. So thanks
again for all support
you
can give in making this a success.
Dario Taraborelli
Senior Research Analyst, Wikimedia Foundation
[1]
http://blog.wikimedia.org/2011/12/08/experiment-decision-making/
[2]
http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Incide…
[3]
http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Search_…
[4]
http://lists.wikimedia.org/pipermail/foundation-l/2011-December/070742.html
[5]
https://lists.wikimedia.org/mailman/private/internal-l/2011-December/018842…
[6]
http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archiv…
[7]
http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interaction…
[8]
http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065580.html
[9]
http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065558.html
http://meta.wikimedia.org/w/index.php?title=CentralNotice/Calendar&oldi…
meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and_Behavi…
Dario, nobody in any of the discussions on the English Wikipedia (whose
editors are the target of this research project) suggested that a *central
site notice* be used for this or any other research project. The
discussion in April 2011 showed consensus opposition to bot-delivered talk
page notices. One editor involved in the discussion suggested "site
notices" (which I believe were interpreted by the participants to mean a
local site notice) and two others mentioned watchlist notices. The
subsequent discussion about central notices discussed the possibility of
developing a narrowcasting ability for such notices, and discussed
specifically notices directly related to WMF projects or activities. It
did not, in any way, address the concept of using a central notice to
promote a non-WMF activity (such as this research project). Indeed, this is
the first use of a central notice for anything not directly related to an
obviously WMF-related activity.
The ability to narrowcast central notices is a positive advancement;
however, the processes for proposing and determining the appropriateness of
a narrowcast are poorly publicized, and some of them don't appear to have
even existed until after this notice was taken down. There are still no
community-approved guidelines for the use of central notices, although a
draft one is currently up for comment.[1] An RFC initiated in August 2010
with respect to "global banners"/central notices, well in advance of the
development of the narrowcasting ability, strongly supported consensus
approval on Meta for non-fundraising global banners.[2] Now that there is
the ability to target central notices to only one project or community, it
is extremely important that that community be directly notified of such
discussion - a discussion that never took place in any public forum that I
can see in advance of this central notice being activated.
The links above include one to a private mailing list that the majority of
readers of this list have no access to. You may want to consider asking the
persons whose contributions are contained in that particular message to
grant permission for it to be reproduced here so that the rest of us aren't
left in the dark about who said what.
I don't begridge scholars carrying out approved research with Wiki?edians
who volunteer to do so; in fact, I've responded to several requests myself.
I do, however, have concerns about any research that expects to contact
40,000 editors and involve 1500 of them; that is a very significant portion
of our active editorship on the English Wikipedia project. I'm curious to
know if scholars have shown much interest in studying some of the other
projects as much as they've initiated studies on enwp.
Risker/Anne
[1]
http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines
http://meta.wikimedia.org/wiki/Requests_for_comment/Global_banners
_______________________________________________
foundation-l mailing list
foundation-l(a)lists.wikimedia.org
Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/foundation-l