The only documentation from the research team that I have seen so far with respect to the target participation is in the initial proposal on enwp back in 2010, when it was proposed to leave 40,000 talk page messages; there was no indication that 30,000 of them would be newly registered users at that time. Not to criticize the genuine attempt at information sharing on Dario's part - it is much appreciated - but there is so much change in what was put forward from what we had initially been approached about that it's preferable to hear it from the researcher's mouth, and to have it well documented.
Reading through the AN discussion again, I don't feel like the basic gist of our research proposal has changed much since March 2010. Even if it does not provide the actual number, one of our posts there does say that " among the 40 000 users or so that we will invite, many will be newly registered users". I do agree that the contact protocol for the study, on the other hand, has changed much since then. This was the result of a longstanding effort that we made in collaboration with WMF in order to try to bring this study to Wikipedians in the most community compliant way possible, following the discussion that we had on the Admin Board. I'm not saying that we may not have made some mistakes. Our first banner proposals (there were 13 of them in total) did not feature our logos. We included them in order to comply with one of WMF's request, with people (in my view somewhat legitimately) being concerned about the fact that if this banner was to redirect people to a third party website for the purpose of performing an experiment, it should be made as clear as possible from the very beginning that this project was not run by WMF. But I can tell for myself as well as all the other researchers who worked on this research project for the past year and a half that we were acting in good faith and remained 100% committed to respecting this community throughout the process. I think we have achieved quite a lot already, developing together a code that could selectively display banners to specific subset of editors based on their user metrics, which is a significant improvement in the flexibility of the banner tools now at the disposal of WMF and the community. I am truly sorry that some community members considered that our banner had to be taken down only a few hours after what was indeed a great moment for us as a research team...
Regards,
Jérôme.
2011/12/10 Risker risker.wp@gmail.com
Hi Jerome -
The only documentation from the research team that I have seen so far with respect to the target participation is in the initial proposal on enwp back in 2010, when it was proposed to leave 40,000 talk page messages; there was no indication that 30,000 of them would be newly registered users at that time. Not to criticize the genuine attempt at information sharing on Dario's part - it is much appreciated - but there is so much change in what was put forward from what we had initially been approached about that it's preferable to hear it from the researcher's mouth, and to have it well documented.
Something that has never been clear is the reason that English Wikipedia editors were identified as the preferred target; there does not appear to be anything in this study that is particularly oriented toward Wikipedia activity.
Risker/Anne
2011/12/10 Jérôme Hergueux jerome.hergueux@gmail.com
This is actually not the case. Those 30,000 users or so are users who registered their Wikipedia account 30 days prior to the launch of the study. There are no other requirements for those users to be eligible to participate. This is in line with Dario's previous message:
the banner has been designed to target a subsample of the English
Wikipedia
registered editor population. Based on estimates by the research team,
the
eligibility criteria apply to about 10,000 very active contributors and about 30,000 new editors of the English Wikipedia.
Regards,
Jérôme.
2011/12/10 Risker risker.wp@gmail.com
Hi Jerome - please show me where it says that; I've not been able to
verify
that interpretation at all. My understanding is that the 30,000 are
users
with fewer than 100 edits per month on average, not that they are new users.
Risker/Anne
2011/12/10 Jérôme Hergueux jerome.hergueux@gmail.com
I do, however, have concerns about any research that expects to
contact
40,000 editors and involve 1500 of them; that is a very significant
portion
of our active editorship on the English Wikipedia project.
Commenting on this: out of those targeted 40,000 editors, 30,000 or
so
are
*newly registered users*, so that the sample remains somewhat representative of the diversity we find on en:wp. The rest of it
indeed
are
active contributors.
Regards,
Jérôme.
2011/12/10 Risker risker.wp@gmail.com
On 9 December 2011 22:51, Dario Taraborelli <
dtaraborelli@wikimedia.org
wrote:
I’d like to give everybody on this list some information on the Berkman/Sciences Po research project that many of you have been
discussing
here.
On Thursday the Wikimedia Foundation announced the launch of a
banner
to
support a study led by a team at the Berkman Center/Sciences Po
and
recruiting participants from the English Wikipedia editor
community
[1].
The banner was taken down within hours of its launch after
concerns
raised
in various community forums (the Admin Noticeboard [2], the
Village
Pump
Tech [3], various IRC channels and mailing lists such as
foundation-l
[4]
and internal-l [5]) that the design was confusing, that it was
perceived
as
a commercial ad and that the community approval process and
privacy
terms
were unclear and hardly visible.
Here’s what happened until the launch, what went wrong after the
launch
and what we are planning to do next.
==The prequel== This proposal went through a long review process, involving
community
forums, the Research Committee and various WMF departments since
early
The Berkman research team first approached WMF to discuss this
study
in
January 2010. They suggested a protocol to recruit English
Wikipedia
contributors to participate in an early version of this study by
March
2010
and posted a proposal to the Administrators’ noticeboard to get
community
feedback [6]. The community response at that time opposed the
proposed
recruitment protocol (posting individual invitation messages on
user
talk
pages). It was suggested instead that the recruitment should be
handled
through a CentralNotice banner to be displayed to registered
editors,
but
concerns were raised on how to minimize the disruption.
To address these concerns, the proposal went through a full
review
with
the Wikimedia Research Committee, that was completed in July
The
RCom
evaluated the methods, the recruitment strategy, the language
used
in
the
survey and approved the proposal pending a final solution for the recruitment taking into account the concerns expressed by the
community
[7].
Based on suggestions made by community members (e.g. [8]) the
research
team started to work on a technical solution to selectively
display a
banner to a subset of registered editors of the English Wikipedia
meeting
certain eligibility conditions. WMF agreed to invest engineering
effort
into a system that would allow CentralNotice to serve contents
to a
specific set of editors – functionality that would benefit
future
campaigns run by the community, chapters or the Foundation [9]
[10].
A new CentralNotice backend was then designed to look up various
editor
metrics (i.e. number of contributions, account registration date
and
editor
privileges) – all public information available from our database
–
and
to
perform a participant eligibility check against these metrics. A
banner
would then be displayed to eligible participants, posting the
above
data
(user ID + editor metrics) along with a unique token to the
server
hosting
the survey upon clicking. On the landing page of the survey,
participants
would have the possibility to read the privacy terms of the
survey
and
decide whether to take it or not.
Throughout the review process of this recruitment protocol, the
research
team received constant feedback from the Foundation’s legal team,
the
community department, the tech department and the communication
team
before
the campaign went live.
The campaign was announced in the CentralNotice calendar one
month
before
its launch [11] and the launch was with a post on the
Foundation’s
blog.
The banner was enabled on December 8 at 11:00pm UTC. 800+
participants
completed the study within a few hours since its launch. The
banner
was
then taken down by a meta-admin a few hours after the launch due
to
the
concerns described above.
So what went wrong?
==A few explanations we owe you==
• Is the Foundation running ads? No, this banner is a recruitment campaign for a research project
that
has
been thoroughly reviewed by the Research Committee. We have a
long
tradition of supporting recruitment for research about our
communities
via
various sitenotices. The methodology of this project is sound and
the
recruitment method less invasive than thousands of individual
messages
posted on user talk pages. We believe this research will help
advance
our
understanding of the dynamics of participation in our projects.
Receiving
support by the Research Committee implies that all published
output
and
anonymized data produced by this study will be made available
under
open
licenses. [12] The banner also received full Wikimedia Foundation
approval
before its launch.
• Is this campaign conflicting with the fundraiser? No, this banner is running only for a subset of logged-in editors
for
whom
the main fundraiser campaign has already been taken down. We
carefully
timed this campaign to minimize the impact on the fundraiser and
we
scheduled it on the CentralNotice calendar with a month notice
for
this
reason.
• Is this campaign running at 100% on the English Wikipedia? No, the banner has been designed to target a subsample of the
English
Wikipedia registered editor population. Based on estimates by the
research
team, the eligibility criteria apply to about 10,000 very active contributors and about 30,000 new editors of the English
Wikipedia.
The
target number of completed responses is 1500.
• Why does the banner include logos of organizations not
affiliated
with
Wikimedia? The design of the banner was based on the decision to give
participants
as
much information as possible about the research team running the
project
and to set accurate expectations about the study.
==What we are doing now==
We realize that despite an extensive review, the launch of this
project
was not fully advertised on community forums. We plan to shortly
resume
the
campaign (for the time needed by the researchers to complete
their
responses) after a full redesign of the recruitment protocol in
order
to
address the concerns raised by many of you over the last 24
hours.
Here’s
what we are doing:
• Provide you with better information about the project We asked the research team to promptly set up a FAQ section on
the
project
page on Meta [13], and to be available to address any concern
about
the
study on the discussion page of this project. The project page on
Meta
will
be linked from the recruitment banner itself.
• Redesign the banner We understand that the banner design has been interpreted by some
as
ad-like (even if the goal was to make clear that this study was
not
being
run by WMF, as it implied a redirection to a third party website
for
performing the experiment). In coordination with the research
team,
we
will
come up with a banner design that will be more in line with the
concerns
expressed by the community (for instance by removing the logos
from
the
banner).
• Make privacy terms as transparent as possible Upon clicking on the banner, participants accept to share their
username,
edit count and user privileges with the research team. The
previous
version
didn’t make it explicit and we are working to address this
problem.
To
make
the process totally transparent we will make the acceptance of
these
terms
explicit in the banner itself.
Once redirected to the landing page, participants will have to
accept
the
terms of participation in order to enter the study. The project
is
funded
by the European Research Council: the data collected in this
study
is
subject to strict European privacy protocols. The research team
will
use
this data for research purposes only. The research team is not
exposed
to
and does not record participants’ IP addresses.
==How you can help==
We would like to hear from you on the redesign of the banner to
make
sure
it meets the expectations of the community and doesn’t lend
itself
to
any
kind of confusion. We will post the new banners to Meta and try
to
address
all pending questions before we resume the campaign.
This is one of the first times we’re supporting a complex,
important
research initiative like this one, and I apologize for the bumps
in
the
road. We believe that supporting research is part of our mission:
it
helps
advance our understanding of ourselves. So thanks again for all
support
you
can give in making this a success.
Dario Taraborelli Senior Research Analyst, Wikimedia Foundation
[1]
http://blog.wikimedia.org/2011/12/08/experiment-decision-making/
[2]
http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Inciden...
[3]
http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Search_b...
[4]
http://lists.wikimedia.org/pipermail/foundation-l/2011-December/070742.html
[5]
https://lists.wikimedia.org/mailman/private/internal-l/2011-December/018842....
[6]
http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archive...
[7]
http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interactions...
[8]
http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065580.html
[9]
http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065558.html
[10]
http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines
[11]
http://meta.wikimedia.org/w/index.php?title=CentralNotice/Calendar&oldid...
[12] http://meta.wikimedia.org/wiki/Research:Subject_recruitment [13]
meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and_Behavior
Dario, nobody in any of the discussions on the English Wikipedia
(whose
editors are the target of this research project) suggested that a
*central
site notice* be used for this or any other research project. The discussion in April 2011 showed consensus opposition to
bot-delivered
talk
page notices. One editor involved in the discussion suggested "site notices" (which I believe were interpreted by the participants to
mean
a
local site notice) and two others mentioned watchlist notices. The subsequent discussion about central notices discussed the
possibility
of
developing a narrowcasting ability for such notices, and discussed specifically notices directly related to WMF projects or
activities.
It
did not, in any way, address the concept of using a central notice
to
promote a non-WMF activity (such as this research project). Indeed,
this
is
the first use of a central notice for anything not directly related
to
an
obviously WMF-related activity.
The ability to narrowcast central notices is a positive
advancement;
however, the processes for proposing and determining the
appropriateness
of
a narrowcast are poorly publicized, and some of them don't appear
to
have
even existed until after this notice was taken down. There are
still
no
community-approved guidelines for the use of central notices,
although
a
draft one is currently up for comment.[1] An RFC initiated in
August
2010
with respect to "global banners"/central notices, well in advance
of
the
development of the narrowcasting ability, strongly supported
consensus
approval on Meta for non-fundraising global banners.[2] Now that
there
is
the ability to target central notices to only one project or
community,
it
is extremely important that that community be directly notified of
such
discussion - a discussion that never took place in any public forum
that
I
can see in advance of this central notice being activated.
The links above include one to a private mailing list that the
majority
of
readers of this list have no access to. You may want to consider
asking
the
persons whose contributions are contained in that particular
message
to
grant permission for it to be reproduced here so that the rest of
us
aren't
left in the dark about who said what.
I don't begridge scholars carrying out approved research with
Wiki?edians
who volunteer to do so; in fact, I've responded to several requests
myself.
I do, however, have concerns about any research that expects to
contact
40,000 editors and involve 1500 of them; that is a very significant
portion
of our active editorship on the English Wikipedia project. I'm
curious
to
know if scholars have shown much interest in studying some of the
other
projects as much as they've initiated studies on enwp.
Risker/Anne
[1] http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines http://meta.wikimedia.org/wiki/Requests_for_comment/Global_banners _______________________________________________ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/foundation-l
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/foundation-l
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l