Hi Jerome -
The only documentation from the research team that I have seen so far with
respect to the target participation is in the initial proposal on enwp back
in 2010, when it was proposed to leave 40,000 talk page messages; there was
no indication that 30,000 of them would be newly registered users at that
time. Not to criticize the genuine attempt at information sharing on
Dario's part - it is much appreciated - but there is so much change in what
was put forward from what we had initially been approached about that it's
preferable to hear it from the researcher's mouth, and to have it well
documented.
Something that has never been clear is the reason that English Wikipedia
editors were identified as the preferred target; there does not appear to
be anything in this study that is particularly oriented toward Wikipedia
activity.
Risker/Anne
2011/12/10 Jérôme Hergueux <jerome.hergueux(a)gmail.com>
This is actually not the case. Those 30,000 users
or so are users who
registered their Wikipedia account 30 days prior to the launch of the
study. There are no other requirements for those users to be eligible to
participate. This is in line with Dario's previous message:
the banner has been designed to target a subsample of the English Wikipedia
registered editor population. Based on estimates by the research team, the
eligibility criteria apply to about 10,000 very active contributors and
about 30,000 new editors of the English Wikipedia.
Regards,
Jérôme.
2011/12/10 Risker <risker.wp(a)gmail.com>
Hi Jerome - please show me where it says that;
I've not been able to
verify
that interpretation at all. My understanding is
that the 30,000 are
users
with fewer than 100 edits per month on average,
not that they are new
users.
Risker/Anne
2011/12/10 Jérôme Hergueux <jerome.hergueux(a)gmail.com>
I do, however, have concerns about any research
that expects to contact
40,000 editors and involve 1500 of them; that is a very significant
portion
of our active editorship on the English Wikipedia
project.
Commenting on this: out of those targeted 40,000 editors, 30,000 or so
are
*newly registered users*, so that the sample
remains somewhat
representative of the diversity we find on en:wp. The rest of it indeed
are
active contributors.
Regards,
Jérôme.
2011/12/10 Risker <risker.wp(a)gmail.com>
> On 9 December 2011 22:51, Dario Taraborelli <
dtaraborelli(a)wikimedia.org
>>> wrote:
>>
>>> I’d like to give everybody on this list some information on the
>>> Berkman/Sciences Po research project that many of you have been
>> discussing
>>> here.
>>>
>>> On Thursday the Wikimedia Foundation announced the launch of a
banner
> to
>>> support a study led by a team at the Berkman Center/Sciences Po and
>>> recruiting participants from the English Wikipedia editor community
> [1].
>>> The banner was taken down within hours of its launch after concerns
>> raised
>>> in various community forums (the Admin Noticeboard [2], the Village
> Pump
>>> Tech [3], various IRC channels and mailing lists such as
foundation-l
[4]
>> and internal-l [5]) that the design was confusing, that it was
perceived
> as
>> a commercial ad and that the community approval process and privacy
terms
>> were unclear and hardly visible.
>>
>> Here’s what happened until the launch, what went wrong after the
launch
>>> and what we are planning to do next.
>>>
>>> ==The prequel==
>>> This proposal went through a long review process, involving
community
>>
forums, the Research Committee and various WMF departments since
early
>> 2010.
>>>
>>> The Berkman research team first approached WMF to discuss this
study
in
>>> January 2010. They suggested a protocol to recruit English
Wikipedia
>>
contributors to participate in an early version of this study by
March
> 2010
>> and posted a proposal to the Administrators’ noticeboard to get
community
>> feedback [6]. The community response at that time opposed the
proposed
>>> recruitment protocol (posting individual invitation messages on
user
talk
>> pages). It was suggested instead that the recruitment should be
handled
>>> through a CentralNotice banner to be displayed to registered
editors,
but
>> concerns were raised on how to minimize the disruption.
>>
>> To address these concerns, the proposal went through a full review
with
>> the Wikimedia Research Committee, that
was completed in July 2011.
The
>> RCom
>>> evaluated the methods, the recruitment strategy, the language used
in
the
>> survey and approved the proposal pending a final solution for the
>> recruitment taking into account the concerns expressed by the
community
> [7].
>>
>> Based on suggestions made by community members (e.g. [8]) the
research
>>> team started to work on a technical solution to selectively
display a
>>
banner to a subset of registered editors of the English Wikipedia
meeting
>> certain eligibility conditions. WMF agreed to invest engineering
effort
>>> into a system that would allow CentralNotice to serve contents to a
>>> specific set of editors – functionality that would benefit future
>>> campaigns run by the community, chapters or the Foundation [9]
[10].
>>
>> A new CentralNotice backend was then designed to look up various
editor
>>> metrics (i.e. number of contributions, account registration date
and
>
editor
>> privileges) – all public information available from our database –
and
to
>> perform a participant eligibility check against these metrics. A
banner
>> would then be displayed to eligible
participants, posting the above
data
>> (user ID + editor metrics) along with a unique token to the server
> hosting
>> the survey upon clicking. On the landing page of the survey,
participants
>> would have the possibility to read the privacy terms of the survey
and
>>> decide whether to take it or not.
>>>
>>> Throughout the review process of this recruitment protocol, the
> research
>>> team received constant feedback from the Foundation’s legal team,
the
>>> community department, the tech
department and the communication
team
>
before
>> the campaign went live.
>>
>> The campaign was announced in the CentralNotice calendar one month
before
>> its launch [11] and the launch was with a post on the Foundation’s
blog.
>> The banner was enabled on December 8 at 11:00pm UTC. 800+
participants
>> completed the study within a few hours
since its launch. The banner
was
>> then taken down by a meta-admin a few
hours after the launch due to
the
>>> concerns described above.
>>>
>>> So what went wrong?
>>>
>>> ==A few explanations we owe you==
>>>
>>> • Is the Foundation running ads?
>>> No, this banner is a recruitment campaign for a research project
that
has
>> been thoroughly reviewed by the Research Committee. We have a long
>> tradition of supporting recruitment for research about our
communities
>> via
>>> various sitenotices. The methodology of this project is sound and
the
>>
recruitment method less invasive than thousands of individual
messages
>>> posted on user talk pages. We believe this research will help
advance
our
>> understanding of the dynamics of participation in our projects.
Receiving
>> support by the Research Committee implies that all published output
and
>>> anonymized data produced by this study will be made available under
> open
>>> licenses. [12] The banner also received full Wikimedia Foundation
>> approval
>>> before its launch.
>>>
>>> • Is this campaign conflicting with the fundraiser?
>>> No, this banner is running only for a subset of logged-in editors
for
> whom
>> the main fundraiser campaign has already been taken down. We
carefully
>> timed this campaign to minimize the
impact on the fundraiser and we
>> scheduled it on the CentralNotice calendar with a month notice for
this
>>> reason.
>>>
>>> • Is this campaign running at 100% on the English Wikipedia?
>>> No, the banner has been designed to target a subsample of the
English
>>
Wikipedia registered editor population. Based on estimates by the
> research
>> team, the eligibility criteria apply to about 10,000 very active
>> contributors and about 30,000 new editors of the English Wikipedia.
The
>> target number of completed responses is
1500.
>>
>> • Why does the banner include logos of organizations not affiliated
with
>> Wikimedia?
>> The design of the banner was based on the decision to give
participants
> as
>> much information as possible about the research team running the
project
>> and to set accurate expectations about the study.
>>
>>
>> ==What we are doing now==
>>
>> We realize that despite an extensive review, the launch of this
project
>> was not fully advertised on community
forums. We plan to shortly
resume
>> the
>>> campaign (for the time needed by the researchers to complete their
>>> responses) after a full redesign of the recruitment protocol in
order
to
>> address the concerns raised by many of you over the last 24 hours.
Here’s
>> what we are doing:
>>
>> • Provide you with better information about the project
>> We asked the research team to promptly set up a FAQ section on the
> project
>> page on Meta [13], and to be available to address any concern about
the
>> study on the discussion page of this
project. The project page on
Meta
>> will
>>> be linked from the recruitment banner itself.
>>>
>>> • Redesign the banner
>>> We understand that the banner design has been interpreted by some
as
>>> ad-like (even if the goal was to make
clear that this study was not
> being
>>> run by WMF, as it implied a redirection to a third party website
for
>>
performing the experiment). In coordination with the research team,
we
> will
>> come up with a banner design that will be more in line with the
concerns
>> expressed by the community (for instance by removing the logos from
the
>> banner).
>>
>> • Make privacy terms as transparent as possible
>> Upon clicking on the banner, participants accept to share their
username,
>> edit count and user privileges with the research team. The previous
> version
>> didn’t make it explicit and we are working to address this problem.
To
>> make
>>> the process totally transparent we will make the acceptance of
these
>> terms
>>> explicit in the banner itself.
>>>
>>> Once redirected to the landing page, participants will have to
accept
> the
>>> terms of participation in order to enter the study. The project is
> funded
>>> by the European Research Council: the data collected in this study
is
>>> subject to strict European privacy
protocols. The research team
will
use
>> this data for research purposes only. The research team is not
exposed
> to
>>> and does not record participants’ IP addresses.
>>>
>>> ==How you can help==
>>>
>>> We would like to hear from you on the redesign of the banner to
make
> sure
>>> it meets the expectations of the community and doesn’t lend itself
to
> any
>>> kind of confusion. We will post the new banners to Meta and try to
>> address
>>> all pending questions before we resume the campaign.
>>>
>>> This is one of the first times we’re supporting a complex,
important
>>
research initiative like this one, and I apologize for the bumps in
the
>>> road. We believe that supporting research is part of our mission:
it
>
helps
>> advance our understanding of ourselves. So thanks again for all
support
>> you
>>> can give in making this a success.
>>>
>>>
>>> Dario Taraborelli
>>> Senior Research Analyst, Wikimedia Foundation
>>>
>>> [1]
http://blog.wikimedia.org/2011/12/08/experiment-decision-making/
http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Incide…
http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Search_…
http://lists.wikimedia.org/pipermail/foundation-l/2011-December/070742.html
https://lists.wikimedia.org/mailman/private/internal-l/2011-December/018842…
http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archiv…
http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interaction…
>>> [8]
>>
http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065580.html
>>> [9]
>>
http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065558.html
>>> [10]
http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines
http://meta.wikimedia.org/w/index.php?title=CentralNotice/Calendar&oldi…
meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and_Behavi…
>>>
>>>
>>
>> Dario, nobody in any of the discussions on the English Wikipedia
(whose
>> editors are the target of this research
project) suggested that a
> *central
>> site notice* be used for this or any other research project. The
>> discussion in April 2011 showed consensus opposition to bot-delivered
> talk
>> page notices. One editor involved in the discussion suggested "site
>> notices" (which I believe were interpreted by the participants to
mean
a
> local site notice) and two others mentioned
watchlist notices. The
> subsequent discussion about central notices discussed the possibility
of
> developing a narrowcasting ability for such
notices, and discussed
> specifically notices directly related to WMF projects or activities.
It
> did not, in any way, address the concept of
using a central notice to
> promote a non-WMF activity (such as this research project). Indeed,
this
> is
>> the first use of a central notice for anything not directly related
to
an
> obviously WMF-related activity.
>
> The ability to narrowcast central notices is a positive advancement;
> however, the processes for proposing and determining the
appropriateness
of
> a narrowcast are poorly publicized, and some of them don't appear to
have
>> even existed until after this notice was taken down. There are still
no
>> community-approved guidelines for the use
of central notices,
although
a
> draft one is currently up for comment.[1] An
RFC initiated in August
2010
> with respect to "global
banners"/central notices, well in advance of
the
>> development of the narrowcasting ability, strongly supported
consensus
>> approval on Meta for non-fundraising
global banners.[2] Now that
there
is
>> the ability to target central notices to only one project or
community,
> it
>> is extremely important that that community be directly notified of
such
>
discussion - a discussion that never took place in any public forum
that
> I
>> can see in advance of this central notice being activated.
>>
>> The links above include one to a private mailing list that the
majority
> of
>> readers of this list have no access to. You may want to consider
asking
> the
>> persons whose contributions are contained in that particular message
to
> grant
permission for it to be reproduced here so that the rest of us
aren't
> left in the dark about who said what.
>
> I don't begridge scholars carrying out approved research with
Wiki?edians
>> who volunteer to do so; in fact, I've responded to several requests
> myself.
>> I do, however, have concerns about any research that expects to
contact
>> 40,000 editors and involve 1500 of them;
that is a very significant
> portion
>> of our active editorship on the English Wikipedia project. I'm
curious
> to
>> know if scholars have shown much interest in studying some of the
other
>> projects as much as they've initiated
studies on enwp.
>>
>> Risker/Anne
>>
>>
>>
>> [1]
http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines
>>
http://meta.wikimedia.org/wiki/Requests_for_comment/Global_banners
>> _______________________________________________
>> foundation-l mailing list
>> foundation-l(a)lists.wikimedia.org
>> Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/foundation-l
>
_______________________________________________
foundation-l mailing list
foundation-l(a)lists.wikimedia.org
Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/foundation-l
_______________________________________________
foundation-l mailing list
foundation-l(a)lists.wikimedia.org
Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/foundation-l
_______________________________________________
foundation-l mailing list
foundation-l(a)lists.wikimedia.org
Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/foundation-l
_______________________________________________
foundation-l mailing list
foundation-l(a)lists.wikimedia.org
Unsubscribe: