[Foundation-l] Regarding Berkman/Sciences Po study

Sat Dec 10 05:42:11 UTC 2011

I do, however, have concerns about any research that expects to contact
40,000 editors and involve 1500 of them; that is a very significant portion
of our active editorship on the English Wikipedia project.

Commenting on this: out of those targeted 40,000 editors, 30,000 or so are
*newly registered users*, so that the sample remains somewhat
representative of the diversity we find on en:wp. The rest of it indeed are
active contributors.

Regards,

Jérôme.

2011/12/10 Risker <risker.wp at gmail.com>

> On 9 December 2011 22:51, Dario Taraborelli <dtaraborelli at wikimedia.org
> >wrote:
>
> > I’d like to give everybody on this list some information on the
> > Berkman/Sciences Po research project that many of you have been
> discussing
> > here.
> >
> > On Thursday the Wikimedia Foundation announced the launch of a banner to
> > support a study led by a team at the Berkman Center/Sciences Po and
> > recruiting participants from the English Wikipedia editor community [1].
> > The banner was taken down within hours of its launch after concerns
> raised
> > in various community forums (the Admin Noticeboard [2], the Village Pump
> > Tech [3], various IRC channels and mailing lists such as foundation-l [4]
> > and internal-l [5]) that the design was confusing, that it was perceived
> as
> > a commercial ad and that the community approval process and privacy terms
> > were unclear and hardly visible.
> >
> > Here’s what happened until the launch, what went wrong after the launch
> > and what we are planning to do next.
> >
> > ==The prequel==
> > This proposal went through a long review process, involving community
> > forums, the Research Committee and various WMF departments since early
> 2010.
> >
> > The Berkman research team first approached WMF to discuss this study in
> > January 2010. They suggested a protocol to recruit English Wikipedia
> > contributors to participate in an early version of this study by March
> 2010
> > and posted a proposal to the Administrators’ noticeboard to get community
> > feedback [6]. The community response at that time opposed the proposed
> > recruitment protocol (posting individual invitation messages on user talk
> > pages). It was suggested instead that the recruitment should be handled
> > through a CentralNotice banner to be displayed to registered editors, but
> > concerns were raised on how to minimize the disruption.
> >
> > To address these concerns, the proposal went through a full review with
> > the Wikimedia Research Committee, that was completed in July 2011. The
> RCom
> > evaluated the methods, the recruitment strategy, the language used in the
> > survey and approved the proposal pending a final solution for the
> > recruitment taking into account the concerns expressed by the community
> [7].
> >
> > Based on suggestions made by community members (e.g. [8]) the research
> > team started to work on a technical solution to selectively display a
> > banner to a subset of registered editors of the English Wikipedia meeting
> > certain eligibility conditions. WMF agreed to invest engineering effort
> > into a system that would allow CentralNotice to serve contents to a
> > specific set of editors –  functionality that would benefit future
> > campaigns run by the community, chapters or the Foundation [9] [10].
> >
> > A new CentralNotice backend was then designed to look up various editor
> > metrics (i.e. number of contributions, account registration date and
> editor
> > privileges) – all public information available from our database – and to
> > perform a participant eligibility check against these metrics. A banner
> > would then be displayed to eligible participants, posting the above data
> > (user ID + editor metrics) along with a unique token to the server
> hosting
> > the survey upon clicking. On the landing page of the survey, participants
> > would have the possibility to read the privacy terms of the survey and
> > decide whether to take it or not.
> >
> > Throughout the review process of this recruitment protocol, the research
> > team received constant feedback from the Foundation’s legal team, the
> > community department, the tech department and the communication team
> before
> > the campaign went live.
> >
> > The campaign was announced in the CentralNotice calendar one month before
> > its launch [11] and the launch was with a post on the Foundation’s blog.
> > The banner was enabled on December 8 at 11:00pm UTC. 800+ participants
> > completed the study within a few hours since its launch. The banner was
> > then taken down by a meta-admin a few hours after the launch due to the
> > concerns described above.
> >
> > So what went wrong?
> >
> > ==A few explanations we owe you==
> >
> > • Is the Foundation running ads?
> > No, this banner is a recruitment campaign for a research project that has
> > been thoroughly reviewed by the Research Committee. We have a long
> > tradition of supporting recruitment for research about our communities
> via
> > various sitenotices. The methodology of this project is sound and the
> > recruitment method less invasive than thousands of individual messages
> > posted on user talk pages. We believe this research will help advance our
> > understanding of the dynamics of participation in our projects. Receiving
> > support by the Research Committee implies that all published output and
> > anonymized data produced by this study will be made available under open
> > licenses. [12] The banner also received full Wikimedia Foundation
> approval
> > before its launch.
> >
> > • Is this campaign conflicting with the fundraiser?
> > No, this banner is running only for a subset of logged-in editors for
> whom
> > the main fundraiser campaign has already been taken down. We carefully
> > timed this campaign to minimize the impact on the fundraiser and we
> > scheduled it on the CentralNotice calendar with a month notice for this
> > reason.
> >
> > • Is this campaign running at 100% on the English Wikipedia?
> > No, the banner has been designed to target a subsample of the English
> > Wikipedia registered editor population. Based on estimates by the
> research
> > team, the eligibility criteria apply to about 10,000 very active
> > contributors and about 30,000 new editors of the English Wikipedia. The
> > target number of completed responses is 1500.
> >
> > • Why does the banner include logos of organizations not affiliated with
> > Wikimedia?
> > The design of the banner was based on the decision to give participants
> as
> > much information as possible about the research team running the project
> > and to set accurate expectations about the study.
> >
> >
> > ==What we are doing now==
> >
> > We realize that despite an extensive review, the launch of this project
> > was not fully advertised on community forums. We plan to shortly resume
> the
> > campaign (for the time needed by the researchers to complete their
> > responses) after a full redesign of the recruitment protocol in order to
> > address the concerns raised by many of you over the last 24 hours. Here’s
> > what we are doing:
> >
> > • Provide you with better information about the project
> > We asked the research team to promptly set up a FAQ section on the
> project
> > page on Meta [13], and to be available to address any concern about the
> > study on the discussion page of this project. The project page on Meta
> will
> > be linked  from the recruitment banner itself.
> >
> > • Redesign the banner
> > We understand that the banner design has been interpreted by some as
> > ad-like (even if the goal was to make clear that this study was not being
> > run by WMF, as it implied a redirection to a third party website for
> > performing the experiment). In coordination with the research team, we
> will
> > come up with a banner design that will be more in line with the concerns
> > expressed by the community (for instance by removing the logos from the
> > banner).
> >
> > • Make privacy terms as transparent as possible
> > Upon clicking on the banner, participants accept to share their username,
> > edit count and user privileges with the research team. The previous
> version
> > didn’t make it explicit and we are working to address this problem. To
> make
> > the process totally transparent we will make the acceptance of these
> terms
> > explicit in the banner itself.
> >
> > Once redirected to the landing page, participants will have to accept the
> > terms of participation in order to enter the study. The project is funded
> > by the European Research Council: the data collected in this study is
> > subject to strict European privacy protocols. The research team will use
> > this data for research purposes only. The research team is not exposed to
> > and does not record participants’ IP addresses.
> >
> > ==How you can help==
> >
> > We would like to hear from you on the redesign of the banner to make sure
> > it meets the expectations of the community and doesn’t lend itself to any
> > kind of confusion. We will post the new banners to Meta and try to
> address
> > all pending questions before we resume the campaign.
> >
> > This is one of the first times we’re supporting a complex, important
> > research initiative like this one, and I apologize for the bumps in the
> > road. We believe that supporting research is part of our mission: it
> helps
> > advance our understanding of ourselves. So thanks again for all support
> you
> > can give in making this a success.
> >
> >
> > Dario Taraborelli
> > Senior Research Analyst, Wikimedia Foundation
> >
> > [1] http://blog.wikimedia.org/2011/12/08/experiment-decision-making/
> > [2]
> >
> http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Incidents#Harvard.2FScience_Po_Adverts
> > [3]
> >
> http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Search_banner_Wikipedia_Research_Committee
> > [4]
> >
> http://lists.wikimedia.org/pipermail/foundation-l/2011-December/070742.html
> > [5]
> >
> https://lists.wikimedia.org/mailman/private/internal-l/2011-December/018842.html
> > [6]
> >
> http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archive222#Researchers_requesting_administrators.E2.80.99_advices_to_launch_a_study
> > [7]
> >
> http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interactions_and_Behavior#RCom_review
> > [8]
> http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065580.html
> > [9]
> http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065558.html
> > [10] http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines
> > [11]
> >
> http://meta.wikimedia.org/w/index.php?title=CentralNotice/Calendar&oldid=3056067
> > [12] http://meta.wikimedia.org/wiki/Research:Subject_recruitment
> > [13]
> >
> meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and_Behavior
> >
> >
>
> Dario, nobody in any of the discussions on the English Wikipedia (whose
> editors are the target of this research project) suggested that a *central
> site notice* be used for this or any other research project.  The
> discussion in April 2011 showed consensus opposition to bot-delivered talk
> page notices. One editor involved in the discussion suggested "site
> notices" (which I believe were interpreted by the participants to mean a
> local site notice) and two others mentioned watchlist notices.  The
> subsequent discussion about central notices discussed the possibility of
> developing a narrowcasting ability for such notices, and discussed
> specifically notices directly related to WMF projects or activities.  It
> did not, in any way, address the concept of using a central notice to
> promote a non-WMF activity (such as this research project). Indeed, this is
> the first use of a central notice for anything not directly related to an
> obviously WMF-related activity.
>
> The ability to narrowcast central notices is a positive advancement;
> however, the processes for proposing and determining the appropriateness of
> a narrowcast are poorly publicized, and some of them don't appear to have
> even existed until after this notice was taken down. There are still no
> community-approved guidelines for the use of central notices, although a
> draft one is currently up for comment.[1] An RFC initiated in August 2010
> with respect to "global banners"/central notices, well in advance of the
> development of the narrowcasting ability, strongly supported consensus
> approval on Meta for non-fundraising global banners.[2] Now that there is
> the ability to target central notices to only one project or community, it
> is extremely important that that community be directly notified of such
> discussion - a discussion that never took place in any public forum that I
> can see in advance of this central notice being activated.
>
> The links above include one to a private mailing list that the majority of
> readers of this list have no access to. You may want to consider asking the
> persons whose contributions are contained in that particular message to
> grant permission for it to be reproduced here so that the rest of us aren't
> left in the dark about who said what.
>
> I don't begridge scholars carrying out approved research with Wiki?edians
> who volunteer to do so; in fact, I've responded to several requests myself.
> I do, however, have concerns about any research that expects to contact
> 40,000 editors and involve 1500 of them; that is a very significant portion
> of our active editorship on the English Wikipedia project.  I'm curious to
> know if scholars have shown much interest in studying some of the other
> projects as much as they've initiated studies on enwp.
>
> Risker/Anne
>
>
>
> [1] http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines
> http://meta.wikimedia.org/wiki/Requests_for_comment/Global_banners
> _______________________________________________
> foundation-l mailing list
> foundation-l at lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>