On 9 December 2011 22:51, Dario Taraborelli <dtaraborelli(a)wikimedia.org discussing > here. > > On Thursday the Wikimedia Foundation announced the launch of a banner

> support a study led by a team at the Berkman Center/Sciences Po and > recruiting participants from the English Wikipedia editor community

raised > in various community forums (the Admin Noticeboard [2], the Village

> Tech [3], various IRC channels and mailing lists such as foundation-l

> and internal-l [5]) that the design was confusing, that it was

as > a commercial ad and that the community approval process and privacy

2010. 2010 > and posted a proposal to the Administrators’ noticeboard to get

> feedback [6]. The community response at that time opposed the proposed > recruitment protocol (posting individual invitation messages on user

> pages). It was suggested instead that the recruitment should be handled > through a CentralNotice banner to be displayed to registered editors,

RCom > evaluated the methods, the recruitment strategy, the language used in

[7]. > > Based on suggestions made by community members (e.g. [8]) the research > team started to work on a technical solution to selectively display a > banner to a subset of registered editors of the English Wikipedia

editor > privileges) – all public information available from our database – and

> perform a participant eligibility check against these metrics. A banner > would then be displayed to eligible participants, posting the above

hosting > the survey upon clicking. On the landing page of the survey,

> would have the possibility to read the privacy terms of the survey and > decide whether to take it or not. > > Throughout the review process of this recruitment protocol, the

before > the campaign went live. > > The campaign was announced in the CentralNotice calendar one month

> its launch [11] and the launch was with a post on the Foundation’s

> The banner was enabled on December 8 at 11:00pm UTC. 800+ participants > completed the study within a few hours since its launch. The banner was > then taken down by a meta-admin a few hours after the launch due to the > concerns described above. > > So what went wrong? > > ==A few explanations we owe you== > > • Is the Foundation running ads? > No, this banner is a recruitment campaign for a research project that

via > various sitenotices. The methodology of this project is sound and the > recruitment method less invasive than thousands of individual messages > posted on user talk pages. We believe this research will help advance

> understanding of the dynamics of participation in our projects.

> support by the Research Committee implies that all published output and > anonymized data produced by this study will be made available under

approval whom research > team, the eligibility criteria apply to about 10,000 very active > contributors and about 30,000 new editors of the English Wikipedia. The > target number of completed responses is 1500. > > • Why does the banner include logos of organizations not affiliated

as > much information as possible about the research team running the

the > campaign (for the time needed by the researchers to complete their > responses) after a full redesign of the recruitment protocol in order

> address the concerns raised by many of you over the last 24 hours.

project will > be linked from the recruitment banner itself. > > • Redesign the banner > We understand that the banner design has been interpreted by some as > ad-like (even if the goal was to make clear that this study was not

will > come up with a banner design that will be more in line with the

> expressed by the community (for instance by removing the logos from the > banner). > > • Make privacy terms as transparent as possible > Upon clicking on the banner, participants accept to share their

version make terms > explicit in the banner itself. > > Once redirected to the landing page, participants will have to accept

> terms of participation in order to enter the study. The project is

> by the European Research Council: the data collected in this study is > subject to strict European privacy protocols. The research team will

> this data for research purposes only. The research team is not exposed

> and does not record participants’ IP addresses. > > ==How you can help== > > We would like to hear from you on the redesign of the banner to make

> it meets the expectations of the community and doesn’t lend itself to

address helps you

http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065580.html http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065558.html

Dario, nobody in any of the discussions on the English Wikipedia (whose editors are the target of this research project) suggested that a

site notice* be used for this or any other research project. The discussion in April 2011 showed consensus opposition to bot-delivered

page notices. One editor involved in the discussion suggested "site notices" (which I believe were interpreted by the participants to mean a local site notice) and two others mentioned watchlist notices. The subsequent discussion about central notices discussed the possibility of developing a narrowcasting ability for such notices, and discussed specifically notices directly related to WMF projects or activities. It did not, in any way, address the concept of using a central notice to promote a non-WMF activity (such as this research project). Indeed, this

the first use of a central notice for anything not directly related to an obviously WMF-related activity. The ability to narrowcast central notices is a positive advancement; however, the processes for proposing and determining the appropriateness

a narrowcast are poorly publicized, and some of them don't appear to have even existed until after this notice was taken down. There are still no community-approved guidelines for the use of central notices, although a draft one is currently up for comment.[1] An RFC initiated in August 2010 with respect to "global banners"/central notices, well in advance of the development of the narrowcasting ability, strongly supported consensus approval on Meta for non-fundraising global banners.[2] Now that there is the ability to target central notices to only one project or community,

is extremely important that that community be directly notified of such discussion - a discussion that never took place in any public forum that

can see in advance of this central notice being activated. The links above include one to a private mailing list that the majority

readers of this list have no access to. You may want to consider asking

persons whose contributions are contained in that particular message to grant permission for it to be reproduced here so that the rest of us

left in the dark about who said what. I don't begridge scholars carrying out approved research with Wiki?edians who volunteer to do so; in fact, I've responded to several requests

I do, however, have concerns about any research that expects to contact 40,000 editors and involve 1500 of them; that is a very significant

of our active editorship on the English Wikipedia project. I'm curious

know if scholars have shown much interest in studying some of the other projects as much as they've initiated studies on enwp. Risker/Anne [1] http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines http://meta.wikimedia.org/wiki/Requests_for_comment/Global_banners _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

Hi Jerome - please show me where it says that; I've not been able to

that interpretation at all. My understanding is that the 30,000 are

with fewer than 100 edits per month on average, not that they are new users. Risker/Anne 2011/12/10 Jérôme Hergueux <jerome.hergueux(a)gmail.com> portion are are dtaraborelli(a)wikimedia.org > > >wrote: > > > > > I’d like to give everybody on this list some information on the > > > Berkman/Sciences Po research project that many of you have been > > discussing > > > here. > > > > > > On Thursday the Wikimedia Foundation announced the launch of a

> to > > > support a study led by a team at the Berkman Center/Sciences Po and > > > recruiting participants from the English Wikipedia editor community > [1]. > > > The banner was taken down within hours of its launch after concerns > > raised > > > in various community forums (the Admin Noticeboard [2], the Village > Pump > > > Tech [3], various IRC channels and mailing lists such as

launch > > > and what we are planning to do next. > > > > > > ==The prequel== > > > This proposal went through a long review process, involving

early > > 2010. > > > > > > The Berkman research team first approached WMF to discuss this

in > > > January 2010. They suggested a protocol to recruit English

March proposed > > > recruitment protocol (posting individual invitation messages on

handled > > > through a CentralNotice banner to be displayed to registered

with The > > RCom > > > evaluated the methods, the recruitment strategy, the language used

community research > > > team started to work on a technical solution to selectively

effort > > > into a system that would allow CentralNotice to serve contents to a > > > specific set of editors – functionality that would benefit future > > > campaigns run by the community, chapters or the Foundation [9]

editor > > > metrics (i.e. number of contributions, account registration date

and banner and > > > decide whether to take it or not. > > > > > > Throughout the review process of this recruitment protocol, the > research > > > team received constant feedback from the Foundation’s legal team,

> > > community department, the tech department and the communication

participants was the > > > concerns described above. > > > > > > So what went wrong? > > > > > > ==A few explanations we owe you== > > > > > > • Is the Foundation running ads? > > > No, this banner is a recruitment campaign for a research project

communities > > via > > > various sitenotices. The methodology of this project is sound and

messages > > > posted on user talk pages. We believe this research will help

and > > > anonymized data produced by this study will be made available under > open > > > licenses. [12] The banner also received full Wikimedia Foundation > > approval > > > before its launch. > > > > > > • Is this campaign conflicting with the fundraiser? > > > No, this banner is running only for a subset of logged-in editors

carefully this > > > reason. > > > > > > • Is this campaign running at 100% on the English Wikipedia? > > > No, the banner has been designed to target a subsample of the

The participants project resume > > the > > > campaign (for the time needed by the researchers to complete their > > > responses) after a full redesign of the recruitment protocol in

the Meta > > will > > > be linked from the recruitment banner itself. > > > > > > • Redesign the banner > > > We understand that the banner design has been interpreted by some

> > > ad-like (even if the goal was to make clear that this study was not > being > > > run by WMF, as it implied a redirection to a third party website

we the To > > make > > > the process totally transparent we will make the acceptance of

> > terms > > > explicit in the banner itself. > > > > > > Once redirected to the landing page, participants will have to

> the > > > terms of participation in order to enter the study. The project is > funded > > > by the European Research Council: the data collected in this study

> > > subject to strict European privacy protocols. The research team

exposed > to > > > and does not record participants’ IP addresses. > > > > > > ==How you can help== > > > > > > We would like to hear from you on the redesign of the banner to

> sure > > > it meets the expectations of the community and doesn’t lend itself

> any > > > kind of confusion. We will post the new banners to Meta and try to > > address > > > all pending questions before we resume the campaign. > > > > > > This is one of the first times we’re supporting a complex,

the > > > road. We believe that supporting research is part of our mission:

support > > you > > > can give in making this a success. > > > > > > > > > Dario Taraborelli > > > Senior Research Analyst, Wikimedia Foundation > > > > > > [1]

> > > [8] > >

> > > [9] > >

> > > [10]

> > > > > > > > > > Dario, nobody in any of the discussions on the English Wikipedia

> > editors are the target of this research project) suggested that a > *central > > site notice* be used for this or any other research project. The > > discussion in April 2011 showed consensus opposition to bot-delivered > talk > > page notices. One editor involved in the discussion suggested "site > > notices" (which I believe were interpreted by the participants to

a of It this > is > > the first use of a central notice for anything not directly related

an appropriateness have > > even existed until after this notice was taken down. There are still

> > community-approved guidelines for the use of central notices,

a 2010 the > > development of the narrowcasting ability, strongly supported

> > approval on Meta for non-fundraising global banners.[2] Now that

is > > the ability to target central notices to only one project or

> it > > is extremely important that that community be directly notified of

that > I > > can see in advance of this central notice being activated. > > > > The links above include one to a private mailing list that the

> of > > readers of this list have no access to. You may want to consider

> the > > persons whose contributions are contained in that particular message

Wiki?edians > > who volunteer to do so; in fact, I've responded to several requests > myself. > > I do, however, have concerns about any research that expects to

> > 40,000 editors and involve 1500 of them; that is a very significant > portion > > of our active editorship on the English Wikipedia project. I'm

> to > > know if scholars have shown much interest in studying some of the

> > projects as much as they've initiated studies on enwp. > > > > Risker/Anne > > > > > > > > [1] http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines > > http://meta.wikimedia.org/wiki/Requests_for_comment/Global_banners > > _______________________________________________ > > foundation-l mailing list > > foundation-l(a)lists.wikimedia.org > > Unsubscribe:

_______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

Hi Jerome - The only documentation from the research team that I have seen so far

respect to the target participation is in the initial proposal on enwp

in 2010, when it was proposed to leave 40,000 talk page messages; there

no indication that 30,000 of them would be newly registered users at that time. Not to criticize the genuine attempt at information sharing on Dario's part - it is much appreciated - but there is so much change in

was put forward from what we had initially been approached about that

preferable to hear it from the researcher's mouth, and to have it well documented. Something that has never been clear is the reason that English Wikipedia editors were identified as the preferred target; there does not appear to be anything in this study that is particularly oriented toward Wikipedia activity. Risker/Anne 2011/12/10 Jérôme Hergueux <jerome.hergueux(a)gmail.com> > This is actually not the case. Those 30,000 users or so are users who > registered their Wikipedia account 30 days prior to the launch of the > study. There are no other requirements for those users to be eligible to > participate. This is in line with Dario's previous message: > > the banner has been designed to target a subsample of the English

> registered editor population. Based on estimates by the research team,

> eligibility criteria apply to about 10,000 very active contributors and > about 30,000 new editors of the English Wikipedia. > > Regards, > > Jérôme. > > 2011/12/10 Risker <risker.wp(a)gmail.com> > >> Hi Jerome - please show me where it says that; I've not been able to > verify >> that interpretation at all. My understanding is that the 30,000 are > users >> with fewer than 100 edits per month on average, not that they are new >> users. >> >> Risker/Anne >> >> 2011/12/10 Jérôme Hergueux <jerome.hergueux(a)gmail.com> >> >>> I do, however, have concerns about any research that expects to

>>> 40,000 editors and involve 1500 of them; that is a very significant >> portion >>> of our active editorship on the English Wikipedia project. >>> >>> Commenting on this: out of those targeted 40,000 editors, 30,000 or so >> are >>> *newly registered users*, so that the sample remains somewhat >>> representative of the diversity we find on en:wp. The rest of it

>> are >>> active contributors. >>> >>> Regards, >>> >>> Jérôme. >>> >>> >>> 2011/12/10 Risker &lt;risker.wp(a)gmail.com&gt; >>> >>>> On 9 December 2011 22:51, Dario Taraborelli < >> dtaraborelli(a)wikimedia.org >>>>> wrote: >>>> >>>>> I’d like to give everybody on this list some information on the >>>>> Berkman/Sciences Po research project that many of you have been >>>> discussing >>>>> here. >>>>> >>>>> On Thursday the Wikimedia Foundation announced the launch of a > banner >>> to >>>>> support a study led by a team at the Berkman Center/Sciences Po and >>>>> recruiting participants from the English Wikipedia editor community >>> [1]. >>>>> The banner was taken down within hours of its launch after concerns >>>> raised >>>>> in various community forums (the Admin Noticeboard [2], the Village >>> Pump >>>>> Tech [3], various IRC channels and mailing lists such as > foundation-l >>> [4] >>>>> and internal-l [5]) that the design was confusing, that it was >>> perceived >>>> as >>>>> a commercial ad and that the community approval process and privacy >>> terms >>>>> were unclear and hardly visible. >>>>> >>>>> Here’s what happened until the launch, what went wrong after the >> launch >>>>> and what we are planning to do next. >>>>> >>>>> ==The prequel== >>>>> This proposal went through a long review process, involving > community >>>>> forums, the Research Committee and various WMF departments since >> early >>>> 2010. >>>>> >>>>> The Berkman research team first approached WMF to discuss this > study >> in >>>>> January 2010. They suggested a protocol to recruit English > Wikipedia >>>>> contributors to participate in an early version of this study by >> March >>>> 2010 >>>>> and posted a proposal to the Administrators’ noticeboard to get >>> community >>>>> feedback [6]. The community response at that time opposed the >> proposed >>>>> recruitment protocol (posting individual invitation messages on > user >>> talk >>>>> pages). It was suggested instead that the recruitment should be >> handled >>>>> through a CentralNotice banner to be displayed to registered > editors, >>> but >>>>> concerns were raised on how to minimize the disruption. >>>>> >>>>> To address these concerns, the proposal went through a full review >> with >>>>> the Wikimedia Research Committee, that was completed in July 2011. >> The >>>> RCom >>>>> evaluated the methods, the recruitment strategy, the language used > in >>> the >>>>> survey and approved the proposal pending a final solution for the >>>>> recruitment taking into account the concerns expressed by the >> community >>>> [7]. >>>>> >>>>> Based on suggestions made by community members (e.g. [8]) the >> research >>>>> team started to work on a technical solution to selectively > display a >>>>> banner to a subset of registered editors of the English Wikipedia >>> meeting >>>>> certain eligibility conditions. WMF agreed to invest engineering >> effort >>>>> into a system that would allow CentralNotice to serve contents to a >>>>> specific set of editors – functionality that would benefit future >>>>> campaigns run by the community, chapters or the Foundation [9] > [10]. >>>>> >>>>> A new CentralNotice backend was then designed to look up various >> editor >>>>> metrics (i.e. number of contributions, account registration date > and >>>> editor >>>>> privileges) – all public information available from our database – >> and >>> to >>>>> perform a participant eligibility check against these metrics. A >> banner >>>>> would then be displayed to eligible participants, posting the above >>> data >>>>> (user ID + editor metrics) along with a unique token to the server >>>> hosting >>>>> the survey upon clicking. On the landing page of the survey, >>> participants >>>>> would have the possibility to read the privacy terms of the survey >> and >>>>> decide whether to take it or not. >>>>> >>>>> Throughout the review process of this recruitment protocol, the >>> research >>>>> team received constant feedback from the Foundation’s legal team, > the >>>>> community department, the tech department and the communication > team >>>> before >>>>> the campaign went live. >>>>> >>>>> The campaign was announced in the CentralNotice calendar one month >>> before >>>>> its launch [11] and the launch was with a post on the Foundation’s >>> blog. >>>>> The banner was enabled on December 8 at 11:00pm UTC. 800+ >> participants >>>>> completed the study within a few hours since its launch. The banner >> was >>>>> then taken down by a meta-admin a few hours after the launch due to >> the >>>>> concerns described above. >>>>> >>>>> So what went wrong? >>>>> >>>>> ==A few explanations we owe you== >>>>> >>>>> • Is the Foundation running ads? >>>>> No, this banner is a recruitment campaign for a research project > that >>> has >>>>> been thoroughly reviewed by the Research Committee. We have a long >>>>> tradition of supporting recruitment for research about our >> communities >>>> via >>>>> various sitenotices. The methodology of this project is sound and > the >>>>> recruitment method less invasive than thousands of individual >> messages >>>>> posted on user talk pages. We believe this research will help > advance >>> our >>>>> understanding of the dynamics of participation in our projects. >>> Receiving >>>>> support by the Research Committee implies that all published output >> and >>>>> anonymized data produced by this study will be made available under >>> open >>>>> licenses. [12] The banner also received full Wikimedia Foundation >>>> approval >>>>> before its launch. >>>>> >>>>> • Is this campaign conflicting with the fundraiser? >>>>> No, this banner is running only for a subset of logged-in editors > for >>>> whom >>>>> the main fundraiser campaign has already been taken down. We >> carefully >>>>> timed this campaign to minimize the impact on the fundraiser and we >>>>> scheduled it on the CentralNotice calendar with a month notice for >> this >>>>> reason. >>>>> >>>>> • Is this campaign running at 100% on the English Wikipedia? >>>>> No, the banner has been designed to target a subsample of the > English >>>>> Wikipedia registered editor population. Based on estimates by the >>>> research >>>>> team, the eligibility criteria apply to about 10,000 very active >>>>> contributors and about 30,000 new editors of the English Wikipedia. >> The >>>>> target number of completed responses is 1500. >>>>> >>>>> • Why does the banner include logos of organizations not affiliated >>> with >>>>> Wikimedia? >>>>> The design of the banner was based on the decision to give >> participants >>>> as >>>>> much information as possible about the research team running the >>> project >>>>> and to set accurate expectations about the study. >>>>> >>>>> >>>>> ==What we are doing now== >>>>> >>>>> We realize that despite an extensive review, the launch of this >> project >>>>> was not fully advertised on community forums. We plan to shortly >> resume >>>> the >>>>> campaign (for the time needed by the researchers to complete their >>>>> responses) after a full redesign of the recruitment protocol in > order >>> to >>>>> address the concerns raised by many of you over the last 24 hours. >>> Here’s >>>>> what we are doing: >>>>> >>>>> • Provide you with better information about the project >>>>> We asked the research team to promptly set up a FAQ section on the >>>> project >>>>> page on Meta [13], and to be available to address any concern about >> the >>>>> study on the discussion page of this project. The project page on >> Meta >>>> will >>>>> be linked from the recruitment banner itself. >>>>> >>>>> • Redesign the banner >>>>> We understand that the banner design has been interpreted by some > as >>>>> ad-like (even if the goal was to make clear that this study was not >>> being >>>>> run by WMF, as it implied a redirection to a third party website > for >>>>> performing the experiment). In coordination with the research team, >> we >>>> will >>>>> come up with a banner design that will be more in line with the >>> concerns >>>>> expressed by the community (for instance by removing the logos from >> the >>>>> banner). >>>>> >>>>> • Make privacy terms as transparent as possible >>>>> Upon clicking on the banner, participants accept to share their >>> username, >>>>> edit count and user privileges with the research team. The previous >>>> version >>>>> didn’t make it explicit and we are working to address this problem. >> To >>>> make >>>>> the process totally transparent we will make the acceptance of > these >>>> terms >>>>> explicit in the banner itself. >>>>> >>>>> Once redirected to the landing page, participants will have to > accept >>> the >>>>> terms of participation in order to enter the study. The project is >>> funded >>>>> by the European Research Council: the data collected in this study > is >>>>> subject to strict European privacy protocols. The research team > will >>> use >>>>> this data for research purposes only. The research team is not >> exposed >>> to >>>>> and does not record participants’ IP addresses. >>>>> >>>>> ==How you can help== >>>>> >>>>> We would like to hear from you on the redesign of the banner to > make >>> sure >>>>> it meets the expectations of the community and doesn’t lend itself > to >>> any >>>>> kind of confusion. We will post the new banners to Meta and try to >>>> address >>>>> all pending questions before we resume the campaign. >>>>> >>>>> This is one of the first times we’re supporting a complex, > important >>>>> research initiative like this one, and I apologize for the bumps in >> the >>>>> road. We believe that supporting research is part of our mission: > it >>>> helps >>>>> advance our understanding of ourselves. So thanks again for all >> support >>>> you >>>>> can give in making this a success. >>>>> >>>>> >>>>> Dario Taraborelli >>>>> Senior Research Analyst, Wikimedia Foundation >>>>> >>>>> [1] > http://blog.wikimedia.org/2011/12/08/experiment-decision-making/ >>>>> [2] >>>>> >>>> >>> >> >

>>>>> [3] >>>>> >>>> >>> >> >

>>>>> [4] >>>>> >>>> >>> >> >

>>>>> [5] >>>>> >>>> >>> >> >

>>>>> [6] >>>>> >>>> >>> >> >

>>>>> [7] >>>>> >>>> >>> >> >

>>>>> [8] >>>> > http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065580.html >>>>> [9] >>>> > http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065558.html >>>>> [10] > http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines >>>>> [11] >>>>> >>>> >>> >> >

>>>>> [12] http://meta.wikimedia.org/wiki/Research:Subject_recruitment >>>>> [13] >>>>> >>>> >>> >> >

>>>>> >>>>> >>>> >>>> Dario, nobody in any of the discussions on the English Wikipedia > (whose >>>> editors are the target of this research project) suggested that a >>> *central >>>> site notice* be used for this or any other research project. The >>>> discussion in April 2011 showed consensus opposition to bot-delivered >>> talk >>>> page notices. One editor involved in the discussion suggested "site >>>> notices" (which I believe were interpreted by the participants to > mean >> a >>>> local site notice) and two others mentioned watchlist notices. The >>>> subsequent discussion about central notices discussed the possibility >> of >>>> developing a narrowcasting ability for such notices, and discussed >>>> specifically notices directly related to WMF projects or activities. >> It >>>> did not, in any way, address the concept of using a central notice to >>>> promote a non-WMF activity (such as this research project). Indeed, >> this >>> is >>>> the first use of a central notice for anything not directly related > to >> an >>>> obviously WMF-related activity. >>>> >>>> The ability to narrowcast central notices is a positive advancement; >>>> however, the processes for proposing and determining the >> appropriateness >>> of >>>> a narrowcast are poorly publicized, and some of them don't appear to >> have >>>> even existed until after this notice was taken down. There are still > no >>>> community-approved guidelines for the use of central notices, > although >> a >>>> draft one is currently up for comment.[1] An RFC initiated in August >> 2010 >>>> with respect to "global banners"/central notices, well in advance of >> the >>>> development of the narrowcasting ability, strongly supported > consensus >>>> approval on Meta for non-fundraising global banners.[2] Now that > there >> is >>>> the ability to target central notices to only one project or > community, >>> it >>>> is extremely important that that community be directly notified of > such >>>> discussion - a discussion that never took place in any public forum >> that >>> I >>>> can see in advance of this central notice being activated. >>>> >>>> The links above include one to a private mailing list that the > majority >>> of >>>> readers of this list have no access to. You may want to consider > asking >>> the >>>> persons whose contributions are contained in that particular message > to >>>> grant permission for it to be reproduced here so that the rest of us >>> aren't >>>> left in the dark about who said what. >>>> >>>> I don't begridge scholars carrying out approved research with >> Wiki?edians >>>> who volunteer to do so; in fact, I've responded to several requests >>> myself. >>>> I do, however, have concerns about any research that expects to > contact >>>> 40,000 editors and involve 1500 of them; that is a very significant >>> portion >>>> of our active editorship on the English Wikipedia project. I'm > curious >>> to >>>> know if scholars have shown much interest in studying some of the > other >>>> projects as much as they've initiated studies on enwp. >>>> >>>> Risker/Anne >>>> >>>> >>>> >>>> [1] http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines >>>> http://meta.wikimedia.org/wiki/Requests_for_comment/Global_banners >>>> _______________________________________________ >>>> foundation-l mailing list >>>> foundation-l(a)lists.wikimedia.org >>>> Unsubscribe: > https://lists.wikimedia.org/mailman/listinfo/foundation-l >>>> >>> _______________________________________________ >>> foundation-l mailing list >>> foundation-l(a)lists.wikimedia.org >>> Unsubscribe:

_______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

Having taken the survey - I honestly don't see what all the fuss is

Sure the banner could have been designed better - but this seems like a disproportionate reaction to that minor mistake in the grand scheme of a one year approval process that obviously was transparent. They're US

organizations, so why are people surprised they'd target en.WP? It's the largest WMF project, so logical to focus on it over say en.WT. Focusing

editors or people with accounts makes a lot of sense. I'm not really

why people seem to be all caught up in the numbers or target audience. I've worked on research that had much higher target numbers than this. This is one of the more bizarre (although not the most bizarre)

I've seen. I have no stake in the matter nor was I involved in its approval process, but as I was taking it I was thinking of many ways WMF and other

could utilize the data. Understanding people's altruistic behavior is vital to volunteer recruitment and fundraising efforts. I was impressed with how well it was put together, the explanations were especially well done. I'm not surprised people on enWP objected. You could suggest we

a search box or print feature and I'm confident at least a small vocal group would express displeasure without realizing they're features

present. I'm not saying they should be dismissed outright, but feel they should be taken in better context. Nothing anyone has said convinces me this is worth such in-depth and cyclical debate. Seems to have gone off the rail a bit... Thank you to the folks involved for providing responses and background information. Already gone beyond what I would have personally expected. Just my two cents - you may now return to finding problems with it. -greg aka varnent ------- Gregory Varnum Lead, Aequalitas Project Founding Principal, VarnEnt @GregVarnum fb.com/GregVarnum On Dec 10, 2011, at 1:27 AM, Risker wrote: with back was > no indication that 30,000 of them would be newly registered users at

what it's > preferable to hear it from the researcher's mouth, and to have it well > documented. > > Something that has never been clear is the reason that English

> editors were identified as the preferred target; there does not appear

> be anything in this study that is particularly oriented toward

> activity. > > Risker/Anne > > 2011/12/10 Jérôme Hergueux &lt;jerome.hergueux(a)gmail.com&gt; > >> This is actually not the case. Those 30,000 users or so are users who >> registered their Wikipedia account 30 days prior to the launch of the >> study. There are no other requirements for those users to be eligible

Wikipedia the >> eligibility criteria apply to about 10,000 very active contributors

contact >>>> 40,000 editors and involve 1500 of them; that is a very significant >>> portion >>>> of our active editorship on the English Wikipedia project. >>>> >>>> Commenting on this: out of those targeted 40,000 editors, 30,000 or

indeed >>> are >>>> active contributors. >>>> >>>> Regards, >>>> >>>> Jérôme. >>>> >>>> >>>> 2011/12/10 Risker &lt;risker.wp(a)gmail.com&gt; >>>> >>>>> On 9 December 2011 22:51, Dario Taraborelli < >>> dtaraborelli(a)wikimedia.org >>>>>> wrote: >>>>> >>>>>> I’d like to give everybody on this list some information on the >>>>>> Berkman/Sciences Po research project that many of you have been >>>>> discussing >>>>>> here. >>>>>> >>>>>> On Thursday the Wikimedia Foundation announced the launch of a >> banner >>>> to >>>>>> support a study led by a team at the Berkman Center/Sciences Po

>>>>>> recruiting participants from the English Wikipedia editor

>>>> [1]. >>>>>> The banner was taken down within hours of its launch after

>>>>> raised >>>>>> in various community forums (the Admin Noticeboard [2], the

>>>> Pump >>>>>> Tech [3], various IRC channels and mailing lists such as >> foundation-l >>>> [4] >>>>>> and internal-l [5]) that the design was confusing, that it was >>>> perceived >>>>> as >>>>>> a commercial ad and that the community approval process and

>>>> terms >>>>>> were unclear and hardly visible. >>>>>> >>>>>> Here’s what happened until the launch, what went wrong after the >>> launch >>>>>> and what we are planning to do next. >>>>>> >>>>>> ==The prequel== >>>>>> This proposal went through a long review process, involving >> community >>>>>> forums, the Research Committee and various WMF departments since >>> early >>>>> 2010. >>>>>> >>>>>> The Berkman research team first approached WMF to discuss this >> study >>> in >>>>>> January 2010. They suggested a protocol to recruit English >> Wikipedia >>>>>> contributors to participate in an early version of this study by >>> March >>>>> 2010 >>>>>> and posted a proposal to the Administrators’ noticeboard to get >>>> community >>>>>> feedback [6]. The community response at that time opposed the >>> proposed >>>>>> recruitment protocol (posting individual invitation messages on >> user >>>> talk >>>>>> pages). It was suggested instead that the recruitment should be >>> handled >>>>>> through a CentralNotice banner to be displayed to registered >> editors, >>>> but >>>>>> concerns were raised on how to minimize the disruption. >>>>>> >>>>>> To address these concerns, the proposal went through a full review >>> with >>>>>> the Wikimedia Research Committee, that was completed in July 2011. >>> The >>>>> RCom >>>>>> evaluated the methods, the recruitment strategy, the language used >> in >>>> the >>>>>> survey and approved the proposal pending a final solution for the >>>>>> recruitment taking into account the concerns expressed by the >>> community >>>>> [7]. >>>>>> >>>>>> Based on suggestions made by community members (e.g. [8]) the >>> research >>>>>> team started to work on a technical solution to selectively >> display a >>>>>> banner to a subset of registered editors of the English Wikipedia >>>> meeting >>>>>> certain eligibility conditions. WMF agreed to invest engineering >>> effort >>>>>> into a system that would allow CentralNotice to serve contents to

>>>>>> specific set of editors – functionality that would benefit future >>>>>> campaigns run by the community, chapters or the Foundation [9] >> [10]. >>>>>> >>>>>> A new CentralNotice backend was then designed to look up various >>> editor >>>>>> metrics (i.e. number of contributions, account registration date >> and >>>>> editor >>>>>> privileges) – all public information available from our database – >>> and >>>> to >>>>>> perform a participant eligibility check against these metrics. A >>> banner >>>>>> would then be displayed to eligible participants, posting the

>>>> data >>>>>> (user ID + editor metrics) along with a unique token to the server >>>>> hosting >>>>>> the survey upon clicking. On the landing page of the survey, >>>> participants >>>>>> would have the possibility to read the privacy terms of the survey >>> and >>>>>> decide whether to take it or not. >>>>>> >>>>>> Throughout the review process of this recruitment protocol, the >>>> research >>>>>> team received constant feedback from the Foundation’s legal team, >> the >>>>>> community department, the tech department and the communication >> team >>>>> before >>>>>> the campaign went live. >>>>>> >>>>>> The campaign was announced in the CentralNotice calendar one month >>>> before >>>>>> its launch [11] and the launch was with a post on the Foundation’s >>>> blog. >>>>>> The banner was enabled on December 8 at 11:00pm UTC. 800+ >>> participants >>>>>> completed the study within a few hours since its launch. The

>>> was >>>>>> then taken down by a meta-admin a few hours after the launch due

>>> the >>>>>> concerns described above. >>>>>> >>>>>> So what went wrong? >>>>>> >>>>>> ==A few explanations we owe you== >>>>>> >>>>>> • Is the Foundation running ads? >>>>>> No, this banner is a recruitment campaign for a research project >> that >>>> has >>>>>> been thoroughly reviewed by the Research Committee. We have a long >>>>>> tradition of supporting recruitment for research about our >>> communities >>>>> via >>>>>> various sitenotices. The methodology of this project is sound and >> the >>>>>> recruitment method less invasive than thousands of individual >>> messages >>>>>> posted on user talk pages. We believe this research will help >> advance >>>> our >>>>>> understanding of the dynamics of participation in our projects. >>>> Receiving >>>>>> support by the Research Committee implies that all published

>>> and >>>>>> anonymized data produced by this study will be made available

>>>> open >>>>>> licenses. [12] The banner also received full Wikimedia Foundation >>>>> approval >>>>>> before its launch. >>>>>> >>>>>> • Is this campaign conflicting with the fundraiser? >>>>>> No, this banner is running only for a subset of logged-in editors >> for >>>>> whom >>>>>> the main fundraiser campaign has already been taken down. We >>> carefully >>>>>> timed this campaign to minimize the impact on the fundraiser and

>>>>>> scheduled it on the CentralNotice calendar with a month notice for >>> this >>>>>> reason. >>>>>> >>>>>> • Is this campaign running at 100% on the English Wikipedia? >>>>>> No, the banner has been designed to target a subsample of the >> English >>>>>> Wikipedia registered editor population. Based on estimates by the >>>>> research >>>>>> team, the eligibility criteria apply to about 10,000 very active >>>>>> contributors and about 30,000 new editors of the English

>>> The >>>>>> target number of completed responses is 1500. >>>>>> >>>>>> • Why does the banner include logos of organizations not

>>>> with >>>>>> Wikimedia? >>>>>> The design of the banner was based on the decision to give >>> participants >>>>> as >>>>>> much information as possible about the research team running the >>>> project >>>>>> and to set accurate expectations about the study. >>>>>> >>>>>> >>>>>> ==What we are doing now== >>>>>> >>>>>> We realize that despite an extensive review, the launch of this >>> project >>>>>> was not fully advertised on community forums. We plan to shortly >>> resume >>>>> the >>>>>> campaign (for the time needed by the researchers to complete their >>>>>> responses) after a full redesign of the recruitment protocol in >> order >>>> to >>>>>> address the concerns raised by many of you over the last 24 hours. >>>> Here’s >>>>>> what we are doing: >>>>>> >>>>>> • Provide you with better information about the project >>>>>> We asked the research team to promptly set up a FAQ section on the >>>>> project >>>>>> page on Meta [13], and to be available to address any concern

>>> the >>>>>> study on the discussion page of this project. The project page on >>> Meta >>>>> will >>>>>> be linked from the recruitment banner itself. >>>>>> >>>>>> • Redesign the banner >>>>>> We understand that the banner design has been interpreted by some >> as >>>>>> ad-like (even if the goal was to make clear that this study was

>>>> being >>>>>> run by WMF, as it implied a redirection to a third party website >> for >>>>>> performing the experiment). In coordination with the research

>>> we >>>>> will >>>>>> come up with a banner design that will be more in line with the >>>> concerns >>>>>> expressed by the community (for instance by removing the logos

>>> the >>>>>> banner). >>>>>> >>>>>> • Make privacy terms as transparent as possible >>>>>> Upon clicking on the banner, participants accept to share their >>>> username, >>>>>> edit count and user privileges with the research team. The

>>>>> version >>>>>> didn’t make it explicit and we are working to address this

>>> To >>>>> make >>>>>> the process totally transparent we will make the acceptance of >> these >>>>> terms >>>>>> explicit in the banner itself. >>>>>> >>>>>> Once redirected to the landing page, participants will have to >> accept >>>> the >>>>>> terms of participation in order to enter the study. The project is >>>> funded >>>>>> by the European Research Council: the data collected in this study >> is >>>>>> subject to strict European privacy protocols. The research team >> will >>>> use >>>>>> this data for research purposes only. The research team is not >>> exposed >>>> to >>>>>> and does not record participants’ IP addresses. >>>>>> >>>>>> ==How you can help== >>>>>> >>>>>> We would like to hear from you on the redesign of the banner to >> make >>>> sure >>>>>> it meets the expectations of the community and doesn’t lend itself >> to >>>> any >>>>>> kind of confusion. We will post the new banners to Meta and try to >>>>> address >>>>>> all pending questions before we resume the campaign. >>>>>> >>>>>> This is one of the first times we’re supporting a complex, >> important >>>>>> research initiative like this one, and I apologize for the bumps

>>>>>> [8] >>>>> >>

>>>>>> [9] >>>>> >>

>>>>>> >>>>>> >>>>> >>>>> Dario, nobody in any of the discussions on the English Wikipedia >> (whose >>>>> editors are the target of this research project) suggested that a >>>> *central >>>>> site notice* be used for this or any other research project. The >>>>> discussion in April 2011 showed consensus opposition to

>>>> talk >>>>> page notices. One editor involved in the discussion suggested "site >>>>> notices" (which I believe were interpreted by the participants to >> mean >>> a >>>>> local site notice) and two others mentioned watchlist notices. The >>>>> subsequent discussion about central notices discussed the

>>> of >>>>> developing a narrowcasting ability for such notices, and discussed >>>>> specifically notices directly related to WMF projects or

>>> It >>>>> did not, in any way, address the concept of using a central notice

>>>>> promote a non-WMF activity (such as this research project). Indeed, >>> this >>>> is >>>>> the first use of a central notice for anything not directly related >> to >>> an >>>>> obviously WMF-related activity. >>>>> >>>>> The ability to narrowcast central notices is a positive

>>>>> however, the processes for proposing and determining the >>> appropriateness >>>> of >>>>> a narrowcast are poorly publicized, and some of them don't appear

>>> have >>>>> even existed until after this notice was taken down. There are

>> no >>>>> community-approved guidelines for the use of central notices, >> although >>> a >>>>> draft one is currently up for comment.[1] An RFC initiated in

>>> 2010 >>>>> with respect to "global banners"/central notices, well in advance

>>> the >>>>> development of the narrowcasting ability, strongly supported >> consensus >>>>> approval on Meta for non-fundraising global banners.[2] Now that >> there >>> is >>>>> the ability to target central notices to only one project or >> community, >>>> it >>>>> is extremely important that that community be directly notified of >> such >>>>> discussion - a discussion that never took place in any public forum >>> that >>>> I >>>>> can see in advance of this central notice being activated. >>>>> >>>>> The links above include one to a private mailing list that the >> majority >>>> of >>>>> readers of this list have no access to. You may want to consider >> asking >>>> the >>>>> persons whose contributions are contained in that particular

>> to >>>>> grant permission for it to be reproduced here so that the rest of

https://lists.wikimedia.org/mailman/listinfo/foundation-l >>>> >>> _______________________________________________ >>> foundation-l mailing list >>> foundation-l(a)lists.wikimedia.org >>> Unsubscribe:

>>> >> _______________________________________________ >> foundation-l mailing list >> foundation-l(a)lists.wikimedia.org >> Unsubscribe:

_______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

With the greatest of respect.. it shouldn't need the whole of RCOM, no statement is needed. Just for someone to go on-wiki and answer queries

if nothing else say - sorry we need to look into this, bear with us). The complaint Kim is making is that no one has done this. Tom

The problem is that the research committee made only a token effort at finding or following relevant onwiki policy or consensus , nor did they try to explain or correct their actions onwiki in a timely manner as per WIARM. Or where they did, they didn't follow up. Any of those 3 elements (Policy, Consensus, WIARM/BRD) each could and still can help bring people up to speed and reduce misunderstandings. That's part of what they're for, after all! I'm sure that people will be more supportive once things are sorted out in that way. Hmm, the research committee still hasn't made any onwiki statement at a relevant location that I can find. If this were a court case, RCom would pretty much have lost by default and/or forfeit already.

As I said, analyze and nitpick things to death. Does any of that above * really* matter?

This is actually not the case. Those 30,000 users or so are users who registered their Wikipedia account 30 days prior to the launch of the study. There are no other requirements for those users to be eligible to participate. This is in line with Dario's previous message: the banner has been designed to target a subsample of the English

registered editor population. Based on estimates by the research team,

eligibility criteria apply to about 10,000 very active contributors and about 30,000 new editors of the English Wikipedia. Regards, Jérôme. 2011/12/10 Risker &lt;risker.wp(a)gmail.com&gt; verify users > with fewer than 100 edits per month on average, not that they are new > users. > > Risker/Anne > > 2011/12/10 Jérôme Hergueux &lt;jerome.hergueux(a)gmail.com&gt; > > > I do, however, have concerns about any research that expects to

> > 40,000 editors and involve 1500 of them; that is a very significant > portion > > of our active editorship on the English Wikipedia project. > > > > Commenting on this: out of those targeted 40,000 editors, 30,000 or

> are > > *newly registered users*, so that the sample remains somewhat > > representative of the diversity we find on en:wp. The rest of it

banner > > to > > > > support a study led by a team at the Berkman Center/Sciences Po

> > > > recruiting participants from the English Wikipedia editor

> > [1]. > > > > The banner was taken down within hours of its launch after

> > > raised > > > > in various community forums (the Admin Noticeboard [2], the

foundation-l > > [4] > > > > and internal-l [5]) that the design was confusing, that it was > > perceived > > > as > > > > a commercial ad and that the community approval process and

community study Wikipedia user editors, > > but > > > > concerns were raised on how to minimize the disruption. > > > > > > > > To address these concerns, the proposal went through a full

> with > > > > the Wikimedia Research Committee, that was completed in July

> The > > > RCom > > > > evaluated the methods, the recruitment strategy, the language

in display a > > > > banner to a subset of registered editors of the English Wikipedia > > meeting > > > > certain eligibility conditions. WMF agreed to invest engineering > effort > > > > into a system that would allow CentralNotice to serve contents

> > > > specific set of editors – functionality that would benefit

[10]. and > > > editor > > > > privileges) – all public information available from our database

> and > > to > > > > perform a participant eligibility check against these metrics. A > banner > > > > would then be displayed to eligible participants, posting the

> > data > > > > (user ID + editor metrics) along with a unique token to the

> > > hosting > > > > the survey upon clicking. On the landing page of the survey, > > participants > > > > would have the possibility to read the privacy terms of the

the team > > > before > > > > the campaign went live. > > > > > > > > The campaign was announced in the CentralNotice calendar one

> > before > > > > its launch [11] and the launch was with a post on the

> > blog. > > > > The banner was enabled on December 8 at 11:00pm UTC. 800+ > participants > > > > completed the study within a few hours since its launch. The

> was > > > > then taken down by a meta-admin a few hours after the launch due

that > > has > > > > been thoroughly reviewed by the Research Committee. We have a

the advance > > our > > > > understanding of the dynamics of participation in our projects. > > Receiving > > > > support by the Research Committee implies that all published

> and > > > > anonymized data produced by this study will be made available

for > > > whom > > > > the main fundraiser campaign has already been taken down. We > carefully > > > > timed this campaign to minimize the impact on the fundraiser and

> > > > scheduled it on the CentralNotice calendar with a month notice

English > > > > Wikipedia registered editor population. Based on estimates by the > > > research > > > > team, the eligibility criteria apply to about 10,000 very active > > > > contributors and about 30,000 new editors of the English

> The > > > > target number of completed responses is 1500. > > > > > > > > • Why does the banner include logos of organizations not

> > with > > > > Wikimedia? > > > > The design of the banner was based on the decision to give > participants > > > as > > > > much information as possible about the research team running the > > project > > > > and to set accurate expectations about the study. > > > > > > > > > > > > ==What we are doing now== > > > > > > > > We realize that despite an extensive review, the launch of this > project > > > > was not fully advertised on community forums. We plan to shortly > resume > > > the > > > > campaign (for the time needed by the researchers to complete

order > > to > > > > address the concerns raised by many of you over the last 24

> > Here’s > > > > what we are doing: > > > > > > > > • Provide you with better information about the project > > > > We asked the research team to promptly set up a FAQ section on

> > > project > > > > page on Meta [13], and to be available to address any concern

as > > > > ad-like (even if the goal was to make clear that this study was

for > > > > performing the experiment). In coordination with the research

> we > > > will > > > > come up with a banner design that will be more in line with the > > concerns > > > > expressed by the community (for instance by removing the logos

> the > > > > banner). > > > > > > > > • Make privacy terms as transparent as possible > > > > Upon clicking on the banner, participants accept to share their > > username, > > > > edit count and user privileges with the research team. The

> > > version > > > > didn’t make it explicit and we are working to address this

these accept > > the > > > > terms of participation in order to enter the study. The project

> > funded > > > > by the European Research Council: the data collected in this

is will make > > sure > > > > it meets the expectations of the community and doesn’t lend

to > > any > > > > kind of confusion. We will post the new banners to Meta and try

important > > > > research initiative like this one, and I apologize for the bumps

it http://blog.wikimedia.org/2011/12/08/experiment-decision-making/

http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065580.html http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065558.html http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines

(whose > > > editors are the target of this research project) suggested that a > > *central > > > site notice* be used for this or any other research project. The > > > discussion in April 2011 showed consensus opposition to

mean > a > > > local site notice) and two others mentioned watchlist notices. The > > > subsequent discussion about central notices discussed the

> of > > > developing a narrowcasting ability for such notices, and discussed > > > specifically notices directly related to WMF projects or

> It > > > did not, in any way, address the concept of using a central notice

to > an > > > obviously WMF-related activity. > > > > > > The ability to narrowcast central notices is a positive

> > > however, the processes for proposing and determining the > appropriateness > > of > > > a narrowcast are poorly publicized, and some of them don't appear

> have > > > even existed until after this notice was taken down. There are

no although > a > > > draft one is currently up for comment.[1] An RFC initiated in

> 2010 > > > with respect to "global banners"/central notices, well in advance

consensus there community, such majority asking > > the > > > persons whose contributions are contained in that particular

to > > > grant permission for it to be reproduced here so that the rest of

contact curious other https://lists.wikimedia.org/mailman/listinfo/foundation-l > > > > > _______________________________________________ > > foundation-l mailing list > > foundation-l(a)lists.wikimedia.org > > Unsubscribe:

_______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

• Is the Foundation running ads? No, this banner is a recruitment campaign for a research project that has been thoroughly reviewed by the Research Committee. We have a long tradition of supporting recruitment for research about our communities via various sitenotices. The methodology of this project is sound and the recruitment method less invasive than thousands of individual messages posted on user talk pages. We believe this research will help advance our understanding of the dynamics of participation in our projects. Receiving support by the Research Committee implies that all published output and anonymized data produced by this study will be made available under open licenses. [12] The banner also received full Wikimedia Foundation approval before its launch.