thanks everybody, good food for thought. Here are my 2 (or 3) cents
==Community research==
I am under the impression that we are currently applying a double standard to community
research vs academic research when it comes to SR requests. To name a case that was
mentioned in this thread, I am very fond of the work Sarah is doing on gender gap but
it's unfortunate that the project wasn't properly documented and reviewed on R:I.
As WSC noted earlier on the list, there are issues of privacy and data retention (on top
of recruitment methods) that community members are often unaware of. There was a recent
case of a community-driven survey (one I was partly involved with as a WMF contact) that
failed to display appropriate terms of participation on the landing page. As a result the
data collected could not be shared with the rest of the community, which generated some
hostile reaction by other individuals who were interested in crunching the data. We should
be pointing community members who wish to run research projects involving SR to RCom in
the same way as we capture any attempts by external researchers to contact editors without
RCom support/approval.
==The ruthless researcher==
A vocal number of community members seem to move from the assumption that research by
community members is by definition healthy, useful and unproblematic and external research
is by definition intrusive and potentially irrelevant to the real problems we should be
focusing on. As a result the RCom's role is often perceived as that of a gatekeeper to
protect the community against the ruthless academic researcher. I'd like to think that
part of the role of RCom is to change that perception and to help push the idea that if
anything we should not stop researchers who want to help understand our communities and
crunch our data but support them. Many WP researchers (especially those who study
community dynamics) are active Wikimedians (I am one of them, since 2004) and are at least
as vital to our community as MediaWiki hackers are. We can decide to introduce measures
that raise the barrier for researchers to study our communities, but let's consider
that what we know about Wikipedia's editor dynamics/motivation/participation comes
almost entirely from scholarly research. What we should start exploring is the idea of
actively pitching research questions to researchers, I'd love to see some of the RCom
members who are more closely involved with the community take the lead on this initiative.
Someone I met at WikiSym was also very excited at this idea and maybe we can start a
dedicated taskforce.
==Surveys, surveys, surveys==
I am not a big fan of surveys, I think there is an annoying imbalance in the SR requests
we have been reviewing so far: with only a few exceptions most of these requests are for
survey participants. I want to make sure that when we say that subject recruitment should
be better controlled we distinguish between surveys and experimentation. The Berkman study
itself includes a survey but is mainly a behavioral experiment. As I said in earlier
discussions I believe the omnibus survey is a theoretical but practically unmanageable
solution (I can give you some background of what it takes to run the editor/editor surveys
in terms of logistics, translations etc if you're interested). If we only support
requests for inclusion in an omnibus survey that will imply the de facto end of surveys by
external researchers, but I want to stress that if we entirely shut down SR we are also
putting an end to experimentation.
==Recruitment vs recruitment methods==
As others already noted, the problem with Berkman was the recruitment method, not
recruitment per se. It's unfortunate that the method (selective displaying CN banners
to eligible users instead of posting user talk messages), timing (after the end of the
fundraiser) and design (displaying logos of the research institutions involved) that were
designed to minimize disruption and increase the transparency of the campaign were
considered unacceptable: we obviously need to take community concerns seriously and
rethink how we communicate and gauge community consensus. At the same time I keep hearing
that research requests are disruptive, but I haven't seen any direct evidence other
than people occasionally reporting this problem. I mean this as a genuine, not a
rhetorical question as I am trying to figure out how someone saying "hey we are
interested in what you are doing as a Wikipedian" is more disruptive than templated
messages telling new users that they did something wrong. I'd like to better
understand which classes of users are under particular stress with SR requests and
whether, depending on what the target population of a study is, we can adjust the
recruitment method (for example is some method acceptable for newbies but disruptive for
admins or highly active editors and vice versa)
Dario
On Dec 14, 2011, at 8:48 AM, Aaron Halfaker wrote:
Mayo, you bring up a very good point. I too feel that
it is the minority of editors who are at all upset with research recruitment on Wikipedia.
Asking about research recruitment on the next editor survey seems like a good idea.
I might also offer that the real problem here was the method of recruitment, not
recruitment itself. This is something I hope to bring up in our meeting. I think that
the study would have gone much more smoothly if we had a mechanism to request the
participation of individual editors that did not appear in such a prominent place like the
central notice banner, but instead was more like a personal request to an individual.
As we consider research recruitment, I want to make sure that the conversation is not
framed around the supposition that recruitment itself is the problem. Instead, I think we
are looking at a problem related to the method of recruitment and ensuring that method
bother editors as little as possible.
-Aaron
On Wed, Dec 14, 2011 at 10:24 AM, Fuster, Mayo <Mayo.Fuster(a)eui.eu> wrote:
Hello!
I hope you are fine.
Dario I already moved in order that Goran has access to the survey.
WSC your comments and suggestions seems to strongly assume that there is a consensus on
the need to "limiting the amount of surveying that Wikimedians are subjected
to". Which is the base for this statement?. Do we have any strong indicator to stand
that there is too much request or that this is not the case?. At least on the base of
Berkman episode, I would not arrive to that conclusion. Certainly, it does not represent
my interpretation: I don't think in the community there is a predominance of a
rejection attitude. To me developing research is a way to contribute and beneficial to
Wikipedia - but again, beyond each impression on community position and each own personal
position on this. we don't have a strong indicator or elaborate analysis of the
approach of the community toward research.
In something I think we need to reflect on is that in this stage of things - and from
Berkman and Sarah experience- researchers can extract the conclusion that it is better to
not get in contact with Rcom and it is better not to consult the community on your
recruitment method - you would save much more time and effort . There is something that
it is not working, if this is the case. In this regard, I would not think in terms of how
to control and limit the amount of research developed (also because it would be very very
difficult) but instead value and incentive that it is done in a way in concordance with
how Wikipedians view about how should be done (in terms of recruitment process, in terms
of open data, in terms of assuring the results arrive to the community, in terms of
addressing questions relevant for wikimedia goals, etc) and that is design in a way that
could be as much beneficial for the community as possible.
Cheers! Mayo
«·´`·.(*·.¸(`·.¸ ¸.·´)¸.·*).·´`·»
«·´¨*·¸¸« Mayo Fuster Morell ».¸.·*¨`·»
«·´`·.(¸.·´(¸.·* *·.¸)`·.¸).·´`·»
Research Digital Commons Governance:
http://www.onlinecreation.info
Fellow Berkman center for Internet and Society. Harvard University.
Postdoctoral Researcher. Institute of Govern and Public Policies. Autonomous University
of Barcelona.
Visiting scholar. Internet Interdisciplinary Institute. Open University of Catalonia
(UOC).
Member Research Committee. Wikimedia Foundation
Ph.D European University Institute
Visiting researcher (2008). School of information. University of California, Berkeley.
E-mail: mayo.fuster(a)eui.eu
E-mail: mayofm(a)cyber.law.harvard.edu
Twitter/Identica: Lilaroja
Skype: mayoneti
Phone United States: 001 - 8576548231
Phone Spanish State: 0034-648877748
Berkman Center
23 Everett Street, 2nd Floor
Cambridge, MA 02138
+1 (617) 495-7547 (Phone)
+1 (617) 495-7641 (Fax)
Personal Postal Address USA:
The Acetarium
http://www.acetarium.com/
265 Elm Street - 4
Somerville, MA, USA
02144
________________________________________
From: rcom-l-bounces(a)lists.wikimedia.org [rcom-l-bounces(a)lists.wikimedia.org] On Behalf
Of WereSpielChequers [werespielchequers(a)gmail.com]
Sent: 14 December 2011 16:09
To: The Wikimedia Foundation Research Committee mailing list
Subject: Re: [RCom-l] The tragedy of the Commons
Hi Yaroslav,
While I didn't see the actual survey I'm aware that it was run. I suspect that
the community would have little problem differentiating between a Wikimedian surveying a
targetted group of Wikimedians on currently contentious matters internal to the community
as opposed to an outside researcher surveying a large proportion of the community and
perhaps asking questions that don't seem very relevant. Sarah's survey could have
been done as part of an Omnibus, and I'm sure if we had an Omnibus survey it would be
an opportunity to do a followup.
Alternatively we could see it as part of my alternative option of targeted research -
unlike the Berkman survey Sarah did her targetting in such a way that she wasn't
blocked as spam.....
WSC
On 14 December 2011 14:58, Yaroslav M. Blanter
<putevod@mccme.ru<mailto:putevod@mccme.ru>> wrote:
On Wed, 14 Dec 2011 13:46:48 +0000, WereSpielChequers
<werespielchequers@gmail.com<mailto:werespielchequers@gmail.com>> wrote:
The controversy over Berkman is not in my view
primarily a communication
issue and it certainly isn't about the legitimacy of that survey. I
believe
that the community trusts RCom as a regulator of
research to know
whether
research is legitimate or not.
A big part of the controversy is over advertising, and I'm not convinced
that you can design a banner ad for a third party research survey that
isn't seen by some as advertising for that third party. An Omnibus
survey
could be a Wikimedia one and therefore I would
argue an internal ad
rather
than a third party one. Perhaps that isn't
our only option, and maybe
there
are alternative ways to solve that, one way would
be to change policy to
allow advertising for bona fide research. But that would be a difficult
one
to sell to the community, particularly on the
heels of a fundraising
drive
where "Wikipedia doesn't take ads"
was a core message.
The other aspect of being a regulator of research is the issue of how we
control the amount of research requests made to the community. To my
mind
that is fundamental to what we should be doing,
and it is a major reason
for my being on this committee. But this is almost an opposite thought
process to "promoting research".
There are two proposals that I've made as to how we do this, one would
be
to contact everyone once a year with an Omnibus
survey, the other rather
more complex one is to throttle back research surveying by volume and
limit
each campaign to a small subset of the community.
The two approaches can
be
hybridised by rewarding institutions that
collaborate by allowing them
to
use our systems to approach a larger proportion
of editors. One reason
why
I was opposed to the Berkman survey was that it
was the worst of both
worlds - one single research project going to all or almost all of our
most
surveyed community.
I'm not convinced that the community currently has confidence in RCom to
regulate the amount of research requests that wikimedians and especially
English language Wikipedians are exposed to. Nor am I convinced that
everyone on this committee regards that as our responsibility. To my
mind
this gives us a couple of possibilities, one
would be to try and agree a
mechanism for limiting the amount of surveying that Wikimedians are
subjected to, and then sell that to the community via a request for
comment. One option in any such request for comment could be for the
community to agree not to put any constraints on researchers, but I'd be
surprised if that option got consensus however strongly it was promoted
by
some members of RCom. The other possibility would
be to clarify that the
remit of this committee is to promote legitimate research by vetting
proposals and otherwise communicating with the community; and to inform
the
community that if it wants to put constraints on
legitimate researchers
contacting wikimedians via the site then it needs a an additional
process
other than RCom.
WereSpielChequers
Thanks for your ideas, which I find very much reasonable. I have an
immediate objection though. Not all research goes through RCom, and we have
no means to stop any single person or organization from sending a hundred
messages to talk pages. For instance, recently it was a survey with the
purpose of understanding the role of the female editors, or whatever the
purpose was (It is difficult for me to find a link immediately, but it can
be done, I guess it was run by Sarah Stierch and colleagues). They did not
bother to go to RCom, and I could imagine what the response were if we
demanded that for instance this survey would become part of Omnibus. Since
it looks almost inevitable that we have to go and ask the community
opinions at some stage, we probably also need to ask this question: Should
every research requiring subject recruitment be regulated (reviewed) by
RCom in advance, or may be the community (first robably of en.wp) just does
not want any regulation of the subject recruitment.
Cheers
Yaroslav
_______________________________________________
RCom-l mailing list
RCom-l@lists.wikimedia.org<mailto:RCom-l@lists.wikimedia.org>
https://lists.wikimedia.org/mailman/listinfo/rcom-l
The information transmitted is intended only for the person or entity to which it is
addressed and may contain confidential and/or privileged material. Any review,
retransmission, dissemination, distribution, forwarding, or other use of, or taking of any
action in reliance upon, this information by persons or entities other than the intended
recipient is prohibited without the express permission of the sender. If you received this
communication in error, please contact the sender and delete the material from any
computer.
_______________________________________________
RCom-l mailing list
RCom-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/rcom-l
_______________________________________________
RCom-l mailing list
RCom-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/rcom-l