Well, it seems they took some of your
advice. It did say “top 500 contributors” in its original form, but now says “any
interested Wikipedia user”.
Kerry
From:
Aaron Halfaker [mailto:aaron.halfaker@gmail.com]
Sent: Thursday, 31 July 2014 10:45
PM
To: Kerry Raymond; Research into
Wikimedia content and communities
Subject: Re: [Wiki-research-l]
discussion about wikipedia surveys
For example, this proposal[1] was sent to me last year. The
researcher's plan was not to sample from the pool of active editors, but
instead to contact the 500 most active
Wikipedians on enwiki to survey them about their motivations.
There were several concerns raised about (1) whether the proposed study
was duplicating prior work, (2) why the busiest editors needed to be surveyed
and (3) whether the researcher's methodology would allow for the intended
insights to be gained.
Regretfully, the researcher decided not to respond to anyone other than
myself and was also unwilling to work through many of the concerns I raised.
-Aaron
On Wed, Jul 30, 2014 at 8:19 PM, Kerry Raymond <kerry.raymond@gmail.com>
wrote:
30? No wonder we are worried about editor attrition J Seriously,
http://stats.wikimedia.org/EN/SummaryEN.htm
shows that in May 2014 on en.WP we had about 32K active editors
(> 5 edits per month) and 3K very active editors (>100 edits per month).
Or have I missed something here? Are researchers only interested in
people who have been on Wikipedia for 10+ years with 10M edits or …?
Kerry
From: wiki-research-l-bounces@lists.wikimedia.org
[mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Lane Rasberry
Sent: Wednesday, 30 July 2014
12:00 PM
To: Research into Wikimedia
content and communities
Subject: Re: [Wiki-research-l]
discussion about wikipedia surveys
Hey guys,
I posted some
thoughts to my own blog and am linking to those posts below. Everything I say
on my blog is captured in the summary below, so feel free to not click through.
----
My biggest worry
is that researchers who recruit human subjects assume that there are huge
numbers of Wikipedians for them to survey, and consequently, they do not need
to do a lot of advance survey preparation because there is no harm from
distracting Wikipedians from their usual volunteer work. This assumption is
wrong because actually almost every researcher recruiting human subjects wants
Wikipedians who are in very short supply. Consequently, researchers do cause
harm to the community by soliciting for volunteer time, and Wikipedia community
benefit is dubious when researchers do not do sufficient preparation for their
work. This is not quite accurate, but if there were one message I could convey
to researchers, it would be "Your research participant pool only consists
of about 30 super busy people and many other volunteers greatly depend on
getting their time. When you take time from a Wikipedian, you are taking that
time away from other volunteers who really need it, so be respectful of your
intervention in our communities."
----
I do not want a
lot of gatekeeping between researchers and the Wikipedia community, but at the
same time, researchers should take professional pride in their work and take
care not to disrupt Wikimedia community activities.
<http://bluerasberry.com/2014/07/request-for-researchers-when-doing-research-on-the-wikimedia-community/>
<http://bluerasberry.com/2014/07/problems-with-research-on-wikipedia/>
I am still
thinking about what should be done with research.
yours,
On Tue, Jul 29,
2014 at 6:00 PM, Dario Taraborelli <dtaraborelli@wikimedia.org>
wrote:
Hi all,
I am a bit late in
the game, but since so many questions were raised about RCom, its scope, its goals,
the source of its authority etc. and I helped coordinate it in the early days I
thought I’d chime in to clear some confusion.
Is
RCom an official WMF body or a group of volunteers?
RCom was created
as a volunteer body to help design policies and best practices around research
on Wikimedia projects. People who joined the committee did so on a volunteer
basis and with a variety of interests by responding to a call for participation
issued by WMF. Despite the fact that the original initiative came from WMF, its
membership almost entirely consisted of non-WMF researchers and community
members (those of us who are now with Wikimedia had no affiliation with the
Foundation when RCom was launched [1]). RCom work was and remains 100%
volunteer-driven, even for those of us who are full-time employees of the
Foundation.
Is
RCom a body regulating subject recruitment?
No, subject
recruitment was only one among many areas of interest identified by its
participants [2]
Is
RCom still alive?
RCom stopped working
a while ago as a group meeting on a regular basis to discuss joint
initiatives. However, it spawned a large number of initiatives and
workgroups that are still alive and kicking, some of which have evolved into
other projects that are now only loosely associated with RCom. These include
reviewing subject recruitment requests, but also the Research Newsletter, which
has been published monthly for the last 3 years; countless initiatives in the
area of open access; initiatives to facilitate Wikimedia data documentation and
data discoverability; hackathons and outreach events aimed at bringing together
researchers and Wikimedia contributors. Subject recruitment reviews and
discussions are still happening, and I believe they provide a valuable service
when you consider that they are entirely run by a microscopic number of
volunteers. I don’t think that the alternative between “either RCom exists and
it functions effectively or reviews should immediately stop” is well framed or
even desirable, for the reasons that I explain below.
What’s
the source of RCom’s authority in reviewing subject recruitment requests?
Despite the
perception that one of RCom’s duties would be to provide formal approval for
research projects, it was never designed to do so and it never had the power to
enforce formal review decisions. Instead, it was offered as a volunteer support
service in an effort to help minimize disruption, improve the relevance of
research involving Wikimedia contributors, sanity check the credentials of the researchers,
create collaborations between researchers working on the same topic. The lack
of community or WMF policies to back subject recruitment caused in the past few
years quite some headaches, particularly in those cases in which recruitment
attempts were blocked and referred to the RCom in order to “obtain formal
approval”. The review process itself was meant to be as inclusive as possible
and not restricted to RCom participants and researchers having their proposal
reviewed were explicitly invited to address any questions or concerns raised by
community members on the talk page. I totally agree that the way in which the
project templates and forms were designed needs some serious overhaul to remove
any indication of a binding review process or a commitment for reviews to be
delivered within a fixed time frame. I cannot think of any example in which the
review process discriminated some type of projects (say qualitative research)
in favor of other types of research, but I am sure different research proposals
attracted different levels of participation and interest in the review process.
My recommendation to anyone interested in designing future subject recruitment
processes is to focus on a lightweight review process open to the largest
possible number of community members but backed by transparent and enforceable policies. It’s a really hard
problem and there is simply no obvious silver bullet solution that can be found
without some experimentation and fault tolerance.
What
about requests for private data?
Private data and
technical support requests from WMF are a different story: they were folded
into the list of frequently asked questions hosted on the RCom section of Meta,
but by definition they require a direct and substantial involvement from the
Foundation: (1) they involve WMF as the legal entity that would be held liable
for disclosing data in breech of its privacy policies and (2) they involve paid
staff resources and need to be prioritized against a lot of other requests.
There are now dedicated sections on private data on the Wikimedia Privacy
Policy [4] and Data Retention guidelines [5]. Many people, including myself and
other members of the Foundation’s Analytics team, believe that we should try
and collect the minimum amount of private data that we need in order to operate
and study our projects and make all those types of aggregate/sanitized data
that we can retain indefinitely publicly available to everyone under open
licenses. We’ve already started a process to do so and to ensure that more data
(for example, data collected via site instrumentation [6]) be exposed via Labs
or other APIs, in the respect of our users’ privacy.
How
can we incentivize researchers to “give back” to the community?
In the early days
we drafted a set of requirements [3] to make sure we could get back as much as
possible from research involving WMF resources. It’s been hard to implement
these requirements without policies to enforce them. The suggestion of having
more researchers apply for a slot at the Research Showcase to present their
work is an excellent idea that we should consider. In general, the Research
team at WMF is always interested in hearing about incentives to drive more
interest towards actionable research on Wikimedia projects.
Dario
On Jul 29, 2014,
at 6:49 PM, Aaron Halfaker <aaron.halfaker@gmail.com> wrote:
Either [RCOM is] functioning or its not, surely?
Well, I explained
that there are functioning sub-committees still. In other words, there
are initiatives that RCOM started that are alive and successful, but we no
longer coordinate as a larger group. I don't know how else to explain it.
I guess you could say that RCOM is still functioning and that we no
longer require/engage in group meetings.
As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
That's a proposed
policy. Until it is passed by consensus, the "must" is a
proposed term. I think that it should be "must", but until that
consensus is reached, I'll continue to say that it "ought to".
Regarding
researchers stating what should be regulated, I think there is a big difference
between deciding what should be regulated
and being involved in the discussion of *how*
it should be regulated. Hence why I welcome participation.
What I'm saying is that you have a vested interest in not being
regulated, but I'd still like to discuss how your activities can be regulated
effectively & efficiently. Does that make sense?
b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
I welcome you to
raise it to them. I don't think it is worth their time, but they might
disagree.
But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I think that the
proposed policy on English Wikipedia does that quite well. That's why I
directed people there. Also, again, I am not working on RCOM or subject
recruitment as a WMF employee. I do this in my volunteer time. This
is true of all of RCOM who happen to also be staff.
if you want
process to be more clearly documented, you also have to address people like
Poitr who would rather not have processes described in detail. When you
guys work out how clearly you want a process to be described, please let me
know. I'm tired of re-spec'ing processes. This is the third
iteration.
If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.
Heather, that is a
proposed policy page on English
Wikipedia. It is not part of RCOM. It would render RCOM irrelevant
for subject recruitment concerns. That's why I started it. I don't
think that RCOM/WMF/researchers should own subject recruitment review. I
think the community being studied should own it and that RCOM/WMF/researchers
should participate.
Also, I am not
your employee. This is my volunteer time. I don't have much of it,
so I focus on keeping the system running -- and it is -- and improving the
system -- which is the proposal I linked to. If you want something done
and other volunteers don't have time to do it. Do it
yourself.
-Aaron
On Tue, Jul 29,
2014 at 10:23 AM, Heather Ford <hfordsa@gmail.com> wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <aaron.halfaker@gmail.com> wrote:
RCOM is not functioning as a complete group anymore.
I'm a little
confused why this wasn't made clear right at the beginning of this thread e.g.
when others suggested this might be the case and you refuted them? Also, I'm
not sure what 'functioning as a complete group' actually means. Either its
functioning or its not, surely?
However, we split into sub-committees while we were still a functioning group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still.
I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.
So, does that mean
that is what the policy *ought to* be now? And do you believe that this should
this be the way that the policy gets decided? Because it isn't right now as far
as I can see. As Kerry noted earlier on, the policy as it stands [1] says that
researchers "must" obtain approval through the process described. If
the wording now needs to be changed to "ought to" then surely this
requires more consensus than your single message here?
re. the comment
that I (and the other researchers?) on this list shouldn't be the ones to
decide what the regulation should be, I disagree on two counts. a) It seems on
the one hand that you want this to be "self-regulation" i.e. you
invited researchers on this list to join R-COM at the beginning of this thread,
but that you don't think that the researchers here should be able to determine
what to regulate. I know that you're looking for an inclusive process but you
can't have it both ways: if we are going to help regulate, then we need to at
least help decide how to regulate. b) Pine suggested a board decision on this
earlier one to obtain clarity and I supported this but it was met with silence,
which is why I followed up.
There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves.
I can understand
your frustration; I really can! I know that you've done a lot of really great,
prior work on this and I don't think any of us are saying that we need to throw
the baby out with the bathwater. But what is clear is that clarification is
required - especially on the distribution of tasks between Foundation employees,
the research community and Wikimedia editors. And this is *especially* true for
people outside this list.
I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation.
I'm frustrated by
this response. If the policy is incorrectly described on the policy pages, then
someone from RCom (or whatever it is now called) should be the one to change
this - preferably with some discussion. I find it frustrating that WMF
employees are often the ones who make the final policy pronouncements but then
tell others to implement it. And if we don't do the work, then we're apparently
not assuming good faith.
This is a great
opportunity to rejuvenate the process; hopefully it will eventually be seen
that way :)
Best,
Heather.
-Aaron
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
--
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l