For example, this proposal[1] was sent to me last year. The researcher's
plan was not to sample from the pool of active editors, but instead to
contact the 500 *most active Wikipedians* on enwiki to survey them about
their motivations. There were several concerns raised about (1) whether
the proposed study was duplicating prior work, (2) why the busiest editors
needed to be surveyed and (3) whether the researcher's methodology would
allow for the intended insights to be gained.
Regretfully, the researcher decided not to respond to anyone other than
myself and was also unwilling to work through many of the concerns I raised.
1.
-Aaron
On Wed, Jul 30, 2014 at 8:19 PM, Kerry Raymond <kerry.raymond(a)gmail.com>
wrote:
30? No wonder we are worried about editor attrition J
Seriously,
http://stats.wikimedia.org/EN/SummaryEN.htm
shows that in May 2014 on en.WP we had about 32K active editors (> 5 edits
per month) and 3K very active editors (>100 edits per month).
Or have I missed something here? Are researchers only interested in people
who have been on Wikipedia for 10+ years with 10M edits or …?
Kerry
------------------------------
*From:* wiki-research-l-bounces(a)lists.wikimedia.org [mailto:
wiki-research-l-bounces(a)lists.wikimedia.org] *On Behalf Of *Lane Rasberry
*Sent:* Wednesday, 30 July 2014 12:00 PM
*To:* Research into Wikimedia content and communities
*Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
Hey guys,
I posted some thoughts to my own blog and am linking to those posts below.
Everything I say on my blog is captured in the summary below, so feel free
to not click through.
----
My biggest worry is that researchers who recruit human subjects assume
that there are huge numbers of Wikipedians for them to survey, and
consequently, they do not need to do a lot of advance survey preparation
because there is no harm from distracting Wikipedians from their usual
volunteer work. This assumption is wrong because actually almost every
researcher recruiting human subjects wants Wikipedians who are in very
short supply. Consequently, researchers do cause harm to the community by
soliciting for volunteer time, and Wikipedia community benefit is dubious
when researchers do not do sufficient preparation for their work. This is
not quite accurate, but if there were one message I could convey to
researchers, it would be "Your research participant pool only consists of
about 30 super busy people and many other volunteers greatly depend on
getting their time. When you take time from a Wikipedian, you are taking
that time away from other volunteers who really need it, so be respectful
of your intervention in our communities."
----
I do not want a lot of gatekeeping between researchers and the Wikipedia
community, but at the same time, researchers should take professional pride
in their work and take care not to disrupt Wikimedia community activities.
<
http://bluerasberry.com/2014/07/request-for-researchers-when-doing-research…
<http://bluerasberry.com/2014/07/problems-with-research-on-wikipedia/>
I am still thinking about what should be done with research.
yours,
On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli <
dtaraborelli(a)wikimedia.org> wrote:
Hi all,
I am a bit late in the game, but since so many questions were raised about
RCom, its scope, its goals, the source of its authority etc. and I helped
coordinate it in the early days I thought I’d chime in to clear some
confusion.
*Is RCom an official WMF body or a group of volunteers?*
RCom was created as a volunteer body to help design policies and best
practices around research on Wikimedia projects. People who joined the
committee did so on a volunteer basis and with a variety of interests by
responding to a call for participation issued by WMF. Despite the fact that
the original initiative came from WMF, its membership almost entirely
consisted of non-WMF researchers and community members (those of us who are
now with Wikimedia had no affiliation with the Foundation when RCom was
launched [1]). RCom work was and remains 100% volunteer-driven, even for
those of us who are full-time employees of the Foundation.
*Is RCom a body regulating subject recruitment?*
No, subject recruitment was only one among many areas of interest
identified by its participants [2]
*Is RCom still alive?*
RCom stopped working a while ago* as a* *group meeting on a regular basis
to discuss joint initiatives*. However, it spawned a large number of
initiatives and workgroups that are still alive and kicking, some of which
have evolved into other projects that are now only loosely associated with
RCom. These include reviewing subject recruitment requests, but also the
Research Newsletter, which has been published monthly for the last 3 years;
countless initiatives in the area of open access; initiatives to facilitate
Wikimedia data documentation and data discoverability; hackathons and
outreach events aimed at bringing together researchers and Wikimedia
contributors. Subject recruitment reviews and discussions are still
happening, and I believe they provide a valuable service when you consider
that they are entirely run by a microscopic number of volunteers. I don’t
think that the alternative between “either RCom exists and it functions
effectively or reviews should immediately stop” is well framed or even
desirable, for the reasons that I explain below.
*What’s the source of RCom’s authority in reviewing subject recruitment
requests?*
Despite the perception that one of RCom’s duties would be to provide
formal approval for research projects, it was never designed to do so and
it never had the power to enforce formal review decisions. Instead, it was
offered as a volunteer support service in an effort to help minimize
disruption, improve the relevance of research involving Wikimedia
contributors, sanity check the credentials of the researchers, create
collaborations between researchers working on the same topic. The lack of
community or WMF policies to back subject recruitment caused in the past
few years quite some headaches, particularly in those cases in which
recruitment attempts were blocked and referred to the RCom in order to
“obtain formal approval”. The review process itself was meant to be as
inclusive as possible and not restricted to RCom participants and
researchers having their proposal reviewed were explicitly invited to
address any questions or concerns raised by community members on the talk
page. I totally agree that the way in which the project templates and forms
were designed needs some serious overhaul to remove any indication of a
binding review process or a commitment for reviews to be delivered within a
fixed time frame. I cannot think of any example in which the review process
discriminated some type of projects (say qualitative research) in favor of
other types of research, but I am sure different research proposals
attracted different levels of participation and interest in the review
process. My recommendation to anyone interested in designing future subject
recruitment processes is to focus on a lightweight review process open to
the largest possible number of community members but backed by transparent
and *enforceable* policies. It’s a really hard problem and there is
simply no obvious silver bullet solution that can be found without some
experimentation and fault tolerance.
*What about requests for private data?*
Private data and technical support requests from WMF are a different
story: they were folded into the list of frequently asked questions hosted
on the RCom section of Meta, but by definition they require a direct and
substantial involvement from the Foundation: (1) they involve WMF as the
legal entity that would be held liable for disclosing data in breech of its
privacy policies and (2) they involve paid staff resources and need to be
prioritized against a lot of other requests. There are now dedicated
sections on private data on the Wikimedia Privacy Policy [4] and Data
Retention guidelines [5]. Many people, including myself and other members
of the Foundation’s Analytics team, believe that we should try and collect
the minimum amount of private data that we need in order to operate and
study our projects and make all those types of aggregate/sanitized data
that we can retain indefinitely publicly available to everyone under open
licenses. We’ve already started a process to do so and to ensure that more
data (for example, data collected via site instrumentation [6]) be exposed
via Labs or other APIs, in the respect of our users’ privacy.
*How can we incentivize researchers to “give back” to the community?*
In the early days we drafted a set of requirements [3] to make sure we
could get back as much as possible from research involving WMF resources.
It’s been hard to implement these requirements without policies to enforce
them. The suggestion of having more researchers apply for a slot at the
Research Showcase to present their work is an excellent idea that we should
consider. In general, the Research team at WMF is always interested in
hearing about incentives to drive more interest towards actionable research
on Wikimedia projects.
Dario
[1]
https://meta.wikimedia.org/w/index.php?title=Research:Committee&oldid=2…
[2]
https://meta.wikimedia.org/wiki/Research:Committee/Areas_of_interest
[3]
https://meta.wikimedia.org/wiki/Research:WMF_support
[4]
https://meta.wikimedia.org/wiki/Privacy_policy
[5]
https://meta.wikimedia.org/wiki/Data_retention_guidelines
[6]
http://meta.wikimedia.org/wiki/Research:Schemas
On Jul 29, 2014, at 6:49 PM, Aaron Halfaker <aaron.halfaker(a)gmail.com>
wrote:
Either [RCOM is] functioning or its not, surely?
Well, I explained that there are functioning sub-committees still. In
other words, there are initiatives that RCOM started that are alive and
successful, but we no longer coordinate as a larger group. I don't know
how else to explain it. I guess you could say that RCOM is still
functioning and that we no longer require/engage in group meetings.
As Kerry noted earlier on, the policy as it stands [1] says that
researchers "must" obtain approval through the process described. If the
wording now needs to be changed to "ought to" then surely this requires
more consensus than your single message here?
That's a proposed policy. Until it is passed by consensus, the "must" is
a proposed term. I think that it should be "must", but until that
consensus is reached, I'll continue to say that it "ought to".
Regarding researchers stating what should be regulated, I think there is a
big difference between *deciding what should be regulated* and *being
involved in the discussion of *how* it should be regulated*. Hence why I
welcome participation. What I'm saying is that you have a vested interest
in not being regulated, but I'd still like to discuss how your activities
can be regulated effectively & efficiently. Does that make sense?
b) Pine suggested a board decision on this earlier one to obtain clarity
and I supported this but it was met with silence, which is why I followed
up.
I welcome you to raise it to them. I don't think it is worth their time,
but they might disagree.
But what is clear is that clarification is required - especially on the
distribution of tasks between Foundation employees, the research community
and Wikimedia editors. And this is *especially* true for people outside
this list.
I think that the proposed policy on English Wikipedia does that quite
well. That's why I directed people there. Also, again, I am not working
on RCOM or subject recruitment as a WMF employee. I do this in my
volunteer time. This is true of all of RCOM who happen to also be staff.
if you want process to be more clearly documented, you also have to
address people like Poitr who would rather not have processes described in
detail. When you guys work out how clearly you want a process to be
described, please let me know. I'm tired of re-spec'ing processes. This
is the third iteration.
If the policy is incorrectly described on the policy pages, then someone
from RCom (or whatever it is now called) should be the one to change this -
preferably with some discussion.
Heather, that is a *proposed *policy page on English Wikipedia. It is
not part of RCOM. It would render RCOM irrelevant for subject recruitment
concerns. That's why I started it. I don't think that
RCOM/WMF/researchers should own subject recruitment review. I think the
community being studied should own it and that RCOM/WMF/researchers should
participate.
Also, I am not your employee. This is my volunteer time. I don't have
much of it, so I focus on keeping the system running -- and it is -- and
improving the system -- which is the proposal I linked to. If you want
something done and other volunteers don't have time to do it. Do it
yourself. <https://en.wikipedia.org/wiki/Wikipedia:SOFIXIT>
-Aaron
On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <hfordsa(a)gmail.com> wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <
aaron.halfaker(a)gmail.com> wrote:
RCOM is not functioning as a complete group anymore.
I'm a little confused why this wasn't made clear right at the beginning of
this thread e.g. when others suggested this might be the case and you
refuted them? Also, I'm not sure what 'functioning as a complete group'
actually means. Either its functioning or its not, surely?
However, we split into sub-committees while we were still a
functioning group. The subject recruitment sub-committee and newsletter
sub-committees are performing vital functions still.
I never stated that research recruiting needs RCOM approval. I definitely
said that it "ought to" have RCOM approval.
So, does that mean that is what the policy *ought to* be now? And do you
believe that this should this be the way that the policy gets decided?
Because it isn't right now as far as I can see. As Kerry noted earlier on,
the policy as it stands [1] says that researchers "must" obtain approval
through the process described. If the wording now needs to be changed to
"ought to" then surely this requires more consensus than your single
message here?
re. the comment that I (and the other researchers?) on this list shouldn't
be the ones to decide what the regulation should be, I disagree on two
counts. a) It seems on the one hand that you want this to be
"self-regulation" i.e. you invited researchers on this list to join R-COM
at the beginning of this thread, but that you don't think that the
researchers here should be able to determine what to regulate. I know that
you're looking for an inclusive process but you can't have it both ways: if
we are going to help regulate, then we need to at least help decide how to
regulate. b) Pine suggested a board decision on this earlier one to obtain
clarity and I supported this but it was met with silence, which is why I
followed up.
There are also more than two "review coordinators" (not not
"reviewers") -- it's just that DarTar and I have accepted the burden of
distributing work. When people are busy, we often coordinate the reviews
ourselves.
I can understand your frustration; I really can! I know that you've done a
lot of really great, prior work on this and I don't think any of us are
saying that we need to throw the baby out with the bathwater. But what is
clear is that clarification is required - especially on the distribution of
tasks between Foundation employees, the research community and Wikimedia
editors. And this is *especially* true for people outside this list.
I welcome your edits to make it clear that review is optional. As you
might imagine, I have plenty of work to do and I appreciate your good-faith
collaboration on improving our research documentation.
I'm frustrated by this response. If the policy is incorrectly described on
the policy pages, then someone from RCom (or whatever it is now called)
should be the one to change this - preferably with some discussion. I find
it frustrating that WMF employees are often the ones who make the final
policy pronouncements but then tell others to implement it. And if we don't
do the work, then we're apparently not assuming good faith.
This is a great opportunity to rejuvenate the process; hopefully it will
eventually be seen that way :)
Best,
Heather.
[1]
https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
-Aaron
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
--
Lane Rasberry
user:bluerasberry on Wikipedia
206.801.0814
lane(a)bluerasberry.com
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l