Hey folks,
I directed Michael to the RCom public archives of this conversation and he
has posted a response to the concerns that we've raised on meta. Please
see:
-Aaron
On Fri, Nov 18, 2011 at 8:10 PM, Dario Taraborelli <
dtaraborelli(a)wikimedia.org> wrote:
Goran,
let me quickly comment on the point you raise, I definitely don't want to
increase the bureaucratization of the RCom review process. I want it to be
lightweight, efficient, effectively distributed across RCom members and
useful for the researcher and for Wikimedia. I don't think asking for the
name of someone's supervisor is a major requirement given the amount of
effort most of us are putting in this process, or is it? If you consider
that we eventually say yes to every single request we receive (I cannot
think of a single SR request that we rejected in the past, anybody?) making
sure there is at least some sane accountability mechanism in place when we
lack other information doesn't strike me as a major selection barrier.
Keep in mind that we are targeting our community, especially new users,
with an increasing number of requests of various kinds. I was not aware of
this problem until I started working at WMF and realized that as a new user
in 2011 you will be contacted by several bots or students/researchers/WMF
staff willing to recruit you for a survey before you get to talk to a
fellow editor welcoming you (if you are lucky) to the project.
I don't want to be fussy about this particular request, and we should move
forward if there is consensus, but the general problem with our passive
support of SR requests is not settled yet as far as I am concerned. I have
limited time and resources to allocate to reviewing research collaboration
requests and it's fair to say that 80-90% of my RCom activity is taken by
reviewing surveys of any kind. I don't think we've ever followed up with
previous surveys to see if they were successfully completed, if they
produced any interesting results, if their results were ever shared with
the community. What I am trying to say is that we could spend our effort
more wisely to maximize the usefulness of our research outreach program and
help researchers help us in this process.
Dario
On Nov 16, 2011, at 2:28 PM, Goran Milovanovic wrote:
Well, when it comes to
"I would be happy to give my approval conditionally on having (at least)
the student's supervisor involved. I actually think this would be a
sensible rule to apply to *all* SR proposals submitted by students."
I have to say I could not agree, and here is the reason: we may stumble
upon many academic environments where students, or PhD candidates, or young
researchers in general are in fact more competent in specific areas of
research then their supervisors. Of course, this is not a common situation,
but if we talk about Internet related research, where new concepts, new
ideas, and new methods are introduced every now and then, we are talking
about a field where the above described asymmetry in competences is likely
to appear.
Personal experience: between 2002. and 2005. a research team that I have
led in Belgrade, Serbia, published four books on Internet behavior,
Internet usage, attitudes towards the Internet and related topics, of which
none were peer-reviewed in Serbia. Why? They were all supported by standard
methodological, theoretical and statistical instruments of social sciences.
Most of the team members already had experience in publishing in
peer-reviewed journals or presented at recognized conferences. Simply,
because that were the first attempts to study anything related to Internet
and society in my country at all, and there were no peer reviewers
available. I am literary saying that back then we were to first to try to
crawl for the relevant literature and references at all.
What I want to add is that most researchers who are into Internet related
research are people who are into new ideas, looking for conceptual
breakthroughs and new theoretical frameworks. Limiting their efforts by
constrains related to the already bureaucratized relations in academia
could end up in actually enlarging the pile of problems on the behalf of
people who simply want to put their new ideas to test. Wikipedia was
developed by people who decided to abandon the classical rules and try with
something previously unimaginable: who would say it work to become of the
landmarks of the Internet? I believe we should keep that spirit when it
comes to research as well.
"I hope it's clear that I don't want to shut down research for anybody but
top researchers with big grants, but we also cannot afford spending time
and effort and community attention for projects of an unclear scientific
value or interest."
I recognize the importance of the issue you're raising, but I do not think
the selection criterion you are proposing will help us sort out the
projects of clear from the projects of unclear value or interest. I
believe, in the spirit of good faith, that task needs to remain with us
completely, as well as with the members of the scientific community who we
might contact and ask for an advice when we are not sure what to do about a
particular project proposal.
Best,
Goran
On Wed, Nov 16, 2011 at 11:03 PM, Dario Taraborelli <
dtaraborelli(a)wikimedia.org> wrote:
ok, here's my dispassionate opinion. I post
on the list because the
problems I refer to below are not specific to this project but to many
other requests.
We have been receiving a *lot* of SR requests from students over the last
few months and I think we should start deciding more aggressively what
research is likely to have an impact and what research won't produce any
major tangible results.
I really like the topic of the Anonymity and Conformity study but I
several concerns with the solidity of the current proposal:
• we will be approving for the first time some kind of large-scale
recruitment approach via user talk pages for a student project: this is
something we've never done before and we should only do it if there's a
good reason.
• the advisor of the proponent doesn't seem to be involved at all in this
project and is not even named in the proposal. Aaron asked the proponent to
share the name of his supervisor in September, but he hasn't done so (yet?)
• the proponent says that no funding is supporting this research and that
this study is "conducted with the author's own efforts"
• no one else other than the applicant will be implied in the data
collection and analysis and the proponent doesn't seem to have an actual
research record
• there is no trace in this proposal of an approval by an ethics
committee. The proponent says that this is not applicable (and it's true
that IRB policy is very different between the US and other countries), but
some official record would help us assess the credibility of the proposal.
for these reasons, I am hesitant whether we should blindly approve this
request. I would be happy to give my approval conditionally on having (at
least) the student's supervisor involved. I actually think this would be a
sensible rule to apply to *all* SR proposals submitted by students.
I hope it's clear that I don't want to shut down research for anybody but
top researchers with big grants, but we also cannot afford spending time
and effort and community attention for projects of an unclear scientific
value or interest.
I'd love to hear your thoughts on this
Dario
On Nov 16, 2011, at 12:30 PM, Aaron Halfaker wrote:
I was hoping to close this poll hours ago, but we only have three members
of RCom participating (thanks Yaroslav and Steven!).
It is absolutely crucial that if we end up technically approving this
study methodology that such approval actually reflects the consensus of
RCom members.
For your benefit, I'll summarize the proposed plan:
A request to participate in a *survey* about enforcing conformance with
community/group outcomes *needs 200-300 responses* from general
Wikipedia editors. Invitations to take the survey will be posted an
editors' User_talk pages. *A pilot set of 15 requests will be posted
immediately* following approval from RCom to test for problems and
determine the expected response rate. Afterwards, *up to 500 User_talk
postings* will be made (depending on response rate) to illicit enough
responses to give statistical confidence.
This is the first proposed project of this scale that we are reviewing
for approval so I really want to make sure we are doing it right.
-Aaron
On Mon, Nov 14, 2011 at 8:49 AM, Aaron Halfaker <aaron.halfaker(a)gmail.com
wrote:
Hey folks,
This proposal is an important milestone for our subject recruitment
processes, since it represents the first mass recruitment request (200-300
responses needed). I'm hoping to either show a high level of support with
this poll or discover what problems still need to be dealt with.
I'd like to close the poll by *Wednesday @ noon UTC*. Please make sure
to chime in.
See poll:
http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_…
-Aaron
_______________________________________________
RCom-l mailing list
RCom-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/rcom-l
_______________________________________________
RCom-l mailing list
RCom-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/rcom-l
--
--------------------------------------------------------------
"Truth is much too complicated to allow
anything but approximations."
:: John von Neumann
--------------------------------------------------------------
http://www.milovanovicresearch.com
_______________________________________________
RCom-l mailing list
RCom-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/rcom-l
_______________________________________________
RCom-l mailing list
RCom-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/rcom-l