Hi all,
FYI, for those who do not regularly follow wikimedia-l, there's a discussion going on there about Wikipedia surveys (sparked off by one particular survey) that may be of interest to this list. See http://lists.wikimedia.org/pipermail/wikimedia-l/2014-July/073367.html
Briefly, the meta-question seems to be: we set up some researcher best practices such that researchers should get approval via RCOM, but that process is now not active. So now what? What should researchers do?
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
-- phoebe
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&hidebots=1&hideredirs=1&limit=500&offset=&namespace=202 It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario ( dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: < https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&...
It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don't think it's any business of WMF's. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF's concerns in relation to recruitment of WPians to research projects) will simply use WMF's channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I've forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up ("too hard to deal with these people") and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It's nice to have a carrot as well as a stick when it comes to onerous processes :-)
Because of the criticism of "not giving back", could we perhaps do things to try to make the researcher feel part of the community to make "giving back" more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here .
Kerry
_____
From: wiki-research-l-bounces@lists.wikimedia.org [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Aaron Halfaker Sent: Thursday, 17 July 2014 6:59 AM To: Research into Wikimedia content and communities Subject: Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario (dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: <https://meta.wikimedia.org/w/index.php?title=Special:NewPages https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&hid ebots=1&hideredirs=1&limit=500&offset=&namespace=202 &feed=atom&hidebots=1&hideredirs=1&limit=500&offset=&namespace=202> It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
WMF does not "own" me as a contributor; it does not decide who can and
cannot recruit me for whatever purposes.
I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-07-17 8:24 GMT+03:00 Kerry Raymond kerry.raymond@gmail.com:
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario ( dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: < https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&...
It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use.
Kerry
_____
From: Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il] Sent: Thursday, 17 July 2014 5:37 PM To: kerry.raymond@gmail.com; Research into Wikimedia content and communities Subject: Re: [Wiki-research-l] discussion about wikipedia surveys
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes.
I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-07-17 8:24 GMT+03:00 Kerry Raymond kerry.raymond@gmail.com:
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes :-)
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
Kerry
_____
From: wiki-research-l-bounces@lists.wikimedia.org [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Aaron Halfaker Sent: Thursday, 17 July 2014 6:59 AM To: Research into Wikimedia content and communities Subject: Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario (dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: <https://meta.wikimedia.org/w/index.php?title=Special:NewPages https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&hidebots=1&hideredirs=1&limit=500&offset=&namespace=202 &feed=atom&hidebots=1&hideredirs=1&limit=500&offset=&namespace=202> It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Agree with Kerry that we really need to have a more flexible process that speaks to the main problem that (I think) RCOM was started to solve i.e. that Wikipedians were getting tired of being continually contacted by researchers to fill out *surveys*. I'm not sure where feelings are about that right now (I certainly haven't seen a huge amount of surveys myself) but I guess the big question right now is whether RCOM is actually active or not. I must say that I was surprised, Aaron, when I read that it is active because I had heard from others in your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed. As Nathan discusses on the Wikimedia-l list, there aren't many indications that RCOM is still active. Perhaps there has been a recent decision to resuscitate it? If that's the case, let us know about it :) And then we can discuss what needs to happen to build a good process.
One immediate requirement that I've been talking to others about is finding ways of making the case to the WMF as a group of researchers for the anonymization of country level data, for example. I've spoken to a few researchers (and I myself made a request about a year ago that hasn't been responded to) and it seems like some work is required by the foundation to do this anonymisation but that there are a few of us who would be really keen to use this data to produce research very valuable to Wikipedia - especially from smaller language versions/developing countries. Having an official process that assesses how worthwhile this investment of time would be to the Foundation would be a great idea, I think, but right now there seems to be a general focus on the research that the Foundation does itself rather than enabling researchers outside. I know how busy Aaron and Dario (and others in the team) are so perhaps this requires a new position to coordinate between researchers and Foundation resources?
Anyway, I think the big question right now is whether there are any plans for RCOM that have been made by the research team and the only people who can answer that are folks in the research team :)
Best, Heather.
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 17 July 2014 08:49, Kerry Raymond kerry.raymond@gmail.com wrote:
Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use.
Kerry
*From:* Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il] *Sent:* Thursday, 17 July 2014 5:37 PM *To:* kerry.raymond@gmail.com; Research into Wikimedia content and communities
*Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
WMF does not "own" me as a contributor; it does not decide who can and
cannot recruit me for whatever purposes.
I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-07-17 8:24 GMT+03:00 Kerry Raymond kerry.raymond@gmail.com:
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario ( dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: < https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&...
It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Aaron, when I read that it is active because I had heard from others in
your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed.
1) If anyone told you that we are no longer active, they were wrong. 2) The "lightweight" process you refer to is what I linked to in enwiki in my previous response. See again: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
Generally, there seems to be a misconception that RCom == paid WMF activities. While RCom involves a relationship with the Wikimedia Foundation, our activities as part of RCom are 100% volunteer and open to participation from other Wikipedians (seriously, let me know if you want to help out!), and as such, our backlog tends to suffer when our available volunteer time does. FWIW, I became involved in this work as a volunteer (before I started working with the WMF). With that in mind, it seems like we are not discussing RCom itself which is mostly inactive -- so much as we are discussing the subject recruitment review process which is still active. Let me state this clearly: *If you send an email to me or Dario about a research project that you would like reviewed, we will help you coordinate a review. *Our job as review coordinators is to make sure that the study is adequately documented and that Wikipedians and other researchers are pulled in to discuss the material. We don't just welcome broad involvement -- we need it! We all suffer from the lack of it. Please show up help us!
To give you some context on the current stats and situation, I should probably give a bit of history. I've been working to improve subject recruitment review -- with the goal of improving interactions between researchers and Wikipedians -- for years. Let me first say that *I'm game to make this better**.* In my experience, the biggest issue to documenting the a review/endorsement/whatever process that I have come across is this: there seems to be a lot of people who feel that minimizing *process description* provides power and adaptability to intended processes[1]. It's these people that I've regularly battled in my frequent efforts to increase the formalization around the subject recruitment proposal vetting process (e.g. SRAG had a structured appeals process and stated timelines). The result of these battles is the severely under-documented process "described" in meta:R:FAQ https://meta.wikimedia.org/wiki/Research:FAQ.
Here's some links to my previous work on subject recruitment process that will show these old discussions about process creep https://en.wikipedia.org/wiki/Wikipedia:Avoid_instruction_creep.
- https://en.wikipedia.org/wiki/Wikipedia:Subject_Recruitment_Approvals_Group - https://en.wikipedia.org/wiki/Wikipedia_talk:Subject_Recruitment_Approvals_G... - https://en.wikipedia.org/w/index.php?title=Wikipedia:Research&oldid=3546... - https://en.wikipedia.org/wiki/Wikipedia_talk:Research/Archive_1 - https://en.wikipedia.org/wiki/Wikipedia_talk:Research/Archive_2 -- Note that this was actually an *enwiki policy* for about 5 hours before the RfC was overturned due to too few editors being involved in the straw poll.
For new work, see my current (but stalled for about 1.5 years) push for a structured process on English Wikipedia. https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment See also the checklist I have been working on with Lane. https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment/Wikipedian_chec...
When you review these docs and the corresponding conversations, please keep in mind that I was a new Wikipedian for the development of WP:SRAG and WP:Research, so I made some really critical mistakes -- like taking hyperbolic criticism of the proposals personally. :\
So what now? Well, in the meantime, if you let me know about some subject recruitment you want to do, I'll help you find someone to coordinate a review that fits within the process described in the RCom docs. In the short term, are any of you folks interested in going through some iterations of the new WP:Research_recruitment policy doc?
-Aaron
On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford hfordsa@gmail.com wrote:
Agree with Kerry that we really need to have a more flexible process that speaks to the main problem that (I think) RCOM was started to solve i.e. that Wikipedians were getting tired of being continually contacted by researchers to fill out *surveys*. I'm not sure where feelings are about that right now (I certainly haven't seen a huge amount of surveys myself) but I guess the big question right now is whether RCOM is actually active or not. I must say that I was surprised, Aaron, when I read that it is active because I had heard from others in your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed. As Nathan discusses on the Wikimedia-l list, there aren't many indications that RCOM is still active. Perhaps there has been a recent decision to resuscitate it? If that's the case, let us know about it :) And then we can discuss what needs to happen to build a good process.
One immediate requirement that I've been talking to others about is finding ways of making the case to the WMF as a group of researchers for the anonymization of country level data, for example. I've spoken to a few researchers (and I myself made a request about a year ago that hasn't been responded to) and it seems like some work is required by the foundation to do this anonymisation but that there are a few of us who would be really keen to use this data to produce research very valuable to Wikipedia - especially from smaller language versions/developing countries. Having an official process that assesses how worthwhile this investment of time would be to the Foundation would be a great idea, I think, but right now there seems to be a general focus on the research that the Foundation does itself rather than enabling researchers outside. I know how busy Aaron and Dario (and others in the team) are so perhaps this requires a new position to coordinate between researchers and Foundation resources?
Anyway, I think the big question right now is whether there are any plans for RCOM that have been made by the research team and the only people who can answer that are folks in the research team :)
Best, Heather.
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 17 July 2014 08:49, Kerry Raymond kerry.raymond@gmail.com wrote:
Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use.
Kerry
*From:* Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il] *Sent:* Thursday, 17 July 2014 5:37 PM *To:* kerry.raymond@gmail.com; Research into Wikimedia content and communities
*Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
WMF does not "own" me as a contributor; it does not decide who can and
cannot recruit me for whatever purposes.
I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-07-17 8:24 GMT+03:00 Kerry Raymond kerry.raymond@gmail.com:
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario ( dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) < nemowiki@gmail.com> wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: < https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&...
It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Kerry said:
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
This is a bit different than the main topic, so I wanted to break it out into another reply.
We just had Nate Matias[0] from the MIT media lab present on his work at the last showcase[1]. We also just sent out a survey about the showcase that includes a call for recommended speakers at future showcases[2]. As for a Wikipedia research conference, see OpenSym[3] (formerly WikiSym) and Wikimania[4] (not as researchy, but a great venue to maximize wiki research impact).
0. http://natematias.com/ 1. https://www.mediawiki.org/wiki/Analytics/Research_and_Data/Showcase#July_201... 2. http://lists.wikimedia.org/pipermail/wiki-research-l/2014-July/003574.html 3. http://www.opensym.org/os2014/ 4. https://wikimania2014.wikimedia.org/wiki/Main_Page
On Thu, Jul 17, 2014 at 8:30 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
Aaron, when I read that it is active because I had heard from others in
your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed.
- If anyone told you that we are no longer active, they were wrong.
- The "lightweight" process you refer to is what I linked to in enwiki
in my previous response. See again: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
Generally, there seems to be a misconception that RCom == paid WMF activities. While RCom involves a relationship with the Wikimedia Foundation, our activities as part of RCom are 100% volunteer and open to participation from other Wikipedians (seriously, let me know if you want to help out!), and as such, our backlog tends to suffer when our available volunteer time does. FWIW, I became involved in this work as a volunteer (before I started working with the WMF). With that in mind, it seems like we are not discussing RCom itself which is mostly inactive -- so much as we are discussing the subject recruitment review process which is still active. Let me state this clearly: *If you send an email to me or Dario about a research project that you would like reviewed, we will help you coordinate a review. *Our job as review coordinators is to make sure that the study is adequately documented and that Wikipedians and other researchers are pulled in to discuss the material. We don't just welcome broad involvement -- we need it! We all suffer from the lack of it. Please show up help us!
To give you some context on the current stats and situation, I should probably give a bit of history. I've been working to improve subject recruitment review -- with the goal of improving interactions between researchers and Wikipedians -- for years. Let me first say that *I'm game to make this better**.* In my experience, the biggest issue to documenting the a review/endorsement/whatever process that I have come across is this: there seems to be a lot of people who feel that minimizing *process description* provides power and adaptability to intended processes[1]. It's these people that I've regularly battled in my frequent efforts to increase the formalization around the subject recruitment proposal vetting process (e.g. SRAG had a structured appeals process and stated timelines). The result of these battles is the severely under-documented process "described" in meta:R:FAQ https://meta.wikimedia.org/wiki/Research:FAQ.
Here's some links to my previous work on subject recruitment process that will show these old discussions about process creep https://en.wikipedia.org/wiki/Wikipedia:Avoid_instruction_creep.
https://en.wikipedia.org/wiki/Wikipedia:Subject_Recruitment_Approvals_Group
https://en.wikipedia.org/w/index.php?title=Wikipedia:Research&oldid=3546...
- https://en.wikipedia.org/wiki/Wikipedia_talk:Research/Archive_1 Note that this was actually an *enwiki policy* for about 5 hours before the RfC was overturned due to too few editors being involved in the straw poll.
For new work, see my current (but stalled for about 1.5 years) push for a structured process on English Wikipedia. https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment See also the checklist I have been working on with Lane. https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment/Wikipedian_chec...
When you review these docs and the corresponding conversations, please keep in mind that I was a new Wikipedian for the development of WP:SRAG and WP:Research, so I made some really critical mistakes -- like taking hyperbolic criticism of the proposals personally. :\
So what now? Well, in the meantime, if you let me know about some subject recruitment you want to do, I'll help you find someone to coordinate a review that fits within the process described in the RCom docs. In the short term, are any of you folks interested in going through some iterations of the new WP:Research_recruitment policy doc?
-Aaron
On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford hfordsa@gmail.com wrote:
Agree with Kerry that we really need to have a more flexible process that speaks to the main problem that (I think) RCOM was started to solve i.e. that Wikipedians were getting tired of being continually contacted by researchers to fill out *surveys*. I'm not sure where feelings are about that right now (I certainly haven't seen a huge amount of surveys myself) but I guess the big question right now is whether RCOM is actually active or not. I must say that I was surprised, Aaron, when I read that it is active because I had heard from others in your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed. As Nathan discusses on the Wikimedia-l list, there aren't many indications that RCOM is still active. Perhaps there has been a recent decision to resuscitate it? If that's the case, let us know about it :) And then we can discuss what needs to happen to build a good process.
One immediate requirement that I've been talking to others about is finding ways of making the case to the WMF as a group of researchers for the anonymization of country level data, for example. I've spoken to a few researchers (and I myself made a request about a year ago that hasn't been responded to) and it seems like some work is required by the foundation to do this anonymisation but that there are a few of us who would be really keen to use this data to produce research very valuable to Wikipedia - especially from smaller language versions/developing countries. Having an official process that assesses how worthwhile this investment of time would be to the Foundation would be a great idea, I think, but right now there seems to be a general focus on the research that the Foundation does itself rather than enabling researchers outside. I know how busy Aaron and Dario (and others in the team) are so perhaps this requires a new position to coordinate between researchers and Foundation resources?
Anyway, I think the big question right now is whether there are any plans for RCOM that have been made by the research team and the only people who can answer that are folks in the research team :)
Best, Heather.
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 17 July 2014 08:49, Kerry Raymond kerry.raymond@gmail.com wrote:
Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use.
Kerry
*From:* Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il] *Sent:* Thursday, 17 July 2014 5:37 PM *To:* kerry.raymond@gmail.com; Research into Wikimedia content and communities
*Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
WMF does not "own" me as a contributor; it does not decide who can
and cannot recruit me for whatever purposes.
I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-07-17 8:24 GMT+03:00 Kerry Raymond kerry.raymond@gmail.com:
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario ( dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) < nemowiki@gmail.com> wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: < https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&...
It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
I guess I was not so much thinking of an general invitation to the R&D Showcase but a specific “expectation” (albeit couched as an invitation) on those given permission to recruit via WMF channels to give a few short (or long as appropriate to the stage of their research) talks on their project. Ditto research projects supported through IEG or similar.
I agree that OpenSym is available as a research conference but it is not run by our community and therefore doesn’t help to create a sense of community with the researchers in question. Wikimania is run by our community but isn’t a research conference (would not count as a publication for academic purposes). But I don’t know if it’s realistic to try to establish another conference in terms of the volunteer effort to run it.
Kerry
_____
From: wiki-research-l-bounces@lists.wikimedia.org [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Aaron Halfaker Sent: Friday, 18 July 2014 1:45 AM To: Research into Wikimedia content and communities Subject: Re: [Wiki-research-l] discussion about wikipedia surveys
Kerry said:
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
This is a bit different than the main topic, so I wanted to break it out into another reply.
We just had Nate Matias[0] from the MIT media lab present on his work at the last showcase[1]. We also just sent out a survey about the showcase that includes a call for recommended speakers at future showcases[2]. As for a Wikipedia research conference, see OpenSym[3] (formerly WikiSym) and Wikimania[4] (not as researchy, but a great venue to maximize wiki research impact).
1. https://www.mediawiki.org/wiki/Analytics/Research_and_Data/Showcase#July_201...
2. http://lists.wikimedia.org/pipermail/wiki-research-l/2014-July/003574.html
3. http://www.opensym.org/os2014/
4. https://wikimania2014.wikimedia.org/wiki/Main_Page
On Thu, Jul 17, 2014 at 8:30 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
Aaron, when I read that it is active because I had heard from others in your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed.
1) If anyone told you that we are no longer active, they were wrong.
2) The "lightweight" process you refer to is what I linked to in enwiki in my previous response. See again: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
Generally, there seems to be a misconception that RCom == paid WMF activities. While RCom involves a relationship with the Wikimedia Foundation, our activities as part of RCom are 100% volunteer and open to participation from other Wikipedians (seriously, let me know if you want to help out!), and as such, our backlog tends to suffer when our available volunteer time does. FWIW, I became involved in this work as a volunteer (before I started working with the WMF). With that in mind, it seems like we are not discussing RCom itself which is mostly inactive -- so much as we are discussing the subject recruitment review process which is still active. Let me state this clearly: If you send an email to me or Dario about a research project that you would like reviewed, we will help you coordinate a review. Our job as review coordinators is to make sure that the study is adequately documented and that Wikipedians and other researchers are pulled in to discuss the material. We don't just welcome broad involvement -- we need it! We all suffer from the lack of it. Please show up help us!
To give you some context on the current stats and situation, I should probably give a bit of history. I've been working to improve subject recruitment review -- with the goal of improving interactions between researchers and Wikipedians -- for years. Let me first say that I'm game to make this better. In my experience, the biggest issue to documenting the a review/endorsement/whatever process that I have come across is this: there seems to be a lot of people who feel that minimizing process description provides power and adaptability to intended processes[1]. It's these people that I've regularly battled in my frequent efforts to increase the formalization around the subject recruitment proposal vetting process (e.g. SRAG had a structured appeals process and stated timelines). The result of these battles is the severely under-documented process "described" in meta:R:FAQ https://meta.wikimedia.org/wiki/Research:FAQ .
Here's some links to my previous work on subject recruitment process that will show these old discussions about process creep https://en.wikipedia.org/wiki/Wikipedia:Avoid_instruction_creep .
* https://en.wikipedia.org/wiki/Wikipedia:Subject_Recruitment_Approvals_Group
o https://en.wikipedia.org/wiki/Wikipedia_talk:Subject_Recruitment_Approvals_G...
* https://en.wikipedia.org/w/index.php?title=Wikipedia:Research https://en.wikipedia.org/w/index.php?title=Wikipedia:Research&oldid=354600173 &oldid=354600173
o https://en.wikipedia.org/wiki/Wikipedia_talk:Research/Archive_1
o https://en.wikipedia.org/wiki/Wikipedia_talk:Research/Archive_2 -- Note that this was actually an enwiki policy for about 5 hours before the RfC was overturned due to too few editors being involved in the straw poll.
For new work, see my current (but stalled for about 1.5 years) push for a structured process on English Wikipedia. https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment See also the checklist I have been working on with Lane. https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment/Wikipedian_chec...
When you review these docs and the corresponding conversations, please keep in mind that I was a new Wikipedian for the development of WP:SRAG and WP:Research, so I made some really critical mistakes -- like taking hyperbolic criticism of the proposals personally. :\
So what now? Well, in the meantime, if you let me know about some subject recruitment you want to do, I'll help you find someone to coordinate a review that fits within the process described in the RCom docs. In the short term, are any of you folks interested in going through some iterations of the new WP:Research_recruitment policy doc?
-Aaron
On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford hfordsa@gmail.com wrote:
Agree with Kerry that we really need to have a more flexible process that speaks to the main problem that (I think) RCOM was started to solve i.e. that Wikipedians were getting tired of being continually contacted by researchers to fill out *surveys*. I'm not sure where feelings are about that right now (I certainly haven't seen a huge amount of surveys myself) but I guess the big question right now is whether RCOM is actually active or not. I must say that I was surprised, Aaron, when I read that it is active because I had heard from others in your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed. As Nathan discusses on the Wikimedia-l list, there aren't many indications that RCOM is still active. Perhaps there has been a recent decision to resuscitate it? If that's the case, let us know about it :) And then we can discuss what needs to happen to build a good process.
One immediate requirement that I've been talking to others about is finding ways of making the case to the WMF as a group of researchers for the anonymization of country level data, for example. I've spoken to a few researchers (and I myself made a request about a year ago that hasn't been responded to) and it seems like some work is required by the foundation to do this anonymisation but that there are a few of us who would be really keen to use this data to produce research very valuable to Wikipedia - especially from smaller language versions/developing countries. Having an official process that assesses how worthwhile this investment of time would be to the Foundation would be a great idea, I think, but right now there seems to be a general focus on the research that the Foundation does itself rather than enabling researchers outside. I know how busy Aaron and Dario (and others in the team) are so perhaps this requires a new position to coordinate between researchers and Foundation resources?
Anyway, I think the big question right now is whether there are any plans for RCOM that have been made by the research team and the only people who can answer that are folks in the research team :)
Best,
Heather.
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford http://www.oii.ox.ac.uk/research/projects/?id=115 Digital Ethnography Group http://hblog.org http://hblog.org/ | @hfordsa http://www.twitter.com/hfordsa
On 17 July 2014 08:49, Kerry Raymond kerry.raymond@gmail.com wrote:
Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use.
Kerry
_____
From: Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il] Sent: Thursday, 17 July 2014 5:37 PM To: kerry.raymond@gmail.com; Research into Wikimedia content and communities
Subject: Re: [Wiki-research-l] discussion about wikipedia surveys
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes.
I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-07-17 8:24 GMT+03:00 Kerry Raymond kerry.raymond@gmail.com:
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes :-)
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
Kerry
_____
From: wiki-research-l-bounces@lists.wikimedia.org [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Aaron Halfaker Sent: Thursday, 17 July 2014 6:59 AM To: Research into Wikimedia content and communities Subject: Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario (dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: <https://meta.wikimedia.org/w/index.php?title=Special:NewPages https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&hidebots=1&hideredirs=1&limit=500&offset=&namespace=202 &feed=atom&hidebots=1&hideredirs=1&limit=500&offset=&namespace=202> It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
If RCOM needs more volunteer Wikimedians, the alive and well IEG Committee includes a Research Working Group that reviews grant proposals for WMF funding through the IEG program, so RCOM could reach out to IEGCom. I'm on IEGCom and the RWG but I can't speak for RCOM. (:
Pine
On Thu, Jul 17, 2014 at 3:10 PM, Kerry Raymond kerry.raymond@gmail.com wrote:
I guess I was not so much thinking of an general invitation to the R&D Showcase but a specific “expectation” (albeit couched as an invitation) on those given permission to recruit via WMF channels to give a few short (or long as appropriate to the stage of their research) talks on their project. Ditto research projects supported through IEG or similar.
I agree that OpenSym is available as a research conference but it is not run by our community and therefore doesn’t help to create a sense of community with the researchers in question. Wikimania is run by our community but isn’t a research conference (would not count as a publication for academic purposes). But I don’t know if it’s realistic to try to establish another conference in terms of the volunteer effort to run it.
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Friday, 18 July 2014 1:45 AM
*To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
Kerry said:
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
This is a bit different than the main topic, so I wanted to break it out into another reply.
We just had Nate Matias[0] from the MIT media lab present on his work at the last showcase[1]. We also just sent out a survey about the showcase that includes a call for recommended speakers at future showcases[2]. As for a Wikipedia research conference, see OpenSym[3] (formerly WikiSym) and Wikimania[4] (not as researchy, but a great venue to maximize wiki research impact).
https://www.mediawiki.org/wiki/Analytics/Research_and_Data/Showcase#July_201...
http://lists.wikimedia.org/pipermail/wiki-research-l/2014-July/003574.html
On Thu, Jul 17, 2014 at 8:30 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
Aaron, when I read that it is active because I had heard from others in
your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed.
If anyone told you that we are no longer active, they were wrong.
The "lightweight" process you refer to is what I linked to in enwiki in
my previous response. See again: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
Generally, there seems to be a misconception that RCom == paid WMF activities. While RCom involves a relationship with the Wikimedia Foundation, our activities as part of RCom are 100% volunteer and open to participation from other Wikipedians (seriously, let me know if you want to help out!), and as such, our backlog tends to suffer when our available volunteer time does. FWIW, I became involved in this work as a volunteer (before I started working with the WMF). With that in mind, it seems like we are not discussing RCom itself which is mostly inactive -- so much as we are discussing the subject recruitment review process which is still active. Let me state this clearly: *If you send an email to me or Dario about a research project that you would like reviewed, we will help you coordinate a review. *Our job as review coordinators is to make sure that the study is adequately documented and that Wikipedians and other researchers are pulled in to discuss the material. We don't just welcome broad involvement -- we need it! We all suffer from the lack of it. Please show up help us!
To give you some context on the current stats and situation, I should probably give a bit of history. I've been working to improve subject recruitment review -- with the goal of improving interactions between researchers and Wikipedians -- for years. Let me first say that *I'm game to make this better.* In my experience, the biggest issue to documenting the a review/endorsement/whatever process that I have come across is this: there seems to be a lot of people who feel that minimizing *process description* provides power and adaptability to intended processes[1]. It's these people that I've regularly battled in my frequent efforts to increase the formalization around the subject recruitment proposal vetting process (e.g. SRAG had a structured appeals process and stated timelines). The result of these battles is the severely under-documented process "described" in meta:R:FAQ https://meta.wikimedia.org/wiki/Research:FAQ.
Here's some links to my previous work on subject recruitment process that will show these old discussions about process creep https://en.wikipedia.org/wiki/Wikipedia:Avoid_instruction_creep.
· https://en.wikipedia.org/wiki/Wikipedia:Subject_Recruitment_Approvals_Group
o https://en.wikipedia.org/wiki/Wikipedia_talk:Subject_Recruitment_Approvals_G...
· https://en.wikipedia.org/w/index.php?title=Wikipedia:Research&oldid=3546...
o https://en.wikipedia.org/wiki/Wikipedia_talk:Research/Archive_1
o https://en.wikipedia.org/wiki/Wikipedia_talk:Research/Archive_2 -- Note that this was actually an *enwiki policy* for about 5 hours before the RfC was overturned due to too few editors being involved in the straw poll.
For new work, see my current (but stalled for about 1.5 years) push for a structured process on English Wikipedia. https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment See also the checklist I have been working on with Lane. https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment/Wikipedian_chec...
When you review these docs and the corresponding conversations, please keep in mind that I was a new Wikipedian for the development of WP:SRAG and WP:Research, so I made some really critical mistakes -- like taking hyperbolic criticism of the proposals personally. :\
So what now? Well, in the meantime, if you let me know about some subject recruitment you want to do, I'll help you find someone to coordinate a review that fits within the process described in the RCom docs. In the short term, are any of you folks interested in going through some iterations of the new WP:Research_recruitment policy doc?
-Aaron
On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford hfordsa@gmail.com wrote:
Agree with Kerry that we really need to have a more flexible process that speaks to the main problem that (I think) RCOM was started to solve i.e. that Wikipedians were getting tired of being continually contacted by researchers to fill out *surveys*. I'm not sure where feelings are about that right now (I certainly haven't seen a huge amount of surveys myself) but I guess the big question right now is whether RCOM is actually active or not. I must say that I was surprised, Aaron, when I read that it is active because I had heard from others in your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed. As Nathan discusses on the Wikimedia-l list, there aren't many indications that RCOM is still active. Perhaps there has been a recent decision to resuscitate it? If that's the case, let us know about it :) And then we can discuss what needs to happen to build a good process.
One immediate requirement that I've been talking to others about is finding ways of making the case to the WMF as a group of researchers for the anonymization of country level data, for example. I've spoken to a few researchers (and I myself made a request about a year ago that hasn't been responded to) and it seems like some work is required by the foundation to do this anonymisation but that there are a few of us who would be really keen to use this data to produce research very valuable to Wikipedia - especially from smaller language versions/developing countries. Having an official process that assesses how worthwhile this investment of time would be to the Foundation would be a great idea, I think, but right now there seems to be a general focus on the research that the Foundation does itself rather than enabling researchers outside. I know how busy Aaron and Dario (and others in the team) are so perhaps this requires a new position to coordinate between researchers and Foundation resources?
Anyway, I think the big question right now is whether there are any plans for RCOM that have been made by the research team and the only people who can answer that are folks in the research team :)
Best,
Heather.
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 17 July 2014 08:49, Kerry Raymond kerry.raymond@gmail.com wrote:
Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use.
Kerry
*From:* Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il] *Sent:* Thursday, 17 July 2014 5:37 PM *To:* kerry.raymond@gmail.com; Research into Wikimedia content and communities
*Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
WMF does not "own" me as a contributor; it does not decide who can and
cannot recruit me for whatever purposes.
I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-07-17 8:24 GMT+03:00 Kerry Raymond kerry.raymond@gmail.com:
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario ( dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: < https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&...
It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
On Thu, Jul 17, 2014 at 7:50 PM, Pine W wiki.pine@gmail.com wrote:
If RCOM needs more volunteer Wikimedians, the alive and well IEG Committee includes a Research Working Group that reviews grant proposals for WMF funding through the IEG program, so RCOM could reach out to IEGCom. I'm on IEGCom and the RWG but I can't speak for RCOM. (:
Thanks, Pine. I'll likely hold you to that offer ;)
On Thu, Jul 17, 2014 at 3:10 PM, Kerry Raymond kerry.raymond@gmail.com wrote:
I guess I was not so much thinking of an general invitation to the R&D Showcase but a specific “expectation” (albeit couched as an invitation) on those given permission to recruit via WMF channels to give a few short (or long as appropriate to the stage of their research) talks on their project. Ditto research projects supported through IEG or similar.
I agree that OpenSym is available as a research conference but it is not run by our community and therefore doesn’t help to create a sense of community with the researchers in question. Wikimania is run by our community but isn’t a research conference (would not count as a publication for academic purposes). But I don’t know if it’s realistic to try to establish another conference in terms of the volunteer effort to run it.
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Friday, 18 July 2014 1:45 AM
*To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
Kerry said:
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
This is a bit different than the main topic, so I wanted to break it out into another reply.
We just had Nate Matias[0] from the MIT media lab present on his work at the last showcase[1]. We also just sent out a survey about the showcase that includes a call for recommended speakers at future showcases[2]. As for a Wikipedia research conference, see OpenSym[3] (formerly WikiSym) and Wikimania[4] (not as researchy, but a great venue to maximize wiki research impact).
https://www.mediawiki.org/wiki/Analytics/Research_and_Data/Showcase#July_201...
http://lists.wikimedia.org/pipermail/wiki-research-l/2014-July/003574.html
On Thu, Jul 17, 2014 at 8:30 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
Aaron, when I read that it is active because I had heard from others
in your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed.
If anyone told you that we are no longer active, they were wrong.
The "lightweight" process you refer to is what I linked to in enwiki
in my previous response. See again: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
Generally, there seems to be a misconception that RCom == paid WMF activities. While RCom involves a relationship with the Wikimedia Foundation, our activities as part of RCom are 100% volunteer and open to participation from other Wikipedians (seriously, let me know if you want to help out!), and as such, our backlog tends to suffer when our available volunteer time does. FWIW, I became involved in this work as a volunteer (before I started working with the WMF). With that in mind, it seems like we are not discussing RCom itself which is mostly inactive -- so much as we are discussing the subject recruitment review process which is still active. Let me state this clearly: *If you send an email to me or Dario about a research project that you would like reviewed, we will help you coordinate a review. *Our job as review coordinators is to make sure that the study is adequately documented and that Wikipedians and other researchers are pulled in to discuss the material. We don't just welcome broad involvement -- we need it! We all suffer from the lack of it. Please show up help us!
To give you some context on the current stats and situation, I should probably give a bit of history. I've been working to improve subject recruitment review -- with the goal of improving interactions between researchers and Wikipedians -- for years. Let me first say that *I'm game to make this better.* In my experience, the biggest issue to documenting the a review/endorsement/whatever process that I have come across is this: there seems to be a lot of people who feel that minimizing *process description* provides power and adaptability to intended processes[1]. It's these people that I've regularly battled in my frequent efforts to increase the formalization around the subject recruitment proposal vetting process (e.g. SRAG had a structured appeals process and stated timelines). The result of these battles is the severely under-documented process "described" in meta:R:FAQ https://meta.wikimedia.org/wiki/Research:FAQ.
Here's some links to my previous work on subject recruitment process that will show these old discussions about process creep https://en.wikipedia.org/wiki/Wikipedia:Avoid_instruction_creep.
· https://en.wikipedia.org/wiki/Wikipedia:Subject_Recruitment_Approvals_Group
o https://en.wikipedia.org/wiki/Wikipedia_talk:Subject_Recruitment_Approvals_G...
· https://en.wikipedia.org/w/index.php?title=Wikipedia:Research&oldid=3546...
o https://en.wikipedia.org/wiki/Wikipedia_talk:Research/Archive_1
o https://en.wikipedia.org/wiki/Wikipedia_talk:Research/Archive_2 -- Note that this was actually an *enwiki policy* for about 5 hours before the RfC was overturned due to too few editors being involved in the straw poll.
For new work, see my current (but stalled for about 1.5 years) push for a structured process on English Wikipedia. https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment See also the checklist I have been working on with Lane. https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment/Wikipedian_chec...
When you review these docs and the corresponding conversations, please keep in mind that I was a new Wikipedian for the development of WP:SRAG and WP:Research, so I made some really critical mistakes -- like taking hyperbolic criticism of the proposals personally. :\
So what now? Well, in the meantime, if you let me know about some subject recruitment you want to do, I'll help you find someone to coordinate a review that fits within the process described in the RCom docs. In the short term, are any of you folks interested in going through some iterations of the new WP:Research_recruitment policy doc?
-Aaron
On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford hfordsa@gmail.com wrote:
Agree with Kerry that we really need to have a more flexible process that speaks to the main problem that (I think) RCOM was started to solve i.e. that Wikipedians were getting tired of being continually contacted by researchers to fill out *surveys*. I'm not sure where feelings are about that right now (I certainly haven't seen a huge amount of surveys myself) but I guess the big question right now is whether RCOM is actually active or not. I must say that I was surprised, Aaron, when I read that it is active because I had heard from others in your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed. As Nathan discusses on the Wikimedia-l list, there aren't many indications that RCOM is still active. Perhaps there has been a recent decision to resuscitate it? If that's the case, let us know about it :) And then we can discuss what needs to happen to build a good process.
One immediate requirement that I've been talking to others about is finding ways of making the case to the WMF as a group of researchers for the anonymization of country level data, for example. I've spoken to a few researchers (and I myself made a request about a year ago that hasn't been responded to) and it seems like some work is required by the foundation to do this anonymisation but that there are a few of us who would be really keen to use this data to produce research very valuable to Wikipedia - especially from smaller language versions/developing countries. Having an official process that assesses how worthwhile this investment of time would be to the Foundation would be a great idea, I think, but right now there seems to be a general focus on the research that the Foundation does itself rather than enabling researchers outside. I know how busy Aaron and Dario (and others in the team) are so perhaps this requires a new position to coordinate between researchers and Foundation resources?
Anyway, I think the big question right now is whether there are any plans for RCOM that have been made by the research team and the only people who can answer that are folks in the research team :)
Best,
Heather.
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 17 July 2014 08:49, Kerry Raymond kerry.raymond@gmail.com wrote:
Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use.
Kerry
*From:* Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il] *Sent:* Thursday, 17 July 2014 5:37 PM *To:* kerry.raymond@gmail.com; Research into Wikimedia content and communities
*Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
WMF does not "own" me as a contributor; it does not decide who can and
cannot recruit me for whatever purposes.
I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-07-17 8:24 GMT+03:00 Kerry Raymond kerry.raymond@gmail.com:
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario ( dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) < nemowiki@gmail.com> wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: < https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&...
It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
You're welcome, J-Mo.
I think it would help if there was a Board resolution authorizing the existence of RCom and outlining its scope and membership. For example, the membership might be something like 9 members with 3 WMF researchers, 3 content volunteers, 2 outside researchers, and 1 member of the WMF Board, plus 6 hours a week of WMF administrative support for handling routine questions and organizing documentation for quick Committee review. Would you, Aaron or Phoebe like to draft something for the Board to consider, or does that need to go through the ED first? I agree with other commentators that having RCOM exist without a clear charter and regular public updates of its membership and work should be remedied, and I think setting up some procedures for how consultations happen could address the issue of people personally approaching you and asking for advice about research projects.
Pine
On Fri, Jul 18, 2014 at 4:49 PM, Jonathan Morgan jmorgan@wikimedia.org wrote:
On Thu, Jul 17, 2014 at 7:50 PM, Pine W wiki.pine@gmail.com wrote:
If RCOM needs more volunteer Wikimedians, the alive and well IEG Committee includes a Research Working Group that reviews grant proposals for WMF funding through the IEG program, so RCOM could reach out to IEGCom. I'm on IEGCom and the RWG but I can't speak for RCOM. (:
Thanks, Pine. I'll likely hold you to that offer ;)
On Thu, Jul 17, 2014 at 3:10 PM, Kerry Raymond kerry.raymond@gmail.com wrote:
I guess I was not so much thinking of an general invitation to the R&D Showcase but a specific “expectation” (albeit couched as an invitation) on those given permission to recruit via WMF channels to give a few short (or long as appropriate to the stage of their research) talks on their project. Ditto research projects supported through IEG or similar.
I agree that OpenSym is available as a research conference but it is not run by our community and therefore doesn’t help to create a sense of community with the researchers in question. Wikimania is run by our community but isn’t a research conference (would not count as a publication for academic purposes). But I don’t know if it’s realistic to try to establish another conference in terms of the volunteer effort to run it.
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Friday, 18 July 2014 1:45 AM
*To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
Kerry said:
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
This is a bit different than the main topic, so I wanted to break it out into another reply.
We just had Nate Matias[0] from the MIT media lab present on his work at the last showcase[1]. We also just sent out a survey about the showcase that includes a call for recommended speakers at future showcases[2]. As for a Wikipedia research conference, see OpenSym[3] (formerly WikiSym) and Wikimania[4] (not as researchy, but a great venue to maximize wiki research impact).
https://www.mediawiki.org/wiki/Analytics/Research_and_Data/Showcase#July_201...
http://lists.wikimedia.org/pipermail/wiki-research-l/2014-July/003574.html
On Thu, Jul 17, 2014 at 8:30 AM, Aaron Halfaker < aaron.halfaker@gmail.com> wrote:
Aaron, when I read that it is active because I had heard from others
in your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed.
If anyone told you that we are no longer active, they were wrong.
The "lightweight" process you refer to is what I linked to in enwiki
in my previous response. See again: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
Generally, there seems to be a misconception that RCom == paid WMF activities. While RCom involves a relationship with the Wikimedia Foundation, our activities as part of RCom are 100% volunteer and open to participation from other Wikipedians (seriously, let me know if you want to help out!), and as such, our backlog tends to suffer when our available volunteer time does. FWIW, I became involved in this work as a volunteer (before I started working with the WMF). With that in mind, it seems like we are not discussing RCom itself which is mostly inactive -- so much as we are discussing the subject recruitment review process which is still active. Let me state this clearly: *If you send an email to me or Dario about a research project that you would like reviewed, we will help you coordinate a review. *Our job as review coordinators is to make sure that the study is adequately documented and that Wikipedians and other researchers are pulled in to discuss the material. We don't just welcome broad involvement -- we need it! We all suffer from the lack of it. Please show up help us!
To give you some context on the current stats and situation, I should probably give a bit of history. I've been working to improve subject recruitment review -- with the goal of improving interactions between researchers and Wikipedians -- for years. Let me first say that *I'm game to make this better.* In my experience, the biggest issue to documenting the a review/endorsement/whatever process that I have come across is this: there seems to be a lot of people who feel that minimizing *process description* provides power and adaptability to intended processes[1]. It's these people that I've regularly battled in my frequent efforts to increase the formalization around the subject recruitment proposal vetting process (e.g. SRAG had a structured appeals process and stated timelines). The result of these battles is the severely under-documented process "described" in meta:R:FAQ https://meta.wikimedia.org/wiki/Research:FAQ .
Here's some links to my previous work on subject recruitment process that will show these old discussions about process creep https://en.wikipedia.org/wiki/Wikipedia:Avoid_instruction_creep.
· https://en.wikipedia.org/wiki/Wikipedia:Subject_Recruitment_Approvals_Group
o https://en.wikipedia.org/wiki/Wikipedia_talk:Subject_Recruitment_Approvals_G...
· https://en.wikipedia.org/w/index.php?title=Wikipedia:Research&oldid=3546...
o https://en.wikipedia.org/wiki/Wikipedia_talk:Research/Archive_1
o https://en.wikipedia.org/wiki/Wikipedia_talk:Research/Archive_2 -- Note that this was actually an *enwiki policy* for about 5 hours before the RfC was overturned due to too few editors being involved in the straw poll.
For new work, see my current (but stalled for about 1.5 years) push for a structured process on English Wikipedia. https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment See also the checklist I have been working on with Lane. https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment/Wikipedian_chec...
When you review these docs and the corresponding conversations, please keep in mind that I was a new Wikipedian for the development of WP:SRAG and WP:Research, so I made some really critical mistakes -- like taking hyperbolic criticism of the proposals personally. :\
So what now? Well, in the meantime, if you let me know about some subject recruitment you want to do, I'll help you find someone to coordinate a review that fits within the process described in the RCom docs. In the short term, are any of you folks interested in going through some iterations of the new WP:Research_recruitment policy doc?
-Aaron
On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford hfordsa@gmail.com wrote:
Agree with Kerry that we really need to have a more flexible process that speaks to the main problem that (I think) RCOM was started to solve i.e. that Wikipedians were getting tired of being continually contacted by researchers to fill out *surveys*. I'm not sure where feelings are about that right now (I certainly haven't seen a huge amount of surveys myself) but I guess the big question right now is whether RCOM is actually active or not. I must say that I was surprised, Aaron, when I read that it is active because I had heard from others in your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed. As Nathan discusses on the Wikimedia-l list, there aren't many indications that RCOM is still active. Perhaps there has been a recent decision to resuscitate it? If that's the case, let us know about it :) And then we can discuss what needs to happen to build a good process.
One immediate requirement that I've been talking to others about is finding ways of making the case to the WMF as a group of researchers for the anonymization of country level data, for example. I've spoken to a few researchers (and I myself made a request about a year ago that hasn't been responded to) and it seems like some work is required by the foundation to do this anonymisation but that there are a few of us who would be really keen to use this data to produce research very valuable to Wikipedia - especially from smaller language versions/developing countries. Having an official process that assesses how worthwhile this investment of time would be to the Foundation would be a great idea, I think, but right now there seems to be a general focus on the research that the Foundation does itself rather than enabling researchers outside. I know how busy Aaron and Dario (and others in the team) are so perhaps this requires a new position to coordinate between researchers and Foundation resources?
Anyway, I think the big question right now is whether there are any plans for RCOM that have been made by the research team and the only people who can answer that are folks in the research team :)
Best,
Heather.
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 17 July 2014 08:49, Kerry Raymond kerry.raymond@gmail.com wrote:
Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use.
Kerry
*From:* Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il] *Sent:* Thursday, 17 July 2014 5:37 PM *To:* kerry.raymond@gmail.com; Research into Wikimedia content and communities
*Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
WMF does not "own" me as a contributor; it does not decide who can
and cannot recruit me for whatever purposes.
I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-07-17 8:24 GMT+03:00 Kerry Raymond kerry.raymond@gmail.com:
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario ( dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) < nemowiki@gmail.com> wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: < https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&...
It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation User:Jmorgan (WMF) https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF) jmorgan@wikimedia.org
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
+1!
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 19 July 2014 08:37, Pine W wiki.pine@gmail.com wrote:
You're welcome, J-Mo.
I think it would help if there was a Board resolution authorizing the existence of RCom and outlining its scope and membership. For example, the membership might be something like 9 members with 3 WMF researchers, 3 content volunteers, 2 outside researchers, and 1 member of the WMF Board, plus 6 hours a week of WMF administrative support for handling routine questions and organizing documentation for quick Committee review. Would you, Aaron or Phoebe like to draft something for the Board to consider, or does that need to go through the ED first? I agree with other commentators that having RCOM exist without a clear charter and regular public updates of its membership and work should be remedied, and I think setting up some procedures for how consultations happen could address the issue of people personally approaching you and asking for advice about research projects.
Pine
On Fri, Jul 18, 2014 at 4:49 PM, Jonathan Morgan jmorgan@wikimedia.org wrote:
On Thu, Jul 17, 2014 at 7:50 PM, Pine W wiki.pine@gmail.com wrote:
If RCOM needs more volunteer Wikimedians, the alive and well IEG Committee includes a Research Working Group that reviews grant proposals for WMF funding through the IEG program, so RCOM could reach out to IEGCom. I'm on IEGCom and the RWG but I can't speak for RCOM. (:
Thanks, Pine. I'll likely hold you to that offer ;)
On Thu, Jul 17, 2014 at 3:10 PM, Kerry Raymond kerry.raymond@gmail.com wrote:
I guess I was not so much thinking of an general invitation to the R&D Showcase but a specific “expectation” (albeit couched as an invitation) on those given permission to recruit via WMF channels to give a few short (or long as appropriate to the stage of their research) talks on their project. Ditto research projects supported through IEG or similar.
I agree that OpenSym is available as a research conference but it is not run by our community and therefore doesn’t help to create a sense of community with the researchers in question. Wikimania is run by our community but isn’t a research conference (would not count as a publication for academic purposes). But I don’t know if it’s realistic to try to establish another conference in terms of the volunteer effort to run it.
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Friday, 18 July 2014 1:45 AM
*To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
Kerry said:
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
This is a bit different than the main topic, so I wanted to break it out into another reply.
We just had Nate Matias[0] from the MIT media lab present on his work at the last showcase[1]. We also just sent out a survey about the showcase that includes a call for recommended speakers at future showcases[2]. As for a Wikipedia research conference, see OpenSym[3] (formerly WikiSym) and Wikimania[4] (not as researchy, but a great venue to maximize wiki research impact).
https://www.mediawiki.org/wiki/Analytics/Research_and_Data/Showcase#July_201...
http://lists.wikimedia.org/pipermail/wiki-research-l/2014-July/003574.html
On Thu, Jul 17, 2014 at 8:30 AM, Aaron Halfaker < aaron.halfaker@gmail.com> wrote:
Aaron, when I read that it is active because I had heard from others
in your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed.
If anyone told you that we are no longer active, they were wrong.
The "lightweight" process you refer to is what I linked to in enwiki
in my previous response. See again: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
Generally, there seems to be a misconception that RCom == paid WMF activities. While RCom involves a relationship with the Wikimedia Foundation, our activities as part of RCom are 100% volunteer and open to participation from other Wikipedians (seriously, let me know if you want to help out!), and as such, our backlog tends to suffer when our available volunteer time does. FWIW, I became involved in this work as a volunteer (before I started working with the WMF). With that in mind, it seems like we are not discussing RCom itself which is mostly inactive -- so much as we are discussing the subject recruitment review process which is still active. Let me state this clearly: *If you send an email to me or Dario about a research project that you would like reviewed, we will help you coordinate a review. *Our job as review coordinators is to make sure that the study is adequately documented and that Wikipedians and other researchers are pulled in to discuss the material. We don't just welcome broad involvement -- we need it! We all suffer from the lack of it. Please show up help us!
To give you some context on the current stats and situation, I should probably give a bit of history. I've been working to improve subject recruitment review -- with the goal of improving interactions between researchers and Wikipedians -- for years. Let me first say that *I'm game to make this better.* In my experience, the biggest issue to documenting the a review/endorsement/whatever process that I have come across is this: there seems to be a lot of people who feel that minimizing *process description* provides power and adaptability to intended processes[1]. It's these people that I've regularly battled in my frequent efforts to increase the formalization around the subject recruitment proposal vetting process (e.g. SRAG had a structured appeals process and stated timelines). The result of these battles is the severely under-documented process "described" in meta:R:FAQ https://meta.wikimedia.org/wiki/Research:FAQ.
Here's some links to my previous work on subject recruitment process that will show these old discussions about process creep https://en.wikipedia.org/wiki/Wikipedia:Avoid_instruction_creep.
· https://en.wikipedia.org/wiki/Wikipedia:Subject_Recruitment_Approvals_Group
o https://en.wikipedia.org/wiki/Wikipedia_talk:Subject_Recruitment_Approvals_G...
· https://en.wikipedia.org/w/index.php?title=Wikipedia:Research&oldid=3546...
o https://en.wikipedia.org/wiki/Wikipedia_talk:Research/Archive_1
o https://en.wikipedia.org/wiki/Wikipedia_talk:Research/Archive_2 -- Note that this was actually an *enwiki policy* for about 5 hours before the RfC was overturned due to too few editors being involved in the straw poll.
For new work, see my current (but stalled for about 1.5 years) push for a structured process on English Wikipedia. https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment See also the checklist I have been working on with Lane. https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment/Wikipedian_chec...
When you review these docs and the corresponding conversations, please keep in mind that I was a new Wikipedian for the development of WP:SRAG and WP:Research, so I made some really critical mistakes -- like taking hyperbolic criticism of the proposals personally. :\
So what now? Well, in the meantime, if you let me know about some subject recruitment you want to do, I'll help you find someone to coordinate a review that fits within the process described in the RCom docs. In the short term, are any of you folks interested in going through some iterations of the new WP:Research_recruitment policy doc?
-Aaron
On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford hfordsa@gmail.com wrote:
Agree with Kerry that we really need to have a more flexible process that speaks to the main problem that (I think) RCOM was started to solve i.e. that Wikipedians were getting tired of being continually contacted by researchers to fill out *surveys*. I'm not sure where feelings are about that right now (I certainly haven't seen a huge amount of surveys myself) but I guess the big question right now is whether RCOM is actually active or not. I must say that I was surprised, Aaron, when I read that it is active because I had heard from others in your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed. As Nathan discusses on the Wikimedia-l list, there aren't many indications that RCOM is still active. Perhaps there has been a recent decision to resuscitate it? If that's the case, let us know about it :) And then we can discuss what needs to happen to build a good process.
One immediate requirement that I've been talking to others about is finding ways of making the case to the WMF as a group of researchers for the anonymization of country level data, for example. I've spoken to a few researchers (and I myself made a request about a year ago that hasn't been responded to) and it seems like some work is required by the foundation to do this anonymisation but that there are a few of us who would be really keen to use this data to produce research very valuable to Wikipedia - especially from smaller language versions/developing countries. Having an official process that assesses how worthwhile this investment of time would be to the Foundation would be a great idea, I think, but right now there seems to be a general focus on the research that the Foundation does itself rather than enabling researchers outside. I know how busy Aaron and Dario (and others in the team) are so perhaps this requires a new position to coordinate between researchers and Foundation resources?
Anyway, I think the big question right now is whether there are any plans for RCOM that have been made by the research team and the only people who can answer that are folks in the research team :)
Best,
Heather.
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 17 July 2014 08:49, Kerry Raymond kerry.raymond@gmail.com wrote:
Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use.
Kerry
*From:* Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il] *Sent:* Thursday, 17 July 2014 5:37 PM *To:* kerry.raymond@gmail.com; Research into Wikimedia content and communities
*Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
WMF does not "own" me as a contributor; it does not decide who can
and cannot recruit me for whatever purposes.
I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-07-17 8:24 GMT+03:00 Kerry Raymond kerry.raymond@gmail.com:
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario ( dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) < nemowiki@gmail.com> wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: < https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&...
It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation User:Jmorgan (WMF) https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF) jmorgan@wikimedia.org
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford hfordsa@gmail.com wrote:
Agree with Kerry that we really need to have a more flexible process that speaks to the main problem that (I think) RCOM was started to solve i.e. that Wikipedians were getting tired of being continually contacted by researchers to fill out *surveys*.
That's correct, afaik that was the original motivation, along with some of the concerns that Lane/Nathan raised in the other list -- i.e. that it was difficult for contributors to tell if a survey was ethical, vetted, etc. Frankly, I think some long-term contributors just felt jaded -- for a while it seemed there were lots of surveys and studies to try to find out things that seemed intuitively obvious if you were a participant in the community. I think Heather is right that it seems like there have been fewer surveys in recent years, for whatever reason.
Part of the problem is a somewhat subtle demographic one: while contributors to Wikipedia do turn over, so newer contributors will not necessarily have seen lots of surveys, very heavy editors and admins (who are often easier to identify) tend to be long-term participants who might have been surveyed many times. Additionally, the people who follow mailing lists, social media, etc. (or at least the people who speak up on those channels) skew towards very-long-term contributors who have strong opinions and have seen it all before. So, if you advertise your survey on the mailing list, that's the population you get, and that's the feedback you get. (But it's a catch-22; there's not really other obvious mass channels).
Anyway, this is a hard problem without super-obvious solutions, and not one that there's a lot of models for -- very few online projects are simultaneously as open with their data and as interesting for research purposes!
best, Phoebe
On 17 July 2014 17:55, phoebe ayers phoebe.wiki@gmail.com wrote:
Part of the problem is a somewhat subtle demographic one: while contributors to Wikipedia do turn over, so newer contributors will not necessarily have seen lots of surveys, very heavy editors and admins (who are often easier to identify) tend to be long-term participants who might have been surveyed many times. Additionally, the people who follow mailing lists, social media, etc. (or at least the people who speak up on those channels) skew towards very-long-term contributors who have strong opinions and have seen it all before. So, if you advertise your survey on the mailing list, that's the population you get, and that's the feedback you get. (But it's a catch-22; there's not really other obvious mass channels).
This is a really important insight, thanks for sharing it, Phoebe. It's important to work out what the problem is that we're trying to solve before we try solving it! If the key problem here is that Wikipedians need to be protected from researchers constantly surveying them, and actually the wide-ranging surveys are really rare these days, then maybe the problem is with heavy editors and admins being constantly 'surveyed' (although I'm guessing that this is not the only research method being used as I talk about below).
Does anyone know whether this is actually a problem with editors these days? I know that I have interviewed a bunch of editors over the years without RCOM approval (some with RCOM approval) and I have only had good experiences. Sure there were people who didn't want to be interviewed, but they just ignored my requests - I'm not sure that they would say that they were bothered enough that an entire process needed to be developed to approve projects.
I think part of the problem here is that there is a bias towards particular types of research projects in the way that RCOM was designed. I do both quantitative and qualitative research on WP and the quantitative research nowadays focuses mostly on capturing large-scale user actions using the API or the dumps - I have a feeling that's why there are fewer surveys these days - more researchers are using the data to conduct research and (right now) that doesn't require any permissions beyond what is required by uni ethics board (and all the problems that come with that!).
The projects I do as a qualitative researcher tend to be exploratory. I will interview people on skype, for example, about their work on particular articles before I know that I have a project. I could certainly develop a proposal to RCOM but it would be so wide-ranging that I'm unsure what the actual benefit was. I think that a much bigger problem is actually developing community guidelines around ethical treatment of subjects who don't often realise that their comments and interactions can be legally (but, I believe not necessarily ethically) used without their permission (I wrote something about my thoughts on this here [1]).
Basically, I think that we need to reassess what kinds of problems are the most important ones right now that we want to solve rather than resuscitating a process that was designed to address a specific type of problem that was prevalent a long time ago. The new problems that I see right now that a research community is best placed to solve are things like:
- developing community guidelines for the representation of editors' identities in research (similar, perhaps to the AOIR guidelines [2]); - finding ways of making responsible requests to the WMF for data that they hold that might benefit research outside the WMF; - developing opportunities for researchers to collaborate and share what they're doing with the wider research community (as Kerry suggests).
[1] http://ethnographymatters.net/blog/2013/06/27/onymous-pseudonymous-neither-o...
[2] http://aoir.org/reports/ethics2.pdf
Best, Heather.
Anyway, this is a hard problem without super-obvious solutions, and not one that there's a lot of models for -- very few online projects are simultaneously as open with their data and as interesting for research purposes!
best, Phoebe
--
- I use this address for lists; send personal messages to phoebe.ayers
<at> gmail.com *
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
"- developing community guidelines for the representation of editors' identities in research (similar, perhaps to the AOIR guidelines [2]); - finding ways of making responsible requests to the WMF for data that they hold that might benefit research outside the WMF; - developing opportunities for researchers to collaborate and share what they're doing with the wider research community (as Kerry suggests). "
Some good ideas here. The first point can be clarified in the ethically researching Wikipedia page. The second point is something that RCOM could do, if it focused more on helping researchers. Third idea is related to the fact that too many researchers don't know about this listerv or other (or any) Wikipedia/meta research pages. As I said two posts before or so, we should reorganize all research pages, creating a well-linked and non-duplicating best practices/list of structure a bit similar to the Wikipedia Global Education program, then try to advertise it to all all past and present researchers of Wikipedia.
--
Piotr Konieczny, PhD http://hanyang.academia.edu/PiotrKonieczny http://scholar.google.com/citations?user=gdV8_AEAAAAJ http://en.wikipedia.org/wiki/User:Piotrus
On 7/18/2014 17:07, Heather Ford wrote:
On 17 July 2014 17:55, phoebe ayers <phoebe.wiki@gmail.com mailto:phoebe.wiki@gmail.com> wrote:
Part of the problem is a somewhat subtle demographic one: while contributors to Wikipedia do turn over, so newer contributors will not necessarily have seen lots of surveys, very heavy editors and admins (who are often easier to identify) tend to be long-term participants who might have been surveyed many times. Additionally, the people who follow mailing lists, social media, etc. (or at least the people who speak up on those channels) skew towards very-long-term contributors who have strong opinions and have seen it all before. So, if you advertise your survey on the mailing list, that's the population you get, and that's the feedback you get. (But it's a catch-22; there's not really other obvious mass channels).This is a really important insight, thanks for sharing it, Phoebe. It's important to work out what the problem is that we're trying to solve before we try solving it! If the key problem here is that Wikipedians need to be protected from researchers constantly surveying them, and actually the wide-ranging surveys are really rare these days, then maybe the problem is with heavy editors and admins being constantly 'surveyed' (although I'm guessing that this is not the only research method being used as I talk about below).
Does anyone know whether this is actually a problem with editors these days? I know that I have interviewed a bunch of editors over the years without RCOM approval (some with RCOM approval) and I have only had good experiences. Sure there were people who didn't want to be interviewed, but they just ignored my requests - I'm not sure that they would say that they were bothered enough that an entire process needed to be developed to approve projects.
I think part of the problem here is that there is a bias towards particular types of research projects in the way that RCOM was designed. I do both quantitative and qualitative research on WP and the quantitative research nowadays focuses mostly on capturing large-scale user actions using the API or the dumps - I have a feeling that's why there are fewer surveys these days - more researchers are using the data to conduct research and (right now) that doesn't require any permissions beyond what is required by uni ethics board (and all the problems that come with that!).
The projects I do as a qualitative researcher tend to be exploratory. I will interview people on skype, for example, about their work on particular articles before I know that I have a project. I could certainly develop a proposal to RCOM but it would be so wide-ranging that I'm unsure what the actual benefit was. I think that a much bigger problem is actually developing community guidelines around ethical treatment of subjects who don't often realise that their comments and interactions can be legally (but, I believe not necessarily ethically) used without their permission (I wrote something about my thoughts on this here [1]).
Basically, I think that we need to reassess what kinds of problems are the most important ones right now that we want to solve rather than resuscitating a process that was designed to address a specific type of problem that was prevalent a long time ago. The new problems that I see right now that a research community is best placed to solve are things like:
- developing community guidelines for the representation of editors'
identities in research (similar, perhaps to the AOIR guidelines [2]);
- finding ways of making responsible requests to the WMF for data that
they hold that might benefit research outside the WMF;
- developing opportunities for researchers to collaborate and share
what they're doing with the wider research community (as Kerry suggests).
[1] http://ethnographymatters.net/blog/2013/06/27/onymous-pseudonymous-neither-o...
[2] http://aoir.org/reports/ethics2.pdf
Best, Heather.
Anyway, this is a hard problem without super-obvious solutions, and not one that there's a lot of models for -- very few online projects are simultaneously as open with their data and as interesting for research purposes! best, Phoebe -- * I use this address for lists; send personal messages to phoebe.ayers <at> gmail.com <http://gmail.com> * _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
"it was difficult for contributors to tell if a survey was ethical, vetted"
Now, that's a problem of bad research design. Survey design 101 requires that an invitation to the survey briefly discusses those issues. It all boils down to the fact that many lazy or inadequately trained researchers don't bother to do what is described at https://en.wikipedia.org/wiki/Wikipedia:Ethically_researching_Wikipedia#Surv...
It is, unfortunately, not in our power to educate those researchers. RCOM cannot do it, because most researchers will never find out it exists, and will send invitations to their surveys or such ignoring any required (or recommended) processes.
There's only one way that a body like RCOM could try to have some real influence among serious Wikipedia researchers who at least have a decent chance to finding out that it exists and what it does (like those of us here). That is, if it had a carrot to go with the stick of (what, exactly, I am still not sure - ban researchers accounts if they don't follow RCOM procedure? Or just frown at them at WikiSym?). The carrot could be a friendly user interface that would give a researcher an easy way to sample population and send surveys to it, in exchange of jumping through the hoops of whatever RCOM procedure creep becomes. People may consider signing up for RCOM review or such if RCOM gives them something of value in return. Until this happens, I don't except RCOM will become more useful or visible than it has been for the past few years; in fact I am predicting the continuation of its decline, as more and more people realize its a toothless and basically unnecessary body.
--
Piotr Konieczny, PhD http://hanyang.academia.edu/PiotrKonieczny http://scholar.google.com/citations?user=gdV8_AEAAAAJ http://en.wikipedia.org/wiki/User:Piotrus
On 7/18/2014 01:55, phoebe ayers wrote:
On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford <hfordsa@gmail.com mailto:hfordsa@gmail.com> wrote:
Agree with Kerry that we really need to have a more flexible process that speaks to the main problem that (I think) RCOM was started to solve i.e. that Wikipedians were getting tired of being continually contacted by researchers to fill out *surveys*.That's correct, afaik that was the original motivation, along with some of the concerns that Lane/Nathan raised in the other list -- i.e. that it was difficult for contributors to tell if a survey was ethical, vetted, etc. Frankly, I think some long-term contributors just felt jaded -- for a while it seemed there were lots of surveys and studies to try to find out things that seemed intuitively obvious if you were a participant in the community. I think Heather is right that it seems like there have been fewer surveys in recent years, for whatever reason.
Part of the problem is a somewhat subtle demographic one: while contributors to Wikipedia do turn over, so newer contributors will not necessarily have seen lots of surveys, very heavy editors and admins (who are often easier to identify) tend to be long-term participants who might have been surveyed many times. Additionally, the people who follow mailing lists, social media, etc. (or at least the people who speak up on those channels) skew towards very-long-term contributors who have strong opinions and have seen it all before. So, if you advertise your survey on the mailing list, that's the population you get, and that's the feedback you get. (But it's a catch-22; there's not really other obvious mass channels).
Anyway, this is a hard problem without super-obvious solutions, and not one that there's a lot of models for -- very few online projects are simultaneously as open with their data and as interesting for research purposes!
best, Phoebe
--
- I use this address for lists; send personal messages to phoebe.ayers
<at> gmail.com http://gmail.com *
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
First, I wanted to highlight the important issue that Heather raises here, because although it's a separate issue, it's an important one:
On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford hfordsa@gmail.com wrote:
...
One immediate requirement that I've been talking to others about is finding ways of making the case to the WMF as a group of researchers for the anonymization of country level data, for example. I've spoken to a few researchers (and I myself made a request about a year ago that hasn't been responded to) and it seems like some work is required by the foundation to do this anonymisation but that there are a few of us who would be really keen to use this data to produce research very valuable to Wikipedia - especially from smaller language versions/developing countries. Having an official process that assesses how worthwhile this investment of time would be to the Foundation would be a great idea, I think, but right now there seems to be a general focus on the research that the Foundation does itself rather than enabling researchers outside. I know how busy Aaron and Dario (and others in the team) are so perhaps this requires a new position to coordinate between researchers and Foundation resources?
Anyway, I think the big question right now is whether there are any plans for RCOM that have been made by the research team and the only people who can answer that are folks in the research team :)
Best, Heather.
As a community-run group, RCOM doesn't have any role in making non-public data available to researchers. However, Aaron and I are putting together a proposal for a workshop https://meta.wikimedia.org/wiki/Research:CSCW15_workshop that would address issues like this. That's work we're doing in an official capacity, as opposed to the RCOM work, which is volunteer.
On RCOM more generally... I think clarifying the role of the committee, and getting a larger and more diverse set of people involved, might help make RCOM work. But as Aaron can attest, it is difficult to get people to agree on what RCOMs role should be, let alone get them to work for RCOM.
I've been involved with RCOM for a while, albeit not very actively. Unfortunately, I think that the fact that the only people who "review" requests *happen to be** WMF staffers contributes to confusion about RCOM's role and it's authority. IMO, if RCOM or any other subject recruitment review process is to succeed, we need:
- more wiki-researchers who are willing to regularly participate in both peer review *and* in developing better process guidelines and standards (it's really just Aaron right now) - more *Wikipedians* who are willing to do the same - some degree of buy-in from the Wikimedia community as a whole. RCOM needs legitimacy. But where, and from whom? Subject recruitment is a global concern, but the proposed subject recruitment process is focused on en-wiki (mostly because that's where most of the relevant research activities *that we are aware of* are happening). How to make RCOM more global?
RCOM is in a tough spot right now. We can't force researchers to submit their proposals, or abide by the suggestions/recommendations/decisions/whatever that result from their review. But because we *look like *an official body, it's easy to blame us for failing to prevent disruptive research (if you're a community member), for "rubber stamping" research that we like (ditto), or for drowning research in red tape (if you're a wiki-researcher).
- J
*we were wiki-researchers first!
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 17 July 2014 08:49, Kerry Raymond kerry.raymond@gmail.com wrote:
Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use.
Kerry
*From:* Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il] *Sent:* Thursday, 17 July 2014 5:37 PM *To:* kerry.raymond@gmail.com; Research into Wikimedia content and communities
*Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
WMF does not "own" me as a contributor; it does not decide who can and
cannot recruit me for whatever purposes.
I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-07-17 8:24 GMT+03:00 Kerry Raymond kerry.raymond@gmail.com:
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario ( dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) < nemowiki@gmail.com> wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: < https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&...
It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Jonathan Morgan, 17/07/2014 23:37:
But because we /look like /an official body, it's easy to blame us for failing to prevent disruptive research (if you're a community member), for "rubber stamping" research that we like (ditto), or for drowning research in red tape (if you're a wiki-researcher).
RCOM doesn't *look like* an official body, it claims to be one. With its current structure, it looks like a WMF staff committee. https://wikimediafoundation.org/wiki/Resolution:Wikimedia_Committees#Staff_c...
If you don't want it to look official, it's easy: call it "interest group", add a "draft" template, add a "under pilot" warning, call it a subcommittee of the communications committee (a rather common format).
Nemo
On Thu, Jul 17, 2014 at 11:36 PM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
Jonathan Morgan, 17/07/2014 23:37:
But because we /look like /an official body, it's easy to blame us for failing to prevent disruptive research (if you're a community member), for "rubber stamping" research that we like (ditto), or for drowning research in red tape (if you're a wiki-researcher).
RCOM doesn't *look like* an official body, it claims to be one. With its current structure, it looks like a WMF staff committee.
https://wikimediafoundation.org/wiki/Resolution:Wikimedia_Committees#Staff_c...
If you don't want it to look official, it's easy: call it "interest group", add a "draft" template, add a "under pilot" warning, call it a subcommittee of the communications committee (a rather common format).
Heh. Well, I wasn't aware it was I was participating in an official body. Does that mean I get to review proposals under my staff account now? :P
I suppose if *I'm *this confused about RCOM's role, I shouldn't be surprised that others are too.
- J
Yep. One of the things that ruffled my feathers about RCOM from early on was that without any official community or WMF support, it (or some of its members, perhaps not expressing themselves clearly) gave the impression that it holds (or should, or want) the power to decide what can and cannot be researched with regards to Wikipedia. So, at least as far as I am concerned, instead of looking like a best-practices-we-want-to-help body, it started to look like IRB/Godking-wannabe, offering nothing but promising to contribute to instruction/procedure creep.
--
Piotr Konieczny, PhD http://hanyang.academia.edu/PiotrKonieczny http://scholar.google.com/citations?user=gdV8_AEAAAAJ http://en.wikipedia.org/wiki/User:Piotrus
On 7/18/2014 15:36, Federico Leva (Nemo) wrote:
Jonathan Morgan, 17/07/2014 23:37:
But because we /look like /an official body, it's easy to blame us for failing to prevent disruptive research (if you're a community member), for "rubber stamping" research that we like (ditto), or for drowning research in red tape (if you're a wiki-researcher).
RCOM doesn't *look like* an official body, it claims to be one. With its current structure, it looks like a WMF staff committee. https://wikimediafoundation.org/wiki/Resolution:Wikimedia_Committees#Staff_c...
If you don't want it to look official, it's easy: call it "interest group", add a "draft" template, add a "under pilot" warning, call it a subcommittee of the communications committee (a rather common format).
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
On 17 July 2014 22:37, Jonathan Morgan jmorgan@wikimedia.org wrote:
First, I wanted to highlight the important issue that Heather raises here, because although it's a separate issue, it's an important one:
On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford hfordsa@gmail.com wrote:
...
One immediate requirement that I've been talking to others about is finding ways of making the case to the WMF as a group of researchers for the anonymization of country level data, for example. I've spoken to a few researchers (and I myself made a request about a year ago that hasn't been responded to) and it seems like some work is required by the foundation to do this anonymisation but that there are a few of us who would be really keen to use this data to produce research very valuable to Wikipedia - especially from smaller language versions/developing countries. Having an official process that assesses how worthwhile this investment of time would be to the Foundation would be a great idea, I think, but right now there seems to be a general focus on the research that the Foundation does itself rather than enabling researchers outside. I know how busy Aaron and Dario (and others in the team) are so perhaps this requires a new position to coordinate between researchers and Foundation resources?
As a community-run group, RCOM doesn't have any role in making non-public data available to researchers. However, Aaron and I are putting together a proposal for a workshop https://meta.wikimedia.org/wiki/Research:CSCW15_workshop that would address issues like this. That's work we're doing in an official capacity, as opposed to the RCOM work, which is volunteer.
Jonathan, it looks like this will be a great workshop and I think CSCW is a great venue! but I don't think it addresses the issue unless there's something I'm missing (like an invitation, for example! ;) I see that the workshop is forward-facing but its aim seems to be to work with a bunch of different communities like Reddit and GalaxyZoo. What we need are better channels as Wikipedia researchers to communicate our needs as researchers operating outside the WMF. And preferably in a way that doesn't require us to have to travel to Canada to a workshop to do it!
And, I offered it as a joke but it reminds me of a small, subtle point, I think it would be nice if you could offer an invitation to the researchers on this list to join the workshop and/or workshop planning when you advertise the work you're doing on this. I know it's a wiki and anyone could probably join, but I feel like there is enormous possibility for the group represented here to feel involved and recognised, and I, for one, would like to be invited sometimes.. to the fun stuff, that is, not just the hard, arduous stuff :)
Best, Heather.
On RCOM more generally... I think clarifying the role of the committee, and getting a larger and more diverse set of people involved, might help make RCOM work. But as Aaron can attest, it is difficult to get people to agree on what RCOMs role should be, let alone get them to work for RCOM.
I've been involved with RCOM for a while, albeit not very actively. Unfortunately, I think that the fact that the only people who "review" requests *happen to be** WMF staffers contributes to confusion about RCOM's role and it's authority. IMO, if RCOM or any other subject recruitment review process is to succeed, we need:
- more wiki-researchers who are willing to regularly participate in
both peer review *and* in developing better process guidelines and standards (it's really just Aaron right now)
- more *Wikipedians* who are willing to do the same
- some degree of buy-in from the Wikimedia community as a whole. RCOM
needs legitimacy. But where, and from whom? Subject recruitment is a global concern, but the proposed subject recruitment process is focused on en-wiki (mostly because that's where most of the relevant research activities *that we are aware of* are happening). How to make RCOM more global?
RCOM is in a tough spot right now. We can't force researchers to submit their proposals, or abide by the suggestions/recommendations/decisions/whatever that result from their review. But because we *look like *an official body, it's easy to blame us for failing to prevent disruptive research (if you're a community member), for "rubber stamping" research that we like (ditto), or for drowning research in red tape (if you're a wiki-researcher).
- J
*we were wiki-researchers first!
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 17 July 2014 08:49, Kerry Raymond kerry.raymond@gmail.com wrote:
Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use.
Kerry
*From:* Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il] *Sent:* Thursday, 17 July 2014 5:37 PM *To:* kerry.raymond@gmail.com; Research into Wikimedia content and communities
*Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
WMF does not "own" me as a contributor; it does not decide who can
and cannot recruit me for whatever purposes.
I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-07-17 8:24 GMT+03:00 Kerry Raymond kerry.raymond@gmail.com:
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario ( dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) < nemowiki@gmail.com> wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: < https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&...
It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation User:Jmorgan (WMF) https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF) jmorgan@wikimedia.org
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Does anyone know whether this is actually a problem with editors these days?
Yes. We regularly see requests to survey the most active Wikipedians about their motivations to edit. These requests are problematic for some very obvious reasons. See this proposal[1] for an example of a study that was halted in review due to the disruption it would have caused.
The projects I do as a qualitative researcher tend to be exploratory. I
will interview people on skype, for example, about their work on particular articles before I know that I have a project.
Do you document your study on wiki and ask for feedback about disruption before moving forward? Regardless of the process around it, I think we might all agree that is good behavior for any research activity. This might be obvious to you as someone who has been doing ethnographic work in Wikimedia communities for a long time, but it is apprently less obvious to more junior wiki researchers.
This good-faith documentation and discussion describes the whole RCom subject recruitment process. You refer to RCom as "heavy weight", but as far as I can tell, the weight is entirely on the RCom coordinator -- a burden I'll gladly accept to help good research take place without disruption. Researchers should have already documented their research and prepared themselves to discuss the work with their subjects before they arrive.
I don't know of a single study that has passed stalled in RCom's process that has resulted in substantial disruption or stalled for more than two weeks. I welcome you to provide counter examples.
I don't think [the CSCW workshop proposal] addresses the issue unless
there's something I'm missing (like an invitation, for example!
One of the ways that researchers can be supported is through groups that help them socialize their research activities with community members (and minimize disruption for community members). Despite the tone of this conversation, we have been highly successful in this regard.
I think it would be nice if you could offer an invitation to the
researchers on this list
That's the plan. We're just getting to a point where we have a solid idea of what we want to accomplish. An announcement will come soon.
Basically, I think that we need to reassess what kinds of problems are the
most important ones right now that we want to solve rather than resuscitating a process that was designed to address a specific type of problem that was prevalent a long time ago
As I pointed out previously, the subject recruitment process is alive and does not need to be "resuscitated ". It is also solving a relevant problem. I welcome Lane Rasberry (if he has time) to share his substantial concerns about undocumented, undiscussed research taking place on-wiki.
1. https://meta.wikimedia.org/wiki/Research:Online_knowledge_sharing 2. https://meta.wikimedia.org/wiki/Category:Uncategorized_support_requests
-Aaron
On Fri, Jul 18, 2014 at 1:27 AM, Heather Ford hfordsa@gmail.com wrote:
On 17 July 2014 22:37, Jonathan Morgan jmorgan@wikimedia.org wrote:
First, I wanted to highlight the important issue that Heather raises here, because although it's a separate issue, it's an important one:
On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford hfordsa@gmail.com wrote:
...
One immediate requirement that I've been talking to others about is finding ways of making the case to the WMF as a group of researchers for the anonymization of country level data, for example. I've spoken to a few researchers (and I myself made a request about a year ago that hasn't been responded to) and it seems like some work is required by the foundation to do this anonymisation but that there are a few of us who would be really keen to use this data to produce research very valuable to Wikipedia - especially from smaller language versions/developing countries. Having an official process that assesses how worthwhile this investment of time would be to the Foundation would be a great idea, I think, but right now there seems to be a general focus on the research that the Foundation does itself rather than enabling researchers outside. I know how busy Aaron and Dario (and others in the team) are so perhaps this requires a new position to coordinate between researchers and Foundation resources?
As a community-run group, RCOM doesn't have any role in making non-public data available to researchers. However, Aaron and I are putting together a proposal for a workshop https://meta.wikimedia.org/wiki/Research:CSCW15_workshop that would address issues like this. That's work we're doing in an official capacity, as opposed to the RCOM work, which is volunteer.
Jonathan, it looks like this will be a great workshop and I think CSCW is a great venue! but I don't think it addresses the issue unless there's something I'm missing (like an invitation, for example! ;) I see that the workshop is forward-facing but its aim seems to be to work with a bunch of different communities like Reddit and GalaxyZoo. What we need are better channels as Wikipedia researchers to communicate our needs as researchers operating outside the WMF. And preferably in a way that doesn't require us to have to travel to Canada to a workshop to do it!
And, I offered it as a joke but it reminds me of a small, subtle point, I think it would be nice if you could offer an invitation to the researchers on this list to join the workshop and/or workshop planning when you advertise the work you're doing on this. I know it's a wiki and anyone could probably join, but I feel like there is enormous possibility for the group represented here to feel involved and recognised, and I, for one, would like to be invited sometimes.. to the fun stuff, that is, not just the hard, arduous stuff :)
Best, Heather.
On RCOM more generally... I think clarifying the role of the committee, and getting a larger and more diverse set of people involved, might help make RCOM work. But as Aaron can attest, it is difficult to get people to agree on what RCOMs role should be, let alone get them to work for RCOM.
I've been involved with RCOM for a while, albeit not very actively. Unfortunately, I think that the fact that the only people who "review" requests *happen to be** WMF staffers contributes to confusion about RCOM's role and it's authority. IMO, if RCOM or any other subject recruitment review process is to succeed, we need:
- more wiki-researchers who are willing to regularly participate in
both peer review *and* in developing better process guidelines and standards (it's really just Aaron right now)
- more *Wikipedians* who are willing to do the same
- some degree of buy-in from the Wikimedia community as a whole. RCOM
needs legitimacy. But where, and from whom? Subject recruitment is a global concern, but the proposed subject recruitment process is focused on en-wiki (mostly because that's where most of the relevant research activities *that we are aware of* are happening). How to make RCOM more global?
RCOM is in a tough spot right now. We can't force researchers to submit their proposals, or abide by the suggestions/recommendations/decisions/whatever that result from their review. But because we *look like *an official body, it's easy to blame us for failing to prevent disruptive research (if you're a community member), for "rubber stamping" research that we like (ditto), or for drowning research in red tape (if you're a wiki-researcher).
- J
*we were wiki-researchers first!
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 17 July 2014 08:49, Kerry Raymond kerry.raymond@gmail.com wrote:
Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use.
Kerry
*From:* Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il] *Sent:* Thursday, 17 July 2014 5:37 PM *To:* kerry.raymond@gmail.com; Research into Wikimedia content and communities
*Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
WMF does not "own" me as a contributor; it does not decide who can
and cannot recruit me for whatever purposes.
I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-07-17 8:24 GMT+03:00 Kerry Raymond kerry.raymond@gmail.com:
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario ( dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) < nemowiki@gmail.com> wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: < https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&...
It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation User:Jmorgan (WMF) https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF) jmorgan@wikimedia.org
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
"We regularly see requests to survey the most active Wikipedians about their motivations to edit."
As I am in the top 100 most active Wikipedians, unless I am an outlier for some reason, very few of those projects come to fruition, as I get no more than 1-2 requests a year, at most.
"See this proposal[1] for an example of a study that was halted in review due to the disruption it would have caused."
So, they would ask 500 people to take part in a 10 minute survey. A bit long, but... so what? I expect they'd get a response ratio of about 10%, so they should contact the Top 5000. Still not seeing a problem. Those who don't want, don't take part in the survey. It would be nice if the researchers promised to do something constructive like improve Wikipedia content, give out random prizes to contributors, or such to "give back" to the community. Perhaps an idea to add to best practices, but... where's that disruption? What am I missing?
--
Piotr Konieczny, PhD http://hanyang.academia.edu/PiotrKonieczny http://scholar.google.com/citations?user=gdV8_AEAAAAJ http://en.wikipedia.org/wiki/User:Piotrus
On 7/19/2014 02:59, Aaron Halfaker wrote:
Does anyone know whether this is actually a problem with editors these days?Yes. We regularly see requests to survey the most active Wikipedians about their motivations to edit. These requests are problematic for some very obvious reasons. See this proposal[1] for an example of a study that was halted in review due to the disruption it would have caused.
The projects I do as a qualitative researcher tend to be exploratory. I will interview people on skype, for example, about their work on particular articles before I know that I have a project.Do you document your study on wiki and ask for feedback about disruption before moving forward? Regardless of the process around it, I think we might all agree that is good behavior for any research activity. This might be obvious to you as someone who has been doing ethnographic work in Wikimedia communities for a long time, but it is apprently less obvious to more junior wiki researchers.
This good-faith documentation and discussion describes the whole RCom subject recruitment process. You refer to RCom as "heavy weight", but as far as I can tell, the weight is entirely on the RCom coordinator -- a burden I'll gladly accept to help good research take place without disruption. Researchers should have already documented their research and prepared themselves to discuss the work with their subjects before they arrive.
I don't know of a single study that has passed stalled in RCom's process that has resulted in substantial disruption or stalled for more than two weeks. I welcome you to provide counter examples.
I don't think [the CSCW workshop proposal] addresses the issue unless there's something I'm missing (like an invitation, for example!One of the ways that researchers can be supported is through groups that help them socialize their research activities with community members (and minimize disruption for community members). Despite the tone of this conversation, we have been highly successful in this regard.
I think it would be nice if you could offer an invitation to the researchers on this listThat's the plan. We're just getting to a point where we have a solid idea of what we want to accomplish. An announcement will come soon.
Basically, I think that we need to reassess what kinds of problems are the most important ones right now that we want to solve rather than resuscitating a process that was designed to address a specific type of problem that was prevalent a long time agoAs I pointed out previously, the subject recruitment process is alive and does not need to be "resuscitated ". It is also solving a relevant problem. I welcome Lane Rasberry (if he has time) to share his substantial concerns about undocumented, undiscussed research taking place on-wiki.
- https://meta.wikimedia.org/wiki/Research:Online_knowledge_sharing
- https://meta.wikimedia.org/wiki/Category:Uncategorized_support_requests
-Aaron
On Fri, Jul 18, 2014 at 1:27 AM, Heather Ford <hfordsa@gmail.com mailto:hfordsa@gmail.com> wrote:
On 17 July 2014 22:37, Jonathan Morgan <jmorgan@wikimedia.org <mailto:jmorgan@wikimedia.org>> wrote: First, I wanted to highlight the important issue that Heather raises here, because although it's a separate issue, it's an important one: On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford <hfordsa@gmail.com <mailto:hfordsa@gmail.com>> wrote: ... One immediate requirement that I've been talking to others about is finding ways of making the case to the WMF as a group of researchers for the anonymization of country level data, for example. I've spoken to a few researchers (and I myself made a request about a year ago that hasn't been responded to) and it seems like some work is required by the foundation to do this anonymisation but that there are a few of us who would be really keen to use this data to produce research very valuable to Wikipedia - especially from smaller language versions/developing countries. Having an official process that assesses how worthwhile this investment of time would be to the Foundation would be a great idea, I think, but right now there seems to be a general focus on the research that the Foundation does itself rather than enabling researchers outside. I know how busy Aaron and Dario (and others in the team) are so perhaps this requires a new position to coordinate between researchers and Foundation resources? As a community-run group, RCOM doesn't have any role in making non-public data available to researchers. However, Aaron and I are putting together a proposal for a workshop <https://meta.wikimedia.org/wiki/Research:CSCW15_workshop> that would address issues like this. That's work we're doing in an official capacity, as opposed to the RCOM work, which is volunteer. Jonathan, it looks like this will be a great workshop and I think CSCW is a great venue! but I don't think it addresses the issue unless there's something I'm missing (like an invitation, for example! ;) I see that the workshop is forward-facing but its aim seems to be to work with a bunch of different communities like Reddit and GalaxyZoo. What we need are better channels as Wikipedia researchers to communicate our needs as researchers operating outside the WMF. And preferably in a way that doesn't require us to have to travel to Canada to a workshop to do it! And, I offered it as a joke but it reminds me of a small, subtle point, I think it would be nice if you could offer an invitation to the researchers on this list to join the workshop and/or workshop planning when you advertise the work you're doing on this. I know it's a wiki and anyone could probably join, but I feel like there is enormous possibility for the group represented here to feel involved and recognised, and I, for one, would like to be invited sometimes.. to the fun stuff, that is, not just the hard, arduous stuff :) Best, Heather. On RCOM more generally... I think clarifying the role of the committee, and getting a larger and more diverse set of people involved, might help make RCOM work. But as Aaron can attest, it is difficult to get people to agree on what RCOMs role should be, let alone get them to work for RCOM. I've been involved with RCOM for a while, albeit not very actively. Unfortunately, I think that the fact that the only people who "review" requests /happen to be*/ WMF staffers contributes to confusion about RCOM's role and it's authority. IMO, if RCOM or any other subject recruitment review process is to succeed, we need: * more wiki-researchers who are willing to regularly participate in both peer review /and/ in developing better process guidelines and standards (it's really just Aaron right now) * more /Wikipedians/ who are willing to do the same * some degree of buy-in from the Wikimedia community as a whole. RCOM needs legitimacy. But where, and from whom? Subject recruitment is a global concern, but the proposed subject recruitment process is focused on en-wiki (mostly because that's where most of the relevant research activities /that we are aware of/ are happening). How to make RCOM more global? RCOM is in a tough spot right now. We can't force researchers to submit their proposals, or abide by the suggestions/recommendations/decisions/whatever that result from their review. But because we /look like /an official body, it's easy to blame us for failing to prevent disruptive research (if you're a community member), for "rubber stamping" research that we like (ditto), or for drowning research in red tape (if you're a wiki-researcher). - J *we were wiki-researchers first! Heather Ford Oxford Internet Institute <http://www.oii.ox.ac.uk> Doctoral Programme EthnographyMatters <http://ethnographymatters.net> | Oxford Digital Ethnography Group <http://www.oii.ox.ac.uk/research/projects/?id=115> http://hblog.org <http://hblog.org/> | @hfordsa <http://www.twitter.com/hfordsa> On 17 July 2014 08:49, Kerry Raymond <kerry.raymond@gmail.com <mailto:kerry.raymond@gmail.com>> wrote: Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use. Kerry ------------------------------------------------------------------------ *From:*Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il <mailto:amir.aharoni@mail.huji.ac.il>] *Sent:* Thursday, 17 July 2014 5:37 PM *To:* kerry.raymond@gmail.com <mailto:kerry.raymond@gmail.com>; Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys > WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed. -- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore 2014-07-17 8:24 GMT+03:00 Kerry Raymond <kerry.raymond@gmail.com <mailto:kerry.raymond@gmail.com>>: Just saying here what I already put on the Talk page: I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments." WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g. "This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]." If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels. Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence. Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here … Kerry ------------------------------------------------------------------------ *From:*wiki-research-l-bounces@lists.wikimedia.org <mailto:wiki-research-l-bounces@lists.wikimedia.org> [mailto:wiki-research-l-bounces@lists.wikimedia.org <mailto:wiki-research-l-bounces@lists.wikimedia.org>] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario (dtaraborelli@wikimedia.org <mailto:dtaraborelli@wikimedia.org>) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) <nemowiki@gmail.com <mailto:nemowiki@gmail.com>> wrote: phoebe ayers, 16/07/2014 19:21: > (Personally, I think the answer should be to resuscitate RCOM, but > that's easy to say and harder to do!) IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: <https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&hidebots=1&hideredirs=1&limit=500&offset=&namespace=202> It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round. Nemo _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l -- Jonathan T. Morgan Learning Strategist Wikimedia Foundation User:Jmorgan (WMF) <https://meta.wikimedia.org/wiki/User:Jmorgan_%28WMF%29> jmorgan@wikimedia.org <mailto:jmorgan@wikimedia.org> _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
On Fri, Jul 18, 2014 at 1:27 AM, Heather Ford hfordsa@gmail.com wrote:
Jonathan, it looks like this will be a great workshop and I think CSCW is a great venue! but I don't think it addresses the issue unless there's something I'm missing (like an invitation, for example! ;) I see that the workshop is forward-facing but its aim seems to be to work with a bunch of different communities like Reddit and GalaxyZoo. What we need are better channels as Wikipedia researchers to communicate our needs as researchers operating outside the WMF. And preferably in a way that doesn't require us to have to travel to Canada to a workshop to do it!
And, I offered it as a joke but it reminds me of a small, subtle point, I think it would be nice if you could offer an invitation to the researchers on this list to join the workshop and/or workshop planning when you advertise the work you're doing on this. I know it's a wiki and anyone could probably join, but I feel like there is enormous possibility for the group represented here to feel involved and recognised, and I, for one, would like to be invited sometimes.. to the fun stuff, that is, not just the hard, arduous stuff :)
For me, it's just a matter of bandwidth. I get a lot of personal requests for advice, pointers, data, etc from wiki-researchers. I try to answer them, but I can't personally work with every researcher who wants special access beyond public dumps, APIs, databases... or who wants an "in" on subject recruitment. I don't scale all that well :)
Plus, that's an opaque and ad-hoc process, and it doesn't contribute to the formation of public standards and guidelines that benefit other researchers. Hence, the workshop. We will try to get a bunch of researchers together in one place, figure out what their needs are, and get them involved in developing a better process for quenching their data thirst!
Not sure I get your invite question, tho? Are you asking to be a co-author on our workshop proposal? ;)
Best,
Heather.
On RCOM more generally... I think clarifying the role of the committee, and getting a larger and more diverse set of people involved, might help make RCOM work. But as Aaron can attest, it is difficult to get people to agree on what RCOMs role should be, let alone get them to work for RCOM.
I've been involved with RCOM for a while, albeit not very actively. Unfortunately, I think that the fact that the only people who "review" requests *happen to be** WMF staffers contributes to confusion about RCOM's role and it's authority. IMO, if RCOM or any other subject recruitment review process is to succeed, we need:
- more wiki-researchers who are willing to regularly participate in
both peer review *and* in developing better process guidelines and standards (it's really just Aaron right now)
- more *Wikipedians* who are willing to do the same
- some degree of buy-in from the Wikimedia community as a whole. RCOM
needs legitimacy. But where, and from whom? Subject recruitment is a global concern, but the proposed subject recruitment process is focused on en-wiki (mostly because that's where most of the relevant research activities *that we are aware of* are happening). How to make RCOM more global?
RCOM is in a tough spot right now. We can't force researchers to submit their proposals, or abide by the suggestions/recommendations/decisions/whatever that result from their review. But because we *look like *an official body, it's easy to blame us for failing to prevent disruptive research (if you're a community member), for "rubber stamping" research that we like (ditto), or for drowning research in red tape (if you're a wiki-researcher).
- J
*we were wiki-researchers first!
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 17 July 2014 08:49, Kerry Raymond kerry.raymond@gmail.com wrote:
Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use.
Kerry
*From:* Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il] *Sent:* Thursday, 17 July 2014 5:37 PM *To:* kerry.raymond@gmail.com; Research into Wikimedia content and communities
*Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
WMF does not "own" me as a contributor; it does not decide who can
and cannot recruit me for whatever purposes.
I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-07-17 8:24 GMT+03:00 Kerry Raymond kerry.raymond@gmail.com:
Just saying here what I already put on the Talk page:
I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments."
WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g.
"This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]."
If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels.
Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence.
Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J
Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here …
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario ( dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) < nemowiki@gmail.com> wrote:
phoebe ayers, 16/07/2014 19:21:
(Personally, I think the answer should be to resuscitate RCOM, but that's easy to say and harder to do!)
IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: < https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&...
It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round.
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation User:Jmorgan (WMF) https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF) jmorgan@wikimedia.org
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Good points. Which is why I suggest refocusing RCOM from trying to address the rants of few malcontents about "too many surveys" (read: more than 0) to doing something more useful for the research community (and Wikipedia one). Reorganize research pages. Advertise the existence of the reorganized site. Develop tools to make research into Wikipedia easier, and/or pressure WMF to develop those tools (and once we have such tools they can be used as a carrot to tempt people into registering their research programs on meta or such).
--
Piotr Konieczny, PhD http://hanyang.academia.edu/PiotrKonieczny http://scholar.google.com/citations?user=gdV8_AEAAAAJ http://en.wikipedia.org/wiki/User:Piotrus
On 7/18/2014 06:37, Jonathan Morgan wrote:
First, I wanted to highlight the important issue that Heather raises here, because although it's a separate issue, it's an important one:
On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford <hfordsa@gmail.com mailto:hfordsa@gmail.com> wrote:
... One immediate requirement that I've been talking to others about is finding ways of making the case to the WMF as a group of researchers for the anonymization of country level data, for example. I've spoken to a few researchers (and I myself made a request about a year ago that hasn't been responded to) and it seems like some work is required by the foundation to do this anonymisation but that there are a few of us who would be really keen to use this data to produce research very valuable to Wikipedia - especially from smaller language versions/developing countries. Having an official process that assesses how worthwhile this investment of time would be to the Foundation would be a great idea, I think, but right now there seems to be a general focus on the research that the Foundation does itself rather than enabling researchers outside. I know how busy Aaron and Dario (and others in the team) are so perhaps this requires a new position to coordinate between researchers and Foundation resources? Anyway, I think the big question right now is whether there are any plans for RCOM that have been made by the research team and the only people who can answer that are folks in the research team :) Best, Heather.As a community-run group, RCOM doesn't have any role in making non-public data available to researchers. However, Aaron and I are putting together a proposal for a workshop https://meta.wikimedia.org/wiki/Research:CSCW15_workshop that would address issues like this. That's work we're doing in an official capacity, as opposed to the RCOM work, which is volunteer.
On RCOM more generally... I think clarifying the role of the committee, and getting a larger and more diverse set of people involved, might help make RCOM work. But as Aaron can attest, it is difficult to get people to agree on what RCOMs role should be, let alone get them to work for RCOM.
I've been involved with RCOM for a while, albeit not very actively. Unfortunately, I think that the fact that the only people who "review" requests /happen to be*/ WMF staffers contributes to confusion about RCOM's role and it's authority. IMO, if RCOM or any other subject recruitment review process is to succeed, we need:
- more wiki-researchers who are willing to regularly participate in both peer review /and/ in developing better process guidelines and standards (it's really just Aaron right now)
- more /Wikipedians/ who are willing to do the same
- some degree of buy-in from the Wikimedia community as a whole. RCOM needs legitimacy. But where, and from whom? Subject recruitment is a global concern, but the proposed subject recruitment process is focused on en-wiki (mostly because that's where most of the relevant research activities /that we are aware of/ are happening). How to make RCOM more global?
RCOM is in a tough spot right now. We can't force researchers to submit their proposals, or abide by the suggestions/recommendations/decisions/whatever that result from their review. But because we /look like /an official body, it's easy to blame us for failing to prevent disruptive research (if you're a community member), for "rubber stamping" research that we like (ditto), or for drowning research in red tape (if you're a wiki-researcher).
- J
*we were wiki-researchers first!
Heather Ford Oxford Internet Institute <http://www.oii.ox.ac.uk> Doctoral Programme EthnographyMatters <http://ethnographymatters.net> | Oxford Digital Ethnography Group <http://www.oii.ox.ac.uk/research/projects/?id=115> http://hblog.org <http://hblog.org/> | @hfordsa <http://www.twitter.com/hfordsa> On 17 July 2014 08:49, Kerry Raymond <kerry.raymond@gmail.com <mailto:kerry.raymond@gmail.com>> wrote: Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use. Kerry ------------------------------------------------------------------------ *From:*Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il <mailto:amir.aharoni@mail.huji.ac.il>] *Sent:* Thursday, 17 July 2014 5:37 PM *To:* kerry.raymond@gmail.com <mailto:kerry.raymond@gmail.com>; Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys >WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed. -- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore 2014-07-17 8:24 GMT+03:00 Kerry Raymond <kerry.raymond@gmail.com <mailto:kerry.raymond@gmail.com>>: Just saying here what I already put on the Talk page: I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments." WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g. "This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]." If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels. Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence. Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here … Kerry ------------------------------------------------------------------------ *From:*wiki-research-l-bounces@lists.wikimedia.org <mailto:wiki-research-l-bounces@lists.wikimedia.org> [mailto:wiki-research-l-bounces@lists.wikimedia.org <mailto:wiki-research-l-bounces@lists.wikimedia.org>] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario (dtaraborelli@wikimedia.org <mailto:dtaraborelli@wikimedia.org>) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) <nemowiki@gmail.com <mailto:nemowiki@gmail.com>> wrote: phoebe ayers, 16/07/2014 19:21: > (Personally, I think the answer should be to resuscitate RCOM, but > that's easy to say and harder to do!) IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: <https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&hidebots=1&hideredirs=1&limit=500&offset=&namespace=202> It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round. Nemo _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation User:Jmorgan (WMF) https://meta.wikimedia.org/wiki/User:Jmorgan_%28WMF%29 jmorgan@wikimedia.org mailto:jmorgan@wikimedia.org
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
The good and bad news is that the status quo with RCOM is likely to remain unless someone in WMF, the Board, or the community is interested enough in addressing the situation to put in some effort to make RCOM a functioning organization.
At the moment I have the impression that WMF researchers are absorbing most of the work that RCOM and some dedicated RCOM admin support could do, like help with lit review and prevent outside researchers from using WMF databases in ways that compromise user privacy. My perception is that the current situation is inefficient for WMF and for outside researchers who want to do good work with WMF or community resources, and also that RCOM lacks the resources to respond in timely ways to requests for help with outside research that could benefit Wikimedia. So, I there are reasons to changs the status quo, and I hope WMF or the Board would be interested in something like the proposal I made previously.
Phoebe, what do you think?
Pine
+1 on Piotr's comments.
And very, very happy to hear about https://en.wikipedia.org/wiki/Wikipedia:Ethically_researching_Wikipedia -- I think this is definitely the way to go: developing guidelines that we *regularly point people to* when they have questions etc. And maybe something that we as a group can work on in the coming months.
I'll reiterate my suggestions for goals here and add some of Piotr's and others' comments:
1. developing ethical research guidelines for Wikipedia research - by building on the WP:Ethically_researching_Wikipedia page and regularly pointing people to it
2. finding ways of making responsible requests to the WMF for data that they hold that might benefit research outside the WMF - through an official process with guidelines from the WMF on response times/ viable requests etc.
3. developing opportunities for researchers to collaborate and share what they're doing with the wider research community - reorganising the research hub and pointing to best case practices etc (similar to the WP Global Education program, as Piotr suggests) - actively recruiting WP researchers to join this list and visit the research hub - some other regular way of involving researchers such as inviting them to showcase their work and have it recognised on the list, on the hub etc - recognising outstanding research (through a prize perhaps as Aaron suggested)
Looking forward to hearing Phoebe's suggestions!
Best, Heather.
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 29 July 2014 09:04, Pine W wiki.pine@gmail.com wrote:
The good and bad news is that the status quo with RCOM is likely to remain unless someone in WMF, the Board, or the community is interested enough in addressing the situation to put in some effort to make RCOM a functioning organization.
At the moment I have the impression that WMF researchers are absorbing most of the work that RCOM and some dedicated RCOM admin support could do, like help with lit review and prevent outside researchers from using WMF databases in ways that compromise user privacy. My perception is that the current situation is inefficient for WMF and for outside researchers who want to do good work with WMF or community resources, and also that RCOM lacks the resources to respond in timely ways to requests for help with outside research that could benefit Wikimedia. So, I there are reasons to changs the status quo, and I hope WMF or the Board would be interested in something like the proposal I made previously.
Phoebe, what do you think?
Pine
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
I don't think that it is appropriate that those who benefit from deregulation (e.g. No oversight for running surveys. No formalized community review process.) make the decisions about what is worth regulating. You'll notice that the proposed policy that Poitr calls "instruction creep" basically states that you do three things:
1. Document your research. Specifically, your methods of recruitment, consent process, data storage and publication strategy. 2. Discuss your research -- with Wikipedians to make sure that you won't cause a disruption 3. Proceed as consensus emerges.
We all seem to agree that this is good practice. Where is the rest of the "instruction creep"? Where is the anti-researcher bend?
Poitr, you speculate about potential problems like people just coming to say "IDONTLIKEIT", but I have yet to see that happen in RCOM's process despite the fact that we invite editors from the population being sampled to the conversation. Even if it was true, I think that if some of your potential participants don't like what you are doing, you ought to address their concerns.
I'm all for developing guidelines (note that Ethically researching Wikipedia IS NOT a guideline). I've wrote my fair share of essays to help researchers & Wikipedians find their way around research projects in Wikipedia. E.g. http://en.wikipedia.org/wiki/Wikipedia:Research and and https://en.wikipedia.org/wiki/User:EpochFail/Don%27t_bite_the_researchers. However, I've watched good research projects fail because researchers didn't have the wikipedian backgrounds that you guys do (Heather and Poitr). See some examples of (IRB approved) studies running into project-halting difficulties: https://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Research#Examples_o... These examples are what got me to start working on developing a process in the first place.
If you really think that documenting your research and having a discussion about it is too much instruction, then maybe you shouldn't be allowed to contact Wikipedians. If you do think that every research project that does recruitment should be documented and discussed, why not just say so?
-Aaron
On Tue, Jul 29, 2014 at 3:50 AM, Heather Ford hfordsa@gmail.com wrote:
+1 on Piotr's comments.
And very, very happy to hear about https://en.wikipedia.org/wiki/Wikipedia:Ethically_researching_Wikipedia -- I think this is definitely the way to go: developing guidelines that we *regularly point people to* when they have questions etc. And maybe something that we as a group can work on in the coming months.
I'll reiterate my suggestions for goals here and add some of Piotr's and others' comments:
- developing ethical research guidelines for Wikipedia research
- by building on the WP:Ethically_researching_Wikipedia page and
regularly pointing people to it
- finding ways of making responsible requests to the WMF for data that
they hold that might benefit research outside the WMF
- through an official process with guidelines from the WMF on response
times/ viable requests etc.
- developing opportunities for researchers to collaborate and share what
they're doing with the wider research community
- reorganising the research hub and pointing to best case practices etc
(similar to the WP Global Education program, as Piotr suggests)
- actively recruiting WP researchers to join this list and visit the
research hub
- some other regular way of involving researchers such as inviting them to
showcase their work and have it recognised on the list, on the hub etc
- recognising outstanding research (through a prize perhaps as Aaron
suggested)
Looking forward to hearing Phoebe's suggestions!
Best, Heather.
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 29 July 2014 09:04, Pine W wiki.pine@gmail.com wrote:
The good and bad news is that the status quo with RCOM is likely to remain unless someone in WMF, the Board, or the community is interested enough in addressing the situation to put in some effort to make RCOM a functioning organization.
At the moment I have the impression that WMF researchers are absorbing most of the work that RCOM and some dedicated RCOM admin support could do, like help with lit review and prevent outside researchers from using WMF databases in ways that compromise user privacy. My perception is that the current situation is inefficient for WMF and for outside researchers who want to do good work with WMF or community resources, and also that RCOM lacks the resources to respond in timely ways to requests for help with outside research that could benefit Wikimedia. So, I there are reasons to changs the status quo, and I hope WMF or the Board would be interested in something like the proposal I made previously.
Phoebe, what do you think?
Pine
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Re. RCOM and review processes, these are two different things. RCOM is an old, defunct WMF sanctioned working group of staff, researchers and Wikipedians. If we want to revive RCOM, it seems like this should be discussed in another thread.
On Tue, Jul 29, 2014 at 8:26 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
I don't think that it is appropriate that those who benefit from deregulation (e.g. No oversight for running surveys. No formalized community review process.) make the decisions about what is worth regulating. You'll notice that the proposed policy that Poitr calls "instruction creep" basically states that you do three things:
- Document your research. Specifically, your methods of recruitment,
consent process, data storage and publication strategy. 2. Discuss your research -- with Wikipedians to make sure that you won't cause a disruption 3. Proceed as consensus emerges.
We all seem to agree that this is good practice. Where is the rest of the "instruction creep"? Where is the anti-researcher bend?
Poitr, you speculate about potential problems like people just coming to say "IDONTLIKEIT", but I have yet to see that happen in RCOM's process despite the fact that we invite editors from the population being sampled to the conversation. Even if it was true, I think that if some of your potential participants don't like what you are doing, you ought to address their concerns.
I'm all for developing guidelines (note that Ethically researching Wikipedia IS NOT a guideline). I've wrote my fair share of essays to help researchers & Wikipedians find their way around research projects in Wikipedia. E.g. http://en.wikipedia.org/wiki/Wikipedia:Research and and https://en.wikipedia.org/wiki/User:EpochFail/Don%27t_bite_the_researchers. However, I've watched good research projects fail because researchers didn't have the wikipedian backgrounds that you guys do (Heather and Poitr). See some examples of (IRB approved) studies running into project-halting difficulties: https://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Research#Examples_o... These examples are what got me to start working on developing a process in the first place.
If you really think that documenting your research and having a discussion about it is too much instruction, then maybe you shouldn't be allowed to contact Wikipedians. If you do think that every research project that does recruitment should be documented and discussed, why not just say so?
-Aaron
On Tue, Jul 29, 2014 at 3:50 AM, Heather Ford hfordsa@gmail.com wrote:
+1 on Piotr's comments.
And very, very happy to hear about https://en.wikipedia.org/wiki/Wikipedia:Ethically_researching_Wikipedia -- I think this is definitely the way to go: developing guidelines that we *regularly point people to* when they have questions etc. And maybe something that we as a group can work on in the coming months.
I'll reiterate my suggestions for goals here and add some of Piotr's and others' comments:
- developing ethical research guidelines for Wikipedia research
- by building on the WP:Ethically_researching_Wikipedia page and
regularly pointing people to it
- finding ways of making responsible requests to the WMF for data that
they hold that might benefit research outside the WMF
- through an official process with guidelines from the WMF on response
times/ viable requests etc.
- developing opportunities for researchers to collaborate and share what
they're doing with the wider research community
- reorganising the research hub and pointing to best case practices etc
(similar to the WP Global Education program, as Piotr suggests)
- actively recruiting WP researchers to join this list and visit the
research hub
- some other regular way of involving researchers such as inviting them
to showcase their work and have it recognised on the list, on the hub etc
- recognising outstanding research (through a prize perhaps as Aaron
suggested)
Looking forward to hearing Phoebe's suggestions!
Best, Heather.
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 29 July 2014 09:04, Pine W wiki.pine@gmail.com wrote:
The good and bad news is that the status quo with RCOM is likely to remain unless someone in WMF, the Board, or the community is interested enough in addressing the situation to put in some effort to make RCOM a functioning organization.
At the moment I have the impression that WMF researchers are absorbing most of the work that RCOM and some dedicated RCOM admin support could do, like help with lit review and prevent outside researchers from using WMF databases in ways that compromise user privacy. My perception is that the current situation is inefficient for WMF and for outside researchers who want to do good work with WMF or community resources, and also that RCOM lacks the resources to respond in timely ways to requests for help with outside research that could benefit Wikimedia. So, I there are reasons to changs the status quo, and I hope WMF or the Board would be interested in something like the proposal I made previously.
Phoebe, what do you think?
Pine
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Hi Aaron, what's the source of authority for RCOM (or its members acting independently) to perform a review procedure and claim it is required?
On Tue, Jul 29, 2014 at 9:27 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
Re. RCOM and review processes, these are two different things. RCOM is an old, defunct WMF sanctioned working group of staff, researchers and Wikipedians. If we want to revive RCOM, it seems like this should be discussed in another thread.
I don't believe there is any claim of authority for RCOM. At least I was not involved in making claims that it is required and I do not see it as such. In fact, I have argued in the past that studies run by Wikipedians won't gain much from the process[1]. However, I do recommend that academics -- especially those who do not otherwise engage with Wikipedians -- to work with an RCOM member to coordinate a review in order to ensure that you won't see massive push-back when you start recruiting on Wikipedia -- as studies tended to see when they were run before the process.
1. https://meta.wikimedia.org/wiki/Grants_talk:IEG/Reimagining_Wikipedia_Mentor...
On Tue, Jul 29, 2014 at 8:35 AM, Nathan nawrich@gmail.com wrote:
Hi Aaron, what's the source of authority for RCOM (or its members acting independently) to perform a review procedure and claim it is required?
On Tue, Jul 29, 2014 at 9:27 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
Re. RCOM and review processes, these are two different things. RCOM is an old, defunct WMF sanctioned working group of staff, researchers and Wikipedians. If we want to revive RCOM, it seems like this should be discussed in another thread.
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Thanks. Can you explain why you continue to solicit submissions for your review, and promise a 1-2 week turn around time, when it appears that the review process rarely occurs and many (if not most) submissions are not reviewed?
On Tue, Jul 29, 2014 at 9:42 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
I don't believe there is any claim of authority for RCOM. At least I was not involved in making claims that it is required and I do not see it as such. In fact, I have argued in the past that studies run by Wikipedians won't gain much from the process[1]. However, I do recommend that academics -- especially those who do not otherwise engage with Wikipedians -- to work with an RCOM member to coordinate a review in order to ensure that you won't see massive push-back when you start recruiting on Wikipedia -- as studies tended to see when they were run before the process.
https://meta.wikimedia.org/wiki/Grants_talk:IEG/Reimagining_Wikipedia_Mentor...
On Tue, Jul 29, 2014 at 8:35 AM, Nathan nawrich@gmail.com wrote:
Hi Aaron, what's the source of authority for RCOM (or its members acting independently) to perform a review procedure and claim it is required?
On Tue, Jul 29, 2014 at 9:27 AM, Aaron Halfaker <aaron.halfaker@gmail.com
wrote:
Re. RCOM and review processes, these are two different things. RCOM is an old, defunct WMF sanctioned working group of staff, researchers and Wikipedians. If we want to revive RCOM, it seems like this should be discussed in another thread.
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
The review process occurs in all instances where review coordination is requested (by emailing me or DarTar). There's only been one case where a review took more than 2 weeks and that was because the researcher didn't respond to requests for more information quickly.
Nathan, I think you are mistakenly thinking that all research needs to be reviewed. Only research that involves the recruitment of Wikipedians as subjects is intended to be reviewed via RCOM's process. Only those studies that request it will be reviewed.
-Aaron
On Tue, Jul 29, 2014 at 8:59 AM, Nathan nawrich@gmail.com wrote:
Thanks. Can you explain why you continue to solicit submissions for your review, and promise a 1-2 week turn around time, when it appears that the review process rarely occurs and many (if not most) submissions are not reviewed?
On Tue, Jul 29, 2014 at 9:42 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
I don't believe there is any claim of authority for RCOM. At least I was not involved in making claims that it is required and I do not see it as such. In fact, I have argued in the past that studies run by Wikipedians won't gain much from the process[1]. However, I do recommend that academics -- especially those who do not otherwise engage with Wikipedians -- to work with an RCOM member to coordinate a review in order to ensure that you won't see massive push-back when you start recruiting on Wikipedia -- as studies tended to see when they were run before the process.
https://meta.wikimedia.org/wiki/Grants_talk:IEG/Reimagining_Wikipedia_Mentor...
On Tue, Jul 29, 2014 at 8:35 AM, Nathan nawrich@gmail.com wrote:
Hi Aaron, what's the source of authority for RCOM (or its members acting independently) to perform a review procedure and claim it is required?
On Tue, Jul 29, 2014 at 9:27 AM, Aaron Halfaker < aaron.halfaker@gmail.com> wrote:
Re. RCOM and review processes, these are two different things. RCOM is an old, defunct WMF sanctioned working group of staff, researchers and Wikipedians. If we want to revive RCOM, it seems like this should be discussed in another thread.
>
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
On Tue, Jul 29, 2014 at 10:11 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
The review process occurs in all instances where review coordination is requested (by emailing me or DarTar). There's only been one case where a review took more than 2 weeks and that was because the researcher didn't respond to requests for more information quickly.
Nathan, I think you are mistakenly thinking that all research needs to be reviewed. Only research that involves the recruitment of Wikipedians as subjects is intended to be reviewed via RCOM's process. Only those studies that request it will be reviewed.
-Aaron
Thanks, perhaps the confusion exists because there is so much apparent infrastructure around the review process (including a big button that creates a research project page, ostensibly to facilitate a review). It might also be that communication from the former RCOM's members is misleading; in one e-mail in this thread you say RCOM is defunct, and in another you suggest that research recruiting Wikipedians needs RCOM's review.
Either there is an RCOM and it functions effectively, or nothing should or must rely on a defunct committee to complete a defunct process. If the committee is indeed defunct, then messaging around the review process should be adjusted to make it clear that it is voluntary, and there are only two reviewers acting on their own initiative. Your insistence on having it both ways is leading to confusion, not just from me but on the part of people proposing research projects and expecting comment from "RCOM."
RCOM is not functioning as a complete group anymore. However, we split into sub-committees while we were still a functioning group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still.
I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval. There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves.
I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation.
-Aaron
On Tue, Jul 29, 2014 at 9:21 AM, Nathan nawrich@gmail.com wrote:
On Tue, Jul 29, 2014 at 10:11 AM, Aaron Halfaker <aaron.halfaker@gmail.com
wrote:
The review process occurs in all instances where review coordination is requested (by emailing me or DarTar). There's only been one case where a review took more than 2 weeks and that was because the researcher didn't respond to requests for more information quickly.
Nathan, I think you are mistakenly thinking that all research needs to be reviewed. Only research that involves the recruitment of Wikipedians as subjects is intended to be reviewed via RCOM's process. Only those studies that request it will be reviewed.
-Aaron
Thanks, perhaps the confusion exists because there is so much apparent infrastructure around the review process (including a big button that creates a research project page, ostensibly to facilitate a review). It might also be that communication from the former RCOM's members is misleading; in one e-mail in this thread you say RCOM is defunct, and in another you suggest that research recruiting Wikipedians needs RCOM's review.
Either there is an RCOM and it functions effectively, or nothing should or must rely on a defunct committee to complete a defunct process. If the committee is indeed defunct, then messaging around the review process should be adjusted to make it clear that it is voluntary, and there are only two reviewers acting on their own initiative. Your insistence on having it both ways is leading to confusion, not just from me but on the part of people proposing research projects and expecting comment from "RCOM."
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
RCOM is not functioning as a complete group anymore. However, we split into sub-committees while we were still a functioning group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still.
I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval. There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves.
I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation.
-Aaron
Thanks for that information. As time permits I'll make some clarifications in the documentation.
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <aaron.halfaker@gmail.com
wrote:
RCOM is not functioning as a complete group anymore.
I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
However, we split into sub-committees while we were still a functioning
group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still.
I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.
So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
There are also more than two "review coordinators" (not not "reviewers")
-- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves.
I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation.
I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith.
This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)
Best, Heather.
[1] https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
-Aaron
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Either [RCOM is] functioning or its not, surely?
Well, I explained that there are functioning sub-committees still. In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group. I don't know how else to explain it. I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings.
As Kerry noted earlier on, the policy as it stands [1] says that
researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
That's a proposed policy. Until it is passed by consensus, the "must" is a proposed term. I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".
Regarding researchers stating what should be regulated, I think there is a big difference between *deciding what should be regulated* and *being involved in the discussion of *how* it should be regulated*. Hence why I welcome participation. What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently. Does that make sense?
b) Pine suggested a board decision on this earlier one to obtain clarity
and I supported this but it was met with silence, which is why I followed up.
I welcome you to raise it to them. I don't think it is worth their time, but they might disagree.
But what is clear is that clarification is required - especially on the
distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I think that the proposed policy on English Wikipedia does that quite well. That's why I directed people there. Also, again, I am not working on RCOM or subject recruitment as a WMF employee. I do this in my volunteer time. This is true of all of RCOM who happen to also be staff.
if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail. When you guys work out how clearly you want a process to be described, please let me know. I'm tired of re-spec'ing processes. This is the third iteration.
If the policy is incorrectly described on the policy pages, then someone
from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.
Heather, that is a *proposed *policy page on English Wikipedia. It is not part of RCOM. It would render RCOM irrelevant for subject recruitment concerns. That's why I started it. I don't think that RCOM/WMF/researchers should own subject recruitment review. I think the community being studied should own it and that RCOM/WMF/researchers should participate.
Also, I am not your employee. This is my volunteer time. I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to. If you want something done and other volunteers don't have time to do it. Do it yourself. https://en.wikipedia.org/wiki/Wikipedia:SOFIXIT
-Aaron
On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford hfordsa@gmail.com wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <aaron.halfaker@gmail.com
wrote:
RCOM is not functioning as a complete group anymore.
I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
However, we split into sub-committees while we were still a functioning
group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still.
I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.
So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
There are also more than two "review coordinators" (not not "reviewers")
-- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves.
I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation.
I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith.
This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)
Best, Heather.
[1] https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
-Aaron
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Hi all,
I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion.
Is RCom an official WMF body or a group of volunteers?
RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.
Is RCom a body regulating subject recruitment?
No, subject recruitment was only one among many areas of interest identified by its participants [2]
Is RCom still alive?
RCom stopped working a while ago as a group meeting on a regular basis to discuss joint initiatives. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below.
What’s the source of RCom’s authority in reviewing subject recruitment requests?
Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and enforceable policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.
What about requests for private data?
Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.
How can we incentivize researchers to “give back” to the community?
In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.
Dario
[1] https://meta.wikimedia.org/w/index.php?title=Research:Committee&oldid=20... [2] https://meta.wikimedia.org/wiki/Research:Committee/Areas_of_interest [3] https://meta.wikimedia.org/wiki/Research:WMF_support [4] https://meta.wikimedia.org/wiki/Privacy_policy [5] https://meta.wikimedia.org/wiki/Data_retention_guidelines [6] http://meta.wikimedia.org/wiki/Research:Schemas
On Jul 29, 2014, at 6:49 PM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
Either [RCOM is] functioning or its not, surely?
Well, I explained that there are functioning sub-committees still. In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group. I don't know how else to explain it. I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings.
As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
That's a proposed policy. Until it is passed by consensus, the "must" is a proposed term. I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".
Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated. Hence why I welcome participation. What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently. Does that make sense?
b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
I welcome you to raise it to them. I don't think it is worth their time, but they might disagree.
But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I think that the proposed policy on English Wikipedia does that quite well. That's why I directed people there. Also, again, I am not working on RCOM or subject recruitment as a WMF employee. I do this in my volunteer time. This is true of all of RCOM who happen to also be staff.
if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail. When you guys work out how clearly you want a process to be described, please let me know. I'm tired of re-spec'ing processes. This is the third iteration.
If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.
Heather, that is a proposed policy page on English Wikipedia. It is not part of RCOM. It would render RCOM irrelevant for subject recruitment concerns. That's why I started it. I don't think that RCOM/WMF/researchers should own subject recruitment review. I think the community being studied should own it and that RCOM/WMF/researchers should participate.
Also, I am not your employee. This is my volunteer time. I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to. If you want something done and other volunteers don't have time to do it. Do it yourself.
-Aaron
On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford hfordsa@gmail.com wrote: On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote: RCOM is not functioning as a complete group anymore.
I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
However, we split into sub-committees while we were still a functioning group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still.
I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.
So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves.
I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation.
I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith.
This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)
Best, Heather.
[1] https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
-Aaron _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Hey guys,
I posted some thoughts to my own blog and am linking to those posts below. Everything I say on my blog is captured in the summary below, so feel free to not click through.
---- My biggest worry is that researchers who recruit human subjects assume that there are huge numbers of Wikipedians for them to survey, and consequently, they do not need to do a lot of advance survey preparation because there is no harm from distracting Wikipedians from their usual volunteer work. This assumption is wrong because actually almost every researcher recruiting human subjects wants Wikipedians who are in very short supply. Consequently, researchers do cause harm to the community by soliciting for volunteer time, and Wikipedia community benefit is dubious when researchers do not do sufficient preparation for their work. This is not quite accurate, but if there were one message I could convey to researchers, it would be "Your research participant pool only consists of about 30 super busy people and many other volunteers greatly depend on getting their time. When you take time from a Wikipedian, you are taking that time away from other volunteers who really need it, so be respectful of your intervention in our communities." ----
I do not want a lot of gatekeeping between researchers and the Wikipedia community, but at the same time, researchers should take professional pride in their work and take care not to disrupt Wikimedia community activities.
< http://bluerasberry.com/2014/07/request-for-researchers-when-doing-research-...
http://bluerasberry.com/2014/07/problems-with-research-on-wikipedia/
I am still thinking about what should be done with research.
yours,
On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
Hi all,
I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion.
*Is RCom an official WMF body or a group of volunteers?*
RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.
*Is RCom a body regulating subject recruitment?*
No, subject recruitment was only one among many areas of interest identified by its participants [2]
*Is RCom still alive?*
RCom stopped working a while ago* as a* *group meeting on a regular basis to discuss joint initiatives*. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below.
*What’s the source of RCom’s authority in reviewing subject recruitment requests?*
Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and *enforceable* policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.
*What about requests for **private data**?*
Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.
*How can we incentivize researchers to “give back” to the community?*
In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.
Dario
[1] https://meta.wikimedia.org/w/index.php?title=Research:Committee&oldid=20... [2] https://meta.wikimedia.org/wiki/Research:Committee/Areas_of_interest [3] https://meta.wikimedia.org/wiki/Research:WMF_support [4] https://meta.wikimedia.org/wiki/Privacy_policy [5] https://meta.wikimedia.org/wiki/Data_retention_guidelines [6] http://meta.wikimedia.org/wiki/Research:Schemas
On Jul 29, 2014, at 6:49 PM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
Either [RCOM is] functioning or its not, surely?
Well, I explained that there are functioning sub-committees still. In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group. I don't know how else to explain it. I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings.
As Kerry noted earlier on, the policy as it stands [1] says that
researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
That's a proposed policy. Until it is passed by consensus, the "must" is a proposed term. I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".
Regarding researchers stating what should be regulated, I think there is a big difference between *deciding what should be regulated* and *being involved in the discussion of *how* it should be regulated*. Hence why I welcome participation. What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently. Does that make sense?
b) Pine suggested a board decision on this earlier one to obtain clarity
and I supported this but it was met with silence, which is why I followed up.
I welcome you to raise it to them. I don't think it is worth their time, but they might disagree.
But what is clear is that clarification is required - especially on the
distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I think that the proposed policy on English Wikipedia does that quite well. That's why I directed people there. Also, again, I am not working on RCOM or subject recruitment as a WMF employee. I do this in my volunteer time. This is true of all of RCOM who happen to also be staff.
if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail. When you guys work out how clearly you want a process to be described, please let me know. I'm tired of re-spec'ing processes. This is the third iteration.
If the policy is incorrectly described on the policy pages, then someone
from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.
Heather, that is a *proposed *policy page on English Wikipedia. It is not part of RCOM. It would render RCOM irrelevant for subject recruitment concerns. That's why I started it. I don't think that RCOM/WMF/researchers should own subject recruitment review. I think the community being studied should own it and that RCOM/WMF/researchers should participate.
Also, I am not your employee. This is my volunteer time. I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to. If you want something done and other volunteers don't have time to do it. Do it yourself. https://en.wikipedia.org/wiki/Wikipedia:SOFIXIT
-Aaron
On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford hfordsa@gmail.com wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <
aaron.halfaker@gmail.com> wrote:
RCOM is not functioning as a complete group anymore.
I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
However, we split into sub-committees while we were still a functioning
group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still.
I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.
So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
There are also more than two "review coordinators" (not not "reviewers")
-- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves.
I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation.
I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith.
This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)
Best, Heather.
[1] https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
-Aaron
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Lane, how many survey requests do you get per year? And how much time do you spend on them?
Because myself, being in the Top 100 most active editors and thus I'd think fitting in your group of " about 30 super busy people", I get about ~2 requests per year and they cost me few minutes at most, which even being "super busy" I find I can afford.
What I'd focus with a call for the researchers (perhaps another idea for best practices) would be to ask people to do proper lit review. I don't think we have too many surveys, but I do think we have a not-too-small percentage of them pointlessly replicating prior research (as in - we probably don't need a n-th paper on Wikipedian's motivations that badly...). Of course, people who can't be bothered to to a proper lit review can't probably be bothered to find out about our best practices guides, even if we clean the mess that our research pages are currently, so... :/
--
Piotr Konieczny, PhD http://hanyang.academia.edu/PiotrKonieczny http://scholar.google.com/citations?user=gdV8_AEAAAAJ http://en.wikipedia.org/wiki/User:Piotrus
On 7/30/2014 10:59, Lane Rasberry wrote:
Hey guys,
I posted some thoughts to my own blog and am linking to those posts below. Everything I say on my blog is captured in the summary below, so feel free to not click through.
My biggest worry is that researchers who recruit human subjects assume that there are huge numbers of Wikipedians for them to survey, and consequently, they do not need to do a lot of advance survey preparation because there is no harm from distracting Wikipedians from their usual volunteer work. This assumption is wrong because actually almost every researcher recruiting human subjects wants Wikipedians who are in very short supply. Consequently, researchers do cause harm to the community by soliciting for volunteer time, and Wikipedia community benefit is dubious when researchers do not do sufficient preparation for their work. This is not quite accurate, but if there were one message I could convey to researchers, it would be "Your research participant pool only consists of about 30 super busy people and many other volunteers greatly depend on getting their time. When you take time from a Wikipedian, you are taking that time away from other volunteers who really need it, so be respectful of your intervention in our communities."
I do not want a lot of gatekeeping between researchers and the Wikipedia community, but at the same time, researchers should take professional pride in their work and take care not to disrupt Wikimedia community activities.
http://bluerasberry.com/2014/07/request-for-researchers-when-doing-research-on-the-wikimedia-community/ http://bluerasberry.com/2014/07/problems-with-research-on-wikipedia/
I am still thinking about what should be done with research.
yours,
On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli <dtaraborelli@wikimedia.org mailto:dtaraborelli@wikimedia.org> wrote:
Hi all, I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion. *Is RCom an official WMF body or a group of volunteers?* RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation. *Is RCom a body regulating subject recruitment?* No, subject recruitment was only one among many areas of interest identified by its participants [2] *Is RCom still alive?* RCom stopped working a while ago/as a/ /group meeting on a regular basis to discuss joint initiatives/. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below. *What’s the source of RCom’s authority in reviewing subject recruitment requests?* Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and /enforceable/ policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance. *What about requests for **private data**?* Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy. *How can we incentivize researchers to “give back” to the community?* In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects. Dario [1] https://meta.wikimedia.org/w/index.php?title=Research:Committee&oldid=2094818 [2] https://meta.wikimedia.org/wiki/Research:Committee/Areas_of_interest [3] https://meta.wikimedia.org/wiki/Research:WMF_support [4] https://meta.wikimedia.org/wiki/Privacy_policy [5] https://meta.wikimedia.org/wiki/Data_retention_guidelines [6] http://meta.wikimedia.org/wiki/Research:Schemas On Jul 29, 2014, at 6:49 PM, Aaron Halfaker <aaron.halfaker@gmail.com <mailto:aaron.halfaker@gmail.com>> wrote:Either [RCOM is] functioning or its not, surely? Well, I explained that there are functioning sub-committees still. In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group. I don't know how else to explain it. I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? That's a proposed policy. Until it is passed by consensus, the "must" is a proposed term. I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to". Regarding researchers stating what should be regulated, I think there is a big difference between *deciding what should be regulated* and *being involved in the discussion of *how* it should be regulated*. Hence why I welcome participation. What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently. Does that make sense? b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up. I welcome you to raise it to them. I don't think it is worth their time, but they might disagree. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list. I think that the proposed policy on English Wikipedia does that quite well. That's why I directed people there. Also, again, I am not working on RCOM or subject recruitment as a WMF employee. I do this in my volunteer time. This is true of all of RCOM who happen to also be staff. if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail. When you guys work out how clearly you want a process to be described, please let me know. I'm tired of re-spec'ing processes. This is the third iteration. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. Heather, that is a *proposed *policy page on English Wikipedia. It is not part of RCOM. It would render RCOM irrelevant for subject recruitment concerns. That's why I started it. I don't think that RCOM/WMF/researchers should own subject recruitment review. I think the community being studied should own it and that RCOM/WMF/researchers should participate. Also, I am not your employee. This is my volunteer time. I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to. If you want something done and other volunteers don't have time to do it. Do it yourself. <https://en.wikipedia.org/wiki/Wikipedia:SOFIXIT> -Aaron On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <hfordsa@gmail.com <mailto:hfordsa@gmail.com>> wrote: On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <aaron.halfaker@gmail.com <mailto:aaron.halfaker@gmail.com>> wrote: RCOM is not functioning as a complete group anymore. I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely? However, we split into sub-committees while we were still a functioning group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still. I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval. So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up. There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves. I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list. I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation. I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith. This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :) Best, Heather. [1] https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment -Aaron _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l-- Lane Rasberry user:bluerasberry on Wikipedia 206.801.0814 lane@bluerasberry.com mailto:lane@bluerasberry.com
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
@Piotr
Lane, how many survey requests do you get per year? And how much time do
you spend on them?
Perhaps I got 10 survey requests, and perhaps 10 interview requests in the past year which were not obviously related to my outreach work or which I sought to have. I work in Wikipedia outreach and talk to a lot of people in that capacity too. I make myself visible.
I rarely complete surveys, and maybe have only done 5 in the last year with most or all of those being ones that I found rather than that found me.
I am contacted for being a certain kind of Wikipedian, and not for being any Wikipedian. People ask me about medicine, LGBT topics, paid editing or Wikipedian in Residence things. Unusually, I give my phone number and Skype out to a lot of people and do more Wikipedia discussion by voice than on-wiki or with typing. I do not know any other Wikipedian who says this.
Those who don't want, don't take part in the survey. where's that
disruption? What am I missing?
There is a historical precedent in human subject research which says that researchers cannot depend on their subjects to protect themselves, because this is presumed to always lead to unacceptable risk of harm to the subjects. Just saying that people can refuse surveys is out of bounds of contemporary research in western cultures. Some third party oversight from somewhere is necessary. I have no opinion about whether the Wikipedia community or WMF could provide that, but my initial thought is that if researchers follow their own institution's guidelines then things should be okay.
The most usual disruption is that the pool of researchers is large in comparison to the pool of people who could respond to surveys and interviews. Practically all researchers assume that it does not disrupt anything to ask, but there is a lot of asking and it is not obvious that Wikimedia community infrastructure should be used to serve people who are using it in ways that might be disturbing advancement of the Wikimedia mission.
@Kerry
32K active editors (> 5 edits per month) 3K very active editors (>100
edits per month).
*>*Or have I missed something here? Are researchers only interested in people who have been on Wikipedia for 10+ years with 10M edits or …?
No, you have it exactly! It is not the "Wikipedia for 10+ years with 10M edits or" that matters, but rather it is making any further distinction. A researcher who wants any demographic, like women, gay people, an ethnic minority, editors in a certain topic area, people who do a certain function like AfD, people who have had a certain problem, or many other things will cut the pool of 3000 down to 300. I would love for researchers to research those people doing 5-100 edits a month, but there is no way to reach this group and researchers rarely or never are interested in this group. Those 3000 are the ones I want to have protection because they are not a single group, but rather are lots of small groups each serving an important role. A typical survey is relevant to at most 10% of that 3000, meaning the potential research pool is rarely above 300 people. If the interviewer is imagining a research pool of 10,000 or more - which is the usual case - then they would be bold in trying to recruit and take time even though only 300 people could possibly respond. If they get 10% of the possible pool, which is an amazing response, then that could mean 30 responses and not enough to have valid results if the researcher hoped for 1% of 10,000. Also, when only 300 people do a task any time away from that is valuable time lost, and if researchers expected a larger pool, they may not be so careful with the volunteer's time. As researchers almost always come to Wikipedia completely as community outsiders, they almost always undervalue volunteer time here.
On Tue, Jul 29, 2014 at 10:39 PM, Piotr Konieczny piokon@post.pl wrote:
Lane, how many survey requests do you get per year? And how much time do you spend on them?
Because myself, being in the Top 100 most active editors and thus I'd think fitting in your group of " about 30 super busy people", I get about ~2 requests per year and they cost me few minutes at most, which even being "super busy" I find I can afford.
What I'd focus with a call for the researchers (perhaps another idea for best practices) would be to ask people to do proper lit review. I don't think we have too many surveys, but I do think we have a not-too-small percentage of them pointlessly replicating prior research (as in - we probably don't need a n-th paper on Wikipedian's motivations that badly...). Of course, people who can't be bothered to to a proper lit review can't probably be bothered to find out about our best practices guides, even if we clean the mess that our research pages are currently, so... :/
--
Piotr Konieczny, PhDhttp://hanyang.academia.edu/PiotrKoniecznyhttp://scholar.google.com/citation...
On 7/30/2014 10:59, Lane Rasberry wrote:
Hey guys,
I posted some thoughts to my own blog and am linking to those posts below. Everything I say on my blog is captured in the summary below, so feel free to not click through.
My biggest worry is that researchers who recruit human subjects assume that there are huge numbers of Wikipedians for them to survey, and consequently, they do not need to do a lot of advance survey preparation because there is no harm from distracting Wikipedians from their usual volunteer work. This assumption is wrong because actually almost every researcher recruiting human subjects wants Wikipedians who are in very short supply. Consequently, researchers do cause harm to the community by soliciting for volunteer time, and Wikipedia community benefit is dubious when researchers do not do sufficient preparation for their work. This is not quite accurate, but if there were one message I could convey to researchers, it would be "Your research participant pool only consists of about 30 super busy people and many other volunteers greatly depend on getting their time. When you take time from a Wikipedian, you are taking that time away from other volunteers who really need it, so be respectful of your intervention in our communities."
I do not want a lot of gatekeeping between researchers and the Wikipedia community, but at the same time, researchers should take professional pride in their work and take care not to disrupt Wikimedia community activities.
< http://bluerasberry.com/2014/07/request-for-researchers-when-doing-research-...
http://bluerasberry.com/2014/07/problems-with-research-on-wikipedia/
I am still thinking about what should be done with research.
yours,
On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
Hi all,
I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion.
*Is RCom an official WMF body or a group of volunteers?*
RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.
*Is RCom a body regulating subject recruitment?*
No, subject recruitment was only one among many areas of interest identified by its participants [2]
*Is RCom still alive?*
RCom stopped working a while ago* as a* *group meeting on a regular basis to discuss joint initiatives*. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below.
*What’s the source of RCom’s authority in reviewing subject recruitment requests?*
Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and *enforceable* policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.
*What about requests for **private data**?*
Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.
*How can we incentivize researchers to “give back” to the community?*
In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.
Dario
[1] https://meta.wikimedia.org/w/index.php?title=Research:Committee&oldid=20... [2] https://meta.wikimedia.org/wiki/Research:Committee/Areas_of_interest [3] https://meta.wikimedia.org/wiki/Research:WMF_support [4] https://meta.wikimedia.org/wiki/Privacy_policy [5] https://meta.wikimedia.org/wiki/Data_retention_guidelines [6] http://meta.wikimedia.org/wiki/Research:Schemas
On Jul 29, 2014, at 6:49 PM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
Either [RCOM is] functioning or its not, surely?
Well, I explained that there are functioning sub-committees still. In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group. I don't know how else to explain it. I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings.
As Kerry noted earlier on, the policy as it stands [1] says that
researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
That's a proposed policy. Until it is passed by consensus, the "must" is a proposed term. I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".
Regarding researchers stating what should be regulated, I think there is a big difference between *deciding what should be regulated* and *being involved in the discussion of *how* it should be regulated*. Hence why I welcome participation. What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently. Does that make sense?
b) Pine suggested a board decision on this earlier one to obtain
clarity and I supported this but it was met with silence, which is why I followed up.
I welcome you to raise it to them. I don't think it is worth their time, but they might disagree.
But what is clear is that clarification is required - especially on the
distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I think that the proposed policy on English Wikipedia does that quite well. That's why I directed people there. Also, again, I am not working on RCOM or subject recruitment as a WMF employee. I do this in my volunteer time. This is true of all of RCOM who happen to also be staff.
if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail. When you guys work out how clearly you want a process to be described, please let me know. I'm tired of re-spec'ing processes. This is the third iteration.
If the policy is incorrectly described on the policy pages, then
someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.
Heather, that is a *proposed *policy page on English Wikipedia. It is not part of RCOM. It would render RCOM irrelevant for subject recruitment concerns. That's why I started it. I don't think that RCOM/WMF/researchers should own subject recruitment review. I think the community being studied should own it and that RCOM/WMF/researchers should participate.
Also, I am not your employee. This is my volunteer time. I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to. If you want something done and other volunteers don't have time to do it. Do it yourself. https://en.wikipedia.org/wiki/Wikipedia:SOFIXIT
-Aaron
On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford hfordsa@gmail.com wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <aaron.halfaker@gmail.com> wrote:
RCOM is not functioning as a complete group anymore.
I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
However, we split into sub-committees while we were still a
functioning group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still.
I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.
So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
There are also more than two "review coordinators" (not not
"reviewers") -- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves.
I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation.
I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith.
This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)
Best, Heather.
[1] https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
-Aaron
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Lane Rasberry user:bluerasberry on Wikipedia 206.801.0814 lane@bluerasberry.com
Wiki-research-l mailing listWiki-research-l@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
I agree that researchers do not value the time of their subjects. But Wikipedia does not own the time of its volunteers; the volunteers choose to give it to Wikipedia and they may choose to give it to surveys and they may choose to watch Game of Thrones. Wikipedia can't lose what it did not own; it can only be grateful for what is freely given. Ditto for researchers.
As I have previously stated, Wikipedia is within its rights to control the use of its communication channels in relation to promoting research surveys, because it does own those resources.
Kerry
_____
From: wiki-research-l-bounces@lists.wikimedia.org [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Lane Rasberry Sent: Saturday, 2 August 2014 12:41 AM To: Research into Wikimedia content and communities Subject: Re: [Wiki-research-l] discussion about wikipedia surveys
@Piotr
Lane, how many survey requests do you get per year? And how much time do
you spend on them?
Perhaps I got 10 survey requests, and perhaps 10 interview requests in the past year which were not obviously related to my outreach work or which I sought to have. I work in Wikipedia outreach and talk to a lot of people in that capacity too. I make myself visible.
I rarely complete surveys, and maybe have only done 5 in the last year with most or all of those being ones that I found rather than that found me.
I am contacted for being a certain kind of Wikipedian, and not for being any Wikipedian. People ask me about medicine, LGBT topics, paid editing or Wikipedian in Residence things. Unusually, I give my phone number and Skype out to a lot of people and do more Wikipedia discussion by voice than on-wiki or with typing. I do not know any other Wikipedian who says this.
Those who don't want, don't take part in the survey. where's that
disruption? What am I missing?
There is a historical precedent in human subject research which says that researchers cannot depend on their subjects to protect themselves, because this is presumed to always lead to unacceptable risk of harm to the subjects. Just saying that people can refuse surveys is out of bounds of contemporary research in western cultures. Some third party oversight from somewhere is necessary. I have no opinion about whether the Wikipedia community or WMF could provide that, but my initial thought is that if researchers follow their own institution's guidelines then things should be okay.
The most usual disruption is that the pool of researchers is large in comparison to the pool of people who could respond to surveys and interviews. Practically all researchers assume that it does not disrupt anything to ask, but there is a lot of asking and it is not obvious that Wikimedia community infrastructure should be used to serve people who are using it in ways that might be disturbing advancement of the Wikimedia mission.
@Kerry
32K active editors (> 5 edits per month) 3K very active editors (>100
edits per month).
Or have I missed something here? Are researchers only interested in people
who have been on Wikipedia for 10+ years with 10M edits or .?
No, you have it exactly! It is not the "Wikipedia for 10+ years with 10M edits or" that matters, but rather it is making any further distinction. A researcher who wants any demographic, like women, gay people, an ethnic minority, editors in a certain topic area, people who do a certain function like AfD, people who have had a certain problem, or many other things will cut the pool of 3000 down to 300. I would love for researchers to research those people doing 5-100 edits a month, but there is no way to reach this group and researchers rarely or never are interested in this group. Those 3000 are the ones I want to have protection because they are not a single group, but rather are lots of small groups each serving an important role. A typical survey is relevant to at most 10% of that 3000, meaning the potential research pool is rarely above 300 people. If the interviewer is imagining a research pool of 10,000 or more - which is the usual case - then they would be bold in trying to recruit and take time even though only 300 people could possibly respond. If they get 10% of the possible pool, which is an amazing response, then that could mean 30 responses and not enough to have valid results if the researcher hoped for 1% of 10,000. Also, when only 300 people do a task any time away from that is valuable time lost, and if researchers expected a larger pool, they may not be so careful with the volunteer's time. As researchers almost always come to Wikipedia completely as community outsiders, they almost always undervalue volunteer time here.
On Tue, Jul 29, 2014 at 10:39 PM, Piotr Konieczny piokon@post.pl wrote:
Lane, how many survey requests do you get per year? And how much time do you spend on them?
Because myself, being in the Top 100 most active editors and thus I'd think fitting in your group of " about 30 super busy people", I get about ~2 requests per year and they cost me few minutes at most, which even being "super busy" I find I can afford.
What I'd focus with a call for the researchers (perhaps another idea for best practices) would be to ask people to do proper lit review. I don't think we have too many surveys, but I do think we have a not-too-small percentage of them pointlessly replicating prior research (as in - we probably don't need a n-th paper on Wikipedian's motivations that badly...). Of course, people who can't be bothered to to a proper lit review can't probably be bothered to find out about our best practices guides, even if we clean the mess that our research pages are currently, so... :/
--
Piotr Konieczny, PhD http://hanyang.academia.edu/PiotrKonieczny http://scholar.google.com/citations?user=gdV8_AEAAAAJ http://en.wikipedia.org/wiki/User:Piotrus
On 7/30/2014 10:59, Lane Rasberry wrote:
Hey guys,
I posted some thoughts to my own blog and am linking to those posts below. Everything I say on my blog is captured in the summary below, so feel free to not click through.
----
My biggest worry is that researchers who recruit human subjects assume that there are huge numbers of Wikipedians for them to survey, and consequently, they do not need to do a lot of advance survey preparation because there is no harm from distracting Wikipedians from their usual volunteer work. This assumption is wrong because actually almost every researcher recruiting human subjects wants Wikipedians who are in very short supply. Consequently, researchers do cause harm to the community by soliciting for volunteer time, and Wikipedia community benefit is dubious when researchers do not do sufficient preparation for their work. This is not quite accurate, but if there were one message I could convey to researchers, it would be "Your research participant pool only consists of about 30 super busy people and many other volunteers greatly depend on getting their time. When you take time from a Wikipedian, you are taking that time away from other volunteers who really need it, so be respectful of your intervention in our communities." ----
I do not want a lot of gatekeeping between researchers and the Wikipedia community, but at the same time, researchers should take professional pride in their work and take care not to disrupt Wikimedia community activities.
http://bluerasberry.com/2014/07/request-for-researchers-when-doing-research -on-the-wikimedia-community/ http://bluerasberry.com/2014/07/problems-with-research-on-wikipedia/
I am still thinking about what should be done with research.
yours,
On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
Hi all,
I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I'd chime in to clear some confusion.
Is RCom an official WMF body or a group of volunteers?
RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.
Is RCom a body regulating subject recruitment?
No, subject recruitment was only one among many areas of interest identified by its participants [2]
Is RCom still alive?
RCom stopped working a while ago as a group meeting on a regular basis to discuss joint initiatives. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don't think that the alternative between "either RCom exists and it functions effectively or reviews should immediately stop" is well framed or even desirable, for the reasons that I explain below.
What's the source of RCom's authority in reviewing subject recruitment requests?
Despite the perception that one of RCom's duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to "obtain formal approval". The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and enforceable policies. It's a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.
What about requests for private data?
Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation's Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We've already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users' privacy.
How can we incentivize researchers to "give back" to the community?
In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It's been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.
Dario
[1] https://meta.wikimedia.org/w/index.php?title=Research:Committee https://meta.wikimedia.org/w/index.php?title=Research:Committee&oldid=20948 18 &oldid=2094818
[2] https://meta.wikimedia.org/wiki/Research:Committee/Areas_of_interest
[3] https://meta.wikimedia.org/wiki/Research:WMF_support
[4] https://meta.wikimedia.org/wiki/Privacy_policy
[5] https://meta.wikimedia.org/wiki/Data_retention_guidelines
[6] http://meta.wikimedia.org/wiki/Research:Schemas
On Jul 29, 2014, at 6:49 PM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
Either [RCOM is] functioning or its not, surely?
Well, I explained that there are functioning sub-committees still. In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group. I don't know how else to explain it. I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings.
As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
That's a proposed policy. Until it is passed by consensus, the "must" is a proposed term. I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".
Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated. Hence why I welcome participation. What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently. Does that make sense?
b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
I welcome you to raise it to them. I don't think it is worth their time, but they might disagree.
But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I think that the proposed policy on English Wikipedia does that quite well. That's why I directed people there. Also, again, I am not working on RCOM or subject recruitment as a WMF employee. I do this in my volunteer time. This is true of all of RCOM who happen to also be staff.
if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail. When you guys work out how clearly you want a process to be described, please let me know. I'm tired of re-spec'ing processes. This is the third iteration.
If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.
Heather, that is a proposed policy page on English Wikipedia. It is not part of RCOM. It would render RCOM irrelevant for subject recruitment concerns. That's why I started it. I don't think that RCOM/WMF/researchers should own subject recruitment review. I think the community being studied should own it and that RCOM/WMF/researchers should participate.
Also, I am not your employee. This is my volunteer time. I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to. If you want something done and other volunteers don't have time to do it. Do it https://en.wikipedia.org/wiki/Wikipedia:SOFIXIT yourself.
-Aaron
On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford hfordsa@gmail.com wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
RCOM is not functioning as a complete group anymore.
I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
However, we split into sub-committees while we were still a functioning group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still.
I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.
So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves.
I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation.
I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith.
This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)
Best,
Heather.
[1] https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
-Aaron
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
30? No wonder we are worried about editor attrition :-) Seriously,
http://stats.wikimedia.org/EN/SummaryEN.htm
shows that in May 2014 on en.WP we had about 32K active editors (> 5 edits per month) and 3K very active editors (>100 edits per month).
Or have I missed something here? Are researchers only interested in people who have been on Wikipedia for 10+ years with 10M edits or .?
Kerry
_____
From: wiki-research-l-bounces@lists.wikimedia.org [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Lane Rasberry Sent: Wednesday, 30 July 2014 12:00 PM To: Research into Wikimedia content and communities Subject: Re: [Wiki-research-l] discussion about wikipedia surveys
Hey guys,
I posted some thoughts to my own blog and am linking to those posts below. Everything I say on my blog is captured in the summary below, so feel free to not click through.
----
My biggest worry is that researchers who recruit human subjects assume that there are huge numbers of Wikipedians for them to survey, and consequently, they do not need to do a lot of advance survey preparation because there is no harm from distracting Wikipedians from their usual volunteer work. This assumption is wrong because actually almost every researcher recruiting human subjects wants Wikipedians who are in very short supply. Consequently, researchers do cause harm to the community by soliciting for volunteer time, and Wikipedia community benefit is dubious when researchers do not do sufficient preparation for their work. This is not quite accurate, but if there were one message I could convey to researchers, it would be "Your research participant pool only consists of about 30 super busy people and many other volunteers greatly depend on getting their time. When you take time from a Wikipedian, you are taking that time away from other volunteers who really need it, so be respectful of your intervention in our communities." ----
I do not want a lot of gatekeeping between researchers and the Wikipedia community, but at the same time, researchers should take professional pride in their work and take care not to disrupt Wikimedia community activities.
http://bluerasberry.com/2014/07/request-for-researchers-when-doing-research -on-the-wikimedia-community/ http://bluerasberry.com/2014/07/problems-with-research-on-wikipedia/
I am still thinking about what should be done with research.
yours,
On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
Hi all,
I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I'd chime in to clear some confusion.
Is RCom an official WMF body or a group of volunteers?
RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.
Is RCom a body regulating subject recruitment?
No, subject recruitment was only one among many areas of interest identified by its participants [2]
Is RCom still alive?
RCom stopped working a while ago as a group meeting on a regular basis to discuss joint initiatives. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don't think that the alternative between "either RCom exists and it functions effectively or reviews should immediately stop" is well framed or even desirable, for the reasons that I explain below.
What's the source of RCom's authority in reviewing subject recruitment requests?
Despite the perception that one of RCom's duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to "obtain formal approval". The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and enforceable policies. It's a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.
What about requests for private data?
Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation's Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We've already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users' privacy.
How can we incentivize researchers to "give back" to the community?
In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It's been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.
Dario
[1] https://meta.wikimedia.org/w/index.php?title=Research:Committee https://meta.wikimedia.org/w/index.php?title=Research:Committee&oldid=20948 18 &oldid=2094818
[2] https://meta.wikimedia.org/wiki/Research:Committee/Areas_of_interest
[3] https://meta.wikimedia.org/wiki/Research:WMF_support
[4] https://meta.wikimedia.org/wiki/Privacy_policy
[5] https://meta.wikimedia.org/wiki/Data_retention_guidelines
[6] http://meta.wikimedia.org/wiki/Research:Schemas
On Jul 29, 2014, at 6:49 PM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
Either [RCOM is] functioning or its not, surely?
Well, I explained that there are functioning sub-committees still. In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group. I don't know how else to explain it. I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings.
As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
That's a proposed policy. Until it is passed by consensus, the "must" is a proposed term. I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".
Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated. Hence why I welcome participation. What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently. Does that make sense?
b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
I welcome you to raise it to them. I don't think it is worth their time, but they might disagree.
But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I think that the proposed policy on English Wikipedia does that quite well. That's why I directed people there. Also, again, I am not working on RCOM or subject recruitment as a WMF employee. I do this in my volunteer time. This is true of all of RCOM who happen to also be staff.
if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail. When you guys work out how clearly you want a process to be described, please let me know. I'm tired of re-spec'ing processes. This is the third iteration.
If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.
Heather, that is a proposed policy page on English Wikipedia. It is not part of RCOM. It would render RCOM irrelevant for subject recruitment concerns. That's why I started it. I don't think that RCOM/WMF/researchers should own subject recruitment review. I think the community being studied should own it and that RCOM/WMF/researchers should participate.
Also, I am not your employee. This is my volunteer time. I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to. If you want something done and other volunteers don't have time to do it. Do it https://en.wikipedia.org/wiki/Wikipedia:SOFIXIT yourself.
-Aaron
On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford hfordsa@gmail.com wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
RCOM is not functioning as a complete group anymore.
I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
However, we split into sub-committees while we were still a functioning group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still.
I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.
So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves.
I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation.
I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith.
This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)
Best,
Heather.
[1] https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
-Aaron
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
For example, this proposal[1] was sent to me last year. The researcher's plan was not to sample from the pool of active editors, but instead to contact the 500 *most active Wikipedians* on enwiki to survey them about their motivations. There were several concerns raised about (1) whether the proposed study was duplicating prior work, (2) why the busiest editors needed to be surveyed and (3) whether the researcher's methodology would allow for the intended insights to be gained.
Regretfully, the researcher decided not to respond to anyone other than myself and was also unwilling to work through many of the concerns I raised.
1. https://meta.wikimedia.org/wiki/Research:Online_knowledge_sharing
-Aaron
On Wed, Jul 30, 2014 at 8:19 PM, Kerry Raymond kerry.raymond@gmail.com wrote:
30? No wonder we are worried about editor attrition J Seriously,
http://stats.wikimedia.org/EN/SummaryEN.htm
shows that in May 2014 on en.WP we had about 32K active editors (> 5 edits per month) and 3K very active editors (>100 edits per month).
Or have I missed something here? Are researchers only interested in people who have been on Wikipedia for 10+ years with 10M edits or …?
Kerry
*From:* wiki-research-l-bounces@lists.wikimedia.org [mailto: wiki-research-l-bounces@lists.wikimedia.org] *On Behalf Of *Lane Rasberry *Sent:* Wednesday, 30 July 2014 12:00 PM
*To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys
Hey guys,
I posted some thoughts to my own blog and am linking to those posts below. Everything I say on my blog is captured in the summary below, so feel free to not click through.
My biggest worry is that researchers who recruit human subjects assume that there are huge numbers of Wikipedians for them to survey, and consequently, they do not need to do a lot of advance survey preparation because there is no harm from distracting Wikipedians from their usual volunteer work. This assumption is wrong because actually almost every researcher recruiting human subjects wants Wikipedians who are in very short supply. Consequently, researchers do cause harm to the community by soliciting for volunteer time, and Wikipedia community benefit is dubious when researchers do not do sufficient preparation for their work. This is not quite accurate, but if there were one message I could convey to researchers, it would be "Your research participant pool only consists of about 30 super busy people and many other volunteers greatly depend on getting their time. When you take time from a Wikipedian, you are taking that time away from other volunteers who really need it, so be respectful of your intervention in our communities."
I do not want a lot of gatekeeping between researchers and the Wikipedia community, but at the same time, researchers should take professional pride in their work and take care not to disrupt Wikimedia community activities.
< http://bluerasberry.com/2014/07/request-for-researchers-when-doing-research-...
http://bluerasberry.com/2014/07/problems-with-research-on-wikipedia/
I am still thinking about what should be done with research.
yours,
On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
Hi all,
I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion.
*Is RCom an official WMF body or a group of volunteers?*
RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.
*Is RCom a body regulating subject recruitment?*
No, subject recruitment was only one among many areas of interest identified by its participants [2]
*Is RCom still alive?*
RCom stopped working a while ago* as a* *group meeting on a regular basis to discuss joint initiatives*. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below.
*What’s the source of RCom’s authority in reviewing subject recruitment requests?*
Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and *enforceable* policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.
*What about requests for private data?*
Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.
*How can we incentivize researchers to “give back” to the community?*
In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.
Dario
[1] https://meta.wikimedia.org/w/index.php?title=Research:Committee&oldid=20...
[2] https://meta.wikimedia.org/wiki/Research:Committee/Areas_of_interest
[3] https://meta.wikimedia.org/wiki/Research:WMF_support
[4] https://meta.wikimedia.org/wiki/Privacy_policy
[5] https://meta.wikimedia.org/wiki/Data_retention_guidelines
[6] http://meta.wikimedia.org/wiki/Research:Schemas
On Jul 29, 2014, at 6:49 PM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
Either [RCOM is] functioning or its not, surely?
Well, I explained that there are functioning sub-committees still. In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group. I don't know how else to explain it. I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings.
As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
That's a proposed policy. Until it is passed by consensus, the "must" is a proposed term. I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".
Regarding researchers stating what should be regulated, I think there is a big difference between *deciding what should be regulated* and *being involved in the discussion of *how* it should be regulated*. Hence why I welcome participation. What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently. Does that make sense?
b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
I welcome you to raise it to them. I don't think it is worth their time, but they might disagree.
But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I think that the proposed policy on English Wikipedia does that quite well. That's why I directed people there. Also, again, I am not working on RCOM or subject recruitment as a WMF employee. I do this in my volunteer time. This is true of all of RCOM who happen to also be staff.
if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail. When you guys work out how clearly you want a process to be described, please let me know. I'm tired of re-spec'ing processes. This is the third iteration.
If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.
Heather, that is a *proposed *policy page on English Wikipedia. It is not part of RCOM. It would render RCOM irrelevant for subject recruitment concerns. That's why I started it. I don't think that RCOM/WMF/researchers should own subject recruitment review. I think the community being studied should own it and that RCOM/WMF/researchers should participate.
Also, I am not your employee. This is my volunteer time. I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to. If you want something done and other volunteers don't have time to do it. Do it yourself. https://en.wikipedia.org/wiki/Wikipedia:SOFIXIT
-Aaron
On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford hfordsa@gmail.com wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker < aaron.halfaker@gmail.com> wrote:
RCOM is not functioning as a complete group anymore.
I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
However, we split into sub-committees while we were still a functioning group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still.
I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.
So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
There are also more than two "review coordinators" (not not"reviewers") -- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves.
I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation.
I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith.
This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)
Best,
Heather.
[1] https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
-Aaron
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
--
Lane Rasberry
user:bluerasberry on Wikipedia
206.801.0814 lane@bluerasberry.com
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Well, it seems they took some of your advice. It did say "top 500 contributors" in its original form, but now says "any interested Wikipedia user".
Kerry
_____
From: Aaron Halfaker [mailto:aaron.halfaker@gmail.com] Sent: Thursday, 31 July 2014 10:45 PM To: Kerry Raymond; Research into Wikimedia content and communities Subject: Re: [Wiki-research-l] discussion about wikipedia surveys
For example, this proposal[1] was sent to me last year. The researcher's plan was not to sample from the pool of active editors, but instead to contact the 500 most active Wikipedians on enwiki to survey them about their motivations. There were several concerns raised about (1) whether the proposed study was duplicating prior work, (2) why the busiest editors needed to be surveyed and (3) whether the researcher's methodology would allow for the intended insights to be gained.
Regretfully, the researcher decided not to respond to anyone other than myself and was also unwilling to work through many of the concerns I raised.
1. https://meta.wikimedia.org/wiki/Research:Online_knowledge_sharing
-Aaron
On Wed, Jul 30, 2014 at 8:19 PM, Kerry Raymond kerry.raymond@gmail.com wrote:
30? No wonder we are worried about editor attrition :-) Seriously,
http://stats.wikimedia.org/EN/SummaryEN.htm
shows that in May 2014 on en.WP we had about 32K active editors (> 5 edits per month) and 3K very active editors (>100 edits per month).
Or have I missed something here? Are researchers only interested in people who have been on Wikipedia for 10+ years with 10M edits or .?
Kerry
_____
From: wiki-research-l-bounces@lists.wikimedia.org [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Lane Rasberry Sent: Wednesday, 30 July 2014 12:00 PM
To: Research into Wikimedia content and communities Subject: Re: [Wiki-research-l] discussion about wikipedia surveys
Hey guys,
I posted some thoughts to my own blog and am linking to those posts below. Everything I say on my blog is captured in the summary below, so feel free to not click through.
----
My biggest worry is that researchers who recruit human subjects assume that there are huge numbers of Wikipedians for them to survey, and consequently, they do not need to do a lot of advance survey preparation because there is no harm from distracting Wikipedians from their usual volunteer work. This assumption is wrong because actually almost every researcher recruiting human subjects wants Wikipedians who are in very short supply. Consequently, researchers do cause harm to the community by soliciting for volunteer time, and Wikipedia community benefit is dubious when researchers do not do sufficient preparation for their work. This is not quite accurate, but if there were one message I could convey to researchers, it would be "Your research participant pool only consists of about 30 super busy people and many other volunteers greatly depend on getting their time. When you take time from a Wikipedian, you are taking that time away from other volunteers who really need it, so be respectful of your intervention in our communities." ----
I do not want a lot of gatekeeping between researchers and the Wikipedia community, but at the same time, researchers should take professional pride in their work and take care not to disrupt Wikimedia community activities.
http://bluerasberry.com/2014/07/request-for-researchers-when-doing-research -on-the-wikimedia-community/ http://bluerasberry.com/2014/07/problems-with-research-on-wikipedia/
I am still thinking about what should be done with research.
yours,
On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
Hi all,
I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I'd chime in to clear some confusion.
Is RCom an official WMF body or a group of volunteers?
RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.
Is RCom a body regulating subject recruitment?
No, subject recruitment was only one among many areas of interest identified by its participants [2]
Is RCom still alive?
RCom stopped working a while ago as a group meeting on a regular basis to discuss joint initiatives. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don't think that the alternative between "either RCom exists and it functions effectively or reviews should immediately stop" is well framed or even desirable, for the reasons that I explain below.
What's the source of RCom's authority in reviewing subject recruitment requests?
Despite the perception that one of RCom's duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to "obtain formal approval". The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and enforceable policies. It's a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.
What about requests for private data?
Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation's Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We've already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users' privacy.
How can we incentivize researchers to "give back" to the community?
In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It's been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.
Dario
[1] https://meta.wikimedia.org/w/index.php?title=Research:Committee https://meta.wikimedia.org/w/index.php?title=Research:Committee&oldid=20948 18 &oldid=2094818
[2] https://meta.wikimedia.org/wiki/Research:Committee/Areas_of_interest
[3] https://meta.wikimedia.org/wiki/Research:WMF_support
[4] https://meta.wikimedia.org/wiki/Privacy_policy
[5] https://meta.wikimedia.org/wiki/Data_retention_guidelines
[6] http://meta.wikimedia.org/wiki/Research:Schemas
On Jul 29, 2014, at 6:49 PM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
Either [RCOM is] functioning or its not, surely?
Well, I explained that there are functioning sub-committees still. In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group. I don't know how else to explain it. I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings.
As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
That's a proposed policy. Until it is passed by consensus, the "must" is a proposed term. I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".
Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated. Hence why I welcome participation. What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently. Does that make sense?
b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
I welcome you to raise it to them. I don't think it is worth their time, but they might disagree.
But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I think that the proposed policy on English Wikipedia does that quite well. That's why I directed people there. Also, again, I am not working on RCOM or subject recruitment as a WMF employee. I do this in my volunteer time. This is true of all of RCOM who happen to also be staff.
if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail. When you guys work out how clearly you want a process to be described, please let me know. I'm tired of re-spec'ing processes. This is the third iteration.
If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.
Heather, that is a proposed policy page on English Wikipedia. It is not part of RCOM. It would render RCOM irrelevant for subject recruitment concerns. That's why I started it. I don't think that RCOM/WMF/researchers should own subject recruitment review. I think the community being studied should own it and that RCOM/WMF/researchers should participate.
Also, I am not your employee. This is my volunteer time. I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to. If you want something done and other volunteers don't have time to do it. Do it https://en.wikipedia.org/wiki/Wikipedia:SOFIXIT yourself.
-Aaron
On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford hfordsa@gmail.com wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
RCOM is not functioning as a complete group anymore.
I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
However, we split into sub-committees while we were still a functioning group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still.
I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.
So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves.
I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation.
I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith.
This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)
Best,
Heather.
[1] https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
-Aaron
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
That's extremely helpful, and I suggest copying it to the https://meta.wikimedia.org/wiki/Research:Committee page
(that page needs many updates) --
Piotr Konieczny, PhD http://hanyang.academia.edu/PiotrKonieczny http://scholar.google.com/citations?user=gdV8_AEAAAAJ http://en.wikipedia.org/wiki/User:Piotrus
On 7/30/2014 07:00, Dario Taraborelli wrote:
Hi all,
I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion.
*Is RCom an official WMF body or a group of volunteers?*
RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.
*Is RCom a body regulating subject recruitment?*
No, subject recruitment was only one among many areas of interest identified by its participants [2]
*Is RCom still alive?*
RCom stopped working a while ago/as a/ /group meeting on a regular basis to discuss joint initiatives/. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below.
*What’s the source of RCom’s authority in reviewing subject recruitment requests?*
Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and /enforceable/ policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.
*What about requests for **private data**?*
Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.
*How can we incentivize researchers to “give back” to the community?*
In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.
Dario
[1] https://meta.wikimedia.org/w/index.php?title=Research:Committee&oldid=20... [2] https://meta.wikimedia.org/wiki/Research:Committee/Areas_of_interest [3] https://meta.wikimedia.org/wiki/Research:WMF_support [4] https://meta.wikimedia.org/wiki/Privacy_policy [5] https://meta.wikimedia.org/wiki/Data_retention_guidelines [6] http://meta.wikimedia.org/wiki/Research:Schemas
On Jul 29, 2014, at 6:49 PM, Aaron Halfaker <aaron.halfaker@gmail.com mailto:aaron.halfaker@gmail.com> wrote:
Either [RCOM is] functioning or its not, surely?Well, I explained that there are functioning sub-committees still. In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group. I don't know how else to explain it. I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings.
As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?That's a proposed policy. Until it is passed by consensus, the "must" is a proposed term. I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".
Regarding researchers stating what should be regulated, I think there is a big difference between *deciding what should be regulated* and *being involved in the discussion of *how* it should be regulated*. Hence why I welcome participation. What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently. Does that make sense?
b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.I welcome you to raise it to them. I don't think it is worth their time, but they might disagree.
But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.I think that the proposed policy on English Wikipedia does that quite well. That's why I directed people there. Also, again, I am not working on RCOM or subject recruitment as a WMF employee. I do this in my volunteer time. This is true of all of RCOM who happen to also be staff.
if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail. When you guys work out how clearly you want a process to be described, please let me know. I'm tired of re-spec'ing processes. This is the third iteration.
If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.Heather, that is a *proposed *policy page on English Wikipedia. It is not part of RCOM. It would render RCOM irrelevant for subject recruitment concerns. That's why I started it. I don't think that RCOM/WMF/researchers should own subject recruitment review. I think the community being studied should own it and that RCOM/WMF/researchers should participate.
Also, I am not your employee. This is my volunteer time. I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to. If you want something done and other volunteers don't have time to do it. Do it yourself. https://en.wikipedia.org/wiki/Wikipedia:SOFIXIT
-Aaron
On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <hfordsa@gmail.com mailto:hfordsa@gmail.com> wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <aaron.halfaker@gmail.com <mailto:aaron.halfaker@gmail.com>> wrote: RCOM is not functioning as a complete group anymore. I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely? However, we split into sub-committees while we were still a functioning group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still. I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval. So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up. There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves. I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list. I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation. I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith. This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :) Best, Heather. [1] https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment -Aaron _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org mailto:Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
That is indeed really helpful, thanks for taking the time, Dario!
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org | @hfordsa http://www.twitter.com/hfordsa
On 29 July 2014 23:00, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
Hi all,
I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion.
*Is RCom an official WMF body or a group of volunteers?*
RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.
*Is RCom a body regulating subject recruitment?*
No, subject recruitment was only one among many areas of interest identified by its participants [2]
*Is RCom still alive?*
RCom stopped working a while ago* as a* *group meeting on a regular basis to discuss joint initiatives*. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below.
*What’s the source of RCom’s authority in reviewing subject recruitment requests?*
Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and *enforceable* policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.
*What about requests for **private data**?*
Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.
*How can we incentivize researchers to “give back” to the community?*
In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.
Dario
[1] https://meta.wikimedia.org/w/index.php?title=Research:Committee&oldid=20... [2] https://meta.wikimedia.org/wiki/Research:Committee/Areas_of_interest [3] https://meta.wikimedia.org/wiki/Research:WMF_support [4] https://meta.wikimedia.org/wiki/Privacy_policy [5] https://meta.wikimedia.org/wiki/Data_retention_guidelines [6] http://meta.wikimedia.org/wiki/Research:Schemas
On Jul 29, 2014, at 6:49 PM, Aaron Halfaker aaron.halfaker@gmail.com wrote:
Either [RCOM is] functioning or its not, surely?
Well, I explained that there are functioning sub-committees still. In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group. I don't know how else to explain it. I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings.
As Kerry noted earlier on, the policy as it stands [1] says that
researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
That's a proposed policy. Until it is passed by consensus, the "must" is a proposed term. I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".
Regarding researchers stating what should be regulated, I think there is a big difference between *deciding what should be regulated* and *being involved in the discussion of *how* it should be regulated*. Hence why I welcome participation. What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently. Does that make sense?
b) Pine suggested a board decision on this earlier one to obtain clarity
and I supported this but it was met with silence, which is why I followed up.
I welcome you to raise it to them. I don't think it is worth their time, but they might disagree.
But what is clear is that clarification is required - especially on the
distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I think that the proposed policy on English Wikipedia does that quite well. That's why I directed people there. Also, again, I am not working on RCOM or subject recruitment as a WMF employee. I do this in my volunteer time. This is true of all of RCOM who happen to also be staff.
if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail. When you guys work out how clearly you want a process to be described, please let me know. I'm tired of re-spec'ing processes. This is the third iteration.
If the policy is incorrectly described on the policy pages, then someone
from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.
Heather, that is a *proposed *policy page on English Wikipedia. It is not part of RCOM. It would render RCOM irrelevant for subject recruitment concerns. That's why I started it. I don't think that RCOM/WMF/researchers should own subject recruitment review. I think the community being studied should own it and that RCOM/WMF/researchers should participate.
Also, I am not your employee. This is my volunteer time. I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to. If you want something done and other volunteers don't have time to do it. Do it yourself. https://en.wikipedia.org/wiki/Wikipedia:SOFIXIT
-Aaron
On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford hfordsa@gmail.com wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <
aaron.halfaker@gmail.com> wrote:
RCOM is not functioning as a complete group anymore.
I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
However, we split into sub-committees while we were still a functioning
group. The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still.
I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.
So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here?
re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
There are also more than two "review coordinators" (not not "reviewers")
-- it's just that DarTar and I have accepted the burden of distributing work. When people are busy, we often coordinate the reviews ourselves.
I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.
I welcome your edits to make it clear that review is optional. As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation.
I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith.
This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)
Best, Heather.
[1] https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
-Aaron
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
This an interesting clarification. I support framing RCOM's mission as educational (teaching researchers about best practices), and even more so clearly stating that its procedures are voluntary. In other words, such a body should have an uncontroversial consultative/advisory role, rather then be a gatekeeper of sorts. That said, I don't know if we need a "body" at all. Why couldn't all of this be done under existing community auspices such as WikiProject Research?
I still think our priority should be to redesign our research pages, create a proper research portal with best practices (and hopefully some carrot-like tools that help researchers, from certificates to how-tos for grants/data to research tools) that we could then advertise among most Wikipedia researchers.
IMHO one of RCOM's biggest fallacies was (is...) trying to frame itself as a gatekeeper then a facilitator.
--
Piotr Konieczny, PhD http://hanyang.academia.edu/PiotrKonieczny http://scholar.google.com/citations?user=gdV8_AEAAAAJ http://en.wikipedia.org/wiki/User:Piotrus
On 7/29/2014 22:42, Aaron Halfaker wrote:
I don't believe there is any claim of authority for RCOM. At least I was not involved in making claims that it is required and I do not see it as such. In fact, I have argued in the past that studies run by Wikipedians won't gain much from the process[1]. However, I do recommend that academics -- especially those who do not otherwise engage with Wikipedians -- to work with an RCOM member to coordinate a review in order to ensure that you won't see massive push-back when you start recruiting on Wikipedia -- as studies tended to see when they were run before the process.
https://meta.wikimedia.org/wiki/Grants_talk:IEG/Reimagining_Wikipedia_Mentor...
On Tue, Jul 29, 2014 at 8:35 AM, Nathan <nawrich@gmail.com mailto:nawrich@gmail.com> wrote:
Hi Aaron, what's the source of authority for RCOM (or its members acting independently) to perform a review procedure and claim it is required? On Tue, Jul 29, 2014 at 9:27 AM, Aaron Halfaker <aaron.halfaker@gmail.com <mailto:aaron.halfaker@gmail.com>> wrote: Re. RCOM and review processes, these are two different things. RCOM is an old, defunct WMF sanctioned working group of staff, researchers and Wikipedians. If we want to revive RCOM, it seems like this should be discussed in another thread. _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
I have replied at https://en.wikipedia.org/wiki/Wikipedia_talk:Research_recruitment#Strong_obj...
While this group is transparent, Wikipedia discussion is even more so, and I prefer to held discussion in a more transparent venue where possible.
For those that don't want to read another site's discussion, a short summary of my points: * I agree that it's good to VOLUNTARILY recommend best practices as Aaron lists in his 1-3 points * I don't recognize RCOM's or Aaron's authority to say things like "maybe you shouldn't be allowed to contact Wikipedians". If a researchers violates Wikipedia rules, our regular policies enforced by regular admin corp, plus in extreme cases potential shaming of unethical research through publicity/contacting unethical researcher departments should be enough.
--
Piotr Konieczny, PhD http://hanyang.academia.edu/PiotrKonieczny http://scholar.google.com/citations?user=gdV8_AEAAAAJ http://en.wikipedia.org/wiki/User:Piotrus
On 7/29/2014 22:26, Aaron Halfaker wrote:
I don't think that it is appropriate that those who benefit from deregulation (e.g. No oversight for running surveys. No formalized community review process.) make the decisions about what is worth regulating. You'll notice that the proposed policy that Poitr calls "instruction creep" basically states that you do three things:
- Document your research. Specifically, your methods of recruitment,
consent process, data storage and publication strategy. 2. Discuss your research -- with Wikipedians to make sure that you won't cause a disruption 3. Proceed as consensus emerges.
We all seem to agree that this is good practice. Where is the rest of the "instruction creep"? Where is the anti-researcher bend?
Poitr, you speculate about potential problems like people just coming to say "IDONTLIKEIT", but I have yet to see that happen in RCOM's process despite the fact that we invite editors from the population being sampled to the conversation. Even if it was true, I think that if some of your potential participants don't like what you are doing, you ought to address their concerns.
I'm all for developing guidelines (note that Ethically researching Wikipedia IS NOT a guideline). I've wrote my fair share of essays to help researchers & Wikipedians find their way around research projects in Wikipedia. E.g. http://en.wikipedia.org/wiki/Wikipedia:Research and and https://en.wikipedia.org/wiki/User:EpochFail/Don%27t_bite_the_researchers. However, I've watched good research projects fail because researchers didn't have the wikipedian backgrounds that you guys do (Heather and Poitr). See some examples of (IRB approved) studies running into project-halting difficulties: https://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Research#Examples_o... These examples are what got me to start working on developing a process in the first place.
If you really think that documenting your research and having a discussion about it is too much instruction, then maybe you shouldn't be allowed to contact Wikipedians. If you do think that every research project that does recruitment should be documented and discussed, why not just say so?
-Aaron
On Tue, Jul 29, 2014 at 3:50 AM, Heather Ford <hfordsa@gmail.com mailto:hfordsa@gmail.com> wrote:
+1 on Piotr's comments. And very, very happy to hear about https://en.wikipedia.org/wiki/Wikipedia:Ethically_researching_Wikipedia -- I think this is definitely the way to go: developing guidelines that we *regularly point people to* when they have questions etc. And maybe something that we as a group can work on in the coming months. I'll reiterate my suggestions for goals here and add some of Piotr's and others' comments: 1. developing ethical research guidelines for Wikipedia research - by building on the WP:Ethically_researching_Wikipedia page and regularly pointing people to it 2. finding ways of making responsible requests to the WMF for data that they hold that might benefit research outside the WMF - through an official process with guidelines from the WMF on response times/ viable requests etc. 3. developing opportunities for researchers to collaborate and share what they're doing with the wider research community - reorganising the research hub and pointing to best case practices etc (similar to the WP Global Education program, as Piotr suggests) - actively recruiting WP researchers to join this list and visit the research hub - some other regular way of involving researchers such as inviting them to showcase their work and have it recognised on the list, on the hub etc - recognising outstanding research (through a prize perhaps as Aaron suggested) Looking forward to hearing Phoebe's suggestions! Best, Heather. Heather Ford Oxford Internet Institute <http://www.oii.ox.ac.uk> Doctoral Programme EthnographyMatters <http://ethnographymatters.net> | Oxford Digital Ethnography Group <http://www.oii.ox.ac.uk/research/projects/?id=115> http://hblog.org <http://hblog.org/> | @hfordsa <http://www.twitter.com/hfordsa> On 29 July 2014 09:04, Pine W <wiki.pine@gmail.com <mailto:wiki.pine@gmail.com>> wrote: The good and bad news is that the status quo with RCOM is likely to remain unless someone in WMF, the Board, or the community is interested enough in addressing the situation to put in some effort to make RCOM a functioning organization. At the moment I have the impression that WMF researchers are absorbing most of the work that RCOM and some dedicated RCOM admin support could do, like help with lit review and prevent outside researchers from using WMF databases in ways that compromise user privacy. My perception is that the current situation is inefficient for WMF and for outside researchers who want to do good work with WMF or community resources, and also that RCOM lacks the resources to respond in timely ways to requests for help with outside research that could benefit Wikimedia. So, I there are reasons to changs the status quo, and I hope WMF or the Board would be interested in something like the proposal I made previously. Phoebe, what do you think? Pine _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
This time I'll respond below.
On 7/29/2014 17:50, Heather Ford wrote:
+1 on Piotr's comments.
And very, very happy to hear about https://en.wikipedia.org/wiki/Wikipedia:Ethically_researching_Wikipedia -- I think this is definitely the way to go: developing guidelines that we *regularly point people to* when they have questions etc. And maybe something that we as a group can work on in the coming months.
I'll reiterate my suggestions for goals here and add some of Piotr's and others' comments:
- developing ethical research guidelines for Wikipedia research
- by building on the WP:Ethically_researching_Wikipedia page and
regularly pointing people to it
Two ideas: * there's a drive to print out leaflets for Wikimania, this page could be advertised there * even better, we should try to advertise it in a leaflet form at Wikisym * WMF could try to create a short handout booklet based on it
- finding ways of making responsible requests to the WMF for data
that they hold that might benefit research outside the WMF
- through an official process with guidelines from the WMF on response
times/ viable requests etc.
It is a good example of an idea that helps rather than hinders researchers, and an area where RCOM-like body assistance would be useful.
- developing opportunities for researchers to collaborate and share
what they're doing with the wider research community
- reorganising the research hub and pointing to best case practices
etc (similar to the WP Global Education program, as Piotr suggests)
- actively recruiting WP researchers to join this list and visit the
research hub
- some other regular way of involving researchers such as inviting
them to showcase their work and have it recognised on the list, on the hub etc
- recognising outstanding research (through a prize perhaps as Aaron
suggested)
All +1
--
Piotr Konieczny, PhD http://hanyang.academia.edu/PiotrKonieczny http://scholar.google.com/citations?user=gdV8_AEAAAAJ http://en.wikipedia.org/wiki/User:Piotrus
With the disclaimer that I've carried out surveys myself in the past, I want to strongly dispute the claim that " Wikipedians were getting tired of being continually contacted by researchers to fill out *surveys*". As an editor who is in the Top 100 most active Wikipedians, I'd think I'd get to see an above average number of requests, where I don't think I get asked more than once a year. Maybe twice, in a "good" year, with half not even on my talk page but something I see at WikiProjects I frequent.
Even Heather says "I certainly haven't seen a huge amount of surveys myself". I don't think anyone has seen any significant amount of surveys (and what would be "huge"? would even getting one request a month be too much, really?).
It is my belief that this type of discussion is driven by a very tiny and completely unrepresentative group of editors who dislike science/research and are very vocal about it (i.e. Wikipedia equivalent of anti-vaccination activists), in other words people who may not get more than one or two survey requests per year but for whom it is an occasion to write long rants about how researchers are wasting everyone's time. Seeing as not taking part in a survey takes a few seconds of reading and forgetting about an invitation, I think that much more time is wasted by giving any attention to such complains in the first place.
Until such a time that someone can show that researchers are indeed affecting the work of volunteers in any meaningful way (as in, imposing on them more than asking for few seconds-a minute or two each year, collectively) I believe this discussion is a storm in a teacup and, indeed, a waste of our time.
--
Piotr Konieczny, PhD http://hanyang.academia.edu/PiotrKonieczny http://scholar.google.com/citations?user=gdV8_AEAAAAJ http://en.wikipedia.org/wiki/User:Piotrus
On 7/17/2014 18:38, Heather Ford wrote:
Agree with Kerry that we really need to have a more flexible process that speaks to the main problem that (I think) RCOM was started to solve i.e. that Wikipedians were getting tired of being continually contacted by researchers to fill out *surveys*. I'm not sure where feelings are about that right now (I certainly haven't seen a huge amount of surveys myself) but I guess the big question right now is whether RCOM is actually active or not. I must say that I was surprised, Aaron, when I read that it is active because I had heard from others in your team about a year or two ago that this wasn't going to be the vehicle for obtaining permission going forward and that a new, more lightweight process was being designed. As Nathan discusses on the Wikimedia-l list, there aren't many indications that RCOM is still active. Perhaps there has been a recent decision to resuscitate it? If that's the case, let us know about it :) And then we can discuss what needs to happen to build a good process.
One immediate requirement that I've been talking to others about is finding ways of making the case to the WMF as a group of researchers for the anonymization of country level data, for example. I've spoken to a few researchers (and I myself made a request about a year ago that hasn't been responded to) and it seems like some work is required by the foundation to do this anonymisation but that there are a few of us who would be really keen to use this data to produce research very valuable to Wikipedia - especially from smaller language versions/developing countries. Having an official process that assesses how worthwhile this investment of time would be to the Foundation would be a great idea, I think, but right now there seems to be a general focus on the research that the Foundation does itself rather than enabling researchers outside. I know how busy Aaron and Dario (and others in the team) are so perhaps this requires a new position to coordinate between researchers and Foundation resources?
Anyway, I think the big question right now is whether there are any plans for RCOM that have been made by the research team and the only people who can answer that are folks in the research team :)
Best, Heather.
Heather Ford Oxford Internet Institute http://www.oii.ox.ac.uk Doctoral Programme EthnographyMatters http://ethnographymatters.net | Oxford Digital Ethnography Group http://www.oii.ox.ac.uk/research/projects/?id=115 http://hblog.org http://hblog.org/ | @hfordsa http://www.twitter.com/hfordsa
On 17 July 2014 08:49, Kerry Raymond <kerry.raymond@gmail.com mailto:kerry.raymond@gmail.com> wrote:
Yes, I meant the community/communities of WMF. But the authority of the community derives from WMF, which chooses to delegate such matters. I think that “advise” is a good word to use. Kerry ------------------------------------------------------------------------ *From:*Amir E. Aharoni [mailto:amir.aharoni@mail.huji.ac.il <mailto:amir.aharoni@mail.huji.ac.il>] *Sent:* Thursday, 17 July 2014 5:37 PM *To:* kerry.raymond@gmail.com <mailto:kerry.raymond@gmail.com>; Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys >WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. I don't think that it really should be about WMF. The WMF shouldn't enforce anything. The community can formulate good practices for researchers and _advise_ community members not to cooperate with researchers who don't follow these practices. Not much more is needed. -- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore 2014-07-17 8:24 GMT+03:00 Kerry Raymond <kerry.raymond@gmail.com <mailto:kerry.raymond@gmail.com>>: Just saying here what I already put on the Talk page: I am a little bothered by the opening sentence "This page documents the process that researchers must follow before asking Wikipedia contributors to participate in research studies such as surveys, interviews and experiments." WMF does not "own" me as a contributor; it does not decide who can and cannot recruit me for whatever purposes. What WMF does own is its communication channels to me as a contributor and WMF has a right to control what occurs on those channels. Also I think WMF probably should be concerned about both its readers and its contributors being recruited through its channels (as either might be being recruited). I think this distinction should be made, e.g. "This page documents the process that researchers must follow if they wish to use Wikipedia's (WMF's?) communication channels to recruit people to participate in research studies such as surveys, interviews and experiments. Communication channels include its mailing lists, its Project pages, Talk pages, and User Talk pages [and whatever else I've forgotten]." If researchers want to recruit WPians via non-WMF means, I don’t think it’s any business of WMF’s. An example might be a researcher who wanted to contact WPians via chapters or thorgs; I would leave it for the chapter/thorg to decide if they wanted to assist the researcher via their communication channels. Of course, the practical reality of it is that some researchers (oblivious of WMF’s concerns in relation to recruitment of WPians to research projects) will simply use WMF’s channels without asking nicely first. Obviously we can remove such requests on-wiki and follow up any email requests with the commentary that this was not an approved request. In my category of [whatever else I’ve forgotten], I guess there are things like Facebook groups and any other social media presence. Also to be practical, if WMF is to have a process to vet research surveys, I think it has to be sufficiently fast and not be overly demanding to avoid the possibility of the researcher giving up (“too hard to deal with these people”) and simply spamming email, project pages, social media in the hope of recruiting some participants regardless. That is, if we make it too slow/hard to do the right thing, we effectively encourage doing the wrong thing. Also, what value-add can we give them to reward those who do the right thing? It’s nice to have a carrot as well as a stick when it comes to onerous processes J Because of the criticism of “not giving back”, could we perhaps do things to try to make the researcher feel part of the community to make “giving back” more likely? For example, could we give them a slot every now and again to talk about their project in the R&D Showcase? Encourage them to be on this mailing list. Are we at a point where it might make sense to organise a Wikipedia research conference to help build a research community? Just thinking aloud here … Kerry ------------------------------------------------------------------------ *From:*wiki-research-l-bounces@lists.wikimedia.org <mailto:wiki-research-l-bounces@lists.wikimedia.org> [mailto:wiki-research-l-bounces@lists.wikimedia.org <mailto:wiki-research-l-bounces@lists.wikimedia.org>] *On Behalf Of *Aaron Halfaker *Sent:* Thursday, 17 July 2014 6:59 AM *To:* Research into Wikimedia content and communities *Subject:* Re: [Wiki-research-l] discussion about wikipedia surveys RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario (dtaraborelli@wikimedia.org <mailto:dtaraborelli@wikimedia.org>) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) <nemowiki@gmail.com <mailto:nemowiki@gmail.com>> wrote: phoebe ayers, 16/07/2014 19:21: > (Personally, I think the answer should be to resuscitate RCOM, but > that's easy to say and harder to do!) IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: <https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&hidebots=1&hideredirs=1&limit=500&offset=&namespace=202> It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round. Nemo _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
[rant - tl;dr]
Ugh, another new instruction creep with an anti-research bent to boot. Thanks, Aaron, for linking it here, as this is the first time I've heard of this - now I actually get to oppose this on record before this is archived :>
That was first third of the problem with RCOM in the first place: next to nobody knew (or knows) about it. When we still get many studies about Wikipedia who clearly display the fact that the researchers fail at basic lit review not citing any prior studies, to expect that most would try to (and be able to) find such pages is nothing but an exercise in bureaucratizing the project. The second third of the problem is that all such policies, if implemented, would make research much more difficult; anytime you add some reviewers to the mix, you add the risk of having good project rejected because of reviewers IDONTLIKEIT, and with the new proposal idea of letting complete amateurs be the reviewers... Fortunately, this doesn't fix the third compound problem of RCOM, which is that a) it had no real power to enforce anything it required and b) next to nobody wanted to invest time into doing the work, because it's a waste of time: non-productive work (not contributing to building an encyclopedia) that very, very few people in our community care about., and that adds an unimportant line to one's professional CV. RCOM is dying of inactivity and of being not needed, we should officially retire it instead of trying to clone it on Wikipedia.
[/rant]
Don't get me wrong, at first RCOM was a nice and noble idea. A guideline page for researchers is helpful, I do like the idea of trying to list and categorize ongoing research (https://meta.wikimedia.org/wiki/Research:Projects), it provides some useful links to data, FAQ and such. However, as in many other places on Wikipedia, this turned into an unnecessary instruction creep, which I very strongly oppose .
A while ago I've contributed to https://en.wikipedia.org/wiki/Wikipedia:Ethically_researching_Wikipedia It's a simple page, the gist of which is that any professional scholar who is researching Wikipedia should already be familiar with their professional codes of ethics, which in turn are perfectly sufficient to protect Wikipedia and its volunteers and users from any abuses. It also doesn't require any policing from the community outside normal scope. Any (extremely rare - can anyone even cite one?) disruptive experiments which breach the professional codes of ethics in the first place should result in bans and WMF official complains. Outside that, Wikipedians can deal with survey/interview requests like everyone else - ignore them if they don't like them. No special body to police researchers is needed. No approval body is needed for anything outside WMF grants, which WMF and/or the existing grant structure can handle.
What we need is for someone to review all research-related pages on Wikipedia and meta, merge any similar ones, and that's it. In other words, we need to condolence and organize the sprawl mess that research pages have become, not to add to them.
--
Piotr Konieczny, PhD http://hanyang.academia.edu/PiotrKonieczny http://scholar.google.com/citations?user=gdV8_AEAAAAJ http://en.wikipedia.org/wiki/User:Piotrus
On 7/17/2014 05:58, Aaron Halfaker wrote:
RCOM review is still alive and looking for new reviewers (really, coordinators). Researchers can be directed to me or Dario (dtaraborelli@wikimedia.org mailto:dtaraborelli@wikimedia.org) to be assigned a reviewer. There is also a proposed policy on enwiki that could use some eyeballs: https://en.wikipedia.org/wiki/Wikipedia:Research_recruitment
On Wed, Jul 16, 2014 at 11:16 AM, Federico Leva (Nemo) <nemowiki@gmail.com mailto:nemowiki@gmail.com> wrote:
phoebe ayers, 16/07/2014 19:21: > (Personally, I think the answer should be to resuscitate RCOM, but > that's easy to say and harder to do!) IMHO in the meanwhile the most useful thing folks can do is subscribing to the feed of new research pages: <https://meta.wikimedia.org/w/index.php?title=Special:NewPages&feed=atom&hidebots=1&hideredirs=1&limit=500&offset=&namespace=202> It's easier to build a functioning RCOM out of an active community of "reviewers", than the other way round. Nemo _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
wiki-research-l@lists.wikimedia.org