Hello,
I am re-posting here because I thought people might be interested in a proposal (http://en.wikipedia.org/wiki/Wikipedia:Village_pump_(proposals)#Wikipedia_re... http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28proposals%29#Wikipedia_research_review) that my lab-mates and I have been working on. The gestalt of the proposal is that Wikipedia needs a review system to both protect itself from questionable research activities and to allow good research to take place within the system. Currently research involving anything other than the database snapshots is difficult because there is no formal policy related to research within Wikipedia.
See failed examples of attempted research here: * Failed mentoring study: http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Inciden... * Failed interface launch: http://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Wikipedia_in...
-Aaron Halfaker GroupLens Research University of Minnesota
On Wed, Jun 3, 2009 at 12:06 PM, Aaron Halfaker half0032@umn.edu wrote:
Hello,
I am re-posting here because I thought people might be interested in a proposal (http://en.wikipedia.org/wiki/Wikipedia:Village_pump_(proposals)#Wikipedia_re... http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28proposals%29#Wikipedia_research_review) that my lab-mates and I have been working on. The gestalt of the proposal is that Wikipedia needs a review system to both protect itself from questionable research activities and to allow good research to take place within the system. Currently research involving anything other than the database snapshots is difficult because there is no formal policy related to research within Wikipedia.
Could you explain a little more what's the exact problem you want to solve?
I followed the links to individual studies, but I only understood the 2nd one about an interface modification. ( http://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Wikipedia_in... )
It sounds like there were two problems here: (1) you didn't explain what your software was -- people wanted assurance that the thing was safe and the like; and (2) people weren't persuaded that participating in the study was worth their time.
I don't understand why having an IRB-like committee will help solve things. In the academic setting, IRB's are good, or at least important, because they try to stop unethical research that harms the overall research community (or at the very least exposes the sponsoring university to lawsuits).
It's not clear to me why Wikipedia research's relationship to Wikipedians is analogous. The bigger problem seems to be persuading people they should participate in the study. Having approval of a committee might have helped a little bit for the first objection you had -- that they have no idea what the software was -- but it wouldn't have helped with the second -- that they didn't think it was worth participating in.
-- Brendan O'Connor - http://anyall.org
See failed examples of attempted research here:
- Failed mentoring study:
http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Inciden...
- Failed interface launch:
http://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Wikipedia_in...
-Aaron Halfaker GroupLens Research University of Minnesota
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
This proposal isn't really about the merits of any particular study. I only offered a link to the discussions about my most recent user study because I felt it was a good example of push-back from Wikipedia editors.
For a better view of the *initial* troubles with the first failed study, see Katherine's talk page http://en.wikipedia.org/wiki/User_talk:KatherinePanciera. I'm most concerned about what happen in that study. After Katherine asked for Wikipedia users to participate simply by taking a survey, her account was nominated for deletion for allegedly violating an obscure policy that did not match he actions. Katherine had no policy to cite in order to defend herself. For simply asking people to participate in a survey, her account was nearly banned.
I'd suggest you have a read through the proposal (it is actually quite small) in order to more clearly understand the problem we wish to solve. The first section is devoted to just that.
Thanks for the comments!
-Aaron
Brendan O'Connor wrote:
On Wed, Jun 3, 2009 at 12:06 PM, Aaron Halfaker half0032@umn.edu wrote:
Hello,
I am re-posting here because I thought people might be interested in a proposal (http://en.wikipedia.org/wiki/Wikipedia:Village_pump_(proposals)#Wikipedia_re... http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28proposals%29#Wikipedia_research_review) that my lab-mates and I have been working on. The gestalt of the proposal is that Wikipedia needs a review system to both protect itself from questionable research activities and to allow good research to take place within the system. Currently research involving anything other than the database snapshots is difficult because there is no formal policy related to research within Wikipedia.
Could you explain a little more what's the exact problem you want to solve?
I followed the links to individual studies, but I only understood the 2nd one about an interface modification. ( http://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Wikipedia_in... )
It sounds like there were two problems here: (1) you didn't explain what your software was -- people wanted assurance that the thing was safe and the like; and (2) people weren't persuaded that participating in the study was worth their time.
I don't understand why having an IRB-like committee will help solve things. In the academic setting, IRB's are good, or at least important, because they try to stop unethical research that harms the overall research community (or at the very least exposes the sponsoring university to lawsuits).
It's not clear to me why Wikipedia research's relationship to Wikipedians is analogous. The bigger problem seems to be persuading people they should participate in the study. Having approval of a committee might have helped a little bit for the first objection you had -- that they have no idea what the software was -- but it wouldn't have helped with the second -- that they didn't think it was worth participating in.
-- Brendan O'Connor - http://anyall.org
See failed examples of attempted research here:
- Failed mentoring study:
http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Inciden...
- Failed interface launch:
http://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Wikipedia_in...
-Aaron Halfaker GroupLens Research University of Minnesota
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Brendan O'Connor - http://anyall.org
On Wed, Jun 3, 2009 at 1:36 PM, Aaron Halfaker half0032@umn.edu wrote:
This proposal isn't really about the merits of any particular study. I only offered a link to the discussions about my most recent user study because I felt it was a good example of push-back from Wikipedia editors.
For a better view of the *initial* troubles with the first failed study, see Katherine's talk page http://en.wikipedia.org/wiki/User_talk:KatherinePanciera. I'm most concerned about what happen in that study. After Katherine asked for Wikipedia users to participate simply by taking a survey, her account was nominated for deletion for allegedly violating an obscure policy that did not match he actions. Katherine had no policy to cite in order to defend herself. For simply asking people to participate in a survey, her account was nearly banned.
Ah OK. It was hard to figure out what the story was based on the links. Thanks for the summary, it makes it clearer.
I'd suggest you have a read through the proposal (it is actually quite
small) in order to more clearly understand the problem we wish to solve. The first section is devoted to just that.
Just a friendly recommendation -- to make the proposal clearer, you should point out reasons people might object to wikipedia research - "it's canvassing", confusion between academic and industrial research, etc.
For the types of research I think of when I think of Wikipedia, I usually think of read-only research so these scenarios didn't occur to me at all. I'm sure you folks in the GroupLens group are quite familiar with a certain set of issues.
Brendan
On Wednesday 03 June 2009, Aaron Halfaker wrote:
This proposal isn't really about the merits of any particular study. I only offered a link to the discussions about my most recent user study because I felt it was a good example of push-back from Wikipedia editors.
I think identifying the particular scenarios (i.e., push-back) would be a good thing within the proposal, and how the proposed solution might provide remedy. Some quick thoughts:
1. This is a challenging problem. (My experience soliciting participation for interviews on WP -- left on a Project page -- was that I got no responses! :) Leaving lots of messages on people's talk pages might have generated more attention, but obviously not all of it good. (I actually didn't make much use of interviews, and got the few I did through personal/f2f contacts rather than online solicitation.) 2. I would not call it a Wikipedia IRB. 3. I think it was reasonable for Wikipedians to object to the many dozens of messages left on User pages. 4. Katherine's messages were fairly good relative to what a IRB notice/consent form would look like, but I don't see any indication of IRB. The NICE notice could be much more specific. An example of an IRB approved solicitation I used can be seen here: http://reagle.org/joseph/2006/disp/9-consent-form.html 5. I expect that: (a) some ornery folks might always complain, even if there was consensus on a Village Research Pump, (b) the Foundation would not in any way want to indemnify the work being done by someone else.
That said, anything that helps researchers develop appropriate instruments, that furthers information within the community, and that is in keeping with policy and community sensibilities is a good thing.
Interesting proposal!
It is one thing for the community to limit the kinds of correspondence that can happen on wiki or on mailing lists if the volume becomes a nuisance for editors, but difficult to imagine prohibiting researchers from contacting individuals directly to ask about their involvement in Wikipedia. I recommend not comparing such an effort to an IRB for that reason. However laudable the intentions, I'm sure many on this list are aware of the problems that bureaucratization of research ethics has sometimes created in social science research, so IRBs (created for medical research) may not be the best model to follow anyhow. :)
Although Wikipedia as a community cannot regulate human-subjects research ethics in the same way that a government-funded agency can, it can create a set of guidelines for on-site research-related communication. My personal recommendation would not be to immediately jump to policy making or review, but to encourage "good behavior" through best practices discussions like this one.
(I have personally never gotten *any* push back in my interview studies of Wikipedians. I also realized long ago that it's not a good idea to leave messages on talk pages because they are public communication and pretty much instantly break or at least weaken confidentiality clauses in my consent forms.)
Andrea
On Wed, Jun 3, 2009 at 5:45 PM, Joseph Reagle reagle@mit.edu wrote:
On Wednesday 03 June 2009, Aaron Halfaker wrote:
This proposal isn't really about the merits of any particular study. I only offered a link to the discussions about my most recent user study because I felt it was a good example of push-back from Wikipedia editors.
I think identifying the particular scenarios (i.e., push-back) would be a good thing within the proposal, and how the proposed solution might provide remedy. Some quick thoughts:
- This is a challenging problem. (My experience soliciting participation for interviews on WP -- left on a Project page -- was that I got no responses! :) Leaving lots of messages on people's talk pages might have generated more attention, but obviously not all of it good. (I actually didn't make much use of interviews, and got the few I did through personal/f2f contacts rather than online solicitation.)
- I would not call it a Wikipedia IRB.
- I think it was reasonable for Wikipedians to object to the many dozens of messages left on User pages.
- Katherine's messages were fairly good relative to what a IRB notice/consent form would look like, but I don't see any indication of IRB. The NICE notice could be much more specific. An example of an IRB approved solicitation I used can be seen here:
http://reagle.org/joseph/2006/disp/9-consent-form.html 5. I expect that: (a) some ornery folks might always complain, even if there was consensus on a Village Research Pump, (b) the Foundation would not in any way want to indemnify the work being done by someone else.
That said, anything that helps researchers develop appropriate instruments, that furthers information within the community, and that is in keeping with policy and community sensibilities is a good thing.
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
I think a better course of action might be to establish a Wikipedia Experiments subgroup of users who opt-in to participate in experiments, much like what Google does with its experimental features. You're limiting the sample quite a bit, and quite possibly only getting involved or heavily involved Wikipedia users, but if your core survey group is editors it would likely be ideal.
Alex Foley
On Jun 3, 2009, at 3:22 PM, Andrea Forte wrote:
Interesting proposal!
It is one thing for the community to limit the kinds of correspondence that can happen on wiki or on mailing lists if the volume becomes a nuisance for editors, but difficult to imagine prohibiting researchers from contacting individuals directly to ask about their involvement in Wikipedia. I recommend not comparing such an effort to an IRB for that reason. However laudable the intentions, I'm sure many on this list are aware of the problems that bureaucratization of research ethics has sometimes created in social science research, so IRBs (created for medical research) may not be the best model to follow anyhow. :)
Although Wikipedia as a community cannot regulate human-subjects research ethics in the same way that a government-funded agency can, it can create a set of guidelines for on-site research-related communication. My personal recommendation would not be to immediately jump to policy making or review, but to encourage "good behavior" through best practices discussions like this one.
(I have personally never gotten *any* push back in my interview studies of Wikipedians. I also realized long ago that it's not a good idea to leave messages on talk pages because they are public communication and pretty much instantly break or at least weaken confidentiality clauses in my consent forms.)
Andrea
On Wed, Jun 3, 2009 at 5:45 PM, Joseph Reagle reagle@mit.edu wrote:
On Wednesday 03 June 2009, Aaron Halfaker wrote:
This proposal isn't really about the merits of any particular study. I only offered a link to the discussions about my most recent user study because I felt it was a good example of push-back from Wikipedia editors.
I think identifying the particular scenarios (i.e., push-back) would be a good thing within the proposal, and how the proposed solution might provide remedy. Some quick thoughts:
- This is a challenging problem. (My experience soliciting
participation for interviews on WP -- left on a Project page -- was that I got no responses! :) Leaving lots of messages on people's talk pages might have generated more attention, but obviously not all of it good. (I actually didn't make much use of interviews, and got the few I did through personal/f2f contacts rather than online solicitation.) 2. I would not call it a Wikipedia IRB. 3. I think it was reasonable for Wikipedians to object to the many dozens of messages left on User pages. 4. Katherine's messages were fairly good relative to what a IRB notice/consent form would look like, but I don't see any indication of IRB. The NICE notice could be much more specific. An example of an IRB approved solicitation I used can be seen here: http://reagle.org/joseph/2006/disp/9-consent-form.html 5. I expect that: (a) some ornery folks might always complain, even if there was consensus on a Village Research Pump, (b) the Foundation would not in any way want to indemnify the work being done by someone else.
That said, anything that helps researchers develop appropriate instruments, that furthers information within the community, and that is in keeping with policy and community sensibilities is a good thing.
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
On 06/03/2009 05:29 PM, Alexander Foley wrote:
I think a better course of action might be to establish a Wikipedia Experiments subgroup of users who opt-in to participate in experiments, much like what Google does with its experimental features. You're limiting the sample quite a bit, and quite possibly only getting involved or heavily involved Wikipedia users, but if your core survey group is editors it would likely be ideal.
I agree. I'd extend this notion a bit: my impression is that most people are more than happy to be solicited for studies (provided it doesn't happen too often). So I'd suggest two components:
1. The opt-in defines specifically how frequently one can be solicited (e.g. N times per year).
2. The opt-in is widely pushed: highly visible on account creation, and all existing users get one (1) invitation to opt in.
I think #2 is important because a de facto policy that only heavily involved Wikipedians participate in research would be severely limiting to the work we do.
Red
Aaron Halfaker wrote:
Hello,
I am re-posting here because I thought people might be interested in a proposal (http://en.wikipedia.org/wiki/Wikipedia:Village_pump_(proposals)#Wikipedia_re... http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28proposals%29#Wikipedia_research_review) that my lab-mates and I have been working on. The gestalt of the proposal is that Wikipedia needs a review system to both protect itself from questionable research activities and to allow good research to take place within the system. Currently research involving anything other than the database snapshots is difficult because there is no formal policy related to research within Wikipedia.
This would be useful if it would be replacing institutional IRBs. Otherwise, its just adding another layer of bureaucracy (and I don't even want to think what would happen if Wikipedia IRB would disagree with my university IRB...).
wiki-research-l@lists.wikimedia.org