I am asking you to share the documentation of the ethical clearance or exemption your institution would have required, not what people did or didn't say to you as part of conference reviewing or at conferences. Ethical clearance is a process that should have been undertaken before your research commenced, not when you are writing the paper or attending a conference. Are you saying you undertook the research without any consideration of the ethics? Does your university have no guidelines about this?

The Wikipedia guidelines about content analysis are not particularly relevant here. You were not analysing existing Wikipedia articles but injecting new articles of dubious quality into Wikipedia.

Nor is the data about individuals my point. If you wasted people's time reacting to the articles created, you did them harm. If people derived incorrect information from reading your articles, you did them harm. None of those people were aware they were part of your research experiment; that means they did not have informed consent in relation to choosing to participate in your experiment. You could have generated the articles and sought the opinions of readers and editors of Wikipedia on those articles without placing them into Wikipedia itself. That way would have enabled informed consent; others not wishing to take part would not be mislead into doing so.

Sent from my iPad

On 12 Aug 2016, at 3:24 PM, siddhartha banerjee sidd2006@gmail.com wrote:

I thought I should add this too as I missed it in the previous email. This link: https://en.wikipedia.org/wiki/Wikipedia:Ethically_researching_Wikipedia talks about the Content Analysis (seeing number of references removed, or content removed)-- which we did (with the few articles) and that is what we followed as it says "generally considered exempt from such requirements and does not require an IRB approval.". My advisor should be able to add more thoughts on it (I have requested him to reply on this thread).

Thanks, Sidd

...

On Thu, Aug 11, 2016 at 9:36 PM, siddhartha banerjee sidd2006@gmail.com wrote: As I have mentioned earlier, this is not the first work on article generation. This is one of the first work we know: https://people.csail.mit.edu/csauper/pubs/sauper-sm-thesis.pdf https://people.csail.mit.edu/regina/my_papers/wiki.pdf All these did not mention anything about human subjects as finally no personal information is used (about the person, who is deleting, etc). Nor did any reviewers/attendees in the conferences in this area question on this aspect. Also, https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2015-01-28/Recent... is relevant here as it talks about our previous work.

if "record of someone doing something" is relevant from human subjects point of view, any data on Wikipedia can be used to find the editors (if not the real person). For example: https://www.aaai.org/ocs/index.php/AAAI/AAAI11/paper/viewFile/3505/3968 https://alchemy.cs.washington.edu/papers/wu08/wu08.pdf I have met several researchers who work using data (revisions from Wikipedia) and nothin on IRB ever came up.

Nevertheless, as I said, if there are concrete rules, I think it would help the research community as a whole to know what can or cannot be done and also ask for permissions. I appreciate the suggestions that Stuart mentioned in a previous email abut experimenting on would be deleted or articles lacking sources. But, as of now we are not planning anything and if we do, we would for sure get in touch with Denny (who had a video chat with me before starting this thread) and would try to know the best ways of doing it.

I have asked my PhD advisor (other author on the paper) to check this thread and he will be able to give more inputs as I am not very qualified to comment on these aspects.

Thanks, Sidd

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

And to its policies

http://guru.psu.edu/policies/RP03.html

With particular reference to

"Intervention includes both physical procedures by which data are gathered (for example, venipuncture) and manipulations of the participant or the participant’s environment that are performed for research purposes."

Putting that articles into Wikipedia manipulated the environment of Wikipedia readers and editors.

Now I am not saying that huge harm was done, you would have to ask those who subsequently edited the articles (a known group) and those who read the articles (an unknown group) to find out if they are unhappy about what took place.

What I am saying is that if consideration had been given to the question who is impacted by this research plan, the maybe the research plan would have been redesigned to prevent the problem, and we would not have to have this conversation.

Kerry

Sent from my iPad

On 12 Aug 2016, at 6:08 PM, Kerry Raymond kerry.raymond@gmail.com wrote:

I draw attention to Penn State's IRB website

https://www.research.psu.edu/irb/submit

Sent from my iPad

...

On 12 Aug 2016, at 6:03 PM, Kerry Raymond kerry.raymond@gmail.com wrote:

I am asking you to share the documentation of the ethical clearance or exemption your institution would have required, not what people did or didn't say to you as part of conference reviewing or at conferences. Ethical clearance is a process that should have been undertaken before your research commenced, not when you are writing the paper or attending a conference. Are you saying you undertook the research without any consideration of the ethics? Does your university have no guidelines about this?

The Wikipedia guidelines about content analysis are not particularly relevant here. You were not analysing existing Wikipedia articles but injecting new articles of dubious quality into Wikipedia.

Nor is the data about individuals my point. If you wasted people's time reacting to the articles created, you did them harm. If people derived incorrect information from reading your articles, you did them harm. None of those people were aware they were part of your research experiment; that means they did not have informed consent in relation to choosing to participate in your experiment. You could have generated the articles and sought the opinions of readers and editors of Wikipedia on those articles without placing them into Wikipedia itself. That way would have enabled informed consent; others not wishing to take part would not be mislead into doing so.

Sent from my iPad

...

On 12 Aug 2016, at 3:24 PM, siddhartha banerjee sidd2006@gmail.com wrote:

I thought I should add this too as I missed it in the previous email. This link: https://en.wikipedia.org/wiki/Wikipedia:Ethically_researching_Wikipedia talks about the Content Analysis (seeing number of references removed, or content removed)-- which we did (with the few articles) and that is what we followed as it says "generally considered exempt from such requirements and does not require an IRB approval.". My advisor should be able to add more thoughts on it (I have requested him to reply on this thread).

Thanks, Sidd

...

On Thu, Aug 11, 2016 at 9:36 PM, siddhartha banerjee sidd2006@gmail.com wrote: As I have mentioned earlier, this is not the first work on article generation. This is one of the first work we know: https://people.csail.mit.edu/csauper/pubs/sauper-sm-thesis.pdf https://people.csail.mit.edu/regina/my_papers/wiki.pdf All these did not mention anything about human subjects as finally no personal information is used (about the person, who is deleting, etc). Nor did any reviewers/attendees in the conferences in this area question on this aspect. Also, https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2015-01-28/Recent... is relevant here as it talks about our previous work.

if "record of someone doing something" is relevant from human subjects point of view, any data on Wikipedia can be used to find the editors (if not the real person). For example: https://www.aaai.org/ocs/index.php/AAAI/AAAI11/paper/viewFile/3505/3968 https://alchemy.cs.washington.edu/papers/wu08/wu08.pdf I have met several researchers who work using data (revisions from Wikipedia) and nothin on IRB ever came up.

Nevertheless, as I said, if there are concrete rules, I think it would help the research community as a whole to know what can or cannot be done and also ask for permissions. I appreciate the suggestions that Stuart mentioned in a previous email abut experimenting on would be deleted or articles lacking sources. But, as of now we are not planning anything and if we do, we would for sure get in touch with Denny (who had a video chat with me before starting this thread) and would try to know the best ways of doing it.

I have asked my PhD advisor (other author on the paper) to check this thread and he will be able to give more inputs as I am not very qualified to comment on these aspects.

Thanks, Sidd

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Hoi, There is plenty of medical information that is thoroughly debunked that is finding its way into Wikidata on the strength of "it is included in another data source and that makes it ok". So when you talk about potential harm, when is it ok to include damaging information based on it blindly copying it from another "trusted" source and when do you consider it medical information that may do harm? Thanks, GerardM

On 12 August 2016 at 10:38, Stuart A. Yeates syeates@gmail.com wrote:

It's worth noting that https://en.wikipedia.org/wiki/Talonid appears (in so far as I can grasp what it's about) to at least verge on being medical information.

Medical information is subject to specific laws and an exceedingly brave place to start a research project like this. In terms of potential harm to research subjects (=readers of wikipedia) it pretty much hits the jackpot.

cheers stuart

-- ...let us be heard from red core to black sky

On Fri, Aug 12, 2016 at 8:22 PM, Kerry Raymond kerry.raymond@gmail.com wrote:

...

And to its policies

http://guru.psu.edu/policies/RP03.html

With particular reference to

"Intervention includes both physical procedures by which data are gathered (for example, venipuncture) and manipulations of the participant or the participant’s environment that are performed for research purposes."

Putting that articles into Wikipedia manipulated the environment of Wikipedia readers and editors.

Now I am not saying that huge harm was done, you would have to ask those who subsequently edited the articles (a known group) and those who read the articles (an unknown group) to find out if they are unhappy about what took place.

What I am saying is that if consideration had been given to the question who is impacted by this research plan, the maybe the research plan would have been redesigned to prevent the problem, and we would not have to have this conversation.

Kerry

Sent from my iPad

On 12 Aug 2016, at 6:08 PM, Kerry Raymond kerry.raymond@gmail.com wrote:

I draw attention to Penn State's IRB website

https://www.research.psu.edu/irb/submit

Sent from my iPad

On 12 Aug 2016, at 6:03 PM, Kerry Raymond kerry.raymond@gmail.com wrote:

I am asking you to share the documentation of the ethical clearance or exemption your institution would have required, not what people did or didn't say to you as part of conference reviewing or at conferences. Ethical clearance is a process that should have been undertaken before your research commenced, not when you are writing the paper or attending a conference. Are you saying you undertook the research without any consideration of the ethics? Does your university have no guidelines about this?

The Wikipedia guidelines about content analysis are not particularly relevant here. You were not analysing existing Wikipedia articles but injecting new articles of dubious quality into Wikipedia.

Nor is the data about individuals my point. If you wasted people's time reacting to the articles created, you did them harm. If people derived incorrect information from reading your articles, you did them harm. None of those people were aware they were part of your research experiment; that means they did not have informed consent in relation to choosing to participate in your experiment. You could have generated the articles and sought the opinions of readers and editors of Wikipedia on those articles without placing them into Wikipedia itself. That way would have enabled informed consent; others not wishing to take part would not be mislead into doing so.

Sent from my iPad

On 12 Aug 2016, at 3:24 PM, siddhartha banerjee sidd2006@gmail.com wrote:

I thought I should add this too as I missed it in the previous email. This link: https://en.wikipedia.org/wiki/Wikipedia:Ethically_rese arching_Wikipedia talks about the Content Analysis (seeing number of references removed, or content removed)-- which we did (with the few articles) and that is what we followed as it says "generally considered exempt from such requirements and does not require an IRB approval.". My advisor should be able to add more thoughts on it (I have requested him to reply on this thread).

Thanks, Sidd

On Thu, Aug 11, 2016 at 9:36 PM, siddhartha banerjee sidd2006@gmail.com wrote:

...

As I have mentioned earlier, this is not the first work on article generation. This is one of the first work we know: https://people.csail.mit.edu/csauper/pubs/sauper-sm-thesis.pdf https://people.csail.mit.edu/regina/my_papers/wiki.pdf All these did not mention anything about human subjects as finally no personal information is used (about the person, who is deleting, etc). Nor did any reviewers/attendees in the conferences in this area question on this aspect. Also, https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Sign post/2015-01-28/Recent_research is relevant here as it talks about our previous work.

if "record of someone doing something" is relevant from human subjects point of view, any data on Wikipedia can be used to find the editors (if not the real person). For example: https://www.aaai.org/ocs/index.php/AAAI/AAAI11/paper/viewFile/3505/3968 https://alchemy.cs.washington.edu/papers/wu08/wu08.pdf I have met several researchers who work using data (revisions from Wikipedia) and nothin on IRB ever came up.

Nevertheless, as I said, if there are concrete rules, I think it would help the research community as a whole to know what can or cannot be done and also ask for permissions. I appreciate the suggestions that Stuart mentioned in a previous email abut experimenting on would be deleted or articles lacking sources. But, as of now we are not planning anything and if we do, we would for sure get in touch with Denny (who had a video chat with me before starting this thread) and would try to know the best ways of doing it.

I have asked my PhD advisor (other author on the paper) to check this thread and he will be able to give more inputs as I am not very qualified to comment on these aspects.

Thanks, Sidd

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Hoi, That this is only relevant to English Wikipedia is new to me. The problem exists for any and all projects. When you think that erroneous information elsewhere does not have the potential to hurt people or that information from Wikidata may not be exposed in English Wikipedia, I am sorry for having an opinion. Thanks, GerardM

On 12 August 2016 at 11:16, Jonathan Cardy werespielchequers@gmail.com wrote:

This is medical information in the English Wikipedia so English Wikipedia policies apply. The rules of Medical sourcing are stricter than the default on Wikipedia, this is due to the extra risk involved in medical matters, as well as the ongoing problems with the alternative health lobby.

That some transgressions have not yet been detected does not justify a transgression as part of a research project.

Regards

Jonathan

On 12 Aug 2016, at 09:50, Gerard Meijssen gerard.meijssen@gmail.com wrote:

Hoi, There is plenty of medical information that is thoroughly debunked that is finding its way into Wikidata on the strength of "it is included in another data source and that makes it ok". So when you talk about potential harm, when is it ok to include damaging information based on it blindly copying it from another "trusted" source and when do you consider it medical information that may do harm? Thanks, GerardM

On 12 August 2016 at 10:38, Stuart A. Yeates syeates@gmail.com wrote:

...

It's worth noting that https://en.wikipedia.org/wiki/Talonid appears (in so far as I can grasp what it's about) to at least verge on being medical information.

Medical information is subject to specific laws and an exceedingly brave place to start a research project like this. In terms of potential harm to research subjects (=readers of wikipedia) it pretty much hits the jackpot.

cheers stuart

-- ...let us be heard from red core to black sky

On Fri, Aug 12, 2016 at 8:22 PM, Kerry Raymond kerry.raymond@gmail.com wrote:

...

And to its policies

http://guru.psu.edu/policies/RP03.html

With particular reference to

"Intervention includes both physical procedures by which data are gathered (for example, venipuncture) and manipulations of the participant or the participant’s environment that are performed for research purposes."

Putting that articles into Wikipedia manipulated the environment of Wikipedia readers and editors.

Now I am not saying that huge harm was done, you would have to ask those who subsequently edited the articles (a known group) and those who read the articles (an unknown group) to find out if they are unhappy about what took place.

What I am saying is that if consideration had been given to the question who is impacted by this research plan, the maybe the research plan would have been redesigned to prevent the problem, and we would not have to have this conversation.

Kerry

Sent from my iPad

On 12 Aug 2016, at 6:08 PM, Kerry Raymond kerry.raymond@gmail.com wrote:

I draw attention to Penn State's IRB website

https://www.research.psu.edu/irb/submit

Sent from my iPad

On 12 Aug 2016, at 6:03 PM, Kerry Raymond kerry.raymond@gmail.com wrote:

I am asking you to share the documentation of the ethical clearance or exemption your institution would have required, not what people did or didn't say to you as part of conference reviewing or at conferences. Ethical clearance is a process that should have been undertaken before your research commenced, not when you are writing the paper or attending a conference. Are you saying you undertook the research without any consideration of the ethics? Does your university have no guidelines about this?

The Wikipedia guidelines about content analysis are not particularly relevant here. You were not analysing existing Wikipedia articles but injecting new articles of dubious quality into Wikipedia.

Nor is the data about individuals my point. If you wasted people's time reacting to the articles created, you did them harm. If people derived incorrect information from reading your articles, you did them harm. None of those people were aware they were part of your research experiment; that means they did not have informed consent in relation to choosing to participate in your experiment. You could have generated the articles and sought the opinions of readers and editors of Wikipedia on those articles without placing them into Wikipedia itself. That way would have enabled informed consent; others not wishing to take part would not be mislead into doing so.

Sent from my iPad

On 12 Aug 2016, at 3:24 PM, siddhartha banerjee sidd2006@gmail.com wrote:

I thought I should add this too as I missed it in the previous email. This link: https://en.wikipedia.org/wiki/Wikipedia:Ethically_rese arching_Wikipedia talks about the Content Analysis (seeing number of references removed, or content removed)-- which we did (with the few articles) and that is what we followed as it says "generally considered exempt from such requirements and does not require an IRB approval.". My advisor should be able to add more thoughts on it (I have requested him to reply on this thread).

Thanks, Sidd

On Thu, Aug 11, 2016 at 9:36 PM, siddhartha banerjee <sidd2006@gmail.com

...

wrote:

...

As I have mentioned earlier, this is not the first work on article generation. This is one of the first work we know: https://people.csail.mit.edu/csauper/pubs/sauper-sm-thesis.pdf https://people.csail.mit.edu/regina/my_papers/wiki.pdf All these did not mention anything about human subjects as finally no personal information is used (about the person, who is deleting, etc). Nor did any reviewers/attendees in the conferences in this area question on this aspect. Also, https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Sign post/2015-01-28/Recent_research is relevant here as it talks about our previous work.

if "record of someone doing something" is relevant from human subjects point of view, any data on Wikipedia can be used to find the editors (if not the real person). For example: https://www.aaai.org/ocs/index.php/AAAI/AAAI11/paper/viewFile/3505/3968 https://alchemy.cs.washington.edu/papers/wu08/wu08.pdf I have met several researchers who work using data (revisions from Wikipedia) and nothin on IRB ever came up.

Nevertheless, as I said, if there are concrete rules, I think it would help the research community as a whole to know what can or cannot be done and also ask for permissions. I appreciate the suggestions that Stuart mentioned in a previous email abut experimenting on would be deleted or articles lacking sources. But, as of now we are not planning anything and if we do, we would for sure get in touch with Denny (who had a video chat with me before starting this thread) and would try to know the best ways of doing it.

I have asked my PhD advisor (other author on the paper) to check this thread and he will be able to give more inputs as I am not very qualified to comment on these aspects.

Thanks, Sidd

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

​Kerry,

I haven't had the time to read the paper itself, but regarding your comments on the need for informed consent, I would like to point out that, at least from what I have gleaned in this thread so far, it seems to me that consent could have probably been waived. Let me quote the relevant regulations here (cf. CFR 46.116(d) http://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr-46/#46.116 ):​

(d) An IRB may approve a consent procedure which does not include, or which

alters, some or all of the elements of informed consent set forth in this section, or waive the requirements to obtain informed consent provided the IRB finds and documents that:

(1) The research involves no more than minimal risk to the subjects;

(2) The waiver or alteration will not adversely affect the rights and welfare of the subjects;

(3) The research could not practicably be carried out without the waiver or alteration; and

(4) Whenever appropriate, the subjects will be provided with additional pertinent information after participation.

I think that given the few article titles seen so far, the research did involve no more than minimal risk to users and editors. Requiring them to ask informed consent from *every* person that came across to those pages without assistance from the developers seems unfeasible; and waiving consent in this case does not seem to adversely affect the rights and welfare of subjects either. Ditto for follow-up information.

Should they have applied to get exempt status from their IRB (i.e. essentially to get a stamp of approval on what I just said)? Yes, they should have. This doesn't change the nature of their research though, which is what matters to the present discussion.

Could they have put the articles in another namespace instead of the main one? Yes, but that is a question of being considerate to other users/editors, not about whether the research is legitimate/ethical or not.

Best

Giovanni

Giovanni Luca Ciampaglia http://glciampaglia.com *∙* Assistant Research Scientist, Indiana University

On Fri, Aug 12, 2016 at 4:22 AM, Kerry Raymond kerry.raymond@gmail.com wrote:

And to its policies

http://guru.psu.edu/policies/RP03.html

With particular reference to

"Intervention includes both physical procedures by which data are gathered (for example, venipuncture) and manipulations of the participant or the participant’s environment that are performed for research purposes."

Putting that articles into Wikipedia manipulated the environment of Wikipedia readers and editors.

Now I am not saying that huge harm was done, you would have to ask those who subsequently edited the articles (a known group) and those who read the articles (an unknown group) to find out if they are unhappy about what took place.

What I am saying is that if consideration had been given to the question who is impacted by this research plan, the maybe the research plan would have been redesigned to prevent the problem, and we would not have to have this conversation.

Kerry

Sent from my iPad

On 12 Aug 2016, at 6:08 PM, Kerry Raymond kerry.raymond@gmail.com wrote:

I draw attention to Penn State's IRB website

https://www.research.psu.edu/irb/submit

Sent from my iPad

On 12 Aug 2016, at 6:03 PM, Kerry Raymond kerry.raymond@gmail.com wrote:

I am asking you to share the documentation of the ethical clearance or exemption your institution would have required, not what people did or didn't say to you as part of conference reviewing or at conferences. Ethical clearance is a process that should have been undertaken before your research commenced, not when you are writing the paper or attending a conference. Are you saying you undertook the research without any consideration of the ethics? Does your university have no guidelines about this?

The Wikipedia guidelines about content analysis are not particularly relevant here. You were not analysing existing Wikipedia articles but injecting new articles of dubious quality into Wikipedia.

Nor is the data about individuals my point. If you wasted people's time reacting to the articles created, you did them harm. If people derived incorrect information from reading your articles, you did them harm. None of those people were aware they were part of your research experiment; that means they did not have informed consent in relation to choosing to participate in your experiment. You could have generated the articles and sought the opinions of readers and editors of Wikipedia on those articles without placing them into Wikipedia itself. That way would have enabled informed consent; others not wishing to take part would not be mislead into doing so.

Sent from my iPad

On 12 Aug 2016, at 3:24 PM, siddhartha banerjee sidd2006@gmail.com wrote:

I thought I should add this too as I missed it in the previous email. This link: https://en.wikipedia.org/wiki/Wikipedia:Ethically_ researching_Wikipedia talks about the Content Analysis (seeing number of references removed, or content removed)-- which we did (with the few articles) and that is what we followed as it says "generally considered exempt from such requirements and does not require an IRB approval.". My advisor should be able to add more thoughts on it (I have requested him to reply on this thread).

Thanks, Sidd

On Thu, Aug 11, 2016 at 9:36 PM, siddhartha banerjee sidd2006@gmail.com wrote:

...

As I have mentioned earlier, this is not the first work on article generation. This is one of the first work we know: https://people.csail.mit.edu/csauper/pubs/sauper-sm-thesis.pdf https://people.csail.mit.edu/regina/my_papers/wiki.pdf All these did not mention anything about human subjects as finally no personal information is used (about the person, who is deleting, etc). Nor did any reviewers/attendees in the conferences in this area question on this aspect. Also, https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Sign post/2015-01-28/Recent_research is relevant here as it talks about our previous work.

if "record of someone doing something" is relevant from human subjects point of view, any data on Wikipedia can be used to find the editors (if not the real person). For example: https://www.aaai.org/ocs/index.php/AAAI/AAAI11/paper/viewFile/3505/3968 https://alchemy.cs.washington.edu/papers/wu08/wu08.pdf I have met several researchers who work using data (revisions from Wikipedia) and nothin on IRB ever came up.

Nevertheless, as I said, if there are concrete rules, I think it would help the research community as a whole to know what can or cannot be done and also ask for permissions. I appreciate the suggestions that Stuart mentioned in a previous email abut experimenting on would be deleted or articles lacking sources. But, as of now we are not planning anything and if we do, we would for sure get in touch with Denny (who had a video chat with me before starting this thread) and would try to know the best ways of doing it.

I have asked my PhD advisor (other author on the paper) to check this thread and he will be able to give more inputs as I am not very qualified to comment on these aspects.

Thanks, Sidd

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Yes, the *IRB* might have chosen to waive the requirement of consent, but it appears the IRB was never given the opportunity to review the proposed research. The student researcher and/or their advisor is not in a position to make that decision on the IRB’s behalf by not submitting an application.

And as per (3), it would appear that recruiting Wikipedia readers and editors to review the generated articles could have been a simple alternative to placing them in Wikipedia main space and avoided the problem being discussed. Perhaps if the application had been properly presented to the IRB, that might have been the outcome.

As a Dean of Research in an IT faculty (prior to my retirement), I am well aware that research students (and sometimes their advisors/supervisors) often take the view that “I’m doing research on software, not people, I don’t need to worry about ethics”. I agree there is no ethics issue in relation to the automatic construction of Wikipedia-like articles, but there is an issue about putting them into Wikipedia mainspace and/or its processes to observe the reaction to them (which did not seem to occur in the prior research mentioned).

Kerry

From: Wiki-research-l [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Giovanni Luca Ciampaglia Sent: Saturday, 13 August 2016 12:57 AM To: Research into Wikimedia content and communities wiki-research-l@lists.wikimedia.org Subject: Re: [Wiki-research-l] Research on automatically created articles

​Kerry,

I haven't had the time to read the paper itself, but regarding your comments on the need for informed consent, I would like to point out that, at least from what I have gleaned in this thread so far, it seems to me that consent could have probably been waived. Let me quote the relevant regulations here (cf. http://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr-46/#46.116 CFR 46.116(d)):​

(d) An IRB may approve a consent procedure which does not include, or which alters, some or all of the elements of informed consent set forth in this section, or waive the requirements to obtain informed consent provided the IRB finds and documents that:

(1) The research involves no more than minimal risk to the subjects;

(2) The waiver or alteration will not adversely affect the rights and welfare of the subjects;

(3) The research could not practicably be carried out without the waiver or alteration; and

(4) Whenever appropriate, the subjects will be provided with additional pertinent information after participation.

I think that given the few article titles seen so far, the research did involve no more than minimal risk to users and editors. Requiring them to ask informed consent from *every* person that came across to those pages without assistance from the developers seems unfeasible; and waiving consent in this case does not seem to adversely affect the rights and welfare of subjects either. Ditto for follow-up information.

Should they have applied to get exempt status from their IRB (i.e. essentially to get a stamp of approval on what I just said)? Yes, they should have. This doesn't change the nature of their research though, which is what matters to the present discussion.

Could they have put the articles in another namespace instead of the main one? Yes, but that is a question of being considerate to other users/editors, not about whether the research is legitimate/ethical or not.

Best

Giovanni

http://glciampaglia.com Giovanni Luca Ciampaglia ∙ Assistant Research Scientist, Indiana University

On Fri, Aug 12, 2016 at 4:22 AM, Kerry Raymond <kerry.raymond@gmail.com mailto:kerry.raymond@gmail.com > wrote:

And to its policies

http://guru.psu.edu/policies/RP03.html

With particular reference to

"Intervention includes both physical procedures by which data are gathered (for example, venipuncture) and manipulations of the participant or the participant’s environment that are performed for research purposes."

Putting that articles into Wikipedia manipulated the environment of Wikipedia readers and editors.

Now I am not saying that huge harm was done, you would have to ask those who subsequently edited the articles (a known group) and those who read the articles (an unknown group) to find out if they are unhappy about what took place.

What I am saying is that if consideration had been given to the question who is impacted by this research plan, the maybe the research plan would have been redesigned to prevent the problem, and we would not have to have this conversation.

Kerry

Sent from my iPad

On 12 Aug 2016, at 6:08 PM, Kerry Raymond <kerry.raymond@gmail.com mailto:kerry.raymond@gmail.com > wrote:

I draw attention to Penn State's IRB website

https://www.research.psu.edu/irb/submit

Sent from my iPad

On 12 Aug 2016, at 6:03 PM, Kerry Raymond <kerry.raymond@gmail.com mailto:kerry.raymond@gmail.com > wrote:

I am asking you to share the documentation of the ethical clearance or exemption your institution would have required, not what people did or didn't say to you as part of conference reviewing or at conferences. Ethical clearance is a process that should have been undertaken before your research commenced, not when you are writing the paper or attending a conference. Are you saying you undertook the research without any consideration of the ethics? Does your university have no guidelines about this?

The Wikipedia guidelines about content analysis are not particularly relevant here. You were not analysing existing Wikipedia articles but injecting new articles of dubious quality into Wikipedia.

Nor is the data about individuals my point. If you wasted people's time reacting to the articles created, you did them harm. If people derived incorrect information from reading your articles, you did them harm. None of those people were aware they were part of your research experiment; that means they did not have informed consent in relation to choosing to participate in your experiment. You could have generated the articles and sought the opinions of readers and editors of Wikipedia on those articles without placing them into Wikipedia itself. That way would have enabled informed consent; others not wishing to take part would not be mislead into doing so.

Sent from my iPad

On 12 Aug 2016, at 3:24 PM, siddhartha banerjee <sidd2006@gmail.com mailto:sidd2006@gmail.com > wrote:

I thought I should add this too as I missed it in the previous email.

This link: https://en.wikipedia.org/wiki/Wikipedia:Ethically_researching_Wikipedia

talks about the Content Analysis (seeing number of references removed, or content removed)-- which we did (with the few articles) and that is what we followed as it says "generally considered exempt from such requirements and does not require an IRB approval.".

My advisor should be able to add more thoughts on it (I have requested him to reply on this thread).

Thanks,

Sidd

On Thu, Aug 11, 2016 at 9:36 PM, siddhartha banerjee <sidd2006@gmail.com mailto:sidd2006@gmail.com > wrote:

As I have mentioned earlier, this is not the first work on article generation. This is one of the first work we know: https://people.csail.mit.edu/csauper/pubs/sauper-sm-thesis.pdf

https://people.csail.mit.edu/regina/my_papers/wiki.pdf

All these did not mention anything about human subjects as finally no personal information is used (about the person, who is deleting, etc). Nor did any reviewers/attendees in the conferences in this area question on this aspect.

Also, https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2015-01-28/Recent... is relevant here as it talks about our previous work.

if "record of someone doing something" is relevant from human subjects point of view, any data on Wikipedia can be used to find the editors (if not the real person). For example:

https://www.aaai.org/ocs/index.php/AAAI/AAAI11/paper/viewFile/3505/3968

https://alchemy.cs.washington.edu/papers/wu08/wu08.pdf

I have met several researchers who work using data (revisions from Wikipedia) and nothin on IRB ever came up.

Nevertheless, as I said, if there are concrete rules, I think it would help the research community as a whole to know what can or cannot be done and also ask for permissions.

I appreciate the suggestions that Stuart mentioned in a previous email abut experimenting on would be deleted or articles lacking sources. But, as of now we are not planning anything and if we do, we would for sure get in touch with Denny (who had a video chat with me before starting this thread) and would try to know the best ways of doing it.

I have asked my PhD advisor (other author on the paper) to check this thread and he will be able to give more inputs as I am not very qualified to comment on these aspects.

Thanks,

Sidd

_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org mailto:Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org mailto:Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l