Hey folks,
This proposal is an important milestone for our subject recruitment processes, since it represents the first mass recruitment request (200-300 responses needed). I'm hoping to either show a high level of support with this poll or discover what problems still need to be dealt with.
I'd like to close the poll by *Wednesday @ noon UTC*. Please make sure to chime in.
See poll: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...
-Aaron
I was hoping to close this poll hours ago, but we only have three members of RCom participating (thanks Yaroslav and Steven!).
It is absolutely crucial that if we end up technically approving this study methodology that such approval actually reflects the consensus of RCom members.
For your benefit, I'll summarize the proposed plan:
A request to participate in a *survey* about enforcing conformance with community/group outcomes *needs 200-300 responses* from general Wikipedia editors. Invitations to take the survey will be posted an editors' User_talk pages. *A pilot set of 15 requests will be posted immediately*following approval from RCom to test for problems and determine the expected response rate. Afterwards, *up to 500 User_talk postings* will be made (depending on response rate) to illicit enough responses to give statistical confidence.
This is the first proposed project of this scale that we are reviewing for approval so I really want to make sure we are doing it right.
-Aaron
On Mon, Nov 14, 2011 at 8:49 AM, Aaron Halfaker aaron.halfaker@gmail.comwrote:
Hey folks,
This proposal is an important milestone for our subject recruitment processes, since it represents the first mass recruitment request (200-300 responses needed). I'm hoping to either show a high level of support with this poll or discover what problems still need to be dealt with.
I'd like to close the poll by *Wednesday @ noon UTC*. Please make sure to chime in.
See poll: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...
-Aaron
Ok Aaron, I am on it. I don't see any problem with this research and my vote to go on with it (changed on the project's talk page already made).
BTW: "...since it represents the first mass recruitment request (200-300 responses needed)." Q: what is "massive" in the recruitment request for 200-300 respondents online?
Best, Goran
On Wed, Nov 16, 2011 at 9:30 PM, Aaron Halfaker aaron.halfaker@gmail.comwrote:
I was hoping to close this poll hours ago, but we only have three members of RCom participating (thanks Yaroslav and Steven!).
It is absolutely crucial that if we end up technically approving this study methodology that such approval actually reflects the consensus of RCom members.
For your benefit, I'll summarize the proposed plan:
A request to participate in a *survey* about enforcing conformance with community/group outcomes *needs 200-300 responses* from general Wikipedia editors. Invitations to take the survey will be posted an editors' User_talk pages. *A pilot set of 15 requests will be posted immediately*following approval from RCom to test for problems and determine the expected response rate. Afterwards, *up to 500 User_talk postings* will be made (depending on response rate) to illicit enough responses to give statistical confidence.
This is the first proposed project of this scale that we are reviewing for approval so I really want to make sure we are doing it right.
-Aaron
On Mon, Nov 14, 2011 at 8:49 AM, Aaron Halfaker aaron.halfaker@gmail.comwrote:
Hey folks,
This proposal is an important milestone for our subject recruitment processes, since it represents the first mass recruitment request (200-300 responses needed). I'm hoping to either show a high level of support with this poll or discover what problems still need to be dealt with.
I'd like to close the poll by *Wednesday @ noon UTC*. Please make sure to chime in.
See poll: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...
-Aaron
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
That's massive as far as editors are concerned in enwiki. The closest applicable policy (http://en.wikipedia.org/wiki/Wikipedia:CANVAS) only makes general statements about what might be considered a "Mass posting", but I'm pretty sure that Wikipedians rarely expect to see a user go from a single digit edit count to 500 from User_talk page postings in the course of a day.
Although I don't expect this to be truly disruptive to editors, I expect that many alarms will go off when the talk postings begin to be posted since the activity will be so unusual.
-Aaron
On Wed, Nov 16, 2011 at 2:41 PM, Goran Milovanovic < goran.s.milovanovic@gmail.com> wrote:
Ok Aaron, I am on it. I don't see any problem with this research and my vote to go on with it (changed on the project's talk page already made).
BTW: "...since it represents the first mass recruitment request (200-300 responses needed)." Q: what is "massive" in the recruitment request for 200-300 respondents online?
Best, Goran
On Wed, Nov 16, 2011 at 9:30 PM, Aaron Halfaker aaron.halfaker@gmail.comwrote:
I was hoping to close this poll hours ago, but we only have three members of RCom participating (thanks Yaroslav and Steven!).
It is absolutely crucial that if we end up technically approving this study methodology that such approval actually reflects the consensus of RCom members.
For your benefit, I'll summarize the proposed plan:
A request to participate in a *survey* about enforcing conformance with community/group outcomes *needs 200-300 responses* from general Wikipedia editors. Invitations to take the survey will be posted an editors' User_talk pages. *A pilot set of 15 requests will be posted immediately* following approval from RCom to test for problems and determine the expected response rate. Afterwards, *up to 500 User_talk postings* will be made (depending on response rate) to illicit enough responses to give statistical confidence.
This is the first proposed project of this scale that we are reviewing for approval so I really want to make sure we are doing it right.
-Aaron
On Mon, Nov 14, 2011 at 8:49 AM, Aaron Halfaker <aaron.halfaker@gmail.com
wrote:
Hey folks,
This proposal is an important milestone for our subject recruitment processes, since it represents the first mass recruitment request (200-300 responses needed). I'm hoping to either show a high level of support with this poll or discover what problems still need to be dealt with.
I'd like to close the poll by *Wednesday @ noon UTC*. Please make sure to chime in.
See poll: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...
-Aaron
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
--
"Truth is much too complicated to allow anything but approximations." :: John von Neumann
http://www.milovanovicresearch.com
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
Thx Aaron - got it. Ok, let's see how this one goes.
BTW, it's been a while since I have started planning a social science study on Wikipiedians, and it might be possible that at some point I will need subject recruiting. I am at the beginning of the process so we're talking months here before I actually decide to conduct the study. Does my membership in RComm interferes with doing a Wikipedia related research on my own? Please advise.
Best, Goran
On Wed, Nov 16, 2011 at 9:54 PM, Aaron Halfaker aaron.halfaker@gmail.comwrote:
That's massive as far as editors are concerned in enwiki. The closest applicable policy (http://en.wikipedia.org/wiki/Wikipedia:CANVAS) only makes general statements about what might be considered a "Mass posting", but I'm pretty sure that Wikipedians rarely expect to see a user go from a single digit edit count to 500 from User_talk page postings in the course of a day.
Although I don't expect this to be truly disruptive to editors, I expect that many alarms will go off when the talk postings begin to be posted since the activity will be so unusual.
-Aaron
On Wed, Nov 16, 2011 at 2:41 PM, Goran Milovanovic < goran.s.milovanovic@gmail.com> wrote:
Ok Aaron, I am on it. I don't see any problem with this research and my vote to go on with it (changed on the project's talk page already made).
BTW: "...since it represents the first mass recruitment request (200-300 responses needed)." Q: what is "massive" in the recruitment request for 200-300 respondents online?
Best, Goran
On Wed, Nov 16, 2011 at 9:30 PM, Aaron Halfaker <aaron.halfaker@gmail.com
wrote:
I was hoping to close this poll hours ago, but we only have three members of RCom participating (thanks Yaroslav and Steven!).
It is absolutely crucial that if we end up technically approving this study methodology that such approval actually reflects the consensus of RCom members.
For your benefit, I'll summarize the proposed plan:
A request to participate in a *survey* about enforcing conformance with community/group outcomes *needs 200-300 responses* from general Wikipedia editors. Invitations to take the survey will be posted an editors' User_talk pages. *A pilot set of 15 requests will be posted immediately* following approval from RCom to test for problems and determine the expected response rate. Afterwards, *up to 500 User_talk postings* will be made (depending on response rate) to illicit enough responses to give statistical confidence.
This is the first proposed project of this scale that we are reviewing for approval so I really want to make sure we are doing it right.
-Aaron
On Mon, Nov 14, 2011 at 8:49 AM, Aaron Halfaker < aaron.halfaker@gmail.com> wrote:
Hey folks,
This proposal is an important milestone for our subject recruitment processes, since it represents the first mass recruitment request (200-300 responses needed). I'm hoping to either show a high level of support with this poll or discover what problems still need to be dealt with.
I'd like to close the poll by *Wednesday @ noon UTC*. Please make sure to chime in.
See poll: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...
-Aaron
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
--
"Truth is much too complicated to allow anything but approximations." :: John von Neumann
http://www.milovanovicresearch.com
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
I think it will be important that you don't participate in your own review as a member of RCom in order to avoid a COI. Otherwise I don't this as a problem. Anyone else have an opinion on the matter?
-Aaron
On Wed, Nov 16, 2011 at 3:05 PM, Goran Milovanovic < goran.s.milovanovic@gmail.com> wrote:
Thx Aaron - got it. Ok, let's see how this one goes.
BTW, it's been a while since I have started planning a social science study on Wikipiedians, and it might be possible that at some point I will need subject recruiting. I am at the beginning of the process so we're talking months here before I actually decide to conduct the study. Does my membership in RComm interferes with doing a Wikipedia related research on my own? Please advise.
Best, Goran
On Wed, Nov 16, 2011 at 9:54 PM, Aaron Halfaker aaron.halfaker@gmail.comwrote:
That's massive as far as editors are concerned in enwiki. The closest applicable policy (http://en.wikipedia.org/wiki/Wikipedia:CANVAS) only makes general statements about what might be considered a "Mass posting", but I'm pretty sure that Wikipedians rarely expect to see a user go from a single digit edit count to 500 from User_talk page postings in the course of a day.
Although I don't expect this to be truly disruptive to editors, I expect that many alarms will go off when the talk postings begin to be posted since the activity will be so unusual.
-Aaron
On Wed, Nov 16, 2011 at 2:41 PM, Goran Milovanovic < goran.s.milovanovic@gmail.com> wrote:
Ok Aaron, I am on it. I don't see any problem with this research and my vote to go on with it (changed on the project's talk page already made).
BTW: "...since it represents the first mass recruitment request (200-300 responses needed)." Q: what is "massive" in the recruitment request for 200-300 respondents online?
Best, Goran
On Wed, Nov 16, 2011 at 9:30 PM, Aaron Halfaker < aaron.halfaker@gmail.com> wrote:
I was hoping to close this poll hours ago, but we only have three members of RCom participating (thanks Yaroslav and Steven!).
It is absolutely crucial that if we end up technically approving this study methodology that such approval actually reflects the consensus of RCom members.
For your benefit, I'll summarize the proposed plan:
A request to participate in a *survey* about enforcing conformance with community/group outcomes *needs 200-300 responses* from general Wikipedia editors. Invitations to take the survey will be posted an editors' User_talk pages. *A pilot set of 15 requests will be posted immediately* following approval from RCom to test for problems and determine the expected response rate. Afterwards, *up to 500 User_talk postings* will be made (depending on response rate) to illicit enough responses to give statistical confidence.
This is the first proposed project of this scale that we are reviewing for approval so I really want to make sure we are doing it right.
-Aaron
On Mon, Nov 14, 2011 at 8:49 AM, Aaron Halfaker < aaron.halfaker@gmail.com> wrote:
Hey folks,
This proposal is an important milestone for our subject recruitment processes, since it represents the first mass recruitment request (200-300 responses needed). I'm hoping to either show a high level of support with this poll or discover what problems still need to be dealt with.
I'd like to close the poll by *Wednesday @ noon UTC*. Please make sure to chime in.
See poll: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...
-Aaron
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
--
"Truth is much too complicated to allow anything but approximations." :: John von Neumann
http://www.milovanovicresearch.com
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
--
"Truth is much too complicated to allow anything but approximations." :: John von Neumann
http://www.milovanovicresearch.com
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
yes, that's the rule we tacitly agreed upon (and applied a few times) in the past
Dario
On Nov 16, 2011, at 1:19 PM, Aaron Halfaker wrote:
I think it will be important that you don't participate in your own review as a member of RCom in order to avoid a COI. Otherwise I don't this as a problem. Anyone else have an opinion on the matter?
-Aaron
On Wed, Nov 16, 2011 at 3:05 PM, Goran Milovanovic goran.s.milovanovic@gmail.com wrote: Thx Aaron - got it. Ok, let's see how this one goes.
BTW, it's been a while since I have started planning a social science study on Wikipiedians, and it might be possible that at some point I will need subject recruiting. I am at the beginning of the process so we're talking months here before I actually decide to conduct the study. Does my membership in RComm interferes with doing a Wikipedia related research on my own? Please advise.
Best, Goran
On Wed, Nov 16, 2011 at 9:54 PM, Aaron Halfaker aaron.halfaker@gmail.com wrote: That's massive as far as editors are concerned in enwiki. The closest applicable policy (http://en.wikipedia.org/wiki/Wikipedia:CANVAS) only makes general statements about what might be considered a "Mass posting", but I'm pretty sure that Wikipedians rarely expect to see a user go from a single digit edit count to 500 from User_talk page postings in the course of a day.
Although I don't expect this to be truly disruptive to editors, I expect that many alarms will go off when the talk postings begin to be posted since the activity will be so unusual.
-Aaron
On Wed, Nov 16, 2011 at 2:41 PM, Goran Milovanovic goran.s.milovanovic@gmail.com wrote: Ok Aaron, I am on it. I don't see any problem with this research and my vote to go on with it (changed on the project's talk page already made).
BTW: "...since it represents the first mass recruitment request (200-300 responses needed)." Q: what is "massive" in the recruitment request for 200-300 respondents online?
Best, Goran
On Wed, Nov 16, 2011 at 9:30 PM, Aaron Halfaker aaron.halfaker@gmail.com wrote: I was hoping to close this poll hours ago, but we only have three members of RCom participating (thanks Yaroslav and Steven!).
It is absolutely crucial that if we end up technically approving this study methodology that such approval actually reflects the consensus of RCom members.
For your benefit, I'll summarize the proposed plan:
A request to participate in a survey about enforcing conformance with community/group outcomes needs 200-300 responses from general Wikipedia editors. Invitations to take the survey will be posted an editors' User_talk pages. A pilot set of 15 requests will be posted immediately following approval from RCom to test for problems and determine the expected response rate. Afterwards, up to 500 User_talk postings will be made (depending on response rate) to illicit enough responses to give statistical confidence.
This is the first proposed project of this scale that we are reviewing for approval so I really want to make sure we are doing it right.
-Aaron
On Mon, Nov 14, 2011 at 8:49 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote: Hey folks,
This proposal is an important milestone for our subject recruitment processes, since it represents the first mass recruitment request (200-300 responses needed). I'm hoping to either show a high level of support with this poll or discover what problems still need to be dealt with.
I'd like to close the poll by Wednesday @ noon UTC. Please make sure to chime in.
See poll: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...
-Aaron
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
--
"Truth is much too complicated to allow anything but approximations." :: John von Neumann
http://www.milovanovicresearch.com
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
--
"Truth is much too complicated to allow anything but approximations." :: John von Neumann
http://www.milovanovicresearch.com
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
Sorry but I'm at the gate in Heathrow and won't have time to get involved in this for another week.
WSC
On 16 November 2011 20:30, Aaron Halfaker aaron.halfaker@gmail.com wrote:
I was hoping to close this poll hours ago, but we only have three members of RCom participating (thanks Yaroslav and Steven!).
It is absolutely crucial that if we end up technically approving this study methodology that such approval actually reflects the consensus of RCom members.
For your benefit, I'll summarize the proposed plan:
A request to participate in a *survey* about enforcing conformance with community/group outcomes *needs 200-300 responses* from general Wikipedia editors. Invitations to take the survey will be posted an editors' User_talk pages. *A pilot set of 15 requests will be posted immediately*following approval from RCom to test for problems and determine the expected response rate. Afterwards, *up to 500 User_talk postings* will be made (depending on response rate) to illicit enough responses to give statistical confidence.
This is the first proposed project of this scale that we are reviewing for approval so I really want to make sure we are doing it right.
-Aaron
On Mon, Nov 14, 2011 at 8:49 AM, Aaron Halfaker aaron.halfaker@gmail.comwrote:
Hey folks,
This proposal is an important milestone for our subject recruitment processes, since it represents the first mass recruitment request (200-300 responses needed). I'm hoping to either show a high level of support with this poll or discover what problems still need to be dealt with.
I'd like to close the poll by *Wednesday @ noon UTC*. Please make sure to chime in.
See poll: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...
-Aaron
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
ok, here's my dispassionate opinion. I post on the list because the problems I refer to below are not specific to this project but to many other requests.
We have been receiving a *lot* of SR requests from students over the last few months and I think we should start deciding more aggressively what research is likely to have an impact and what research won't produce any major tangible results.
I really like the topic of the Anonymity and Conformity study but I several concerns with the solidity of the current proposal:
• we will be approving for the first time some kind of large-scale recruitment approach via user talk pages for a student project: this is something we've never done before and we should only do it if there's a good reason.
• the advisor of the proponent doesn't seem to be involved at all in this project and is not even named in the proposal. Aaron asked the proponent to share the name of his supervisor in September, but he hasn't done so (yet?)
• the proponent says that no funding is supporting this research and that this study is "conducted with the author's own efforts"
• no one else other than the applicant will be implied in the data collection and analysis and the proponent doesn't seem to have an actual research record
• there is no trace in this proposal of an approval by an ethics committee. The proponent says that this is not applicable (and it's true that IRB policy is very different between the US and other countries), but some official record would help us assess the credibility of the proposal.
for these reasons, I am hesitant whether we should blindly approve this request. I would be happy to give my approval conditionally on having (at least) the student's supervisor involved. I actually think this would be a sensible rule to apply to *all* SR proposals submitted by students.
I hope it's clear that I don't want to shut down research for anybody but top researchers with big grants, but we also cannot afford spending time and effort and community attention for projects of an unclear scientific value or interest.
I'd love to hear your thoughts on this
Dario
On Nov 16, 2011, at 12:30 PM, Aaron Halfaker wrote:
I was hoping to close this poll hours ago, but we only have three members of RCom participating (thanks Yaroslav and Steven!).
It is absolutely crucial that if we end up technically approving this study methodology that such approval actually reflects the consensus of RCom members.
For your benefit, I'll summarize the proposed plan:
A request to participate in a survey about enforcing conformance with community/group outcomes needs 200-300 responses from general Wikipedia editors. Invitations to take the survey will be posted an editors' User_talk pages. A pilot set of 15 requests will be posted immediately following approval from RCom to test for problems and determine the expected response rate. Afterwards, up to 500 User_talk postings will be made (depending on response rate) to illicit enough responses to give statistical confidence.
This is the first proposed project of this scale that we are reviewing for approval so I really want to make sure we are doing it right.
-Aaron
On Mon, Nov 14, 2011 at 8:49 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote: Hey folks,
This proposal is an important milestone for our subject recruitment processes, since it represents the first mass recruitment request (200-300 responses needed). I'm hoping to either show a high level of support with this poll or discover what problems still need to be dealt with.
I'd like to close the poll by Wednesday @ noon UTC. Please make sure to chime in.
See poll: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...
-Aaron
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
Well, when it comes to
"I would be happy to give my approval conditionally on having (at least) the student's supervisor involved. I actually think this would be a sensible rule to apply to *all* SR proposals submitted by students."
I have to say I could not agree, and here is the reason: we may stumble upon many academic environments where students, or PhD candidates, or young researchers in general are in fact more competent in specific areas of research then their supervisors. Of course, this is not a common situation, but if we talk about Internet related research, where new concepts, new ideas, and new methods are introduced every now and then, we are talking about a field where the above described asymmetry in competences is likely to appear.
Personal experience: between 2002. and 2005. a research team that I have led in Belgrade, Serbia, published four books on Internet behavior, Internet usage, attitudes towards the Internet and related topics, of which none were peer-reviewed in Serbia. Why? They were all supported by standard methodological, theoretical and statistical instruments of social sciences. Most of the team members already had experience in publishing in peer-reviewed journals or presented at recognized conferences. Simply, because that were the first attempts to study anything related to Internet and society in my country at all, and there were no peer reviewers available. I am literary saying that back then we were to first to try to crawl for the relevant literature and references at all.
What I want to add is that most researchers who are into Internet related research are people who are into new ideas, looking for conceptual breakthroughs and new theoretical frameworks. Limiting their efforts by constrains related to the already bureaucratized relations in academia could end up in actually enlarging the pile of problems on the behalf of people who simply want to put their new ideas to test. Wikipedia was developed by people who decided to abandon the classical rules and try with something previously unimaginable: who would say it work to become of the landmarks of the Internet? I believe we should keep that spirit when it comes to research as well.
"I hope it's clear that I don't want to shut down research for anybody but top researchers with big grants, but we also cannot afford spending time and effort and community attention for projects of an unclear scientific value or interest."
I recognize the importance of the issue you're raising, but I do not think the selection criterion you are proposing will help us sort out the projects of clear from the projects of unclear value or interest. I believe, in the spirit of good faith, that task needs to remain with us completely, as well as with the members of the scientific community who we might contact and ask for an advice when we are not sure what to do about a particular project proposal.
Best, Goran
On Wed, Nov 16, 2011 at 11:03 PM, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
ok, here's my dispassionate opinion. I post on the list because the problems I refer to below are not specific to this project but to many other requests.
We have been receiving a *lot* of SR requests from students over the last few months and I think we should start deciding more aggressively what research is likely to have an impact and what research won't produce any major tangible results.
I really like the topic of the Anonymity and Conformity study but I several concerns with the solidity of the current proposal:
• we will be approving for the first time some kind of large-scale recruitment approach via user talk pages for a student project: this is something we've never done before and we should only do it if there's a good reason.
• the advisor of the proponent doesn't seem to be involved at all in this project and is not even named in the proposal. Aaron asked the proponent to share the name of his supervisor in September, but he hasn't done so (yet?)
• the proponent says that no funding is supporting this research and that this study is "conducted with the author's own efforts"
• no one else other than the applicant will be implied in the data collection and analysis and the proponent doesn't seem to have an actual research record
• there is no trace in this proposal of an approval by an ethics committee. The proponent says that this is not applicable (and it's true that IRB policy is very different between the US and other countries), but some official record would help us assess the credibility of the proposal.
for these reasons, I am hesitant whether we should blindly approve this request. I would be happy to give my approval conditionally on having (at least) the student's supervisor involved. I actually think this would be a sensible rule to apply to *all* SR proposals submitted by students.
I hope it's clear that I don't want to shut down research for anybody but top researchers with big grants, but we also cannot afford spending time and effort and community attention for projects of an unclear scientific value or interest.
I'd love to hear your thoughts on this
Dario
On Nov 16, 2011, at 12:30 PM, Aaron Halfaker wrote:
I was hoping to close this poll hours ago, but we only have three members of RCom participating (thanks Yaroslav and Steven!).
It is absolutely crucial that if we end up technically approving this study methodology that such approval actually reflects the consensus of RCom members.
For your benefit, I'll summarize the proposed plan:
A request to participate in a *survey* about enforcing conformance with community/group outcomes *needs 200-300 responses* from general Wikipedia editors. Invitations to take the survey will be posted an editors' User_talk pages. *A pilot set of 15 requests will be posted immediately*following approval from RCom to test for problems and determine the expected response rate. Afterwards, *up to 500 User_talk postings* will be made (depending on response rate) to illicit enough responses to give statistical confidence.
This is the first proposed project of this scale that we are reviewing for approval so I really want to make sure we are doing it right.
-Aaron
On Mon, Nov 14, 2011 at 8:49 AM, Aaron Halfaker aaron.halfaker@gmail.comwrote:
Hey folks,
This proposal is an important milestone for our subject recruitment processes, since it represents the first mass recruitment request (200-300 responses needed). I'm hoping to either show a high level of support with this poll or discover what problems still need to be dealt with.
I'd like to close the poll by *Wednesday @ noon UTC*. Please make sure to chime in.
See poll: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...
-Aaron
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
This is discussion is awesome. I'd like to propose two options for moving forward from this point:
1. Allow the study to continue and pick this issue up in parallel. - Pro: We don't delay the study of a researcher who has been waiting. - Con: We set a precedent that future researchers would (possibly) not be able to follow. 2. Delay this study until we can come to a general conclusion about our requirements for this scale of recruitment - Pro: We wait until approving anything until we have reached a consensus of what projects should be approved. - Con: Michael Tsikerdekis has to wait an undetermined amount of time before continuing with his work.
I'd like to capture our deliberations in a more public forum. What do you folks think about moving this discussion to the Wiki (whether it is under this project proposal or not)?
-Aaron
On Wed, Nov 16, 2011 at 4:28 PM, Goran Milovanovic < goran.s.milovanovic@gmail.com> wrote:
Well, when it comes to
"I would be happy to give my approval conditionally on having (at least) the student's supervisor involved. I actually think this would be a sensible rule to apply to *all* SR proposals submitted by students."
I have to say I could not agree, and here is the reason: we may stumble upon many academic environments where students, or PhD candidates, or young researchers in general are in fact more competent in specific areas of research then their supervisors. Of course, this is not a common situation, but if we talk about Internet related research, where new concepts, new ideas, and new methods are introduced every now and then, we are talking about a field where the above described asymmetry in competences is likely to appear.
Personal experience: between 2002. and 2005. a research team that I have led in Belgrade, Serbia, published four books on Internet behavior, Internet usage, attitudes towards the Internet and related topics, of which none were peer-reviewed in Serbia. Why? They were all supported by standard methodological, theoretical and statistical instruments of social sciences. Most of the team members already had experience in publishing in peer-reviewed journals or presented at recognized conferences. Simply, because that were the first attempts to study anything related to Internet and society in my country at all, and there were no peer reviewers available. I am literary saying that back then we were to first to try to crawl for the relevant literature and references at all.
What I want to add is that most researchers who are into Internet related research are people who are into new ideas, looking for conceptual breakthroughs and new theoretical frameworks. Limiting their efforts by constrains related to the already bureaucratized relations in academia could end up in actually enlarging the pile of problems on the behalf of people who simply want to put their new ideas to test. Wikipedia was developed by people who decided to abandon the classical rules and try with something previously unimaginable: who would say it work to become of the landmarks of the Internet? I believe we should keep that spirit when it comes to research as well.
"I hope it's clear that I don't want to shut down research for anybody but top researchers with big grants, but we also cannot afford spending time and effort and community attention for projects of an unclear scientific value or interest."
I recognize the importance of the issue you're raising, but I do not think the selection criterion you are proposing will help us sort out the projects of clear from the projects of unclear value or interest. I believe, in the spirit of good faith, that task needs to remain with us completely, as well as with the members of the scientific community who we might contact and ask for an advice when we are not sure what to do about a particular project proposal.
Best, Goran
On Wed, Nov 16, 2011 at 11:03 PM, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
ok, here's my dispassionate opinion. I post on the list because the problems I refer to below are not specific to this project but to many other requests.
We have been receiving a *lot* of SR requests from students over the last few months and I think we should start deciding more aggressively what research is likely to have an impact and what research won't produce any major tangible results.
I really like the topic of the Anonymity and Conformity study but I several concerns with the solidity of the current proposal:
• we will be approving for the first time some kind of large-scale recruitment approach via user talk pages for a student project: this is something we've never done before and we should only do it if there's a good reason.
• the advisor of the proponent doesn't seem to be involved at all in this project and is not even named in the proposal. Aaron asked the proponent to share the name of his supervisor in September, but he hasn't done so (yet?)
• the proponent says that no funding is supporting this research and that this study is "conducted with the author's own efforts"
• no one else other than the applicant will be implied in the data collection and analysis and the proponent doesn't seem to have an actual research record
• there is no trace in this proposal of an approval by an ethics committee. The proponent says that this is not applicable (and it's true that IRB policy is very different between the US and other countries), but some official record would help us assess the credibility of the proposal.
for these reasons, I am hesitant whether we should blindly approve this request. I would be happy to give my approval conditionally on having (at least) the student's supervisor involved. I actually think this would be a sensible rule to apply to *all* SR proposals submitted by students.
I hope it's clear that I don't want to shut down research for anybody but top researchers with big grants, but we also cannot afford spending time and effort and community attention for projects of an unclear scientific value or interest.
I'd love to hear your thoughts on this
Dario
On Nov 16, 2011, at 12:30 PM, Aaron Halfaker wrote:
I was hoping to close this poll hours ago, but we only have three members of RCom participating (thanks Yaroslav and Steven!).
It is absolutely crucial that if we end up technically approving this study methodology that such approval actually reflects the consensus of RCom members.
For your benefit, I'll summarize the proposed plan:
A request to participate in a *survey* about enforcing conformance with community/group outcomes *needs 200-300 responses* from general Wikipedia editors. Invitations to take the survey will be posted an editors' User_talk pages. *A pilot set of 15 requests will be posted immediately* following approval from RCom to test for problems and determine the expected response rate. Afterwards, *up to 500 User_talk postings* will be made (depending on response rate) to illicit enough responses to give statistical confidence.
This is the first proposed project of this scale that we are reviewing for approval so I really want to make sure we are doing it right.
-Aaron
On Mon, Nov 14, 2011 at 8:49 AM, Aaron Halfaker <aaron.halfaker@gmail.com
wrote:
Hey folks,
This proposal is an important milestone for our subject recruitment processes, since it represents the first mass recruitment request (200-300 responses needed). I'm hoping to either show a high level of support with this poll or discover what problems still need to be dealt with.
I'd like to close the poll by *Wednesday @ noon UTC*. Please make sure to chime in.
See poll: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...
-Aaron
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
--
"Truth is much too complicated to allow anything but approximations." :: John von Neumann
http://www.milovanovicresearch.com
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
Hi All, sorry just a quick note and offer my option. I 2nd Goran's comment about perhaps this is a PhD student/young researcher starting a project and usually we are more knowledgeable than our PIs...but if this is so it might be wise for us to know that the PI has in fact OKed this project and it has gone through another set of eyes than Michael and us. This is also so that we are aware that he has some type of guidance.
In response to Aaron's options, which is good to outline these: I would unfortunately vote for option 2 to delay the project?...Yes, it is disappointing to Michael, but that is research and this might give him the opportunity to really hone in the project. To do a bit more background research and address a few of our concerns. And just a quick question: he hasn't responded with who his PI is since Sept? It doesn't sound like he is too much in an hurry...(or am I getting projects/proposals mixed up?)
C
On 11/16/11 6:07 PM, Aaron Halfaker wrote:
This is discussion is awesome. I'd like to propose two options for moving forward from this point:
- Allow the study to continue and pick this issue up in parallel.
- Pro: We don't delay the study of a researcher who has been waiting.
- Con: We set a precedent that future researchers would (possibly) not be able to follow.
- Delay this study until we can come to a general conclusion about our requirements for this scale of recruitment
- Pro: We wait until approving anything until we have reached a consensus of what projects should be approved.
- Con: Michael Tsikerdekis has to wait an undetermined amount of time before continuing with his work.
I'd like to capture our deliberations in a more public forum. What do you folks think about moving this discussion to the Wiki (whether it is under this project proposal or not)?
-Aaron
On Wed, Nov 16, 2011 at 4:28 PM, Goran Milovanovic <goran.s.milovanovic@gmail.com mailto:goran.s.milovanovic@gmail.com> wrote:
Well, when it comes to "I would be happy to give my approval conditionally on having (at least) the student's supervisor involved. I actually think this would be a sensible rule to apply to *all* SR proposals submitted by students." I have to say I could not agree, and here is the reason: we may stumble upon many academic environments where students, or PhD candidates, or young researchers in general are in fact more competent in specific areas of research then their supervisors. Of course, this is not a common situation, but if we talk about Internet related research, where new concepts, new ideas, and new methods are introduced every now and then, we are talking about a field where the above described asymmetry in competences is likely to appear. Personal experience: between 2002. and 2005. a research team that I have led in Belgrade, Serbia, published four books on Internet behavior, Internet usage, attitudes towards the Internet and related topics, of which none were peer-reviewed in Serbia. Why? They were all supported by standard methodological, theoretical and statistical instruments of social sciences. Most of the team members already had experience in publishing in peer-reviewed journals or presented at recognized conferences. Simply, because that were the first attempts to study anything related to Internet and society in my country at all, and there were no peer reviewers available. I am literary saying that back then we were to first to try to crawl for the relevant literature and references at all. What I want to add is that most researchers who are into Internet related research are people who are into new ideas, looking for conceptual breakthroughs and new theoretical frameworks. Limiting their efforts by constrains related to the already bureaucratized relations in academia could end up in actually enlarging the pile of problems on the behalf of people who simply want to put their new ideas to test. Wikipedia was developed by people who decided to abandon the classical rules and try with something previously unimaginable: who would say it work to become of the landmarks of the Internet? I believe we should keep that spirit when it comes to research as well. "I hope it's clear that I don't want to shut down research for anybody but top researchers with big grants, but we also cannot afford spending time and effort and community attention for projects of an unclear scientific value or interest." I recognize the importance of the issue you're raising, but I do not think the selection criterion you are proposing will help us sort out the projects of clear from the projects of unclear value or interest. I believe, in the spirit of good faith, that task needs to remain with us completely, as well as with the members of the scientific community who we might contact and ask for an advice when we are not sure what to do about a particular project proposal. Best, Goran On Wed, Nov 16, 2011 at 11:03 PM, Dario Taraborelli <dtaraborelli@wikimedia.org <mailto:dtaraborelli@wikimedia.org>> wrote: ok, here's my dispassionate opinion. I post on the list because the problems I refer to below are not specific to this project but to many other requests. We have been receiving a *lot* of SR requests from students over the last few months and I think we should start deciding more aggressively what research is likely to have an impact and what research won't produce any major tangible results. I really like the topic of the Anonymity and Conformity study but I several concerns with the solidity of the current proposal: . we will be approving for the first time some kind of large-scale recruitment approach via user talk pages for a student project: this is something we've never done before and we should only do it if there's a good reason. . the advisor of the proponent doesn't seem to be involved at all in this project and is not even named in the proposal. Aaron asked the proponent to share the name of his supervisor in September, but he hasn't done so (yet?) . the proponent says that no funding is supporting this research and that this study is "conducted with the author's own efforts" . no one else other than the applicant will be implied in the data collection and analysis and the proponent doesn't seem to have an actual research record . there is no trace in this proposal of an approval by an ethics committee. The proponent says that this is not applicable (and it's true that IRB policy is very different between the US and other countries), but some official record would help us assess the credibility of the proposal. for these reasons, I am hesitant whether we should blindly approve this request. I would be happy to give my approval conditionally on having (at least) the student's supervisor involved. I actually think this would be a sensible rule to apply to *all* SR proposals submitted by students. I hope it's clear that I don't want to shut down research for anybody but top researchers with big grants, but we also cannot afford spending time and effort and community attention for projects of an unclear scientific value or interest. I'd love to hear your thoughts on this Dario On Nov 16, 2011, at 12:30 PM, Aaron Halfaker wrote:
I was hoping to close this poll hours ago, but we only have three members of RCom participating (thanks Yaroslav and Steven!). It is absolutely crucial that if we end up technically approving this study methodology that such approval actually reflects the consensus of RCom members. For your benefit, I'll summarize the proposed plan: A request to participate in a *survey* about enforcing conformance with community/group outcomes *needs 200-300 responses* from general Wikipedia editors. Invitations to take the survey will be posted an editors' User_talk pages. *A pilot set of 15 requests will be posted immediately* following approval from RCom to test for problems and determine the expected response rate. Afterwards, *up to 500 User_talk postings* will be made (depending on response rate) to illicit enough responses to give statistical confidence. This is the first proposed project of this scale that we are reviewing for approval so I really want to make sure we are doing it right. -Aaron On Mon, Nov 14, 2011 at 8:49 AM, Aaron Halfaker <aaron.halfaker@gmail.com <mailto:aaron.halfaker@gmail.com>> wrote: Hey folks, This proposal is an important milestone for our subject recruitment processes, since it represents the first mass recruitment request (200-300 responses needed). I'm hoping to either show a high level of support with this poll or discover what problems still need to be dealt with. I'd like to close the poll by *Wednesday @ noon UTC*. Please make sure to chime in. See poll: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_the_net#Poll:_RCom_support_for_this_project -Aaron _______________________________________________ RCom-l mailing list RCom-l@lists.wikimedia.org <mailto:RCom-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/rcom-l
_______________________________________________ RCom-l mailing list RCom-l@lists.wikimedia.org <mailto:RCom-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/rcom-l -- -------------------------------------------------------------- "Truth is much too complicated to allow anything but approximations." :: John von Neumann -------------------------------------------------------------- http://www.milovanovicresearch.com _______________________________________________ RCom-l mailing list RCom-l@lists.wikimedia.org <mailto:RCom-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
On Wed, 16 Nov 2011 23:28:28 -0500, "c.note.lilly" c.note.lilly@gmail.com wrote:
C
It is not clear for me who you are, would you please indicate. The mail and the signature do not ring a bell. Sorry for that.
Now, I noticed that Michael just added the contact of his supervisor, so I would be for letting him go and simultaneously starting the discussion on wiki.
Cheers Yaroslav
sorry, this is Cheryl (MichChemGSI).
On 11/17/11 5:20 AM, Yaroslav M. Blanter wrote:
On Wed, 16 Nov 2011 23:28:28 -0500, "c.note.lilly" c.note.lilly@gmail.com wrote:
C
It is not clear for me who you are, would you please indicate. The mail and the signature do not ring a bell. Sorry for that.
Now, I noticed that Michael just added the contact of his supervisor, so I would be for letting him go and simultaneously starting the discussion on wiki.
Cheers Yaroslav
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
Goran,
let me quickly comment on the point you raise, I definitely don't want to increase the bureaucratization of the RCom review process. I want it to be lightweight, efficient, effectively distributed across RCom members and useful for the researcher and for Wikimedia. I don't think asking for the name of someone's supervisor is a major requirement given the amount of effort most of us are putting in this process, or is it? If you consider that we eventually say yes to every single request we receive (I cannot think of a single SR request that we rejected in the past, anybody?) making sure there is at least some sane accountability mechanism in place when we lack other information doesn't strike me as a major selection barrier.
Keep in mind that we are targeting our community, especially new users, with an increasing number of requests of various kinds. I was not aware of this problem until I started working at WMF and realized that as a new user in 2011 you will be contacted by several bots or students/researchers/WMF staff willing to recruit you for a survey before you get to talk to a fellow editor welcoming you (if you are lucky) to the project.
I don't want to be fussy about this particular request, and we should move forward if there is consensus, but the general problem with our passive support of SR requests is not settled yet as far as I am concerned. I have limited time and resources to allocate to reviewing research collaboration requests and it's fair to say that 80-90% of my RCom activity is taken by reviewing surveys of any kind. I don't think we've ever followed up with previous surveys to see if they were successfully completed, if they produced any interesting results, if their results were ever shared with the community. What I am trying to say is that we could spend our effort more wisely to maximize the usefulness of our research outreach program and help researchers help us in this process.
Dario
On Nov 16, 2011, at 2:28 PM, Goran Milovanovic wrote:
Well, when it comes to
"I would be happy to give my approval conditionally on having (at least) the student's supervisor involved. I actually think this would be a sensible rule to apply to *all* SR proposals submitted by students."
I have to say I could not agree, and here is the reason: we may stumble upon many academic environments where students, or PhD candidates, or young researchers in general are in fact more competent in specific areas of research then their supervisors. Of course, this is not a common situation, but if we talk about Internet related research, where new concepts, new ideas, and new methods are introduced every now and then, we are talking about a field where the above described asymmetry in competences is likely to appear.
Personal experience: between 2002. and 2005. a research team that I have led in Belgrade, Serbia, published four books on Internet behavior, Internet usage, attitudes towards the Internet and related topics, of which none were peer-reviewed in Serbia. Why? They were all supported by standard methodological, theoretical and statistical instruments of social sciences. Most of the team members already had experience in publishing in peer-reviewed journals or presented at recognized conferences. Simply, because that were the first attempts to study anything related to Internet and society in my country at all, and there were no peer reviewers available. I am literary saying that back then we were to first to try to crawl for the relevant literature and references at all.
What I want to add is that most researchers who are into Internet related research are people who are into new ideas, looking for conceptual breakthroughs and new theoretical frameworks. Limiting their efforts by constrains related to the already bureaucratized relations in academia could end up in actually enlarging the pile of problems on the behalf of people who simply want to put their new ideas to test. Wikipedia was developed by people who decided to abandon the classical rules and try with something previously unimaginable: who would say it work to become of the landmarks of the Internet? I believe we should keep that spirit when it comes to research as well.
"I hope it's clear that I don't want to shut down research for anybody but top researchers with big grants, but we also cannot afford spending time and effort and community attention for projects of an unclear scientific value or interest."
I recognize the importance of the issue you're raising, but I do not think the selection criterion you are proposing will help us sort out the projects of clear from the projects of unclear value or interest. I believe, in the spirit of good faith, that task needs to remain with us completely, as well as with the members of the scientific community who we might contact and ask for an advice when we are not sure what to do about a particular project proposal.
Best, Goran
On Wed, Nov 16, 2011 at 11:03 PM, Dario Taraborelli dtaraborelli@wikimedia.org wrote: ok, here's my dispassionate opinion. I post on the list because the problems I refer to below are not specific to this project but to many other requests.
We have been receiving a *lot* of SR requests from students over the last few months and I think we should start deciding more aggressively what research is likely to have an impact and what research won't produce any major tangible results.
I really like the topic of the Anonymity and Conformity study but I several concerns with the solidity of the current proposal:
• we will be approving for the first time some kind of large-scale recruitment approach via user talk pages for a student project: this is something we've never done before and we should only do it if there's a good reason.
• the advisor of the proponent doesn't seem to be involved at all in this project and is not even named in the proposal. Aaron asked the proponent to share the name of his supervisor in September, but he hasn't done so (yet?)
• the proponent says that no funding is supporting this research and that this study is "conducted with the author's own efforts"
• no one else other than the applicant will be implied in the data collection and analysis and the proponent doesn't seem to have an actual research record
• there is no trace in this proposal of an approval by an ethics committee. The proponent says that this is not applicable (and it's true that IRB policy is very different between the US and other countries), but some official record would help us assess the credibility of the proposal.
for these reasons, I am hesitant whether we should blindly approve this request. I would be happy to give my approval conditionally on having (at least) the student's supervisor involved. I actually think this would be a sensible rule to apply to *all* SR proposals submitted by students.
I hope it's clear that I don't want to shut down research for anybody but top researchers with big grants, but we also cannot afford spending time and effort and community attention for projects of an unclear scientific value or interest.
I'd love to hear your thoughts on this
Dario
On Nov 16, 2011, at 12:30 PM, Aaron Halfaker wrote:
I was hoping to close this poll hours ago, but we only have three members of RCom participating (thanks Yaroslav and Steven!).
It is absolutely crucial that if we end up technically approving this study methodology that such approval actually reflects the consensus of RCom members.
For your benefit, I'll summarize the proposed plan:
A request to participate in a survey about enforcing conformance with community/group outcomes needs 200-300 responses from general Wikipedia editors. Invitations to take the survey will be posted an editors' User_talk pages. A pilot set of 15 requests will be posted immediately following approval from RCom to test for problems and determine the expected response rate. Afterwards, up to 500 User_talk postings will be made (depending on response rate) to illicit enough responses to give statistical confidence.
This is the first proposed project of this scale that we are reviewing for approval so I really want to make sure we are doing it right.
-Aaron
On Mon, Nov 14, 2011 at 8:49 AM, Aaron Halfaker aaron.halfaker@gmail.com wrote: Hey folks,
This proposal is an important milestone for our subject recruitment processes, since it represents the first mass recruitment request (200-300 responses needed). I'm hoping to either show a high level of support with this poll or discover what problems still need to be dealt with.
I'd like to close the poll by Wednesday @ noon UTC. Please make sure to chime in.
See poll: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...
-Aaron
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
--
"Truth is much too complicated to allow anything but approximations." :: John von Neumann
http://www.milovanovicresearch.com
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
Hey folks,
I directed Michael to the RCom public archives of this conversation and he has posted a response to the concerns that we've raised on meta. Please see: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...
-Aaron
On Fri, Nov 18, 2011 at 8:10 PM, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
Goran,
let me quickly comment on the point you raise, I definitely don't want to increase the bureaucratization of the RCom review process. I want it to be lightweight, efficient, effectively distributed across RCom members and useful for the researcher and for Wikimedia. I don't think asking for the name of someone's supervisor is a major requirement given the amount of effort most of us are putting in this process, or is it? If you consider that we eventually say yes to every single request we receive (I cannot think of a single SR request that we rejected in the past, anybody?) making sure there is at least some sane accountability mechanism in place when we lack other information doesn't strike me as a major selection barrier.
Keep in mind that we are targeting our community, especially new users, with an increasing number of requests of various kinds. I was not aware of this problem until I started working at WMF and realized that as a new user in 2011 you will be contacted by several bots or students/researchers/WMF staff willing to recruit you for a survey before you get to talk to a fellow editor welcoming you (if you are lucky) to the project.
I don't want to be fussy about this particular request, and we should move forward if there is consensus, but the general problem with our passive support of SR requests is not settled yet as far as I am concerned. I have limited time and resources to allocate to reviewing research collaboration requests and it's fair to say that 80-90% of my RCom activity is taken by reviewing surveys of any kind. I don't think we've ever followed up with previous surveys to see if they were successfully completed, if they produced any interesting results, if their results were ever shared with the community. What I am trying to say is that we could spend our effort more wisely to maximize the usefulness of our research outreach program and help researchers help us in this process.
Dario
On Nov 16, 2011, at 2:28 PM, Goran Milovanovic wrote:
Well, when it comes to
"I would be happy to give my approval conditionally on having (at least) the student's supervisor involved. I actually think this would be a sensible rule to apply to *all* SR proposals submitted by students."
I have to say I could not agree, and here is the reason: we may stumble upon many academic environments where students, or PhD candidates, or young researchers in general are in fact more competent in specific areas of research then their supervisors. Of course, this is not a common situation, but if we talk about Internet related research, where new concepts, new ideas, and new methods are introduced every now and then, we are talking about a field where the above described asymmetry in competences is likely to appear.
Personal experience: between 2002. and 2005. a research team that I have led in Belgrade, Serbia, published four books on Internet behavior, Internet usage, attitudes towards the Internet and related topics, of which none were peer-reviewed in Serbia. Why? They were all supported by standard methodological, theoretical and statistical instruments of social sciences. Most of the team members already had experience in publishing in peer-reviewed journals or presented at recognized conferences. Simply, because that were the first attempts to study anything related to Internet and society in my country at all, and there were no peer reviewers available. I am literary saying that back then we were to first to try to crawl for the relevant literature and references at all.
What I want to add is that most researchers who are into Internet related research are people who are into new ideas, looking for conceptual breakthroughs and new theoretical frameworks. Limiting their efforts by constrains related to the already bureaucratized relations in academia could end up in actually enlarging the pile of problems on the behalf of people who simply want to put their new ideas to test. Wikipedia was developed by people who decided to abandon the classical rules and try with something previously unimaginable: who would say it work to become of the landmarks of the Internet? I believe we should keep that spirit when it comes to research as well.
"I hope it's clear that I don't want to shut down research for anybody but top researchers with big grants, but we also cannot afford spending time and effort and community attention for projects of an unclear scientific value or interest."
I recognize the importance of the issue you're raising, but I do not think the selection criterion you are proposing will help us sort out the projects of clear from the projects of unclear value or interest. I believe, in the spirit of good faith, that task needs to remain with us completely, as well as with the members of the scientific community who we might contact and ask for an advice when we are not sure what to do about a particular project proposal.
Best, Goran
On Wed, Nov 16, 2011 at 11:03 PM, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
ok, here's my dispassionate opinion. I post on the list because the problems I refer to below are not specific to this project but to many other requests.
We have been receiving a *lot* of SR requests from students over the last few months and I think we should start deciding more aggressively what research is likely to have an impact and what research won't produce any major tangible results.
I really like the topic of the Anonymity and Conformity study but I several concerns with the solidity of the current proposal:
• we will be approving for the first time some kind of large-scale recruitment approach via user talk pages for a student project: this is something we've never done before and we should only do it if there's a good reason.
• the advisor of the proponent doesn't seem to be involved at all in this project and is not even named in the proposal. Aaron asked the proponent to share the name of his supervisor in September, but he hasn't done so (yet?)
• the proponent says that no funding is supporting this research and that this study is "conducted with the author's own efforts"
• no one else other than the applicant will be implied in the data collection and analysis and the proponent doesn't seem to have an actual research record
• there is no trace in this proposal of an approval by an ethics committee. The proponent says that this is not applicable (and it's true that IRB policy is very different between the US and other countries), but some official record would help us assess the credibility of the proposal.
for these reasons, I am hesitant whether we should blindly approve this request. I would be happy to give my approval conditionally on having (at least) the student's supervisor involved. I actually think this would be a sensible rule to apply to *all* SR proposals submitted by students.
I hope it's clear that I don't want to shut down research for anybody but top researchers with big grants, but we also cannot afford spending time and effort and community attention for projects of an unclear scientific value or interest.
I'd love to hear your thoughts on this
Dario
On Nov 16, 2011, at 12:30 PM, Aaron Halfaker wrote:
I was hoping to close this poll hours ago, but we only have three members of RCom participating (thanks Yaroslav and Steven!).
It is absolutely crucial that if we end up technically approving this study methodology that such approval actually reflects the consensus of RCom members.
For your benefit, I'll summarize the proposed plan:
A request to participate in a *survey* about enforcing conformance with community/group outcomes *needs 200-300 responses* from general Wikipedia editors. Invitations to take the survey will be posted an editors' User_talk pages. *A pilot set of 15 requests will be posted immediately* following approval from RCom to test for problems and determine the expected response rate. Afterwards, *up to 500 User_talk postings* will be made (depending on response rate) to illicit enough responses to give statistical confidence.
This is the first proposed project of this scale that we are reviewing for approval so I really want to make sure we are doing it right.
-Aaron
On Mon, Nov 14, 2011 at 8:49 AM, Aaron Halfaker <aaron.halfaker@gmail.com
wrote:
Hey folks,
This proposal is an important milestone for our subject recruitment processes, since it represents the first mass recruitment request (200-300 responses needed). I'm hoping to either show a high level of support with this poll or discover what problems still need to be dealt with.
I'd like to close the poll by *Wednesday @ noon UTC*. Please make sure to chime in.
See poll: http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...
-Aaron
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
--
"Truth is much too complicated to allow anything but approximations." :: John von Neumann
http://www.milovanovicresearch.com
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l