All,
here's a message from Siko, WMF Head of Community Fellowships. As with the 2011 Summer of Research, WMF is willing to fund research (both in the form of individual fellowships and small grants) to contribute to a better understanding of our community and projects. While there are existing procedures for community fellowships and grants, we don't have guidelines to apply for research fellowships/research grants.
Some community members have started submitting research proposals for RCom review and I thought this could be a good chance to get Siko and Asaf (Head of the WMF grants program) to help us draft guidelines for the evaluation of research fellowship/research grant proposals, which are currently missing from http://meta.wikimedia.org/wiki/Research:FAQ
What I envision is a two-tiered process:
(1) RCom will first review proposals based on its standard procedures, regardless of funding requests. We will solicit the opinion of external referees via a single-blind review process when needed (we did this for the EPIC/Oxford proposal). We will then write our recommendations whether a specific proposal is methodologically sound, relevant and non-disruptive to our community to help WMF make a funding decision .
(2) WMF will request supplementary information to projects applying for funding and use this information, feedback from RCom and its internal assessment of the priority/usefulness of the proposal to make a funding decision.
This will help RCom focus on the research value of the proposal per se while leaving to the WMF fellowship/grant program the actual funding decision. On a related note, I am working closely with Philippe Beaudette to configure SugarCRM to help us triage, handle and assign requests for RCom review.
Please let me know if you have any comments or concerns on the overall proposal. As Siko notes, the Dispute Resolution project below is a research proposal from a community member asking for regular SR support/review, not a WMF-sponsored project, and potentially a good case to get this process started.
Dario
Begin forwarded message:
From: Siko Bouterse sbouterse@wikimedia.org Date: November 22, 2011 10:33:58 AM PST To: Dario Taraborelli dtaraborelli@wikimedia.org Subject: surveys by community members
Hi Dario,
This is a survey request from a community member interested in learning more about his Wikipedia projects, for RCom's review: http://meta.wikimedia.org/wiki/Research:Wikipedia_Dispute_Resolution
Background: Steven Zhang is active in MedCab and the creator of some other DR pages and processes on EN:WP. I've been speaking with him about the possibility of doing a fellowship on dispute resolution, though an exact project is still to be pinned down and nothing is approved for fellowship at this point. Although the survey is not an official WMF project, and Steven is acting in the capacity of community member, I am interested in the results of his survey to learn more about current issues with DR and see if there are projects that we should support in the form of a fellowship.
This may be a growing need, I've gotten a couple of similar inquiries so far and expect they will increase as we ramp up community fellowships. I'm curious to know what the RCom process looks like for surveys run by community members, some of whom might not have the same research background or methodological training as academic researchers, but are motivated to learn and share understanding about their community and projects. Is this something worth asking about on the RCom list? (If so, feel free to forward my message).
In this case, its a relatively small sample size so hopefully not too disruptive. I think Steven could also use some guidance about what free survey collector would be recommended for use - is RCom ok with something simple like Google Forms or have other recommendations?
Thanks! Siko
-- Siko Bouterse Head of Community Fellowships Wikimedia Foundation, Inc.
sbouterse@wikimedia.org
It seems to me that RCom is currently suffering from having too many responsibilities that are not fulfilled. I'm not sure we are ready to add a new responsibility.
We should carefully consider whether RCom can effectively fulfill this additional role.
-Aaron
On Tue, Nov 22, 2011 at 6:43 PM, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
All,
here's a message from Siko, WMF Head of Community Fellowships. As with the 2011 Summer of Research, WMF is willing to fund research (both in the form of individual fellowships and small grants) to contribute to a better understanding of our community and projects. While there are existing procedures for community fellowships and grants, we don't have guidelines to apply for research fellowships/research grants.
Some community members have started submitting research proposals for RCom review and I thought this could be a good chance to get Siko and Asaf (Head of the WMF grants program) to help us draft guidelines for the evaluation of research fellowship/research grant proposals, which are currently missing from http://meta.wikimedia.org/wiki/Research:FAQ
What I envision is a two-tiered process:
(1) RCom will first review proposals based on its standard procedures, regardless of funding requests. We will solicit the opinion of external referees via a single-blind review process when needed (we did this for the EPIC/Oxford proposal). We will then write our recommendations whether a specific proposal is methodologically sound, relevant and non-disruptive to our community to help WMF make a funding decision .
(2) WMF will request supplementary information to projects applying for funding and use this information, feedback from RCom and its internal assessment of the priority/usefulness of the proposal to make a funding decision.
This will help RCom focus on the research value of the proposal per se while leaving to the WMF fellowship/grant program the actual funding decision. On a related note, I am working closely with Philippe Beaudette to configure SugarCRM to help us triage, handle and assign requests for RCom review.
Please let me know if you have any comments or concerns on the overall proposal. As Siko notes, the Dispute Resolution project below is a research proposal from a community member asking for regular SR support/review, not a WMF-sponsored project, and potentially a good case to get this process started.
Dario
Begin forwarded message:
*From: *Siko Bouterse sbouterse@wikimedia.org *Date: *November 22, 2011 10:33:58 AM PST *To: *Dario Taraborelli dtaraborelli@wikimedia.org *Subject: **surveys by community members*
Hi Dario,
This is a survey request from a community member interested in learning more about his Wikipedia projects, for RCom's review: http://meta.wikimedia.org/wiki/Research:Wikipedia_Dispute_Resolution
Background: Steven Zhang is active in MedCab and the creator of some other DR pages and processes on EN:WP. I've been speaking with him about the possibility of doing a fellowship on dispute resolution, though an exact project is still to be pinned down and nothing is approved for fellowship at this point. Although the survey is not an official WMF project, and Steven is acting in the capacity of community member, I am interested in the results of his survey to learn more about current issues with DR and see if there are projects that we should support in the form of a fellowship.
This may be a growing need, I've gotten a couple of similar inquiries so far and expect they will increase as we ramp up community fellowships. I'm curious to know what the RCom process looks like for surveys run by community members, some of whom might not have the same research background or methodological training as academic researchers, but are motivated to learn and share understanding about their community and projects. Is this something worth asking about on the RCom list? (If so, feel free to forward my message).
In this case, its a relatively small sample size so hopefully not too disruptive. I think Steven could also use some guidance about what free survey collector would be recommended for use - is RCom ok with something simple like Google Forms or have other recommendations?
Thanks! Siko
-- Siko Bouterse Head of Community Fellowships Wikimedia Foundation, Inc.
sbouterse@wikimedia.org
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
This is very much focussed on the model where a community member identifies a problem area which would benefit from research, and then they want to be paid to research it.
My concern about such applications are that you risk blurring the line between what the community expects volunteers to do for free and what it is willing to pay for; Also we need to be wary of pushpolling, I've already seen research done by community members being criticised for that and I think we need very clear rules to prevent that happening.
The advantage of an independent researcher is independence, hopefully neutrality and if we request it a certain verifiable professional standard. The disadvantage is that they cost money and may misunderstand the community.
The advantages of a Wikimedian researcher are that they are presumably a volunteer and that they understand what it is that they are studying. The disadvantages include their possibly wanting to research an area of the wiki that they have strong opinions on, and that they are unlikely to have verified credentials for holding personal data for research purposes.
I really like the idea of using research to investigate some of the problems of the community. There are several areas where community opinion is deadlocked and independent neutral research might possibly supply the information that could break that deadlock. But I'm uncomfortable about the model of commissioning existing volunteers to investigate areas where they have strong opinions. I would prefer that we have something of a Chinese wall between topic and researcher. So community members would be welcome to propose research projects, and also to offer to undertake research projects, but not to be both the proposer and the performer of the same paid for research. Or at the least we should avoid paying editors to undertake research in an area where they are known to have strong views.
For example; The theory that I find most credible as an explanation of the decline of the community since 2007 is the end of the "SoFixIt" culture and its replacement by the templating culture which some consider newbie biting and which has lead to hundreds of thousands of articles disfigured by garish templates calling attention to problems that somebody hopes someone else will understand and fix. One possible partial solution to that would be to replace some of the maintenance categories with unobtrusive hidden categories, but to do so we need research to establish whether or not these templates succeed in getting newbies to edit. Currently the community is too divided as to whether we think the templates work as a call to action for us to agree a change, but a small amount of independent research should be able to establish which view is more correct and thereby enable the community to make an informed consensus decision.
WSC
On 23 November 2011 00:43, Dario Taraborelli dtaraborelli@wikimedia.orgwrote:
All,
here's a message from Siko, WMF Head of Community Fellowships. As with the 2011 Summer of Research, WMF is willing to fund research (both in the form of individual fellowships and small grants) to contribute to a better understanding of our community and projects. While there are existing procedures for community fellowships and grants, we don't have guidelines to apply for research fellowships/research grants.
Some community members have started submitting research proposals for RCom review and I thought this could be a good chance to get Siko and Asaf (Head of the WMF grants program) to help us draft guidelines for the evaluation of research fellowship/research grant proposals, which are currently missing from http://meta.wikimedia.org/wiki/Research:FAQ
What I envision is a two-tiered process:
(1) RCom will first review proposals based on its standard procedures, regardless of funding requests. We will solicit the opinion of external referees via a single-blind review process when needed (we did this for the EPIC/Oxford proposal). We will then write our recommendations whether a specific proposal is methodologically sound, relevant and non-disruptive to our community to help WMF make a funding decision .
(2) WMF will request supplementary information to projects applying for funding and use this information, feedback from RCom and its internal assessment of the priority/usefulness of the proposal to make a funding decision.
This will help RCom focus on the research value of the proposal per se while leaving to the WMF fellowship/grant program the actual funding decision. On a related note, I am working closely with Philippe Beaudette to configure SugarCRM to help us triage, handle and assign requests for RCom review.
Please let me know if you have any comments or concerns on the overall proposal. As Siko notes, the Dispute Resolution project below is a research proposal from a community member asking for regular SR support/review, not a WMF-sponsored project, and potentially a good case to get this process started.
Dario
Begin forwarded message:
*From: *Siko Bouterse sbouterse@wikimedia.org *Date: *November 22, 2011 10:33:58 AM PST *To: *Dario Taraborelli dtaraborelli@wikimedia.org *Subject: **surveys by community members*
Hi Dario,
This is a survey request from a community member interested in learning more about his Wikipedia projects, for RCom's review: http://meta.wikimedia.org/wiki/Research:Wikipedia_Dispute_Resolution
Background: Steven Zhang is active in MedCab and the creator of some other DR pages and processes on EN:WP. I've been speaking with him about the possibility of doing a fellowship on dispute resolution, though an exact project is still to be pinned down and nothing is approved for fellowship at this point. Although the survey is not an official WMF project, and Steven is acting in the capacity of community member, I am interested in the results of his survey to learn more about current issues with DR and see if there are projects that we should support in the form of a fellowship.
This may be a growing need, I've gotten a couple of similar inquiries so far and expect they will increase as we ramp up community fellowships. I'm curious to know what the RCom process looks like for surveys run by community members, some of whom might not have the same research background or methodological training as academic researchers, but are motivated to learn and share understanding about their community and projects. Is this something worth asking about on the RCom list? (If so, feel free to forward my message).
In this case, its a relatively small sample size so hopefully not too disruptive. I think Steven could also use some guidance about what free survey collector would be recommended for use - is RCom ok with something simple like Google Forms or have other recommendations?
Thanks! Siko
-- Siko Bouterse Head of Community Fellowships Wikimedia Foundation, Inc.
sbouterse@wikimedia.org
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
I should clarify that in Steven Z's case, we aren't discussing paying him to do a survey - as I do agree with WSC that this falls within the realm of what the community expects volunteers to do, it is something he's undertaking on his own initiative, he wasn't commissioned to do this research.
Even with the potential for bias, I don't know if it is practical to stop community members from undertaking research projects to help them understand how well their own projects are working. In online community projects I've worked on outside of Wikimedia, my teams (volunteers and paid staff) have often needed to use tools like surveys and other "research-like" feedback mechanisms, A/B testing, etc to help us understand the successes and failures of our own projects - perhaps it would be better to commission unbiased outsiders to do this research, but its usually not very feasible.
Although as I said before, I am somewhat interested in the survey results to learn if there are projects the foundation could support related to Dispute Resolution, but if Steven wasn't planning to do this research right now we will not be hiring an independent researcher to take it on instead. (If this is confusing in the proposal itself, please let me know - I think sometimes volunteers believe that suggesting "official WMF support" will help their cause when in fact the opposite is often true!)
He got pointed to RCom for approval to run a survey, and I guess what is still unclear to me from the comments in this thread is should he be going though RCom's process to get approval, or not?
Siko
On Sun, Nov 27, 2011 at 10:07 PM, WereSpielChequers < werespielchequers@gmail.com> wrote:
This is very much focussed on the model where a community member identifies a problem area which would benefit from research, and then they want to be paid to research it.
My concern about such applications are that you risk blurring the line between what the community expects volunteers to do for free and what it is willing to pay for; Also we need to be wary of pushpolling, I've already seen research done by community members being criticised for that and I think we need very clear rules to prevent that happening.
The advantage of an independent researcher is independence, hopefully neutrality and if we request it a certain verifiable professional standard. The disadvantage is that they cost money and may misunderstand the community.
The advantages of a Wikimedian researcher are that they are presumably a volunteer and that they understand what it is that they are studying. The disadvantages include their possibly wanting to research an area of the wiki that they have strong opinions on, and that they are unlikely to have verified credentials for holding personal data for research purposes.
I really like the idea of using research to investigate some of the problems of the community. There are several areas where community opinion is deadlocked and independent neutral research might possibly supply the information that could break that deadlock. But I'm uncomfortable about the model of commissioning existing volunteers to investigate areas where they have strong opinions. I would prefer that we have something of a Chinese wall between topic and researcher. So community members would be welcome to propose research projects, and also to offer to undertake research projects, but not to be both the proposer and the performer of the same paid for research. Or at the least we should avoid paying editors to undertake research in an area where they are known to have strong views.
For example; The theory that I find most credible as an explanation of the decline of the community since 2007 is the end of the "SoFixIt" culture and its replacement by the templating culture which some consider newbie biting and which has lead to hundreds of thousands of articles disfigured by garish templates calling attention to problems that somebody hopes someone else will understand and fix. One possible partial solution to that would be to replace some of the maintenance categories with unobtrusive hidden categories, but to do so we need research to establish whether or not these templates succeed in getting newbies to edit. Currently the community is too divided as to whether we think the templates work as a call to action for us to agree a change, but a small amount of independent research should be able to establish which view is more correct and thereby enable the community to make an informed consensus decision.
WSC
On 23 November 2011 00:43, Dario Taraborelli dtaraborelli@wikimedia.orgwrote:
All,
here's a message from Siko, WMF Head of Community Fellowships. As with the 2011 Summer of Research, WMF is willing to fund research (both in the form of individual fellowships and small grants) to contribute to a better understanding of our community and projects. While there are existing procedures for community fellowships and grants, we don't have guidelines to apply for research fellowships/research grants.
Some community members have started submitting research proposals for RCom review and I thought this could be a good chance to get Siko and Asaf (Head of the WMF grants program) to help us draft guidelines for the evaluation of research fellowship/research grant proposals, which are currently missing from http://meta.wikimedia.org/wiki/Research:FAQ
What I envision is a two-tiered process:
(1) RCom will first review proposals based on its standard procedures, regardless of funding requests. We will solicit the opinion of external referees via a single-blind review process when needed (we did this for the EPIC/Oxford proposal). We will then write our recommendations whether a specific proposal is methodologically sound, relevant and non-disruptive to our community to help WMF make a funding decision .
(2) WMF will request supplementary information to projects applying for funding and use this information, feedback from RCom and its internal assessment of the priority/usefulness of the proposal to make a funding decision.
This will help RCom focus on the research value of the proposal per se while leaving to the WMF fellowship/grant program the actual funding decision. On a related note, I am working closely with Philippe Beaudette to configure SugarCRM to help us triage, handle and assign requests for RCom review.
Please let me know if you have any comments or concerns on the overall proposal. As Siko notes, the Dispute Resolution project below is a research proposal from a community member asking for regular SR support/review, not a WMF-sponsored project, and potentially a good case to get this process started.
Dario
Begin forwarded message:
*From: *Siko Bouterse sbouterse@wikimedia.org *Date: *November 22, 2011 10:33:58 AM PST *To: *Dario Taraborelli dtaraborelli@wikimedia.org *Subject: **surveys by community members*
Hi Dario,
This is a survey request from a community member interested in learning more about his Wikipedia projects, for RCom's review: http://meta.wikimedia.org/wiki/Research:Wikipedia_Dispute_Resolution
Background: Steven Zhang is active in MedCab and the creator of some other DR pages and processes on EN:WP. I've been speaking with him about the possibility of doing a fellowship on dispute resolution, though an exact project is still to be pinned down and nothing is approved for fellowship at this point. Although the survey is not an official WMF project, and Steven is acting in the capacity of community member, I am interested in the results of his survey to learn more about current issues with DR and see if there are projects that we should support in the form of a fellowship.
This may be a growing need, I've gotten a couple of similar inquiries so far and expect they will increase as we ramp up community fellowships. I'm curious to know what the RCom process looks like for surveys run by community members, some of whom might not have the same research background or methodological training as academic researchers, but are motivated to learn and share understanding about their community and projects. Is this something worth asking about on the RCom list? (If so, feel free to forward my message).
In this case, its a relatively small sample size so hopefully not too disruptive. I think Steven could also use some guidance about what free survey collector would be recommended for use - is RCom ok with something simple like Google Forms or have other recommendations?
Thanks! Siko
-- Siko Bouterse Head of Community Fellowships Wikimedia Foundation, Inc.
sbouterse@wikimedia.org
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
OK in that case it is like any other survey, except that from an anonymous community member there are unlikely to be the safeguards re personal data that an Academic at a University could offer. So my presumption would be against any collection of personally identifiable information, even if only stored by the researcher.
WSC
On 28 November 2011 19:22, Siko Bouterse sbouterse@wikimedia.org wrote:
I should clarify that in Steven Z's case, we aren't discussing paying him to do a survey - as I do agree with WSC that this falls within the realm of what the community expects volunteers to do, it is something he's undertaking on his own initiative, he wasn't commissioned to do this research.
Even with the potential for bias, I don't know if it is practical to stop community members from undertaking research projects to help them understand how well their own projects are working. In online community projects I've worked on outside of Wikimedia, my teams (volunteers and paid staff) have often needed to use tools like surveys and other "research-like" feedback mechanisms, A/B testing, etc to help us understand the successes and failures of our own projects - perhaps it would be better to commission unbiased outsiders to do this research, but its usually not very feasible.
Although as I said before, I am somewhat interested in the survey results to learn if there are projects the foundation could support related to Dispute Resolution, but if Steven wasn't planning to do this research right now we will not be hiring an independent researcher to take it on instead. (If this is confusing in the proposal itself, please let me know - I think sometimes volunteers believe that suggesting "official WMF support" will help their cause when in fact the opposite is often true!)
He got pointed to RCom for approval to run a survey, and I guess what is still unclear to me from the comments in this thread is should he be going though RCom's process to get approval, or not?
Siko
On Sun, Nov 27, 2011 at 10:07 PM, WereSpielChequers < werespielchequers@gmail.com> wrote:
This is very much focussed on the model where a community member identifies a problem area which would benefit from research, and then they want to be paid to research it.
My concern about such applications are that you risk blurring the line between what the community expects volunteers to do for free and what it is willing to pay for; Also we need to be wary of pushpolling, I've already seen research done by community members being criticised for that and I think we need very clear rules to prevent that happening.
The advantage of an independent researcher is independence, hopefully neutrality and if we request it a certain verifiable professional standard. The disadvantage is that they cost money and may misunderstand the community.
The advantages of a Wikimedian researcher are that they are presumably a volunteer and that they understand what it is that they are studying. The disadvantages include their possibly wanting to research an area of the wiki that they have strong opinions on, and that they are unlikely to have verified credentials for holding personal data for research purposes.
I really like the idea of using research to investigate some of the problems of the community. There are several areas where community opinion is deadlocked and independent neutral research might possibly supply the information that could break that deadlock. But I'm uncomfortable about the model of commissioning existing volunteers to investigate areas where they have strong opinions. I would prefer that we have something of a Chinese wall between topic and researcher. So community members would be welcome to propose research projects, and also to offer to undertake research projects, but not to be both the proposer and the performer of the same paid for research. Or at the least we should avoid paying editors to undertake research in an area where they are known to have strong views.
For example; The theory that I find most credible as an explanation of the decline of the community since 2007 is the end of the "SoFixIt" culture and its replacement by the templating culture which some consider newbie biting and which has lead to hundreds of thousands of articles disfigured by garish templates calling attention to problems that somebody hopes someone else will understand and fix. One possible partial solution to that would be to replace some of the maintenance categories with unobtrusive hidden categories, but to do so we need research to establish whether or not these templates succeed in getting newbies to edit. Currently the community is too divided as to whether we think the templates work as a call to action for us to agree a change, but a small amount of independent research should be able to establish which view is more correct and thereby enable the community to make an informed consensus decision.
WSC
On 23 November 2011 00:43, Dario Taraborelli dtaraborelli@wikimedia.orgwrote:
All,
here's a message from Siko, WMF Head of Community Fellowships. As with the 2011 Summer of Research, WMF is willing to fund research (both in the form of individual fellowships and small grants) to contribute to a better understanding of our community and projects. While there are existing procedures for community fellowships and grants, we don't have guidelines to apply for research fellowships/research grants.
Some community members have started submitting research proposals for RCom review and I thought this could be a good chance to get Siko and Asaf (Head of the WMF grants program) to help us draft guidelines for the evaluation of research fellowship/research grant proposals, which are currently missing from http://meta.wikimedia.org/wiki/Research:FAQ
What I envision is a two-tiered process:
(1) RCom will first review proposals based on its standard procedures, regardless of funding requests. We will solicit the opinion of external referees via a single-blind review process when needed (we did this for the EPIC/Oxford proposal). We will then write our recommendations whether a specific proposal is methodologically sound, relevant and non-disruptive to our community to help WMF make a funding decision .
(2) WMF will request supplementary information to projects applying for funding and use this information, feedback from RCom and its internal assessment of the priority/usefulness of the proposal to make a funding decision.
This will help RCom focus on the research value of the proposal per se while leaving to the WMF fellowship/grant program the actual funding decision. On a related note, I am working closely with Philippe Beaudette to configure SugarCRM to help us triage, handle and assign requests for RCom review.
Please let me know if you have any comments or concerns on the overall proposal. As Siko notes, the Dispute Resolution project below is a research proposal from a community member asking for regular SR support/review, not a WMF-sponsored project, and potentially a good case to get this process started.
Dario
Begin forwarded message:
*From: *Siko Bouterse sbouterse@wikimedia.org *Date: *November 22, 2011 10:33:58 AM PST *To: *Dario Taraborelli dtaraborelli@wikimedia.org *Subject: **surveys by community members*
Hi Dario,
This is a survey request from a community member interested in learning more about his Wikipedia projects, for RCom's review: http://meta.wikimedia.org/wiki/Research:Wikipedia_Dispute_Resolution
Background: Steven Zhang is active in MedCab and the creator of some other DR pages and processes on EN:WP. I've been speaking with him about the possibility of doing a fellowship on dispute resolution, though an exact project is still to be pinned down and nothing is approved for fellowship at this point. Although the survey is not an official WMF project, and Steven is acting in the capacity of community member, I am interested in the results of his survey to learn more about current issues with DR and see if there are projects that we should support in the form of a fellowship.
This may be a growing need, I've gotten a couple of similar inquiries so far and expect they will increase as we ramp up community fellowships. I'm curious to know what the RCom process looks like for surveys run by community members, some of whom might not have the same research background or methodological training as academic researchers, but are motivated to learn and share understanding about their community and projects. Is this something worth asking about on the RCom list? (If so, feel free to forward my message).
In this case, its a relatively small sample size so hopefully not too disruptive. I think Steven could also use some guidance about what free survey collector would be recommended for use - is RCom ok with something simple like Google Forms or have other recommendations?
Thanks! Siko
-- Siko Bouterse Head of Community Fellowships Wikimedia Foundation, Inc.
sbouterse@wikimedia.org
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
-- Siko Bouterse Head of Community Fellowships Wikimedia Foundation, Inc.
sbouterse@wikimedia.org
What would you all think about the idea of having an academic researcher consult with a community member like this to help with personal data safeguarding best practices, and other methodological issues? I had thought that RCom might provide this sort of help or specific recommendations, but in absence of that (I know you all are very busy), if, for example, one of our PhD research fellows was available to give feedback on survey questions and discuss methodology with this community member, would that make the committee feel more comfortable in a case like this?
Siko
On Tue, Nov 29, 2011 at 2:00 AM, WereSpielChequers < werespielchequers@gmail.com> wrote:
OK in that case it is like any other survey, except that from an anonymous community member there are unlikely to be the safeguards re personal data that an Academic at a University could offer. So my presumption would be against any collection of personally identifiable information, even if only stored by the researcher.
WSC
On 28 November 2011 19:22, Siko Bouterse sbouterse@wikimedia.org wrote:
I should clarify that in Steven Z's case, we aren't discussing paying him to do a survey - as I do agree with WSC that this falls within the realm of what the community expects volunteers to do, it is something he's undertaking on his own initiative, he wasn't commissioned to do this research.
Even with the potential for bias, I don't know if it is practical to stop community members from undertaking research projects to help them understand how well their own projects are working. In online community projects I've worked on outside of Wikimedia, my teams (volunteers and paid staff) have often needed to use tools like surveys and other "research-like" feedback mechanisms, A/B testing, etc to help us understand the successes and failures of our own projects - perhaps it would be better to commission unbiased outsiders to do this research, but its usually not very feasible.
Although as I said before, I am somewhat interested in the survey results to learn if there are projects the foundation could support related to Dispute Resolution, but if Steven wasn't planning to do this research right now we will not be hiring an independent researcher to take it on instead. (If this is confusing in the proposal itself, please let me know - I think sometimes volunteers believe that suggesting "official WMF support" will help their cause when in fact the opposite is often true!)
He got pointed to RCom for approval to run a survey, and I guess what is still unclear to me from the comments in this thread is should he be going though RCom's process to get approval, or not?
Siko
On Sun, Nov 27, 2011 at 10:07 PM, WereSpielChequers < werespielchequers@gmail.com> wrote:
This is very much focussed on the model where a community member identifies a problem area which would benefit from research, and then they want to be paid to research it.
My concern about such applications are that you risk blurring the line between what the community expects volunteers to do for free and what it is willing to pay for; Also we need to be wary of pushpolling, I've already seen research done by community members being criticised for that and I think we need very clear rules to prevent that happening.
The advantage of an independent researcher is independence, hopefully neutrality and if we request it a certain verifiable professional standard. The disadvantage is that they cost money and may misunderstand the community.
The advantages of a Wikimedian researcher are that they are presumably a volunteer and that they understand what it is that they are studying. The disadvantages include their possibly wanting to research an area of the wiki that they have strong opinions on, and that they are unlikely to have verified credentials for holding personal data for research purposes.
I really like the idea of using research to investigate some of the problems of the community. There are several areas where community opinion is deadlocked and independent neutral research might possibly supply the information that could break that deadlock. But I'm uncomfortable about the model of commissioning existing volunteers to investigate areas where they have strong opinions. I would prefer that we have something of a Chinese wall between topic and researcher. So community members would be welcome to propose research projects, and also to offer to undertake research projects, but not to be both the proposer and the performer of the same paid for research. Or at the least we should avoid paying editors to undertake research in an area where they are known to have strong views.
For example; The theory that I find most credible as an explanation of the decline of the community since 2007 is the end of the "SoFixIt" culture and its replacement by the templating culture which some consider newbie biting and which has lead to hundreds of thousands of articles disfigured by garish templates calling attention to problems that somebody hopes someone else will understand and fix. One possible partial solution to that would be to replace some of the maintenance categories with unobtrusive hidden categories, but to do so we need research to establish whether or not these templates succeed in getting newbies to edit. Currently the community is too divided as to whether we think the templates work as a call to action for us to agree a change, but a small amount of independent research should be able to establish which view is more correct and thereby enable the community to make an informed consensus decision.
WSC
On 23 November 2011 00:43, Dario Taraborelli <dtaraborelli@wikimedia.org
wrote:
All,
here's a message from Siko, WMF Head of Community Fellowships. As with the 2011 Summer of Research, WMF is willing to fund research (both in the form of individual fellowships and small grants) to contribute to a better understanding of our community and projects. While there are existing procedures for community fellowships and grants, we don't have guidelines to apply for research fellowships/research grants.
Some community members have started submitting research proposals for RCom review and I thought this could be a good chance to get Siko and Asaf (Head of the WMF grants program) to help us draft guidelines for the evaluation of research fellowship/research grant proposals, which are currently missing from http://meta.wikimedia.org/wiki/Research:FAQ
What I envision is a two-tiered process:
(1) RCom will first review proposals based on its standard procedures, regardless of funding requests. We will solicit the opinion of external referees via a single-blind review process when needed (we did this for the EPIC/Oxford proposal). We will then write our recommendations whether a specific proposal is methodologically sound, relevant and non-disruptive to our community to help WMF make a funding decision .
(2) WMF will request supplementary information to projects applying for funding and use this information, feedback from RCom and its internal assessment of the priority/usefulness of the proposal to make a funding decision.
This will help RCom focus on the research value of the proposal per se while leaving to the WMF fellowship/grant program the actual funding decision. On a related note, I am working closely with Philippe Beaudette to configure SugarCRM to help us triage, handle and assign requests for RCom review.
Please let me know if you have any comments or concerns on the overall proposal. As Siko notes, the Dispute Resolution project below is a research proposal from a community member asking for regular SR support/review, not a WMF-sponsored project, and potentially a good case to get this process started.
Dario
Begin forwarded message:
*From: *Siko Bouterse sbouterse@wikimedia.org *Date: *November 22, 2011 10:33:58 AM PST *To: *Dario Taraborelli dtaraborelli@wikimedia.org *Subject: **surveys by community members*
Hi Dario,
This is a survey request from a community member interested in learning more about his Wikipedia projects, for RCom's review: http://meta.wikimedia.org/wiki/Research:Wikipedia_Dispute_Resolution
Background: Steven Zhang is active in MedCab and the creator of some other DR pages and processes on EN:WP. I've been speaking with him about the possibility of doing a fellowship on dispute resolution, though an exact project is still to be pinned down and nothing is approved for fellowship at this point. Although the survey is not an official WMF project, and Steven is acting in the capacity of community member, I am interested in the results of his survey to learn more about current issues with DR and see if there are projects that we should support in the form of a fellowship.
This may be a growing need, I've gotten a couple of similar inquiries so far and expect they will increase as we ramp up community fellowships. I'm curious to know what the RCom process looks like for surveys run by community members, some of whom might not have the same research background or methodological training as academic researchers, but are motivated to learn and share understanding about their community and projects. Is this something worth asking about on the RCom list? (If so, feel free to forward my message).
In this case, its a relatively small sample size so hopefully not too disruptive. I think Steven could also use some guidance about what free survey collector would be recommended for use - is RCom ok with something simple like Google Forms or have other recommendations?
Thanks! Siko
-- Siko Bouterse Head of Community Fellowships Wikimedia Foundation, Inc.
sbouterse@wikimedia.org
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
-- Siko Bouterse Head of Community Fellowships Wikimedia Foundation, Inc.
sbouterse@wikimedia.org
I think it is a good idea to have an academic researcher consult, but we need to be careful about who is given these responsibilities: do they have a social science background, what is their background in designing questionnaires, how familiar they are with academic regulations about conducting research with human subjects etc.
On Tue, Nov 29, 2011 at 5:16 PM, Siko Bouterse sbouterse@wikimedia.orgwrote:
What would you all think about the idea of having an academic researcher consult with a community member like this to help with personal data safeguarding best practices, and other methodological issues? I had thought that RCom might provide this sort of help or specific recommendations, but in absence of that (I know you all are very busy), if, for example, one of our PhD research fellows was available to give feedback on survey questions and discuss methodology with this community member, would that make the committee feel more comfortable in a case like this?
Siko
On Tue, Nov 29, 2011 at 2:00 AM, WereSpielChequers < werespielchequers@gmail.com> wrote:
OK in that case it is like any other survey, except that from an anonymous community member there are unlikely to be the safeguards re personal data that an Academic at a University could offer. So my presumption would be against any collection of personally identifiable information, even if only stored by the researcher.
WSC
On 28 November 2011 19:22, Siko Bouterse sbouterse@wikimedia.org wrote:
I should clarify that in Steven Z's case, we aren't discussing paying him to do a survey - as I do agree with WSC that this falls within the realm of what the community expects volunteers to do, it is something he's undertaking on his own initiative, he wasn't commissioned to do this research.
Even with the potential for bias, I don't know if it is practical to stop community members from undertaking research projects to help them understand how well their own projects are working. In online community projects I've worked on outside of Wikimedia, my teams (volunteers and paid staff) have often needed to use tools like surveys and other "research-like" feedback mechanisms, A/B testing, etc to help us understand the successes and failures of our own projects - perhaps it would be better to commission unbiased outsiders to do this research, but its usually not very feasible.
Although as I said before, I am somewhat interested in the survey results to learn if there are projects the foundation could support related to Dispute Resolution, but if Steven wasn't planning to do this research right now we will not be hiring an independent researcher to take it on instead. (If this is confusing in the proposal itself, please let me know
- I think sometimes volunteers believe that suggesting "official WMF
support" will help their cause when in fact the opposite is often true!)
He got pointed to RCom for approval to run a survey, and I guess what is still unclear to me from the comments in this thread is should he be going though RCom's process to get approval, or not?
Siko
On Sun, Nov 27, 2011 at 10:07 PM, WereSpielChequers < werespielchequers@gmail.com> wrote:
This is very much focussed on the model where a community member identifies a problem area which would benefit from research, and then they want to be paid to research it.
My concern about such applications are that you risk blurring the line between what the community expects volunteers to do for free and what it is willing to pay for; Also we need to be wary of pushpolling, I've already seen research done by community members being criticised for that and I think we need very clear rules to prevent that happening.
The advantage of an independent researcher is independence, hopefully neutrality and if we request it a certain verifiable professional standard. The disadvantage is that they cost money and may misunderstand the community.
The advantages of a Wikimedian researcher are that they are presumably a volunteer and that they understand what it is that they are studying. The disadvantages include their possibly wanting to research an area of the wiki that they have strong opinions on, and that they are unlikely to have verified credentials for holding personal data for research purposes.
I really like the idea of using research to investigate some of the problems of the community. There are several areas where community opinion is deadlocked and independent neutral research might possibly supply the information that could break that deadlock. But I'm uncomfortable about the model of commissioning existing volunteers to investigate areas where they have strong opinions. I would prefer that we have something of a Chinese wall between topic and researcher. So community members would be welcome to propose research projects, and also to offer to undertake research projects, but not to be both the proposer and the performer of the same paid for research. Or at the least we should avoid paying editors to undertake research in an area where they are known to have strong views.
For example; The theory that I find most credible as an explanation of the decline of the community since 2007 is the end of the "SoFixIt" culture and its replacement by the templating culture which some consider newbie biting and which has lead to hundreds of thousands of articles disfigured by garish templates calling attention to problems that somebody hopes someone else will understand and fix. One possible partial solution to that would be to replace some of the maintenance categories with unobtrusive hidden categories, but to do so we need research to establish whether or not these templates succeed in getting newbies to edit. Currently the community is too divided as to whether we think the templates work as a call to action for us to agree a change, but a small amount of independent research should be able to establish which view is more correct and thereby enable the community to make an informed consensus decision.
WSC
On 23 November 2011 00:43, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
All,
here's a message from Siko, WMF Head of Community Fellowships. As with the 2011 Summer of Research, WMF is willing to fund research (both in the form of individual fellowships and small grants) to contribute to a better understanding of our community and projects. While there are existing procedures for community fellowships and grants, we don't have guidelines to apply for research fellowships/research grants.
Some community members have started submitting research proposals for RCom review and I thought this could be a good chance to get Siko and Asaf (Head of the WMF grants program) to help us draft guidelines for the evaluation of research fellowship/research grant proposals, which are currently missing from http://meta.wikimedia.org/wiki/Research:FAQ
What I envision is a two-tiered process:
(1) RCom will first review proposals based on its standard procedures, regardless of funding requests. We will solicit the opinion of external referees via a single-blind review process when needed (we did this for the EPIC/Oxford proposal). We will then write our recommendations whether a specific proposal is methodologically sound, relevant and non-disruptive to our community to help WMF make a funding decision .
(2) WMF will request supplementary information to projects applying for funding and use this information, feedback from RCom and its internal assessment of the priority/usefulness of the proposal to make a funding decision.
This will help RCom focus on the research value of the proposal per se while leaving to the WMF fellowship/grant program the actual funding decision. On a related note, I am working closely with Philippe Beaudette to configure SugarCRM to help us triage, handle and assign requests for RCom review.
Please let me know if you have any comments or concerns on the overall proposal. As Siko notes, the Dispute Resolution project below is a research proposal from a community member asking for regular SR support/review, not a WMF-sponsored project, and potentially a good case to get this process started.
Dario
Begin forwarded message:
*From: *Siko Bouterse sbouterse@wikimedia.org *Date: *November 22, 2011 10:33:58 AM PST *To: *Dario Taraborelli dtaraborelli@wikimedia.org *Subject: **surveys by community members*
Hi Dario,
This is a survey request from a community member interested in learning more about his Wikipedia projects, for RCom's review: http://meta.wikimedia.org/wiki/Research:Wikipedia_Dispute_Resolution
Background: Steven Zhang is active in MedCab and the creator of some other DR pages and processes on EN:WP. I've been speaking with him about the possibility of doing a fellowship on dispute resolution, though an exact project is still to be pinned down and nothing is approved for fellowship at this point. Although the survey is not an official WMF project, and Steven is acting in the capacity of community member, I am interested in the results of his survey to learn more about current issues with DR and see if there are projects that we should support in the form of a fellowship.
This may be a growing need, I've gotten a couple of similar inquiries so far and expect they will increase as we ramp up community fellowships. I'm curious to know what the RCom process looks like for surveys run by community members, some of whom might not have the same research background or methodological training as academic researchers, but are motivated to learn and share understanding about their community and projects. Is this something worth asking about on the RCom list? (If so, feel free to forward my message).
In this case, its a relatively small sample size so hopefully not too disruptive. I think Steven could also use some guidance about what free survey collector would be recommended for use - is RCom ok with something simple like Google Forms or have other recommendations?
Thanks! Siko
-- Siko Bouterse Head of Community Fellowships Wikimedia Foundation, Inc.
sbouterse@wikimedia.org
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
-- Siko Bouterse Head of Community Fellowships Wikimedia Foundation, Inc.
sbouterse@wikimedia.org
-- Siko Bouterse Head of Community Fellowships Wikimedia Foundation, Inc.
sbouterse@wikimedia.org
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
I know that on at least one occasion the Foundation has signed a non-disclosure agreement with an editor and given them access to raw survey information. But as I understand it that editor is a retired Academic and I assume has confirmed that to the Foundation. In my view it would be wrong to divulge identifiable personal data with fewer safeguards than that.
This still leaves us several potential avenues to follow. One based on Mani Pande's suggestion would be to pair a Wikimedian with a credentialled Academic researcher on the proviso that the credentialled researcher is the only one with access to personally identifiable information, and only passes the data to their wikimedia collaborator after anonymisation.
In my view a neater route would be to stipulate that unless the Foundation has verified that the researcher is an appropriate person to hold indentifiable data, any survey needs to be anonymous. This is of course a little more complex than it sounds as edit count, date of first edit and time as admin are all things that could uniquely identify an editor. So an anonymous survey will often need banded questions especially if the raw data will be released and not just a statistical abstract.
WSC
|On 30 November 2011 01:23, Mani Pande mpande@wikimedia.org wrote:
I think it is a good idea to have an academic researcher consult, but we need to be careful about who is given these responsibilities: do they have a social science background, what is their background in designing questionnaires, how familiar they are with academic regulations about conducting research with human subjects etc.
On Tue, Nov 29, 2011 at 5:16 PM, Siko Bouterse sbouterse@wikimedia.orgwrote:
What would you all think about the idea of having an academic researcher consult with a community member like this to help with personal data safeguarding best practices, and other methodological issues? I had thought that RCom might provide this sort of help or specific recommendations, but in absence of that (I know you all are very busy), if, for example, one of our PhD research fellows was available to give feedback on survey questions and discuss methodology with this community member, would that make the committee feel more comfortable in a case like this?
Siko
On Tue, Nov 29, 2011 at 2:00 AM, WereSpielChequers < werespielchequers@gmail.com> wrote:
OK in that case it is like any other survey, except that from an anonymous community member there are unlikely to be the safeguards re personal data that an Academic at a University could offer. So my presumption would be against any collection of personally identifiable information, even if only stored by the researcher.
WSC
On 28 November 2011 19:22, Siko Bouterse sbouterse@wikimedia.orgwrote:
I should clarify that in Steven Z's case, we aren't discussing paying him to do a survey - as I do agree with WSC that this falls within the realm of what the community expects volunteers to do, it is something he's undertaking on his own initiative, he wasn't commissioned to do this research.
Even with the potential for bias, I don't know if it is practical to stop community members from undertaking research projects to help them understand how well their own projects are working. In online community projects I've worked on outside of Wikimedia, my teams (volunteers and paid staff) have often needed to use tools like surveys and other "research-like" feedback mechanisms, A/B testing, etc to help us understand the successes and failures of our own projects - perhaps it would be better to commission unbiased outsiders to do this research, but its usually not very feasible.
Although as I said before, I am somewhat interested in the survey results to learn if there are projects the foundation could support related to Dispute Resolution, but if Steven wasn't planning to do this research right now we will not be hiring an independent researcher to take it on instead. (If this is confusing in the proposal itself, please let me know
- I think sometimes volunteers believe that suggesting "official WMF
support" will help their cause when in fact the opposite is often true!)
He got pointed to RCom for approval to run a survey, and I guess what is still unclear to me from the comments in this thread is should he be going though RCom's process to get approval, or not?
Siko
On Sun, Nov 27, 2011 at 10:07 PM, WereSpielChequers < werespielchequers@gmail.com> wrote:
This is very much focussed on the model where a community member identifies a problem area which would benefit from research, and then they want to be paid to research it.
My concern about such applications are that you risk blurring the line between what the community expects volunteers to do for free and what it is willing to pay for; Also we need to be wary of pushpolling, I've already seen research done by community members being criticised for that and I think we need very clear rules to prevent that happening.
The advantage of an independent researcher is independence, hopefully neutrality and if we request it a certain verifiable professional standard. The disadvantage is that they cost money and may misunderstand the community.
The advantages of a Wikimedian researcher are that they are presumably a volunteer and that they understand what it is that they are studying. The disadvantages include their possibly wanting to research an area of the wiki that they have strong opinions on, and that they are unlikely to have verified credentials for holding personal data for research purposes.
I really like the idea of using research to investigate some of the problems of the community. There are several areas where community opinion is deadlocked and independent neutral research might possibly supply the information that could break that deadlock. But I'm uncomfortable about the model of commissioning existing volunteers to investigate areas where they have strong opinions. I would prefer that we have something of a Chinese wall between topic and researcher. So community members would be welcome to propose research projects, and also to offer to undertake research projects, but not to be both the proposer and the performer of the same paid for research. Or at the least we should avoid paying editors to undertake research in an area where they are known to have strong views.
For example; The theory that I find most credible as an explanation of the decline of the community since 2007 is the end of the "SoFixIt" culture and its replacement by the templating culture which some consider newbie biting and which has lead to hundreds of thousands of articles disfigured by garish templates calling attention to problems that somebody hopes someone else will understand and fix. One possible partial solution to that would be to replace some of the maintenance categories with unobtrusive hidden categories, but to do so we need research to establish whether or not these templates succeed in getting newbies to edit. Currently the community is too divided as to whether we think the templates work as a call to action for us to agree a change, but a small amount of independent research should be able to establish which view is more correct and thereby enable the community to make an informed consensus decision.
WSC
On 23 November 2011 00:43, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
All,
here's a message from Siko, WMF Head of Community Fellowships. As with the 2011 Summer of Research, WMF is willing to fund research (both in the form of individual fellowships and small grants) to contribute to a better understanding of our community and projects. While there are existing procedures for community fellowships and grants, we don't have guidelines to apply for research fellowships/research grants.
Some community members have started submitting research proposals for RCom review and I thought this could be a good chance to get Siko and Asaf (Head of the WMF grants program) to help us draft guidelines for the evaluation of research fellowship/research grant proposals, which are currently missing from http://meta.wikimedia.org/wiki/Research:FAQ
What I envision is a two-tiered process:
(1) RCom will first review proposals based on its standard procedures, regardless of funding requests. We will solicit the opinion of external referees via a single-blind review process when needed (we did this for the EPIC/Oxford proposal). We will then write our recommendations whether a specific proposal is methodologically sound, relevant and non-disruptive to our community to help WMF make a funding decision .
(2) WMF will request supplementary information to projects applying for funding and use this information, feedback from RCom and its internal assessment of the priority/usefulness of the proposal to make a funding decision.
This will help RCom focus on the research value of the proposal per se while leaving to the WMF fellowship/grant program the actual funding decision. On a related note, I am working closely with Philippe Beaudette to configure SugarCRM to help us triage, handle and assign requests for RCom review.
Please let me know if you have any comments or concerns on the overall proposal. As Siko notes, the Dispute Resolution project below is a research proposal from a community member asking for regular SR support/review, not a WMF-sponsored project, and potentially a good case to get this process started.
Dario
Begin forwarded message:
*From: *Siko Bouterse sbouterse@wikimedia.org *Date: *November 22, 2011 10:33:58 AM PST *To: *Dario Taraborelli dtaraborelli@wikimedia.org *Subject: **surveys by community members*
Hi Dario,
This is a survey request from a community member interested in learning more about his Wikipedia projects, for RCom's review: http://meta.wikimedia.org/wiki/Research:Wikipedia_Dispute_Resolution
Background: Steven Zhang is active in MedCab and the creator of some other DR pages and processes on EN:WP. I've been speaking with him about the possibility of doing a fellowship on dispute resolution, though an exact project is still to be pinned down and nothing is approved for fellowship at this point. Although the survey is not an official WMF project, and Steven is acting in the capacity of community member, I am interested in the results of his survey to learn more about current issues with DR and see if there are projects that we should support in the form of a fellowship.
This may be a growing need, I've gotten a couple of similar inquiries so far and expect they will increase as we ramp up community fellowships. I'm curious to know what the RCom process looks like for surveys run by community members, some of whom might not have the same research background or methodological training as academic researchers, but are motivated to learn and share understanding about their community and projects. Is this something worth asking about on the RCom list? (If so, feel free to forward my message).
In this case, its a relatively small sample size so hopefully not too disruptive. I think Steven could also use some guidance about what free survey collector would be recommended for use - is RCom ok with something simple like Google Forms or have other recommendations?
Thanks! Siko
-- Siko Bouterse Head of Community Fellowships Wikimedia Foundation, Inc.
sbouterse@wikimedia.org
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
-- Siko Bouterse Head of Community Fellowships Wikimedia Foundation, Inc.
sbouterse@wikimedia.org
-- Siko Bouterse Head of Community Fellowships Wikimedia Foundation, Inc.
sbouterse@wikimedia.org
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
-- Mani Pande, PhD Head of Global Development Research Twitter: manipande Skype: manipande
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
On Tue, 29 Nov 2011 17:23:05 -0800, Mani Pande mpande@wikimedia.org wrote:
I think it is a good idea to have an academic researcher consult, but we need to be careful about who is given these responsibilities: do they
have
a social science background, what is their background in designing questionnaires, how familiar they are with academic regulations about conducting research with human subjects etc.
In case of multiple concerns we could contact several persons or even form a team that they decice between themselves.
Cheers Yaroslav