All,
I received an urgent request by a team of researchers who have been waiting for several months for an approval to run a survey of Wikipedia editors [1]. The team includes researchers based at Harvard and Sciences Po. One RCom member (Mayo) is also involved in this project.
The project aims to recruit participants by running banners for logged-in users on the English Wikipedia and the team previously sought community consensus for this recruitment method via a discussion on the Admin Noticeboard [2]. The planning of this project predates the creation of the Research Committee and the community discussion was not explicitly mentioned in the project page, as a result most people on RCom (myself included) were entirely unaware of it.
Over the last weeks we have successfully been able to channel new recruitment requests to Meta, where RCom and community members can discuss proposed recruitment methods for various studies. I told Jérôme and Mayo from the Sciences Po/Harvard team that their project should be no exception and I would like to solicit RCom members to comment on this proposal over the next days.
The request is sensitive not only for its tight timeline (there was apparently a commitment to get the banner campaign started today and the team has already allocated engineers to this project next week), but also because it would be the first time ever that we use the CentralNotice for research projects, and this is a decision that may have implications for future studies.
I'd like to have your thoughts on this proposal via its discussion page by Monday night (Pacific time) at the latest. I am particularly interested in hearing on this from community members (such as Ziko, WereSpielChequers, Milos, Steven). Because of her direct involvement in the project, Mayo won't be participating in the discussion.
I attach below a letter that Jérôme addressed to the Research Committee to document the history of this project. I look forward to hearing from you.
Best Dario
[1] http://meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and_... [2] http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archive...
Begin forwarded message:
From: Jérôme HERGUEUX jerome.hergueux@sciences-po.org Date: June 22, 2011 6:28:19 PM PDT To: dtaraborelli@wikimedia.org Cc: Mayo.Fuster@eui.eu Subject: Follow up Mayo chat Reply-To: Jérôme HERGUEUX jerome.hergueux@sciences-po.org x-mailer: ContactOffice Mail
Dear Dario,
Thank you for helping us out with this!
Please find attached:
- A document explaining rapidly the history of our research project and where we stand now.
- The original code of the banner we prepared to advertize the study (normally the folks coding the banners at the Foundation should be aware of it).
I've linked to our Early research protocol discussion on AN in the link section of our Talk Page on Meta: http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interactions... I'll try to add some relevant info to our research project template very soon.
Best,
Jérôme (User:SalimJah).
-------------------------------------------------------------------------- Tous les courriers électroniques émis depuis la messagerie de Sciences Po doivent respecter des conditions d'usages. Pour les consulter rendez-vous sur http://www.ressources-numeriques.sciences-po.fr/confidentialite_courriel.htm
Dear all,
I had a look at the documents and find their approach reasonable. What I am not convinced about is the necessity of running the notice for two weeks - I would expect that getting 2000 completed questionnaires (even for 25min) out of the whole population of all Wikimedians should take less time.
Anyway, I volunteer to take the survey and to report back to RCom as to how fit I see the survey design for the intended research purpose.
As for the implications of using CentralNotice this time, we could think of limiting the number and length of slots available for such purposes for a certain period of time.
Cheers,
Daniel
On Thu, Jun 23, 2011 at 7:44 PM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
All, I received an urgent request by a team of researchers who have been waiting for several months for an approval to run a survey of Wikipedia editors [1]. The team includes researchers based at Harvard and Sciences Po. One RCom member (Mayo) is also involved in this project. The project aims to recruit participants by running banners for logged-in users on the English Wikipedia and the team previously sought community consensus for this recruitment method via a discussion on the Admin Noticeboard [2]. The planning of this project predates the creation of the Research Committee and the community discussion was not explicitly mentioned in the project page, as a result most people on RCom (myself included) were entirely unaware of it. Over the last weeks we have successfully been able to channel new recruitment requests to Meta, where RCom and community members can discuss proposed recruitment methods for various studies. I told Jérôme and Mayo from the Sciences Po/Harvard team that their project should be no exception and I would like to solicit RCom members to comment on this proposal over the next days. The request is sensitive not only for its tight timeline (there was apparently a commitment to get the banner campaign started today and the team has already allocated engineers to this project next week), but also because it would be the first time ever that we use the CentralNotice for research projects, and this is a decision that may have implications for future studies. I'd like to have your thoughts on this proposal via its discussion page by Monday night (Pacific time) at the latest. I am particularly interested in hearing on this from community members (such as Ziko, WereSpielChequers, Milos, Steven). Because of her direct involvement in the project, Mayo won't be participating in the discussion. I attach below a letter that Jérôme addressed to the Research Committee to document the history of this project. I look forward to hearing from you. Best Dario [1] http://meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and_... [2] http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archive... Begin forwarded message:
From: Jérôme HERGUEUX jerome.hergueux@sciences-po.org Date: June 22, 2011 6:28:19 PM PDT To: dtaraborelli@wikimedia.org Cc: Mayo.Fuster@eui.eu Subject: Follow up Mayo chat Reply-To: Jérôme HERGUEUX jerome.hergueux@sciences-po.org x-mailer: ContactOffice Mail
Dear Dario,
Thank you for helping us out with this!
Please find attached:
- A document explaining rapidly the history of our research project and
where we stand now. 2. The original code of the banner we prepared to advertize the study (normally the folks coding the banners at the Foundation should be aware of it).
I've linked to our Early research protocol discussion on AN in the link section of our Talk Page on Meta: http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interactions... I'll try to add some relevant info to our research project template very soon.
Best,
Jérôme (User:SalimJah).
Tous les courriers électroniques émis depuis la messagerie de Sciences Po doivent respecter des conditions d'usages. Pour les consulter rendez-vous sur http://www.ressources-numeriques.sciences-po.fr/confidentialite_courriel.htm
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
Forgot to mention: Another question I am not clear about is whether the survey is available in all the languages covered by CentralNotice.
On Mon, Jun 27, 2011 at 12:20 AM, Daniel Mietchen daniel.mietchen@googlemail.com wrote:
Dear all,
I had a look at the documents and find their approach reasonable. What I am not convinced about is the necessity of running the notice for two weeks - I would expect that getting 2000 completed questionnaires (even for 25min) out of the whole population of all Wikimedians should take less time.
Anyway, I volunteer to take the survey and to report back to RCom as to how fit I see the survey design for the intended research purpose.
As for the implications of using CentralNotice this time, we could think of limiting the number and length of slots available for such purposes for a certain period of time.
Cheers,
Daniel
On Thu, Jun 23, 2011 at 7:44 PM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
All, I received an urgent request by a team of researchers who have been waiting for several months for an approval to run a survey of Wikipedia editors [1]. The team includes researchers based at Harvard and Sciences Po. One RCom member (Mayo) is also involved in this project. The project aims to recruit participants by running banners for logged-in users on the English Wikipedia and the team previously sought community consensus for this recruitment method via a discussion on the Admin Noticeboard [2]. The planning of this project predates the creation of the Research Committee and the community discussion was not explicitly mentioned in the project page, as a result most people on RCom (myself included) were entirely unaware of it. Over the last weeks we have successfully been able to channel new recruitment requests to Meta, where RCom and community members can discuss proposed recruitment methods for various studies. I told Jérôme and Mayo from the Sciences Po/Harvard team that their project should be no exception and I would like to solicit RCom members to comment on this proposal over the next days. The request is sensitive not only for its tight timeline (there was apparently a commitment to get the banner campaign started today and the team has already allocated engineers to this project next week), but also because it would be the first time ever that we use the CentralNotice for research projects, and this is a decision that may have implications for future studies. I'd like to have your thoughts on this proposal via its discussion page by Monday night (Pacific time) at the latest. I am particularly interested in hearing on this from community members (such as Ziko, WereSpielChequers, Milos, Steven). Because of her direct involvement in the project, Mayo won't be participating in the discussion. I attach below a letter that Jérôme addressed to the Research Committee to document the history of this project. I look forward to hearing from you. Best Dario [1] http://meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and_... [2] http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archive... Begin forwarded message:
From: Jérôme HERGUEUX jerome.hergueux@sciences-po.org Date: June 22, 2011 6:28:19 PM PDT To: dtaraborelli@wikimedia.org Cc: Mayo.Fuster@eui.eu Subject: Follow up Mayo chat Reply-To: Jérôme HERGUEUX jerome.hergueux@sciences-po.org x-mailer: ContactOffice Mail
Dear Dario,
Thank you for helping us out with this!
Please find attached:
- A document explaining rapidly the history of our research project and
where we stand now. 2. The original code of the banner we prepared to advertize the study (normally the folks coding the banners at the Foundation should be aware of it).
I've linked to our Early research protocol discussion on AN in the link section of our Talk Page on Meta: http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interactions... I'll try to add some relevant info to our research project template very soon.
Best,
Jérôme (User:SalimJah).
Tous les courriers électroniques émis depuis la messagerie de Sciences Po doivent respecter des conditions d'usages. Pour les consulter rendez-vous sur http://www.ressources-numeriques.sciences-po.fr/confidentialite_courriel.htm
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
I understand they are only targeting enwiki (Mayo, can you confirm?) and CentralNotice can be configured to serve language and project-specific banners.
Dario
On Jun 26, 2011, at 3:21 PM, Daniel Mietchen wrote:
Forgot to mention: Another question I am not clear about is whether the survey is available in all the languages covered by CentralNotice.
Hi all,
I just took the survey. Here are my observations:
Paypal account: why ask before the survey is completed? And why make entering an email address compulsory if there is an option to donate the money to the WMF, under which there would be no need for an email address (other than to control for duplicate entries, perhaps).
There is ambiguity in questions of the "did you have an idea" kind about other participants' behaviour: Although it sounded more like "did you have some imagination as to the probability distribution of play behaviour amongst other players", I took it to mean "did you *know* how the others played?". In my view, either question only makes sense as a way to estimate how carefully one has read the instructions (in which the correct answer was given, although only roughly for the first).
I was surprised (and disappointed) that the game theoretical questions ended so quickly after all the effort that went into explaining (and understanding) the rules. Also, such decision-making scenarios tend to be different between single-play and repeated-play conditions, and in the latter case, depending upon whether the players (or their behaviour) change or not, and whether this is known to the others.
The demographic part is fairly standard, though some questions (e.g. on country of citizenship, birth, birth of mother, birth of father) seem a bit far-fetched for an analysis of this kind (and it's annoying to enter if all four are the same). So here we have indeed the case of "repeated surveys with similar questions", as WereSpielChequers had put it.
The questions right after the demographic part are probably the most relevant ones from an RCom perspective, and they are posed mostly such that their analysis promises interesting results. There was one that I found a bit odd, but can't remember any details now.
The question on monthly gross income could do with a currency converter (not everyone, even on enwiki, thinks in USD).
Typo: * Where you in a calm environment when you answered the questions?
The technical implementation is excellent, and it took me exactly the 25min that had been advertised.
Daniel
On Mon, Jun 27, 2011 at 7:54 PM, Fuster, Mayo Mayo.Fuster@eui.eu wrote:
Hello!
I understand they are only targeting enwiki (Mayo, can you confirm?)
Yes, that's right.
Have a nice day, Mayo
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
thanks Daniel, fancy copying this to the talk page of the project?
Dario
On Jun 27, 2011, at 1:33 PM, Daniel Mietchen wrote:
Hi all,
I just took the survey. Here are my observations:
Paypal account: why ask before the survey is completed? And why make entering an email address compulsory if there is an option to donate the money to the WMF, under which there would be no need for an email address (other than to control for duplicate entries, perhaps).
There is ambiguity in questions of the "did you have an idea" kind about other participants' behaviour: Although it sounded more like "did you have some imagination as to the probability distribution of play behaviour amongst other players", I took it to mean "did you *know* how the others played?". In my view, either question only makes sense as a way to estimate how carefully one has read the instructions (in which the correct answer was given, although only roughly for the first).
I was surprised (and disappointed) that the game theoretical questions ended so quickly after all the effort that went into explaining (and understanding) the rules. Also, such decision-making scenarios tend to be different between single-play and repeated-play conditions, and in the latter case, depending upon whether the players (or their behaviour) change or not, and whether this is known to the others.
The demographic part is fairly standard, though some questions (e.g. on country of citizenship, birth, birth of mother, birth of father) seem a bit far-fetched for an analysis of this kind (and it's annoying to enter if all four are the same). So here we have indeed the case of "repeated surveys with similar questions", as WereSpielChequers had put it.
The questions right after the demographic part are probably the most relevant ones from an RCom perspective, and they are posed mostly such that their analysis promises interesting results. There was one that I found a bit odd, but can't remember any details now.
The question on monthly gross income could do with a currency converter (not everyone, even on enwiki, thinks in USD).
Typo: * Where you in a calm environment when you answered the questions?
The technical implementation is excellent, and it took me exactly the 25min that had been advertised.
Daniel
On Mon, Jun 27, 2011 at 7:54 PM, Fuster, Mayo Mayo.Fuster@eui.eu wrote:
Hello!
I understand they are only targeting enwiki (Mayo, can you confirm?)
Yes, that's right.
Have a nice day, Mayo
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
I've made some specific comments on the talkpage, but hadn't appreciated that they were planning to run this for two weeks. Perhaps naively I'd assumed the plan was to switch this off once they'd got 2,000 responses.
The more I think about this and the numbers involved, the more I move to the idea that we should be expanding our existing survey into an omnibus survey that carries questions for legitimate researchers like these.
That way the researchers would benefit from a larger more robust dataset and the editors benefit from questions not being duplicated, research not becoming spam and from greater confidence re privacy. The researchers would get more data - perhaps 5,000 individual but anonymised records with common questions such as age band, education level, motive for editing etc, and of course their own question or questions - dress size, operating system, number of times editor has been abducted by UFOs or whatever.
My experience is that if people are persuaded to do a survey they won't be bothered at the length so much as they are by questions they can't answer without looking something up or talking to someone else. But repeated surveys with similar questions are a turnoff.
WSC
On 26 June 2011 23:20, Daniel Mietchen daniel.mietchen@googlemail.com wrote:
Dear all,
I had a look at the documents and find their approach reasonable. What I am not convinced about is the necessity of running the notice for two weeks - I would expect that getting 2000 completed questionnaires (even for 25min) out of the whole population of all Wikimedians should take less time.
Anyway, I volunteer to take the survey and to report back to RCom as to how fit I see the survey design for the intended research purpose.
As for the implications of using CentralNotice this time, we could think of limiting the number and length of slots available for such purposes for a certain period of time.
Cheers,
Daniel
On Thu, Jun 23, 2011 at 7:44 PM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
All, I received an urgent request by a team of researchers who have been waiting for several months for an approval to run a survey of Wikipedia editors [1]. The team includes researchers based at Harvard and Sciences Po. One RCom member (Mayo) is also involved in this project. The project aims to recruit participants by running banners for logged-in users on the English Wikipedia and the team previously sought community consensus for this recruitment method via a discussion on the Admin Noticeboard [2]. The planning of this project predates the creation of the Research Committee and the community discussion was not explicitly mentioned in the project page, as a result most people on RCom (myself included) were entirely unaware of it. Over the last weeks we have successfully been able to channel new recruitment requests to Meta, where RCom and community members can discuss proposed recruitment methods for various studies. I told Jérôme and Mayo from the Sciences Po/Harvard team that their project should be no exception and I would like to solicit RCom members to comment on this proposal over the next days. The request is sensitive not only for its tight timeline (there was apparently a commitment to get the banner campaign started today and the team has already allocated engineers to this project next week), but also because it would be the first time ever that we use the CentralNotice for research projects, and this is a decision that may have implications for future studies. I'd like to have your thoughts on this proposal via its discussion page by Monday night (Pacific time) at the latest. I am particularly interested in hearing on this from community members (such as Ziko, WereSpielChequers, Milos, Steven). Because of her direct involvement in the project, Mayo won't be participating in the discussion. I attach below a letter that Jérôme addressed to the Research Committee to document the history of this project. I look forward to hearing from you. Best Dario [1] http://meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and_... [2] http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archive... Begin forwarded message:
From: Jérôme HERGUEUX jerome.hergueux@sciences-po.org Date: June 22, 2011 6:28:19 PM PDT To: dtaraborelli@wikimedia.org Cc: Mayo.Fuster@eui.eu Subject: Follow up Mayo chat Reply-To: Jérôme HERGUEUX jerome.hergueux@sciences-po.org x-mailer: ContactOffice Mail
Dear Dario,
Thank you for helping us out with this!
Please find attached:
- A document explaining rapidly the history of our research project and
where we stand now. 2. The original code of the banner we prepared to advertize the study (normally the folks coding the banners at the Foundation should be aware of it).
I've linked to our Early research protocol discussion on AN in the link section of our Talk Page on Meta: http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interactions... I'll try to add some relevant info to our research project template very soon.
Best,
Jérôme (User:SalimJah).
Tous les courriers électroniques émis depuis la messagerie de Sciences Po doivent respecter des conditions d'usages. Pour les consulter rendez-vous sur http://www.ressources-numeriques.sciences-po.fr/confidentialite_courriel.htm
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
Daniel, WSC – thanks for your feedback: I agree there is no reason for the banner to run for 2 weeks if they get their complete 2K sample before then.
The omnibus survey also sounds like an interesting plan, but I suspect many researchers may feel uncomfortable not to have control on the exact design of the survey. On the other hand we definitely need to find ways to avoid redundant recruitment requests. I'll add a specific item on this issue to the agenda of the next RCom meeting.
Dario
On Jun 26, 2011, at 3:46 PM, WereSpielChequers wrote:
I've made some specific comments on the talkpage, but hadn't appreciated that they were planning to run this for two weeks. Perhaps naively I'd assumed the plan was to switch this off once they'd got 2,000 responses.
The more I think about this and the numbers involved, the more I move to the idea that we should be expanding our existing survey into an omnibus survey that carries questions for legitimate researchers like these.
That way the researchers would benefit from a larger more robust dataset and the editors benefit from questions not being duplicated, research not becoming spam and from greater confidence re privacy. The researchers would get more data - perhaps 5,000 individual but anonymised records with common questions such as age band, education level, motive for editing etc, and of course their own question or questions - dress size, operating system, number of times editor has been abducted by UFOs or whatever.
My experience is that if people are persuaded to do a survey they won't be bothered at the length so much as they are by questions they can't answer without looking something up or talking to someone else. But repeated surveys with similar questions are a turnoff.
WSC
On 26 June 2011 23:20, Daniel Mietchen daniel.mietchen@googlemail.com wrote:
Dear all,
I had a look at the documents and find their approach reasonable. What I am not convinced about is the necessity of running the notice for two weeks - I would expect that getting 2000 completed questionnaires (even for 25min) out of the whole population of all Wikimedians should take less time.
Anyway, I volunteer to take the survey and to report back to RCom as to how fit I see the survey design for the intended research purpose.
As for the implications of using CentralNotice this time, we could think of limiting the number and length of slots available for such purposes for a certain period of time.
Cheers,
Daniel
On Thu, Jun 23, 2011 at 7:44 PM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
All, I received an urgent request by a team of researchers who have been waiting for several months for an approval to run a survey of Wikipedia editors [1]. The team includes researchers based at Harvard and Sciences Po. One RCom member (Mayo) is also involved in this project. The project aims to recruit participants by running banners for logged-in users on the English Wikipedia and the team previously sought community consensus for this recruitment method via a discussion on the Admin Noticeboard [2]. The planning of this project predates the creation of the Research Committee and the community discussion was not explicitly mentioned in the project page, as a result most people on RCom (myself included) were entirely unaware of it. Over the last weeks we have successfully been able to channel new recruitment requests to Meta, where RCom and community members can discuss proposed recruitment methods for various studies. I told Jérôme and Mayo from the Sciences Po/Harvard team that their project should be no exception and I would like to solicit RCom members to comment on this proposal over the next days. The request is sensitive not only for its tight timeline (there was apparently a commitment to get the banner campaign started today and the team has already allocated engineers to this project next week), but also because it would be the first time ever that we use the CentralNotice for research projects, and this is a decision that may have implications for future studies. I'd like to have your thoughts on this proposal via its discussion page by Monday night (Pacific time) at the latest. I am particularly interested in hearing on this from community members (such as Ziko, WereSpielChequers, Milos, Steven). Because of her direct involvement in the project, Mayo won't be participating in the discussion. I attach below a letter that Jérôme addressed to the Research Committee to document the history of this project. I look forward to hearing from you. Best Dario [1] http://meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and_... [2] http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archive... Begin forwarded message:
From: Jérôme HERGUEUX jerome.hergueux@sciences-po.org Date: June 22, 2011 6:28:19 PM PDT To: dtaraborelli@wikimedia.org Cc: Mayo.Fuster@eui.eu Subject: Follow up Mayo chat Reply-To: Jérôme HERGUEUX jerome.hergueux@sciences-po.org x-mailer: ContactOffice Mail
Dear Dario,
Thank you for helping us out with this!
Please find attached:
- A document explaining rapidly the history of our research project and
where we stand now. 2. The original code of the banner we prepared to advertize the study (normally the folks coding the banners at the Foundation should be aware of it).
I've linked to our Early research protocol discussion on AN in the link section of our Talk Page on Meta: http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interactions... I'll try to add some relevant info to our research project template very soon.
Best,
Jérôme (User:SalimJah).
Tous les courriers électroniques émis depuis la messagerie de Sciences Po doivent respecter des conditions d'usages. Pour les consulter rendez-vous sur http://www.ressources-numeriques.sciences-po.fr/confidentialite_courriel.htm
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
With Omnibus surveys there are tradeoffs - some people may have incompatible questions, though things like age and education bands should be able to resolve that.
The targetting can also be different - but a filter should resolve that - if one researcher doesn't want a subset of participants then providing their selection criteria is a survey variable they should be able to exclude the ones they don't want.
And I would predict that a big advantage to the researcher would be a greater response to a WMF officail survey - especially if it was an annual event.
WSC
On 27 June 2011 18:25, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
Daniel, WSC – thanks for your feedback: I agree there is no reason for the banner to run for 2 weeks if they get their complete 2K sample before then.
The omnibus survey also sounds like an interesting plan, but I suspect many researchers may feel uncomfortable not to have control on the exact design of the survey. On the other hand we definitely need to find ways to avoid redundant recruitment requests. I'll add a specific item on this issue to the agenda of the next RCom meeting.
Dario
On Jun 26, 2011, at 3:46 PM, WereSpielChequers wrote:
I've made some specific comments on the talkpage, but hadn't appreciated that they were planning to run this for two weeks. Perhaps naively I'd assumed the plan was to switch this off once they'd got 2,000 responses.
The more I think about this and the numbers involved, the more I move to the idea that we should be expanding our existing survey into an omnibus survey that carries questions for legitimate researchers like these.
That way the researchers would benefit from a larger more robust dataset and the editors benefit from questions not being duplicated, research not becoming spam and from greater confidence re privacy. The researchers would get more data - perhaps 5,000 individual but anonymised records with common questions such as age band, education level, motive for editing etc, and of course their own question or questions - dress size, operating system, number of times editor has been abducted by UFOs or whatever.
My experience is that if people are persuaded to do a survey they won't be bothered at the length so much as they are by questions they can't answer without looking something up or talking to someone else. But repeated surveys with similar questions are a turnoff.
WSC
On 26 June 2011 23:20, Daniel Mietchen daniel.mietchen@googlemail.com wrote:
Dear all,
I had a look at the documents and find their approach reasonable. What I am not convinced about is the necessity of running the notice for two weeks - I would expect that getting 2000 completed questionnaires (even for 25min) out of the whole population of all Wikimedians should take less time.
Anyway, I volunteer to take the survey and to report back to RCom as to how fit I see the survey design for the intended research purpose.
As for the implications of using CentralNotice this time, we could think of limiting the number and length of slots available for such purposes for a certain period of time.
Cheers,
Daniel
On Thu, Jun 23, 2011 at 7:44 PM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
All, I received an urgent request by a team of researchers who have been waiting for several months for an approval to run a survey of Wikipedia editors [1]. The team includes researchers based at Harvard and Sciences Po. One RCom member (Mayo) is also involved in this project. The project aims to recruit participants by running banners for logged-in users on the English Wikipedia and the team previously sought community consensus for this recruitment method via a discussion on the Admin Noticeboard [2]. The planning of this project predates the creation of the Research Committee and the community discussion was not explicitly mentioned in the project page, as a result most people on RCom (myself included) were entirely unaware of it. Over the last weeks we have successfully been able to channel new recruitment requests to Meta, where RCom and community members can discuss proposed recruitment methods for various studies. I told Jérôme and Mayo from the Sciences Po/Harvard team that their project should be no exception and I would like to solicit RCom members to comment on this proposal over the next days. The request is sensitive not only for its tight timeline (there was apparently a commitment to get the banner campaign started today and the team has already allocated engineers to this project next week), but also because it would be the first time ever that we use the CentralNotice for research projects, and this is a decision that may have implications for future studies. I'd like to have your thoughts on this proposal via its discussion page by Monday night (Pacific time) at the latest. I am particularly interested in hearing on this from community members (such as Ziko, WereSpielChequers, Milos, Steven). Because of her direct involvement in the project, Mayo won't be participating in the discussion. I attach below a letter that Jérôme addressed to the Research Committee to document the history of this project. I look forward to hearing from you. Best Dario [1] http://meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and_... [2] http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archive... Begin forwarded message:
From: Jérôme HERGUEUX jerome.hergueux@sciences-po.org Date: June 22, 2011 6:28:19 PM PDT To: dtaraborelli@wikimedia.org Cc: Mayo.Fuster@eui.eu Subject: Follow up Mayo chat Reply-To: Jérôme HERGUEUX jerome.hergueux@sciences-po.org x-mailer: ContactOffice Mail
Dear Dario,
Thank you for helping us out with this!
Please find attached:
- A document explaining rapidly the history of our research project and
where we stand now. 2. The original code of the banner we prepared to advertize the study (normally the folks coding the banners at the Foundation should be aware of it).
I've linked to our Early research protocol discussion on AN in the link section of our Talk Page on Meta: http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interactions... I'll try to add some relevant info to our research project template very soon.
Best,
Jérôme (User:SalimJah).
Tous les courriers électroniques émis depuis la messagerie de Sciences Po doivent respecter des conditions d'usages. Pour les consulter rendez-vous sur http://www.ressources-numeriques.sciences-po.fr/confidentialite_courriel.htm
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
On Thu, Jun 23, 2011 at 10:44 AM, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
The request is sensitive not only for its tight timeline (there was apparently a commitment to get the banner campaign started today and the team has already allocated engineers to this project next week), but also because it would be the first time ever that we use the CentralNotice for research projects, and this is a decision that may have implications for future studies.
I'd like to have your thoughts on this proposal via its discussion page *by Monday night (Pacific time)* at the latest. I am particularly interested in hearing on this from community members (such as Ziko, WereSpielChequers, Milos, Steven). Because of her direct involvement in the project, Mayo won't be participating in the discussion.
I attach below a letter that Jérôme addressed to the Research Committee to document the history of this project. I look forward to hearing from you.
Best Dario
[1] http://meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and_... [2] http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archive...
Hi everyone,
As a preliminary comment about approval of such a request, I wanted to reiterate that English Wikipedia community members commented in favor of the idea of briefly running a banner, as an alternative to the initial idea of using talk page messages. It should be acceptable if we notify people beforehand and point to that discussion.
I also spoke today with the folks who ran our fundraising banners and the recent WMF survey, and confirmed the following things
1. *When*: Based on our schedule and the lull in community banners after the Board election, etc. it is much better to get something up sooner rather than later. The longer we wait the more conflicts there may be. I am proposed we should run it starting on Tuesday, July 5th, and I and Philippe Beaudette can handle implementing the campaign for the sake of simplicity. 2. *For how long and at what capacity: *The WMF editor survey received about 1,000 responses a day running at 100% capacity, with 600 or so of those being totally clean, usable responses. That suggests we don't need to run any banner more than a week, and that ideally for study it should run about 20% for that week (to logged in users only, as stated before). That capacity will give the study a wider range of responses from timezones, rather than all of the response being from one geography. 20% will also reduce the impact on the community. 3. *What*: The banner design that was attached is generally just fine, but the current text of the proposed banner is:
Berkman Center for Internet & Society at Harvard University - Sciences Po Paris Please help advance research by participating in a quick online experiment [That is the link.] You can gain up to $50 from your participation and decide to donate them to the Wikimedia Foundation if you wish to do so
That amount of text needs to be reduced for usability reasons -- at many common browser sizes almost all of the text would not be visible. As alternate I'm proposing:
Please help with a quick online experiment by Wikipedia researchers. You can gain up to $50 for your participation and can donate it if you choose. Whatever variation we use, that's a large amount of text already for an effective banner, so less it more.
As for future cases, I think we can take this opportunity to draft a policy for similar requests going forward, which I would suggest begins with a requirement for community approval prior to running any survey for external researchers. I think this case is special since we have such a close relationship with the research team already, plus several Wikipedians encouraged them to run a banner to meet their needs.
Thanks,