Re: [Wiki-research-l] Grant Proposal: Request for Feedback

Greeting Christina! Thanks for sharing this and notifying us on the list. Overall, I am very supportive of additional attempts to do more rigorous survey research on Wikipedians. Some questions that I think you could try to address in the proposal: - *Sampling*: You mention that you plan to stratify your sample based on past edit history and recruit via talk page messages. However, beyond this you say nothing about the logistics of subject sampling, recruitment, or any approaches you will take to address the fact that conducting representative surveys in online communities is very, very difficult. Can you elaborate on this aspect of your study? In particular, how will your approach address shortcomings in data and sample quality that have affected previous surveys <http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0065782> of Wikipedia contributors? - *Self-report measures of edit history: *Why ask the respondents to self-report their edit histories (this kind of thing is notoriously hard to do accurately) when you could ask them to provide their usernames or at least link their usernames to their survey responses (since you're recruiting via talk page messages anyway)? - *Collaboration w related studies: *There are several other ongoing efforts to survey wikipedians -- even at least one other one <http://dl.acm.org/citation.cfm?id=2145264>(link to "official" publication is gated, but other versions are available for free) focused on social psychological concerns. Also, my impression is that the WMF is involved in planning another editor survey in the near future. How will your approach complement/extend/overlap with these other efforts? Will you make any effort to collaborate with these ongoing studies? How will your study avoid subject exhaustion -- especially among more active wikipedians who may find themselves invited to participate in many surveys? - *Missing measures and missing people:* Previous studies have shown that a variety of additional factors may figure in shaping the participation practices of Wikipedians as well as those who might edit Wikipedia but choose not to do so. For example, in a recent paper <http://www.tandfonline.com/doi/abs/10.1080/1369118X.2014.957711#.VSqb244adE4> (again, gated link, but I am also happy to provide copies to those who would like access) that I co-wrote w Eszter Hargittai, we find that web use skills are, in some ways, even more robust predictors of wikipedia contribution than gender. There are many other examples of important measures that predict participation in various ways as well, whether it be individual's trust/caution attitudes, newcomer experiences, etc. Which of these measures will you include? How will you ensure that you have included the most important measures in this survey study since survey results are otherwise quitre prone to omitted variable bias? *Missing people and sampling on the dependent variable: *Maybe most importantly, insofar as you say that you are interested in understanding factors that determine who edits, you are selecting on the dependent variable (wikipedia editing) by limiting your study to individuals who have accounts on the encyclopedia and edit already. It strikes me as especially egregious that you are requiring survey respondents to read and reply to the survey recruitment materials via talk page message. This means that precisely those individuals who participate least (and who would provide your study with necessary variation on the outcome of interest) are the least likely to respond and to be included in the study. As a result, I fear that your findings will not speak to these questions effectively unless you find an alternative method of sampling and recruitment. I hope that these comments are helpful for you as you continue to refine the study design. I really think you're pursuing a critical set of concerns in this study and I am eager to see it succeed in the most effective way possible! yours, Aaron On Sun, Apr 12, 2015 at 8:44 AM, Christina Shane-Simpson < christinam.shane(a)gmail.com&gt; wrote: