Dear Christina,

1 are you defining your super editors by total or recent edits? Whilst we have pretty good editor retention amongst high edit count editors, even amongst those with over a 100,000 edits there are inactive and semi active editors.

2 how are you going to ensure that talkpage invites are only responded to by the targeted editors? 

3 have you considered emailing your survey? Yes that loses you at least the 30% who haven't set an email, but you are much more likely to get your responses from the intended target group, also it is quite an effective way to contact the inactive and former editors who might not see a talkpage note.

4 What are you going to do to avoid trying to survey deceased Wikipedians? Especially with talkpage notes.

5 how does one make requests to add other questions to your survey?

6 you mention using census categories to ask the ethnicity question, may one ask whose census, Australia, Canada, India, the UK or the USA? Also are you intending to replicate the census questions or base your questions literally on the census categories generated from those questions?

Regards

Jonathan Cardy

On 12 Apr 2015, at 20:49, Christina Shane-Simpson <christinam.shane@gmail.com> wrote:

Hello Aaron and Other Wiki Researchers,

Thank you for responding so quickly and thoroughly to my recent proposal!  Many of your concerns align with issues I’ve been discussing with my research team, so I’m glad to hear that we’re overlapping in that sense.  Apologies in advance for the length of the following:

-         - Sampling:  I completely agree with your concerns in response to the (relatively) recent revisit to the original Gender Gap results.  As an exploratory study, I don’t think we could accurately represent the entire Wikipedia community or make causal inferences about the community as a whole due to the voluntary nature of the survey and the potential for inaccuracies in self-reporting.  However, I’m hoping that this preliminary project could reveal a few new patterns that might be explored in greater depth at a later date.

Based on the Wikipedia editor rankings, I’d planned to pull the top 20% of editors and post on their Talk Pages, giving us the “super-editor” sample.  Since the two remaining samples are more difficult to recruit, I’m currently exploring the most effective way to obtain a randomized sample of the active (moderate) and inactive editors (infrequent edits) – this will likely be developed with the assistance of someone more skilled in programming than myself.  I’ve also been speaking with a statistician about alternative methods, beyond propensity-matching, where we might account for response biases that are likely to occur.  However, I’d be very open to suggestions from this community about effectively sampling from Wikipedia and methods you’ve used to account for biases common in these surveys.  

-         - Self-Report Measures of Edit History:  This would only serve to verify the editor ranking and provide a more thorough context by which the editor feels he/she makes contributions to the Wikipedia community.  Since we’ll have usernames – via Talk Pages – as you suggested, I’d like to explore actual editing behaviors given that we’d have the resources to do so.

-        - Collaboration:  Participant fatigue is a huge concern with all of these online surveys targeting active editors.  I believe you’re correct that the WMF is planning another editor survey, but I had hoped to provide some foundation for other themes that might be explored in these larger surveys.  The prior WMF surveys didn’t provide as much depth as we might need to reveal any patterns in editing behaviors.  I’ve also reached out to a couple of other proposals, with similar interests, to determine whether we can compliment each other’s efforts.  I think these types of collaborations are very do-able and may help us to limit the frequency of Wikipedia editor surveys.

        - Missing Measures and People:  I was able to access your article, so thank you for linking it!  I’ve been reviewing the literature to clarify variables (such as the web use you identify) to determine which should be included in the survey.  In order to keep the survey at a reasonable length, I’d hoped to capture some of these editing barriers via themes captured in the open-ended responses.  This might be particularly relevant in the context of editors’ perceived barriers, which might vary based on the aforementioned traits.  However, I agree that the study would likely benefit form some further questioning about editing experiences and I’ll be adding this into the proposal.

- Missing People and Sampling:  Your main concern also parallels the concerns of my research team.  I’ve been speaking with my team about potentially recruiting a passive Wikipedia user sample that would serve as a comparison.  It was my original hope that a small incentive would encourage even the infrequent editors to complete the survey measure, but in the event that they don’t we’ll need that comparison group.  Our greatest barrier would be matching the “pertinent” comparison sample characteristics with our super-editors.   I’m not sure that we can achieve this yet, but more to come as I explore this option.

Thank you again Aaron for your thorough feedback!  As I’ve been following this listserv, I’m incredibly grateful that we have developed such a strong research-oriented Wikipedia community.  

Sincerely,

Christina

_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l