On Tue, Feb 17, 2015 at 8:23 AM, aaron shaw <aaronshaw@northwestern.edu> wrote:

...

On Mon, Feb 16, 2015 at 11:38 PM, Federico Leva (Nemo) <nemowiki@gmail.com> wrote:

Speaking of which, the WMF doesn't have resources to appropriately
process the 2012 survey data, so results aren't available yet. Did you
consider offering them to take care of it, at least for the gendergap
number? You would then be able to publish an update.
https://meta.wikimedia.org/wiki/Research_talk:Wikipedia_Editor_Survey_2012#Looking_for_survey_results

As before, my understanding is that the method by which respondents were selected to participate in the survey does not meet standard methods of survey sampling (see this chunk of the description of the survey). As a result, I do not trust the results of the 2012 survey to generate precise estimates of the gender gap or other demographic details about participation.

To clarify, as this has since led to misunderstandings elsewhere: The 2012 editor survey used the same sampling method with the same limitations as the April 2011 and November/December 2011 editor surveys. (By design, as one of the main objectives was to gain comparable data and identify trends.) And the 2008 UNU-MERIT survey that forms the basis of Aaron's and Mako's paper essentially used the same method too - with the exception of the aspects (1) and (2) Aaron described below, namely that it sampled both readers and editors, and that there happened to be a comparable survey, conducted by Pew around the same time, that could be assumed not to exhibit the same type of participation bias.

Thanks to Aaron's and Mako's paper, the limitations of this method and the fact that it likely leads to an underestimation of the female ratio among Wikipedia editors are now better understood. But I'm not aware of research that has used a fundamentally more reliable method to investigate Wikipedia's gender gap; so for now such web-based volunteer surveys continue to inform our awareness of the topic.

After Aaron's and Mako's research became available, I read the 2011 paper which their correction method is based on (Valliant and Dever, "Estimating propensity adjustments for volunteer web surveys"), and Aaron and I have talked several times about the possibility of finding a weaker form of that method that - by extrapolating some of the 2008 information - could be applied to the 2012 survey despite the lack of comparison data (i.e. (1) & (2)). But as he said below, we are not aware of a good option for doing that.

I have myself been nudging some other people in the Foundation who were preparing or considering more specialized user surveys to look at the option of constructing them in a way that enables the use of Aaron's and Mako's method, but it has to be said that the requirement to include a reader sample (i.e. (1)) can come at a cost, and that the equivalent of that Pew survey ((2)) might not be available in many of the countries that one is interested in.

I've spoken to some very receptive folks at the foundation about this and I hope that they/we will be able to improve it in the future. I'm eager to help improve the survey data collection procedures. Unfortunately, I do not have the capacity to analyze the current survey data in greater depth.

The thing that allowed Mako and I to do the study that we published in PLOSONE was the fact that (1) the old UNU-Merit & WMF survey sought to include readers as well as editors; *and* (2) at the exact same time Pew carried out a survey in which they asked a nearly identical question about readership. We used the overlapping results about WP readership from both surveys to generate a correction for the data about editorship. Without similar data on readership and similar data from a representative sample of some reference population (in the case of the pew survey, US adults), we cannot perform the same correction. As a result, I do not feel comfortable estimating how biased (or unbiased) the 2012 survey results may be.

a

On Tue, Feb 17, 2015 at 10:31 AM, Michael Restivo <marestivo@gmail.com> wrote:

Wikipedia Signpost had a discussion of this question, including data on English Wikipedians' gender by edits: https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2011-02-14/News_and_notes

Yes, but as I wrote in that Signpost article, that data relied on the gender editors state in their user preferences and "this information is optional and the majority of accounts do not state it". There a good reasons to assume that the differing incentives distort that data even more than the anonymous responses to banner-advertised surveys. For example, the user has to be comfortable with stating their gender in public, and in several languages female users have to set that user preference if they want the word "user" next to their nick show up in female instead of male grammatical gender form (e.g. "Benutzerin" vs. "Benutzer" in German) - male users do not have that incentive.

Tilman Bayer
Senior Analyst
Wikimedia Foundation
IRC (Freenode): HaeB