On Tue, Feb 17, 2015 at 8:23 AM, aaron shaw <aaronshaw(a)northwestern.edu
wrote:
...
> On Mon, Feb 16, 2015 at 11:38 PM, Federico Leva (Nemo) <nemowiki(a)gmail.com
> > wrote:
>
>> Speaking of which, the WMF doesn't have resources to appropriately
>> process the 2012 survey data, so results aren't available yet. Did you
>> consider offering them to take care of it, at least for the gendergap
>> number? You would then be able to publish an update.
>
>>
https://meta.wikimedia.org/wiki/Research_talk:Wikipedia_Editor_Survey_2012#…
> As before, my understanding is that the
method by which respondents were
> selected to participate in the survey does not meet standard methods of
> survey sampling (see this chunk
>
<https://meta.wikimedia.org/wiki/Research:Wikipedia_Editor_Survey_2012#When.2C_and_how_often_will_the_survey_be_conducted.3F>
of
> the description of the survey). As a result, I do not trust the results of
> the 2012 survey to generate precise estimates of the gender gap or other
> demographic details about participation.
To clarify, as this has since led to misunderstandings elsewhere: The 2012
editor survey used the same sampling method with the same limitations as
the April 2011 and November/December 2011 editor surveys. (By design, as
one of the main objectives was to gain comparable data and identify
trends.) And the 2008 UNU-MERIT survey that forms the basis of Aaron's and
Mako's paper essentially used the same method too - with the exception of
the aspects (1) and (2) Aaron described below, namely that it sampled both
readers and editors, and that there happened to be a comparable survey,
conducted by Pew around the same time, that could be assumed not to exhibit
the same type of participation bias.
Thanks to Aaron's and Mako's paper, the limitations of this method and the
fact that it likely leads to an underestimation of the female ratio among
Wikipedia editors are now better understood. But I'm not aware of research
that has used a fundamentally more reliable method to investigate
Wikipedia's gender gap; so for now such web-based volunteer surveys
continue to inform our awareness of the topic.
After Aaron's and Mako's research became available, I read the 2011 paper
which their correction method is based on (Valliant and Dever, "Estimating
propensity adjustments for volunteer web surveys"), and Aaron and I have
talked several times about the possibility of finding a weaker form of that
method that - by extrapolating some of the 2008 information - could be
applied to the 2012 survey despite the lack of comparison data (i.e. (1) &
(2)). But as he said below, we are not aware of a good option for doing
that.
I have myself been nudging some other people in the Foundation who were
preparing or considering more specialized user surveys to look at the
option of constructing them in a way that enables the use of Aaron's and
Mako's method, but it has to be said that the requirement to include a
reader sample (i.e. (1)) can come at a cost, and that the equivalent of
that Pew survey ((2)) might not be available in many of the countries that
one is interested in.
> I've spoken to some very receptive folks at the foundation about this and
> I hope that they/we will be able to improve it in the future. I'm eager to
> help improve the survey data collection procedures. Unfortunately, I do not
> have the capacity to analyze the current survey data in greater depth.
> The thing that allowed Mako and I to do the
study that we published in
> PLOSONE was the fact that (1) the old UNU-Merit & WMF survey sought to
> include readers as well as editors; *and* (2) at the exact same time Pew
> carried out a survey in which they asked a nearly identical question about
> readership. We used the overlapping results about WP readership from both
> surveys to generate a correction for the data about editorship. Without
> similar data on readership and similar data from a representative sample of
> some reference population (in the case of the pew survey, US adults), we
> cannot perform the same correction. As a result, I do not feel comfortable
> estimating how biased (or unbiased) the 2012 survey results may be.
> a
On Tue, Feb 17, 2015 at 10:31 AM, Michael Restivo <marestivo(a)gmail.com
wrote:
> Wikipedia Signpost had a discussion of this question, including data on
> English Wikipedians' gender by edits:
>
https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2011-02-14/News_…
Yes, but as I wrote in that Signpost article,
that data relied on the
gender editors state in their user preferences and "this information is
optional and the majority of accounts do not state it". There a good
reasons to assume that the differing incentives distort that data even more
than the anonymous responses to banner-advertised surveys. For example, the
user has to be comfortable with stating their gender in public, and in several
languages
<https://en.wikipedia.org/wiki/Gender_neutrality_in_languages_with_grammatical_gender
female
users have to set that user preference if they want the word "user" next to
their nick show up in female instead of male grammatical gender form (e.g.
"Benutzerin" vs. "Benutzer" in German) - male users do not have that
incentive.
--
Tilman Bayer
Senior Analyst
Wikimedia Foundation
IRC (Freenode): HaeB