I've had this conversation repeatedly regarding Wikimedia related data.  Surveys that I have seen on Wikimedia are often self selecting: people answer because they have a specific motivation to do so.  There are no apparent efforts to find out how to make sure results are proportionally relevant.  Thus, in a global survey of women on WMF, if there are 5 USA women responses about non-involvement and 1 in Indian woman response, the Indian woman's response really should be weighted at least FIVE TIMES the response of the USA responses based on proportionality, and the Australian response should be 1/15th the USA response when drawing nationality conclusions...  Good sampling techniques are often not discussed.  Beyond that, bad sampling techniques are often given critical support.  If 27% percent of your female respondents identify as non-heterosexuals?  Is that because Wikipedia attracts MORE lesbians or because the population answering was self selecting and more lesbians were motivated to respond?  (I'd guess 10% would be about right, if not lower for the percentage of lesbian contributors on Wikipedia.)

Beyond that, non-answers in surveys are not counted.  Rarely do you see some one mention "non-response" as a category of response, or see follow ups that encourage people to fill in the blanks. This can be especially problematic if you consider that some people are culturally indoctrinated not to say anything mean, especially if they fear their survey responses may be made public.

I sat through some presentation on the gender gap in Argentina.  The gender gap research done by a group with their own gender gap problems (not a single woman on the research team) never really explained this.  I feel they made some rather faulty assumptions in their research, especially when they assumed there was equal non-gender identifying between genders with out explaining this... and this was then followed up by doing more extensive research based on userbox identification.  The last time I looked, the number of men versus the number of women 486 to 90: http://toolserver.org/~jarry/templatecount/index.php?lang=en&name=Template%3AUser+male#bottom vs http://toolserver.org/~jarry/templatecount/index.php?lang=en&name=Template%3AUser+female#bottom .  That puts total female users at 15%, not 9%.

At its simplest, it is really hard to define the population characteristics of English Wikipedia.


 

I like methodology discussions. :D Research design is fun. :D 

There is a research list, but it isn't particularly active and I haven't found them to be that interested in engaging in methodological discussions.  It tends to be more calls for papers.