<quote who="Pine W" date="Thu, Sep 18, 2014 at 01:49:13PM -0700">
> Yes, but supposedly phone survey companies are able to get
> representative samples of broad populations despite many people
> refusing to respond to phone surveys. If opt-in users were chosen
> using similar methods, could arguably representative data be
> obtained?
The way that people build representative surveys from
non-representative data is by understanding quite a lot about the
nature and structure of the bias in your sample. You might want to
think about how people do this as trying to create a very complicated
system of weights.
Folks who do this for US phone surveys, for example, have spent many
decades and many millions of dollars on research to understand how to
get reliable results and even then it's a quickly moving target. They
still routinely sometimes miss things and get things wrong.
That said, there are certainly things we can learn. Aaron Shaw and I
actually did something related with one of the big Wikipedia surveys
in this article:
http://www.plosone.org/article/info:doi/10.1371/journal.pone.0065782
In our case, our study was only possible because (a) we had very good
luck finding "ground truth" data from the right point in time, (b) we
had detailed demographic data on folks from the WP survey, and (c) we
make a series of untestable assumptions. After all that work, we still
can't know that we've got it right. We really can only suggest that
there are reasons to believe our estimates are better that pretending
that the opt-in survey is unbiased.
In the case of signing up for a Wikipedia toolbar, we might not even
attract a sub-population that even /can/ reliably used to build
representative estimates. :-(
Regards,
Mako
--
Benjamin Mako Hill
http://mako.cc/
Creativity can be a social contribution, but only in so far
as society is free to use the results. --GNU Manifesto
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l