2008/11/1 Thomas Dalton thomas.dalton@gmail.com:
You don't need someone that's good at designing surveys (well you do, but not to spot most of these problems), you just need to try the survey out on a few people first.
The survey was tried out on a group of testers and translators. You only get so much useful feedback - the feedback that we're getting from actually running the survey is much more detailed and valuable for future surveys.
I was under the impression it was done with the support of experts - if that's the case, pick better experts next time!
It was developed by the UNU-Merit Collaborative Creativity Group, who have developed and run in-depth, multilingual surveys on the free software movement, probably one of the most comparable specialized communities. It's a first run, and the results will be imperfect and need to be interpreted very carefully -- but we'll get some basic, useful data, and we have a huge amount of feedback that will help with the design of future surveys. I don't think we could have done much better, especially given that the only resources we spent on this project are staff time to shepherd it.