On Fri, Sep 14, 2012 at 9:35 AM, Bence Damokos bdamokos@gmail.com wrote:
Hi,
On Fri, Sep 14, 2012 at 6:08 PM, Jan-bart de Vreede <jdevreede@wikimedia.org
wrote:
Hey
So someone sent me the internal-l mail and I do think that the "zero real-world" thing is taken out of context here. But a few points
- I am sure that the data set allows us to see chapter's individual
responses, depending on whether or not we know the country (I would figure we do?)
The survey asks indeed for the country where the respondent lives, and using that data, we plan to evaluate the results of the performance rating question a little differently this time. On Meta, I have said more about the objections against the part asking about editors' perception of chapters.
I believe the biggest issue from a methodological point of view, once you filter out responses from non-chapter countries, is that the question asks for a composite average rating on chapterS performance. There is doubt that the results obtained in this way would be valid or reliable measures of chapters ratings. (On the face of it, if one has to rate together a well-performing known organization and 38 unknown ones, it is unlikely that he would adjust his rating of the known organization upwards to get to an average, it seems more likely that he would do a downwards adjustment and perhaps vice-versa in case of a low-performing local organization; different people would apply different adjustments and it becomes doubtful whether we can ever get an accurate average from a simple question asking for opinion about 39 organizations.
Interesting thoughts, but I did not see an argument here why a downwards adjustment should be considered more likely than an upwards adjustment. Anyway, every opinion survey, say the many Gallup polls, has to deal with the basic fact that the respondents can have very different levels of knowledge about the subject which they are asked to rate. Rather than blindly ignoring this, many readers of opinion surveys are actively aware of it and for example, are interested in how opinions change when knowledge improves or decreases among respondents. And yes, these opinions may be subjective, or unfair, having been unduly influenced by isolated success stories or scandals. (In our own opinion.)
Or to put it differently, with these ratings of individual chapters - or indeed the rating of the Foundation in the same question - you would likewise have the issue that respondents judge the whole entity's work despite often knowing only small parts of it; any kind of such question could be attacked for merely asking about a "composite average" where parts are missing from the respondent's knowledge. In reality, at one point one just has to define a certain number as representing the respondent's opinion about a certain thing, and measure it without second-guessing the (non-existing) formula by which the respondent calculates it in their heads as an average of other ratings.
Asking individuals to rate one chapter if relevant and than taking the [weighed] average of the ratings from the 39 relevant countries would be more accurate perhaps. And as a control group, people from outside chapter countries could be asked to rate all chapters in one question to see if there is anything to learn there from the answers.) In general, when a concept is complex, surveys tend to ask more questions to get a better picture, just as the WESI-score on Wikipedia satisfaction is a composite from a number of questions on this survey.
This is often called a latent variable (https://en.wikipedia.org/wiki/Latent_variable ). It is used here because some questions are difficult to ask directly for (mainly) language reasons - e.g., "how satisfied are you with... " would be too ambiguous.
On Fri, Sep 14, 2012 at 12:57 AM, charles andrès charles.andres@wikimedia.ch wrote:
Dear Jan-Bart, Unfortunately the exact wordings was "I'll point out also that there are zero real-world implications for the survey results".
Because we all agree that there is now such thing like a zero real-world impact survey, we really hope that the raw results of this survey will be made as public as possible (privacy issue), and that in the future , survey including question about WMF partners (chapters are not the only ones) will be done since the very beginning in collaboration with all the partners involved.
Yes, we agree about the benefits of making the anonymized raw data from such surveys available to everyone. As it has been noted in the Q&A on Meta since July, this will be done for this survey as it was done for its two predecessors (see http://dumps.wikimedia.org/other/surveys/ - would love to see more such datasets from other surveys in the movement, btw, and more people using them to do their own analyses).
Regarding the "real world": Of course in a strict interpretation, everything written somewhere has real-world impact, even if it is just to change the color of a few pixels on a screen ;) And surely we hope that the results of the survey will be useful to many people. But I think that part of what Sue meant there is that while in the past some organizations in the Wikimedia movement have been conducting surveys or other research projects whose primary purpose was to serve as input for specific important ongoing debates/decision processes, this present survey is not one of them. The prime example of such a debate/decision process would be the fundraising debates from summer 2011 to this spring (cf. the list of pages at https://meta.wikimedia.org/wiki/Template:SuesFundingTemplate ), and the research reports or surveys committed in this context, like WMF's "Survey of how money should be spent" or WMDE's "Spendwerk report". Personally, I was mostly not involved in that process, but I have a feeling that it is still on the minds of many chapter functionaries. So it seems to me that it was worthwhile to point out that the present survey is not intended as input for a concrete upcoming funding decision, for example, and perhaps Sue's remark has already become clearer in recent weeks since the last post in this thread.
On Fri, Sep 14, 2012 at 2:09 PM, Frédéric Schütz schutz@mathgen.ch wrote:
On 14/09/12 18:08, Jan-bart de Vreede wrote:
- Doesn't every survey contain questions that don't apply to the whole
responding audience?
No, because the answer would not be terribly useful. In all surveys I have seen in my work life [statistics work -- I don't design much surveys myself, but reply to a lot of them from different horizons :-], this does never happen.
On the contrary. One doesn't have to look far for such an example - in fact the Pew survey which I cited in an earlier discussions (regarding the notion that there is some kind of rule saying that survey respondents must not be asked about their opinion about actors in faraway countries which they may or may not be informed about) will do for this as well: http://www.pewglobal.org/2012/06/13/chapter-5-rating-world-leaders/ - for example, they asked a question that did not apply to 73% of the respondents in India. (Or perhaps you meant something else, but then I don't understand how it would apply to the present question in the editor survey.)
Typically, the filtering for a question about chapters would be something like this (simplified and written quickly):
- Do you know any organization active within the Wikimedia movement ?
(free text fields allow people to enter names)
Nice idea, but free-form text fields are notoriously hard to parse exactly. (You would either need to program something that e.g. maps "WMCH", "Wikimedia Suisse", "Wikimedia Svizzera", "Das Schweizer Chapter", "Die Wikimedia-Ländervereinigung in der Schweiz" all to the same entity, or code thousands of such replies manually.)
- This page contain a list of organizations active within the Wikimedia
movement; please tick all the ones you know (even if you have only heard the name)
(a list follows, with maybe "WMF", "Local chapters in general", and a list of individual chapters)
Yes, that should be feasible; could be an idea for a future survey.
- People are then asked to rate and/or comment each entity that was
mentioned under (1) or (2).
This way, not only do we avoid having people give their opinion on a topic they've never heard about, but we also get two different levels of knowledge on the topic (either the user knew it well enough to list the name, or he had to be reminded).
But regarding "Local chapters in general" or the contested question, we already avoid this by offering the "don't know" option. Unless one assumes that a significant number of respondents is lying about their knowledge when they enter a rating instead of using that option. But if someone really wants to go that far, they could also just tick an "I know this organization" checkbox even though they never heard of it.