[Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Tilman Bayer]

On Fri, Sep 14, 2012 at 9:35 AM, Bence Damokos <bdamokos at gmail.com> wrote:
> Hi,
>
> On Fri, Sep 14, 2012 at 6:08 PM, Jan-bart de Vreede <jdevreede at wikimedia.org
>> wrote:
>
>> Hey
>>
>> So someone sent me the internal-l mail and I do think that the "zero
>> real-world" thing is taken out of context here. But a few points
>>

>> 3) I am sure that the data set allows us to see chapter's individual
>> responses, depending on whether or not we know the country (I would figure
>> we do?)
The survey asks indeed for the country where the respondent lives, and
using that data, we plan to evaluate the results of the performance
rating question a little differently this time. On Meta, I have said
more about the objections against the part asking about editors'
perception of chapters.

>>
> I believe the biggest issue from a methodological point of view, once you
> filter out responses from non-chapter countries, is that the question asks
> for a composite average rating on chapterS performance. There is doubt that
> the results obtained in this way would be valid or reliable measures of
> chapters ratings.
> (On the face of it, if one has to rate together a well-performing known
> organization and 38 unknown ones, it is unlikely that he would adjust his
> rating of the known organization upwards to get to an average, it seems
> more likely that he would do a downwards adjustment and perhaps vice-versa
> in case of a low-performing local organization; different people would
> apply different adjustments and it becomes doubtful whether we can ever get
> an accurate average from a simple question asking for opinion about 39
> organizations.
Interesting thoughts, but I did not see an argument here why a
downwards adjustment should be considered more likely than an upwards
adjustment. Anyway, every opinion survey, say the many Gallup polls,
has to deal with the basic fact that the respondents can have very
different levels of knowledge about the subject which they are asked
to rate. Rather than blindly ignoring this, many readers of opinion
surveys are actively aware of it and for example, are interested in
how opinions change when knowledge improves or decreases among
respondents. And yes, these opinions may be subjective, or unfair,
having been unduly influenced by isolated success stories or scandals.
(In our own opinion.)

Or to put it differently, with these ratings of individual chapters -
or indeed the rating of the Foundation in the same question - you
would likewise have the issue that respondents judge the whole
entity's work despite often knowing only small parts of it; any kind
of such question could be attacked for merely asking about a
"composite average" where parts are missing from the respondent's
knowledge. In reality, at one point one just has to define a certain
number as representing the respondent's opinion about a certain thing,
and measure it without second-guessing the (non-existing) formula by
which the respondent calculates it in their heads as an average of
other ratings.

> Asking individuals to rate one chapter if relevant and than taking the
> [weighed] average of the ratings from the 39 relevant countries would be
> more accurate perhaps. And as a control group, people from outside chapter
> countries could be asked to rate all chapters in one question to see if
> there is anything to learn there from the answers.)
>  In general, when a concept is complex, surveys tend to ask more questions
> to get a better picture, just as the WESI-score on Wikipedia satisfaction
> is a composite from a number of questions on this survey.
>
This is often called a latent variable
(https://en.wikipedia.org/wiki/Latent_variable ). It is used here
because some questions are difficult to ask directly for (mainly)
language reasons - e.g., "how satisfied are you with... " would be too
ambiguous.

On Fri, Sep 14, 2012 at 12:57 AM, charles andrès
<charles.andres at wikimedia.ch> wrote:
> Dear Jan-Bart,
> Unfortunately the exact wordings was "I'll point out also that there are zero real-world implications for the survey results".
>
> Because we all agree that there is now such thing like a zero real-world impact survey, we really hope that the raw results of this survey will be made as public as possible (privacy issue), and that in the future , survey including question about WMF partners (chapters are not the only ones) will be done since the very beginning in collaboration with all the partners involved.

Yes, we agree about the benefits of making the anonymized raw data
from such surveys available to everyone. As it has been noted in the
Q&A on Meta since July, this will be done for this survey as it was
done for its two predecessors (see
http://dumps.wikimedia.org/other/surveys/ - would love to see more
such datasets from other surveys in the movement, btw, and more people
using them to do their own analyses).

Regarding the "real world": Of course in a strict interpretation,
everything written somewhere has real-world impact, even if it is just
to change the color of a few pixels on a screen ;) And surely we hope
that the results of the survey will be useful to many people. But I
think that part of what Sue meant there is that while in the past some
organizations in the Wikimedia movement have been conducting surveys
or other research projects whose primary purpose was to serve as input
for specific important ongoing debates/decision processes, this
present survey is not one of them. The prime example of such a
debate/decision process would be the fundraising debates from summer
2011 to this spring (cf. the list of pages at
https://meta.wikimedia.org/wiki/Template:SuesFundingTemplate ), and
the research reports or surveys committed in this context, like WMF's
"Survey of how money should be spent" or WMDE's "Spendwerk report".
Personally, I was mostly not involved in that process, but I have a
feeling that it is still on the minds of many chapter functionaries.
So it seems to me that it was worthwhile to point out that the present
survey is not intended as input for a concrete upcoming funding
decision, for example, and perhaps Sue's remark has already become
clearer in recent weeks since the last post in this thread.

On Fri, Sep 14, 2012 at 2:09 PM, Frédéric Schütz <schutz at mathgen.ch> wrote:
> On 14/09/12 18:08, Jan-bart de Vreede wrote:
>
>> 4) Doesn't every survey contain questions that don't apply to the whole
>> responding audience?
>
>
> No, because the answer would not be terribly useful. In all surveys I have
> seen in my work life [statistics work -- I don't design much surveys myself,
> but reply to a lot of them from different horizons :-], this does never
> happen.
On the contrary. One doesn't have to look far for such an example - in
fact the Pew survey which I cited in an earlier  discussions
(regarding the notion that there is some kind of rule saying that
survey respondents must not be asked about their opinion about actors
in faraway countries which they may or may not be informed about) will
do for this as well:
http://www.pewglobal.org/2012/06/13/chapter-5-rating-world-leaders/ -
for example, they asked a question that did not apply to 73% of the
respondents in India. (Or perhaps you meant something else, but then I
don't understand how it would apply to the present question in the
editor survey.)
>
> Typically, the filtering for a question about chapters would be something
> like this (simplified and written quickly):
>
> 1) Do you know any organization active within the Wikimedia movement ?
> (free text fields allow people to enter names)
Nice idea, but free-form text fields are notoriously hard to parse
exactly. (You would either need to program something that e.g. maps
"WMCH", "Wikimedia Suisse", "Wikimedia Svizzera", "Das Schweizer
Chapter", "Die Wikimedia-Ländervereinigung in der Schweiz" all to the
same entity, or code thousands of such replies manually.)
>
> 2) This page contain a list of organizations active within the Wikimedia
> movement; please tick all the ones you know (even if you have only heard the
> name)
>
> (a list follows, with maybe "WMF", "Local chapters in general", and a list
> of individual chapters)
Yes, that should be feasible; could be an idea for a future survey.
>
> 3) People are then asked to rate and/or comment each entity that was
> mentioned under (1) or (2).
>
> This way, not only do we avoid having people give their opinion on a topic
> they've never heard about, but we also get two different levels of knowledge
> on the topic (either the user knew it well enough to list the name, or he
> had to be reminded).
But regarding "Local chapters in general" or the contested question,
we already avoid this by offering the "don't know" option. Unless one
assumes that a significant number of respondents is lying about their
knowledge when they enter a rating instead of using that option. But
if someone really wants to go that far, they could also just tick an
"I know this organization" checkbox even though they never heard of
it.
-- 
Tilman Bayer
Senior Operations Analyst (Movement Communications)
Wikimedia Foundation
IRC (Freenode): HaeB