[Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Delphine Ménard]

On Tue, Aug 21, 2012 at 3:32 PM, Tilman Bayer <tbayer at wikimedia.org> wrote:
> I'm not sure she actually said that. In any case, it is wrong -
> question D6 in the last survey asked if there was a Wikimedia chapter
> in the country where the respondent lived
> (https://meta.wikimedia.org/w/index.php?title=File:December_2011_Wikipedia_Editor_Survey_topline.pdf&page=5
> ).

My bad, glad it was there. And yes she did. Maybe not in those exact
words, but she did put the results into context.
>
>> This was already quite criticized last time, yet
>> the question is still there, unchanged, and without more context than
>> it had last time. (under FINAL THOUGHTS).
> We tried to avoid modifying questions in order to preserve consistence
> and be able  to do some longitudinal analysis (as it was already begun
> for that question in http://blog.wikimedia.org/?p=14296 ).

So if a question is poorly phrased, we'll continue having it till the
end of time to preserve consistency? Mind you, I do want that question
in, I just want it within the same context frame that is given to the
same question about the Foundation. And I'm also missing a question
about other entities that might actually help Wikimedians that we're
or we're not aware of.

>
> Still, I was aware that there had been some objections to that
> question by chapter representatives (which I don't assume have to do
> with the fact that respondents rated chapters' performance lower than
> that of other entities in the two previous surveys), and looked into
> these concerns while the present questionnaire was prepared; I also
> reached out to one of the critics in person at Wikimania. But I still
> haven't seen a compelling argument why the way the question is asked
> should be biased against chapters. The argument that the opinion of
> Wikimedians who live in countries without chapter should not count
> seems weak to me, e.g. because the projects that the work of chapters
> aims to support are international, and because the question asked
> about chapters in general, not one particular chapter.

That is not the argument I was trying to make (ie. voices of
Wikimedians in a country without chapter don't count). Rather, there
is a long list of things the Foundation does, where people are asked
whether they knew about it, or not. And after that, right when people
have been made aware of everything the Foundation does, they are asked
to rate the work of the Foundation. The same question about the
chapters comes after absolutely nothing has been said about chapter
work, which, I believe, does introduce a bias. In short, people are
being asked to rate something they *at this point in the survey* have
an idea about (for the WMF) although they might have had no idea about
it before starting the survey.
All I'm asking is that we review the context in which this question is
being asked so results make more sense.

>
> In any case, in parallel to preparing the upcoming survey, we have
> recently been compiling the public dataset with the anonymized
> responses from the last survey, which was uploaded yesterday here:
> http://dumps.wikimedia.org/other/surveys/editorsurveynov-dec2011/
> Using this, anyone can do the analysis you suggest and check if
> ratings differed significantly in chapter/non-chapter countries.

That's great. Thanks. For the record, I'm not expecting the results to
be so extremely different, but I think the fact that they might be or
might not be is extremely important to know.



>> Finally, what I regret most, is that o little time was allocated to
>> reviewing this survey collaboratively. There was about 20 days to
>> review, comment, give feedback, translate etc. and this in the mist of
>> the summer holiday for a big part of our community. When I see such
>> processes as the ToS revision [1] conducted successfully allowing for
>> 120 days of discussion, and such surveys rushed through the summer,
>> when their data will probably be used to decide much of the strategic
>> orientation of the next year, I'm thinking we're missing out on trying
>> to collect data that is truly relevant to the work we do and that
>> helps us to review our orientations and adjust how well we support our
>> editing community.

> Of course there are lots of interesting and important questions that
> had to be left out of this survey. As said earlier, the idea is to run
> the editor survey more frequently from now on, probably quarterly and
> in a more lightweight version, with a different focus in each.


That's good news and I hope the collaborative process can be
reinforced and more time is allowed for comments, reviews, changes and
finetuning.

Best,

Delphine

-- 
@notafish

NB. This gmail address is used for mailing lists. Personal emails will get lost.
Intercultural musings: Ceci n'est pas une endive - http://blog.notanendive.org
Photos with simple eyes: notaphoto - http://photo.notafish.org