[Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Bence Damokos]

Hi,

On Fri, Sep 14, 2012 at 6:08 PM, Jan-bart de Vreede <jdevreede at wikimedia.org
> wrote:

> Hey
>
> So someone sent me the internal-l mail and I do think that the "zero
> real-world" thing is taken out of context here. But a few points
>
> 1) Lets not have these discussions on internal-l, there is no reason to
> not have those in public
> 2) See 1)
> 3) I am sure that the data set allows us to see chapter's individual
> responses, depending on whether or not we know the country (I would figure
> we do?)
>
I believe the biggest issue from a methodological point of view, once you
filter out responses from non-chapter countries, is that the question asks
for a composite average rating on chapterS performance. There is doubt that
the results obtained in this way would be valid or reliable measures of
chapters ratings.
(On the face of it, if one has to rate together a well-performing known
organization and 38 unknown ones, it is unlikely that he would adjust his
rating of the known organization upwards to get to an average, it seems
more likely that he would do a downwards adjustment and perhaps vice-versa
in case of a low-performing local organization; different people would
apply different adjustments and it becomes doubtful whether we can ever get
an accurate average from a simple question asking for opinion about 39
organizations.
Asking individuals to rate one chapter if relevant and than taking the
[weighed] average of the ratings from the 39 relevant countries would be
more accurate perhaps. And as a control group, people from outside chapter
countries could be asked to rate all chapters in one question to see if
there is anything to learn there from the answers.)
 In general, when a concept is complex, surveys tend to ask more questions
to get a better picture, just as the WESI-score on Wikipedia satisfaction
is a composite from a number of questions on this survey.

> 4) Doesn't every survey contain questions that don't apply to the whole
> responding audience?
>
Probably not - depends on the survey design and goals.

Best regards,
Bence

>
> Jan-Bart
>
>
>
>
> On 14 sep. 2012, at 17:05, Florence Devouard <anthere9 at yahoo.com> wrote:
>
> >
> >
> > On 9/12/12 4:14 PM, Jan-Bart de Vreede wrote:
> >> Hey
> >>
> >> So I might have missed some mails on this thread (perhaps because they
> were not posted on this public list) but I highly doubt that Sue perform
> surveys that do not have a real-world impact on our operations. I know that
> the results of the previous surveys were used in several discussions
> (including at a board level) in order to provide more insight….
> >
> >
> > Indeed. I agree. These surveys do have real-world impact, which is why
> we objected to a survey asking people from all over the world how they
> would rate Wikimedia Chapters activities when
> > 1) there is likely no chapter in their country
> > 2) they may have no idea that a "wikimedia chapter" is for example
> "Wikimedia Washington DC" or "Wikimedia Israel"
> > 3) all chapters are collectively considered regardless of individual
> differences
> >
> > And since you comment on that specific sentence
> >
> > "I'll point out also that there are zero real-world implications for the
> survey results."
> >
> > I'd like to clarify that these exact words come from Sue herself in an
> email sent on the 10th of September on internal-l.
> >
> > I am glad to read that you disagree with that statement and recognize
> that there is real-world impact.
> >
> >
> > (did not want to comment any further on that problematic survey, but
> wanted to attribute statement properly)
> >
> >
> > Flo
> >
> >
> >> On the other hand, using these surveys to gain more insight is not the
> same as using them to "hold each other accountable" which is sometimes easy
> to do. Every survey (and questions) has a lots of interpretation magic
> which can easily lead you astray, but I don't have to tell you (the
> community) this :)
> >
> >
> >> Jan-Bart
> >>
> >>
> >> On 10 Sep 2012, at 23:01, Delphine Ménard <
> notafishz-Re5JQEeQqe8AvxtiuMwx3w at public.gmane.org> wrote:
> >>
> >>> Contrarily to Sue, I do
> >>> think that these surveys (should) have a real-world impact and
> >>> (should) keep us all on our toes, fine tuned to the critisicism, needs
> >>> and wishes of the editors of the WIkimedia projects. As such I expect
> >>> us to make sure that we do get as precise a picture as possible of
> >>> what those are.
> >>
> >> _______________________________________________
> >> Wikimedia-l mailing list
> >> Wikimedia-l-RusutVdil2icGmH+5r0DM0B+6BGkLq7r at public.gmane.org
> >> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
> >>
> >
> >
> >
> > _______________________________________________
> > Wikimedia-l mailing list
> > Wikimedia-l at lists.wikimedia.org
> > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
>
>
> _______________________________________________
> Wikimedia-l mailing list
> Wikimedia-l at lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
>