On 8 May 2014 01:00, Andreas Kolbe <jayen466(a)gmail.com> wrote:
As for study design, I'd suggest you begin with a
*random* sample of
frequently-viewed Wikipedia articles in a given topic area (e.g. those
within the purview of WikiProject Medicine), have them assessed by an
independent panel of academic experts, and let them publish their results.
No control, no calibration. Without those you can't really be sure what
you've measured. While academic attitudes to Wikipedia may be of some
interest they are not a proxy for quality.
You can try and rigidly and arbitrarily define quality. This is what Hasty
et al were trying to do with their rather non standard definition of error.
However as the variation in their numbers show this is really rather hard
to do. You've also got the problems that your definition of quality may up
with little relation to anything anyone cares about and of course you can
pretty much get whatever result you want by selecting definitions.
If you allow more subjective judgement (or accept that you can't entirely
kill it off) you are going to need something to compare your results to.
Unfortunately wikipedia has largely killed off encyclopedia like works
which makes it harder to find things to calibrate it against. Still I guess
you could construct something ranging from tabloid journalism through
broadsheet journalism through mass market books onto text books and
conference papers through to journals with various impact factors
(technically you could calculate an impact factor for wikipedia. This does
not strike me as a good idea).
Since wikipedia's style is fairly obvious you can't do any blinding so you
are going to need a bunch of factors to correct for that. Writing a novel
article in an encyclopedic but non wikipedian style and getting a series of
experts to rate it (half of whom are told it is a wikipedia article and
half of whom are told it comes from somewhere else) might get you some data
but doesn't leave you any way to correct for any encyclopedia vs paper in
journal bias. Sure you can attack that by writing out the same information
in various different styles but that requires yet more preliminary testing.
If you are interested in accuracy rather than quality it gets even worse.
Have to adjust for effects of say diagrams vs photos, fonts (sure you can
standardise those but how to people perceive Wikipedia when its in a
different font from the one they are used to?), layout, british vs american
english. Sure you can adjust for all this but you need yet more preliminary
data.
Probably a load of factors that you miss the first time so you are going to
need pilot studies.
Which is why you don't start with medicine. The time of medical experts is
expensive. Postdocs in other areas can be have almost depressingly cheaply.
All of that is quite doable.
Doable but far more difficult than you seem to think.
If the results are good, it redounds to Wikipedia's
credit.
No if the results were good we risk having more people rely on wikipedia
rather than going to see a medical professional. This is not a good outcome.
--
geni