On 8 May 2014 01:00, Andreas Kolbe jayen466@gmail.com wrote:
As for study design, I'd suggest you begin with a *random* sample of frequently-viewed Wikipedia articles in a given topic area (e.g. those within the purview of WikiProject Medicine), have them assessed by an independent panel of academic experts, and let them publish their results.
No control, no calibration. Without those you can't really be sure what you've measured. While academic attitudes to Wikipedia may be of some interest they are not a proxy for quality.
You can try and rigidly and arbitrarily define quality. This is what Hasty et al were trying to do with their rather non standard definition of error. However as the variation in their numbers show this is really rather hard to do. You've also got the problems that your definition of quality may up with little relation to anything anyone cares about and of course you can pretty much get whatever result you want by selecting definitions.
If you allow more subjective judgement (or accept that you can't entirely kill it off) you are going to need something to compare your results to. Unfortunately wikipedia has largely killed off encyclopedia like works which makes it harder to find things to calibrate it against. Still I guess you could construct something ranging from tabloid journalism through broadsheet journalism through mass market books onto text books and conference papers through to journals with various impact factors (technically you could calculate an impact factor for wikipedia. This does not strike me as a good idea).
Since wikipedia's style is fairly obvious you can't do any blinding so you are going to need a bunch of factors to correct for that. Writing a novel article in an encyclopedic but non wikipedian style and getting a series of experts to rate it (half of whom are told it is a wikipedia article and half of whom are told it comes from somewhere else) might get you some data but doesn't leave you any way to correct for any encyclopedia vs paper in journal bias. Sure you can attack that by writing out the same information in various different styles but that requires yet more preliminary testing.
If you are interested in accuracy rather than quality it gets even worse. Have to adjust for effects of say diagrams vs photos, fonts (sure you can standardise those but how to people perceive Wikipedia when its in a different font from the one they are used to?), layout, british vs american english. Sure you can adjust for all this but you need yet more preliminary data.
Probably a load of factors that you miss the first time so you are going to need pilot studies.
Which is why you don't start with medicine. The time of medical experts is expensive. Postdocs in other areas can be have almost depressingly cheaply.
All of that is quite doable.
Doable but far more difficult than you seem to think.
If the results are good, it redounds to Wikipedia's credit.
No if the results were good we risk having more people rely on wikipedia rather than going to see a medical professional. This is not a good outcome.