On 7/29/05, Jimmy Wales jwales@wikia.com wrote:
Ingo Frost wrote:
[snip]
My question: Is there a scientific study on the quality of the Wikipedia ariticles? Does anyone work on that problems? What methods could be used to analyse the Quality?
I think this is a fantastic question and what I hope this list can foster.
It's an enormously difficult problem to get right, and you've identified some of the tough problems here. Despite my criticism (highly technical and based on internal jargon) of what you said about NPOV, I do think that it is quite hard to judge the quality of certain contentious articles because there is no simple "gold standard" to which we can refer.
In many cases, and I say this with full awareness that it is also not true in many other cases, our articles on contentious or controversial topics are _the best in existence_ simply because they are the _most free from bias_.
It's easy to compare a wikipedia chart of the periodic table of elements against a standard source and measure if it accurately reflects received science. It is much harder in areas where the only reliably objective presentation one can find _at all_ is in Wikipedia in the first place. :-)
This gets us into some potentially insoluble philosophical issues with measuring "quality" so what I recommend is that we remain steadfastly practical, thinking of things which we actually can measure and test.
--Jimbo
To get around these philosophical issues, I believe that the only way to measure quality of articles (especially contentious ones) is qualitatively, ie by asking people/experts their opinions of articles, their experience of the community etc and analysing the types of reactions, the emotional resonance (or lack thereof), the language they used etc. So far, I haven't seen many qualitative studies of Wikipedia - I did one last Christmas as a kind of pilot study for my dissertation, which you can see here: http://wikisource.org/wiki/A_small_scale_study_of_Wikipedia and some others, from Wikimania, include: http://en.wikibooks.org/wiki/Wikimania05/Paper-PA1 http://en.wikibooks.org/wiki/Wikimania05/Paper-JT1 Note: almost all Wikimania papers are still works in progress, including mine :)
The Brockhaus and New York Times studies are examples of qualitative studies. However there are many quantitative studies, again from Wikimania, including Andreas Brandle's paper, which Jakob already mentioned in this thread and user:Boud's study of NPOV and meme evolution: http://en.wikibooks.org/wiki/Wikimania05/Paper-BO1 also upcoming data on article validation: http://meta.wikimedia.org/wiki/Article_validation and many more from: http://en.wikipedia.org/wiki/Wikipedia:Wikipedia_in_academic_studies
A widespread notion is that only quantitative studies are scientific, although this debate itself is fraught with philosophical as well as methodological issues. I think that qualitative studies are extremely valid and I'd like to see a few more of them on Wikipedia. As we all know, there's a great spirit of openness here and, as a fresh researcher here, I was thrilled at such clarity and honesty in the answers I got back (see my pilot study and its appendices). Certainly qualitative data can be large and unwieldy, but this is a separate matter. It's all down to what you want to find out at the end.
Cormac / Cormaggio