Dear Andrew Lih, dear scientific community,
I am a bit disappointed about the available material that tries to measure the quality of Wikipedia articles.
The quoted newspaper article of the Wall street journal for example just analyses technical topics but it would be a dangerous claim to assume that quality is equally distributed over the different fields and topics. But you need that claim as condition for the method of randomly picking articles and conclude for the rest.
There was another attempt to compare the Quality of Wikipedia with other Encyclopedias in the German Computer Newspaper C't with the same random approach.
(1) But there is a problem since it is a random way of choosing articles to compare or to analyse. I see some problems in non technical fields such as soft sciences (in social science for example every theory on society redefines all concepts of society on it's own: how can an encyclopedia claim to have a definition?).
(2) Political terms are sometimes very complex topics where the NPOV may not work, because there is no right nor wrong. It is often a question of opinion and majority that sometimes changes reality. I observed a discussion and an edit war on the article about Direct Democracy (in the Germen Wikipedia: article "Direkte Demokratie") that led to a loss of quality: only a minimal and weak consens survived the different opinions: the evolutionary process did not improve quality in that case.
(3) The third problem is the tendency of specific groups that lead to vandalism. There are groups that use values or ideologies and reject a neutral or scientific view (moralists, religious groups, nationalists, neocapitalists etc.). What about articles that are important for these groups? Are these article tendentious?
My question: Is there a scientific study on the quality of the Wikipedia ariticles? Does anyone work on that problems? What methods could be used to analyse the Quality?
Ingo Frost (studies Wikipedia from a social system science view)
Ingo Frost wrote:
Dear Andrew Lih, dear scientific community,
I am a bit disappointed about the available material that tries to measure the quality of Wikipedia articles.
The quoted newspaper article of the Wall street journal for example just analyses technical topics but it would be a dangerous claim to assume that quality is equally distributed over the different fields and topics. But you need that claim as condition for the method of randomly picking articles and conclude for the rest.
Journalists aren't searching for knowledge - they just want to tell a story.
My question: Is there a scientific study on the quality of the Wikipedia ariticles? Does anyone work on that problems? What methods could be used to analyse the Quality?
The only serious and promising attemt I know of is Andreas Brändle's Masters thesis: * http://en.wikibooks.org/wiki/Wikimania05/Paper-AB1 * http://editthispage.blogspot.com/
Ulrich Fuchs did a little test on vandalising articles: http://de.wikipedia.org/wiki/Wikipedia:Wikipedistik/Vandalismusanf%C3%A4llig...
His approach is not fully scientific but still better then any Journalist's or Wikipedian's attemt. See also http://alex.halavais.net/news/index.php?p=794
Greetings, Jakob
Ingo Frost wrote:
I am a bit disappointed about the available material that tries to measure the quality of Wikipedia articles.
Me too, but let's all agree that it is a difficult thing to do.
(2) Political terms are sometimes very complex topics where the NPOV may not work, because there is no right nor wrong.
I think this is a serious misunderstanding of NPOV and of what it might mean _for an encyclopedia_ to be "right or wrong_.
First, not everyone believes (and I certainly don't) that on political topics there is no right nor wrong. But some do. And NPOV has to deal with all of us. The point is that we can typicall "go meta" and avoid taking a controversial stand ourselves. NPOV does not require us to choose which of two sides is right or wrong on complex topics, but rather requires us to describe the controversy.
Therefore, _for an encyclopedia_, it is quite possible to get it right (or get it wrong) even when the underlying issue is complex and not readily amenable to a final judgment.
I observed a discussion and an edit war on the article about Direct Democracy (in the Germen Wikipedia: article "Direkte Demokratie") that led to a loss of quality: only a minimal and weak consens survived the different opinions: the evolutionary process did not improve quality in that case.
This can certainly be true in any given case. But I wonder if you aren't showing your own POV here -- I often wish articles read differently, but often -- when I'm fully honest with myself -- this is because I wish my own view were more prominently reflected, even if it should not be.
My question: Is there a scientific study on the quality of the Wikipedia ariticles? Does anyone work on that problems? What methods could be used to analyse the Quality?
I think this is a fantastic question and what I hope this list can foster.
It's an enormously difficult problem to get right, and you've identified some of the tough problems here. Despite my criticism (highly technical and based on internal jargon) of what you said about NPOV, I do think that it is quite hard to judge the quality of certain contentious articles because there is no simple "gold standard" to which we can refer.
In many cases, and I say this with full awareness that it is also not true in many other cases, our articles on contentious or controversial topics are _the best in existence_ simply because they are the _most free from bias_.
It's easy to compare a wikipedia chart of the periodic table of elements against a standard source and measure if it accurately reflects received science. It is much harder in areas where the only reliably objective presentation one can find _at all_ is in Wikipedia in the first place. :-)
This gets us into some potentially insoluble philosophical issues with measuring "quality" so what I recommend is that we remain steadfastly practical, thinking of things which we actually can measure and test.
--Jimbo
On 7/29/05, Jimmy Wales jwales@wikia.com wrote:
Ingo Frost wrote:
[snip]
My question: Is there a scientific study on the quality of the Wikipedia ariticles? Does anyone work on that problems? What methods could be used to analyse the Quality?
I think this is a fantastic question and what I hope this list can foster.
It's an enormously difficult problem to get right, and you've identified some of the tough problems here. Despite my criticism (highly technical and based on internal jargon) of what you said about NPOV, I do think that it is quite hard to judge the quality of certain contentious articles because there is no simple "gold standard" to which we can refer.
In many cases, and I say this with full awareness that it is also not true in many other cases, our articles on contentious or controversial topics are _the best in existence_ simply because they are the _most free from bias_.
It's easy to compare a wikipedia chart of the periodic table of elements against a standard source and measure if it accurately reflects received science. It is much harder in areas where the only reliably objective presentation one can find _at all_ is in Wikipedia in the first place. :-)
This gets us into some potentially insoluble philosophical issues with measuring "quality" so what I recommend is that we remain steadfastly practical, thinking of things which we actually can measure and test.
--Jimbo
To get around these philosophical issues, I believe that the only way to measure quality of articles (especially contentious ones) is qualitatively, ie by asking people/experts their opinions of articles, their experience of the community etc and analysing the types of reactions, the emotional resonance (or lack thereof), the language they used etc. So far, I haven't seen many qualitative studies of Wikipedia - I did one last Christmas as a kind of pilot study for my dissertation, which you can see here: http://wikisource.org/wiki/A_small_scale_study_of_Wikipedia and some others, from Wikimania, include: http://en.wikibooks.org/wiki/Wikimania05/Paper-PA1 http://en.wikibooks.org/wiki/Wikimania05/Paper-JT1 Note: almost all Wikimania papers are still works in progress, including mine :)
The Brockhaus and New York Times studies are examples of qualitative studies. However there are many quantitative studies, again from Wikimania, including Andreas Brandle's paper, which Jakob already mentioned in this thread and user:Boud's study of NPOV and meme evolution: http://en.wikibooks.org/wiki/Wikimania05/Paper-BO1 also upcoming data on article validation: http://meta.wikimedia.org/wiki/Article_validation and many more from: http://en.wikipedia.org/wiki/Wikipedia:Wikipedia_in_academic_studies
A widespread notion is that only quantitative studies are scientific, although this debate itself is fraught with philosophical as well as methodological issues. I think that qualitative studies are extremely valid and I'd like to see a few more of them on Wikipedia. As we all know, there's a great spirit of openness here and, as a fresh researcher here, I was thrilled at such clarity and honesty in the answers I got back (see my pilot study and its appendices). Certainly qualitative data can be large and unwieldy, but this is a separate matter. It's all down to what you want to find out at the end.
Cormac / Cormaggio
Cormac Lawler wrote:
To get around these philosophical issues, I believe that the only way to measure quality of articles (especially contentious ones) is qualitatively, ie by asking people/experts their opinions of articles, their experience of the community etc and analysing the types of reactions, the emotional resonance (or lack thereof), the language they used etc. So far, I haven't seen many qualitative studies of Wikipedia - I did one last Christmas as a kind of pilot study for my dissertation, which you can see here: http://wikisource.org/wiki/A_small_scale_study_of_Wikipedia and some others, from Wikimania, include: http://en.wikibooks.org/wiki/Wikimania05/Paper-PA1 http://en.wikibooks.org/wiki/Wikimania05/Paper-JT1 Note: almost all Wikimania papers are still works in progress, including mine :)
The best study I know is done by Andreas Brändle:
http://en.wikibooks.org/w/index.php?title=Wikimania05/AB1
He shows that the number of authors is the most important variable to predict the quality.
Jakob
wiki-research-l@lists.wikimedia.org