Hello! Thanks for the context :) I thought it was an interesting study and results, even in the limited english version.
It would be interesting to see replications of this type of study across languages for several reasons, I think, not the least of which is the potential effect on public awareness of Wikipedia quality and issues. I was especially glad that there was a note at the end of this article about getting involved as a contributor.
best, Phoebe
On Fri, Dec 6, 2013 at 5:01 PM, Arto Lanamäki arto.lanamaki@uia.no wrote:
Hi,
I'll comment this as I am the researcher who was interviewed and consulted for this.
This Helsinki Times article is an English summary of a set of (Finnish language) articles that were published in the biggest newspaper in Finland, Helsingin Sanomat, last weekend. The article series was written by journalist Olavi Koistinen, with the help of several of his colleagues.
I think the Finnish article series was great, but the English summary loses some of its context in translation.
The article claims that it is the "world's largest study on Wikipedia". What this means is that it has the biggest sample of articles (134) of all studies that have assessed Wikipedia content quality/credibility.
The result of this study was a positive one: Finnish Wikipedia content is surprisingly good, even though the quality does vary a lot.
The consequences of this study seem to be even more positive, I think. Similarly to what happened after the Nature study, the community started improving on these articles immediately. It seems this has created lots of positive buzz, attention and activity in Finnish Wikipedia.
This is the main article (in Finnish): http://www.hs.fi/tiede/HS+selvitti+N%C3%A4in+luotettava+Wikipedia+on/a130575... Here is the data as Excel sheet: http://files.snstatic.fi/HS/2013/11/wikipedia/hs-wikipedia-selvitys.xls
With kind regards, Arto Lanamäki
Lähettäjä: wiki-research-l-bounces@lists.wikimedia.org [wiki-research-l-bounces@lists.wikimedia.org] käyttäjän phoebe ayers [phoebe.wiki@gmail.com] puolesta Lähetetty: 5. joulukuuta 2013 21:26 Vastaanottaja: Research into Wikimedia content and communities Aihe: [Wiki-research-l] Fwd: the Helsinki Times evaluates the Finnish Wikipedia
(a la the old Nature study)
http://www.helsinkitimes.fi/finland/finland-news/domestic/8619-world-s-large...
World’s largest study on Wikipedia: Better than it’s reputation
05 Dec 2013
Wikipedia is the most popular encyclopedia in the world, and now in many countries it is also the only one. But can the information in Wikipedia be trusted at all?
Helsingin Sanomat newspaper evaluated 134 articles in the Finnish-language version of Wikipedia with 96 experts. As far as we know, our study is the most extensive individual investigation on the trustworthiness of Wikipedia in the whole world.
We found out that Wikipedia is better than it’s reputation: seventy per cent of the articles got good points for accuracy. But in many ways the Finnish Wikipedia is also far from flawless.
Olavi Koistinen HS
Hundreds of thousands of people use the Finnish-language version of Wikipedia every day. Its articles rank at the top in Google search results, whether you are looking for information on climate change, Michael Jackson, Sydney, ball bearings, cancer, allosaurus, or any number of other topics.
Wikipedia is the only Finnish encyclopedia that is still being updated. The golden age of printed encyclopedias was in the 1990s, but with the advent of the internet, their sales collapsed. The last traditional Finnish encyclopedia, the online version of WSOY’s Facta, was closed down in 2011.
The reliability of the English language Wikipedia has been studied for years. The most famous study was published by the science magazine Nature in 2005, comparing the number of errors in Wikipedia with those in the Encyclopaedia Britannica. The result of the comparison was that Wikipedia was almost as accurate as the Encyclopaedia Britannica. The study involved 42 articles from each encyclopaedia.
However, no thorough investigations have been made about the reliability of the Finnish Wikipedia – until now.
We first chose 134 articles in the Finnish Wikipedia, covering different areas of life, to be studied. Then we asked an expert with a thorough knowledge on the subject matter to evaluate each article. There were a total of 96 people making assessments, most of them professors or other university researchers. They work in eight different Finnish universities.
The Finnish Wikipedia currently has nearly 340 000 articles, so the random selection of 134 articles naturally does not give a complete picture of the reliability of the encyclopedia. Our study is nevertheless the most extensive individual investigation or report on the trustworthiness of Wikipedia in the whole world, as far as we know, if the matter is measured on the basis of the number of articles examined.
“With this measure the study is unique in its scale”, says researcher Arto Lanamäki of the University of Bergen in Norway. Not even the English-language Wikipedia has undergone such an extensive single survey.
The experts read the articles and scored them based on six different indicators, which were lack of errors, coverage and balance, sourcing, topicality, neutrality, and clarity. The articles were graded on a scale of 1-5, in which 1 was the worst mark and 5 was the best.
The entire result dataset was published as an open data, and is available here.
First the good news: The Finnish Wikipedia is largely error-free. The lack of errors is the area in which Wikipedia clearly got its best score. For instance, if someone uses Wikipedia to find the year in which Christopher Columbus discovered the America, the information in the article is most likely to be correct.
No less than 70 per cent of the articles were judged to be good (4) or excellent (5) with respect to lack of errors. According to the indicative evaluation scale a four means that that the article has only “scattered small errors, no big ones”.
This is how we evaluated the Finnish language Wikipedia
Journalists of Helsingin Sanomat first listed 150 topics from different walks of life. Only after that did we check if any Wikipedia articles actually existed on the topic (the vast majority did exist). 134 Wikipedia articles were selected for the final evaluation. To evaluate each article we chose an university-level researcher with knowledge on the subject matter to be an evaluator. There were a total of 96 evaluators, of which 94 were researchers. On top of that, two experienced sport journalists did evaluate 8 Wikipedia articles about sports. Some of the evaluators assessed more than one article. The evaluation was done via a web form. The basic principle of the evaluation was: If an amateur would read the article, would he or she get a truthful impression of the subject matter? Each evaluator scored the article on the basis of six criteria: lack of errors, coverage and balance, sourcing, topicality, neutrality, and clarity. The scale of the points was 5=excellent, 4=good, 3=adequate, 2=tolerable, 1=poor. The evaluators were also given the chance to comment the article based on each criteria. Most of the evaluations were made in early November. They were based on copies of Wikipedia articles as they were on October 22, 2013.
On the question of lack of errors the most common mark was four. What does it mean in practice? According to the evaluators, factors such as these:
“No misleading errors, but some imprecision.” (Higgs boson)
“Quite proficient text, hardly any actual factual errors. However, odd transliteration stands out.” (Icon)
“Geological periods do not quite hit the mark, the factual errors are insignificant.” (Climate change)
However, the conclusion should not be drawn that Wikipedia can be trusted naively. A total of 14 articles were graded as poor (1) or tolerable (2).
“Already in the first sentence the definition of photosynthesis is odd, and several mistakes were found.” (Photosynthesis)
“The article is very uneven in quality, and at times quite propagandistic.” (Syrian civil war)
Many of the articles containing the most errors were about a complicated or abstract phenomena, such as the eurozone debt crisis, evolution, or the Syrian civil war. On the other hand, several articles dealing with equally complex subjects got high marks for lack of errors. For instance, the articles for the big bang and climate change were praised for their precision.
Although most of the articles were extolled for their accuracy, it can be concluded that there is more variation in the quality of articles in Finnish Wikipedia than in traditional encyclopedias. According to studies, the same also applies to English Wikipedia: the best articles are brilliant, but many are weak.
How can a reader assess the quality of a Wikipedia article?
A good way to examine the reliability of any particular article is to check it’s factual sources. If the sources of information are listed at the end of an entry or inside the text itself, the reader can become acquainted with them and form an opinion of the credibility of the information on that basis.
According to our study there are serious shortcomings in the Finnish Wikipedia specifically in the sourcing of articles. No less than 38 per cent of the articles got a grade of weak (1) or tolerable (2). The evaluators had harsh criticism for these articles.
“Sources were not used at all and it shows.” (Internal combustion engine)
“On the basis of the text it would seem that sources have mainly included TV documentaries or children’s dinosaur books.” (Allosaurus)
“Based primarily on a single disputed work. Sources from antiquity are considered inadequate and hostile.” (Caligula)
The last comment highlights a broader problem. The evaluators were critical that many articles were based mostly or partly on one source. This sometimes causes problems: the information from individual works can be selective, and the interpretations biased.
“The article seeks to be neutral. However, the points of emphasis of the material that it was based on can clearly be seen.” (Protestant Reformation)
Of Wikipedia’s own ideals, neutrality is one of the most important. Wikipedia emphasises that the articles need to be written from a “neutral” point of view.
On the basis of our survey the Finnish Wikipedia has been written mainly with a balanced approach, with 56 per cent of the articles getting good (4) or excellent (5) marks. If some articles in the sample were slanted in their statements or points of view, it usually was not attributable to the writer’s deliberate partiality.
“The article is positive in its attitude and aims at neutrality. The problems in the content are connected with the source material and the writer’s basic knowledge: it is clearly not a result of anything deliberate.” (Middle Ages)
Of the 134 articles that were surveyed, only six were found to have been written with a clear bias, in the opinion of the reviewer. The articles were connected with politics of the United States (Osama bin Laden’s death, the US Democratic Party, the 9/11 terror attacks, Alan Greenspan) and events in the Middle East (The civil war in Syria, the second intifada).
“In conditions of war and with an intense conflict dominating, the neutrality requirement for the article is difficult. Occasionally, when reading this article it seems, however, that no actual attempt was made to attain neutrality.” (Civil war in Syria)
“A fairly flattering article for the Democrats. Other points of view would also exist.” (US Democratic Party)
It is interesting that the experts felt that the wrong kind of “neutrality” and avoiding expressing points of view is also a problem. If the writer is afraid to make interpretations of any kind, the article sometimes ends up being superficial.
“There are no interpretations and consequently no points of view either.” (Kingdom of Mali)
“Generally a text only lists things instead of pondering them and presenting well-founded evaluations.” (Finnish composer Kaija Saariaho)
“As the article seeks to be very objective, it starts strangely by describing Pinochet as a ‘president’ – after all, he was one of the best-known dictators of the 20th century.” (Augusto Pinochet)
Sometimes individual facts in an article are correct, but the text fails to mention relevant aspects of the topic, or mentions them too briefly. The article might also ramble on excessively about insignificant details. In such a situation the lay reader might get an inaccurate image of, for instance, what the most important turning points of a country’s history or what the most important achievements of a researcher might be.
This is a quality factor which the evaluators measured by scoring each article based on “coverage and balance”. It has an impact on how well Wikipedia can convey an overall image of the matter to the reader.
The experts’ marks on coverage and balance were divided almost equally between good and bad. So at least half of the articles could have been more comprehensive and balanced.
“The article primarily tells about marriage based on Western – mainly Roman law and the Christian tradition. Other cultures and religions have been left completely outside the examination.” (Marriage)
“The article puts far too much emphasis on personal history and even on related insignificant details. The presentation of the main topic, the scientific work, is far too short and superficial compared with the rest of the material.” (Albert Einstein)
Printed encyclopedias were often criticised as containing obsolete information even when they were fresh off the press. Wikipedia has a better chance to stay topical, since articles can be updated at any time.
The evaluators gave Finnish Wikipedia fairly high marks for topicality, with 43 per cent of the articles getting either excellent or good marks, but on the other hand, 31 per cent were poor or just tolerable in terms of freshness. When something big happens, breaking news are often updated into Wikipedia, but the follow-up on events is poor.
“All sources are old, which means that the content of the text has not been updated after 2011 to any practical degree.” (Osama bin Laden’s death)
On the other hand, articles can become obsolete even if the topic is a mountain range that rose from inside the earth in prehistoric times, or a notable person who died in the 1930s; research on these topics often still goes on.
“The biggest problem of the article is that it does not reflect the current situation of international research.” (Middle Ages)
“Within psychoanalysis, much has happened since Freud, and continues to happen, and Freud’s ideas have been re-evaluated many times. There is almost nothing about this in the article.” (Sigmund Freud)
Table of results on the study of Finnish Wikipedia 1: Lack of errors 2: Coverage and balance 3. Sourcing 4: Topicality 5: Neutrality 6. Clarity 1 (poor) 3 % 5 % 17 % 13 % 5 % 2 % 2 (tolerable) 7 % 24 % 21 % 19 % 16 % 11 % 3 (adequate) 19 % 41 % 34 % 25 % 22 % 31 % 4 (good) 47 % 25 % 21 % 30 % 34 % 42 % 5 (excellent) 23 % 5 % 7 % 13 % 22 % 13 %
% = amount of articles inside the whole sample, which was 134 articles
So what are we supposed to think about all of these results? Can information from Finnish Wikipedia be trusted or not?
There are two points of view on this matter.
Those who have felt so far that everything that is in Wikipedia is true would do well to re-examine their naiveté to a certain degree.
Those who have felt that Wikipedia’s content is nothing but inaccurate and biased pseudo-information, should ease up a little. Based on our survey, this is not the case; as a source of information, Wikipedia is a better than it’s reputation – which is not a particularly good one.
Arto Lanamäki, who has studied Wikipedia at the University of Bergen, says that people often take a very suspicious view on Wikipedia, even if they themselves use it regularly for seeking information.
“In studies, the same article has been placed in the framework of the Encyclopaedia Britannica and Wikipedia, and brought to different people for evaluation. It is quite common for people to take a more suspicious view of the article when it is presented in the framework of Wikipedia”, Lanamäki says.
On the other hand, there are some good reasons for the doubts: the quality of Wikipedia fluctuates considerably. Seventy per cent of the articles got good points for accuracy in our study. If the bar is lowered slightly, then 90 per cent of the articles were at least “adequate” in the view of the reviewers. That is a vast majority, but you might also ask yourself a question: would you trust a printed encyclopedia if you knew that every tenth article in it was inaccurate?
Wikipedia’s undeniable strength, however, is that the information is updated and upgraded all the time. If you paid good money in the early 2000s for a set of books for your bookshelf, it is already obsolete in many respects. Wikipedia’s massive popularity indicates that our view of factual information is in flux. Rising alongside information confirmed by experts and printed in dignified books, is peer-produced information that is constantly spreading and accumulating on the internet.
If you see that something written in Wikipedia makes no sense, do everyone a favour and edit the article so that it is better. You don’t even need a Wikipedia user account for it.
Ask not how Wikipedia can help you – ask how you can help Wikipedia.
(This article was originally published in Finnish in Helsingin Sanomat at November 30th. The HS working group taking part in the drafting of the survey included Riikka Haikarainen, Tuomas Kaseva, Niko Kettunen, Olavi Koistinen, Veikko Lautsi, Siri Markula, Sami Simola and Timo Paukku.)
Finnish Wikipedia has only a few hundred regular writers
By Olavi Koistinen HS
Wikipedia can be written or edited by anyone. About 70,000 people have done one or more edits on the Finnish Wikipedia.
However, most of the content comes from a much smaller group of active contributors.
About 200 people write or edit Finnish Wikipedia regularly, says Joonas Lyytinen, a veteran Wikipedia contributor. He also says that there is an even more active core group consisting of only 20–30 people who produce very large amount of content and take care of many administrative tasks in Finnish Wikipedia. Being part of that group, Lyytinen himself has written about 2000 articles for Finnish version of Wikipedia.
A typical Wikipedia author is a male university student. Often he is a layman who enjoys writing Wikipedia articles because it is a nice way to learn new things – by researching and writing about them.
Researchers also write in Wikipedia, but less than laypeople do. At least some of the researchers feel hesitant with respect to Wikipedia, says University of Bergen researcher Arto Lanamäki, who has studied the Finnish Wikipedia.
The articles in Wikipedia are often the result of teamwork, and there is an ongoing debate about the content in the discussion pages of the Wikipedia. In this debate, a researcher thoroughly familiar with the subject does not get any credibility over a layman just because of his or her position or title – the best argument wins.
Arguing with hobbyists can be stressful for many researchers. “Often a writer has tried to explain his or her point of view by saying ‘don’t you understand that I have a doctorate and I have researched this topic?’ However, this will not work with Wikipedia”, Lanamäki says.
There is no advance censorship, but active users of Wikipedia monitor new updates constantly to counter vandalism. Sometimes, for instance, middle school students tend to sabotage articles.
Finnish Wikipedia would benefit from having more writers. There are just 5 million Finnish speakers and only a few real experts in some fields of knowledge. Wikipedia writers also suffer from a somewhat negative image among occasional users of the service. This might even hinder some people from participating in writing Wikipedia articles.
“It has been observed in academic studies that an impediment to participation in Wikipedia is that people do not want to be labelled Wikipedia nerds”, Lanamäki says.
Olavi Koistinen HS
--
- I use this address for lists; send personal messages to phoebe.ayers <at>
gmail.com *
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l