A similar paper on 39 gastroenterology/ hepatology articles on the
English Wikipedia came to different conclusions:
https://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Medicine#Paper:_.22Evaluation_of_gastroenterology_and_hepatology_articles_on_Wikipedia:_Are_they_suitable_as_learning_resources_for_medical_students.3F.22

Daniel

On Sat, Dec 7, 2013 at 2:20 AM, phoebe ayers <phoebe.wiki@gmail.com> wrote:
> Hello! Thanks for the context :) I thought it was an interesting study
> and results, even in the limited english version.
>
> It would be interesting to see replications of this type of study
> across languages for several reasons, I think, not the least of which
> is the potential effect on public awareness of Wikipedia quality and
> issues. I was especially glad that there was a note at the end of this
> article about getting involved as a contributor.
>
> best,
> Phoebe
>
> On Fri, Dec 6, 2013 at 5:01 PM, Arto Lanamäki <arto.lanamaki@uia.no> wrote:
>> Hi,
>>
>> I'll comment this as I am the researcher who was interviewed and consulted
>> for this.
>>
>> This Helsinki Times article is an English summary of a set of (Finnish
>> language) articles that were published in the biggest newspaper in Finland,
>> Helsingin Sanomat, last weekend. The article series was written by
>> journalist Olavi Koistinen, with the help of several of his colleagues.
>>
>> I think the Finnish article series was great, but the English summary loses
>> some of its context in translation.
>>
>> The article claims that it is the "world's largest study on Wikipedia". What
>> this means is that it has the biggest sample of articles (134) of all
>> studies that have assessed Wikipedia content quality/credibility.
>>
>> The result of this study was a positive one: Finnish Wikipedia content is
>> surprisingly good, even though the quality does vary a lot.
>>
>> The consequences of this study seem to be even more positive, I think.
>> Similarly to what happened after the Nature study, the community started
>> improving on these articles immediately. It seems this has created lots of
>> positive buzz, attention and activity in Finnish Wikipedia.
>>
>> This is the main article (in Finnish):
>> http://www.hs.fi/tiede/HS+selvitti+N%C3%A4in+luotettava+Wikipedia+on/a1305754303586
>> Here is the data as Excel sheet:
>> http://files.snstatic.fi/HS/2013/11/wikipedia/hs-wikipedia-selvitys.xls
>>
>> With kind regards,
>> Arto Lanamäki
>>
>>
>> ________________________________
>> Lähettäjä: wiki-research-l-bounces@lists.wikimedia.org
>> [wiki-research-l-bounces@lists.wikimedia.org] käyttäjän phoebe ayers
>> [phoebe.wiki@gmail.com] puolesta
>> Lähetetty: 5. joulukuuta 2013 21:26
>> Vastaanottaja: Research into Wikimedia content and communities
>> Aihe: [Wiki-research-l] Fwd: the Helsinki Times evaluates the Finnish
>> Wikipedia
>>
>>
>> (a la the old Nature study)
>>
>> http://www.helsinkitimes.fi/finland/finland-news/domestic/8619-world-s-largest-study-on-wikipedia-better-than-it-s-reputation.html
>>
>> World’s largest study on Wikipedia: Better than it’s reputation
>>
>> 05 Dec 2013
>>
>> Wikipedia is the most popular encyclopedia in the world, and now in many
>> countries it is also the only one. But can the information in Wikipedia be
>> trusted at all?
>>
>> Helsingin Sanomat newspaper evaluated 134 articles in the Finnish-language
>> version of Wikipedia with 96 experts. As far as we know, our study is the
>> most extensive individual investigation on the trustworthiness of Wikipedia
>> in the whole world.
>>
>> We found out that Wikipedia is better than it’s reputation: seventy per cent
>> of the articles got good points for accuracy. But in many ways the Finnish
>> Wikipedia is also far from flawless.
>>
>> Olavi Koistinen HS
>>
>> Hundreds of thousands of people use the Finnish-language version of
>> Wikipedia every day. Its articles rank at the top in Google search results,
>> whether you are looking for information on climate change, Michael Jackson,
>> Sydney, ball bearings, cancer, allosaurus, or any number of other topics.
>>
>> Wikipedia is the only Finnish encyclopedia that is still being updated. The
>> golden age of printed encyclopedias was in the 1990s, but with the advent of
>> the internet, their sales collapsed. The last traditional Finnish
>> encyclopedia, the online version of WSOY’s Facta, was closed down in 2011.
>>
>> The reliability of the English language Wikipedia has been studied for
>> years. The most famous study was published by the science magazine Nature in
>> 2005, comparing the number of errors in Wikipedia with those in the
>> Encyclopaedia Britannica. The result of the comparison was that Wikipedia
>> was almost as accurate as the Encyclopaedia Britannica. The study involved
>> 42 articles from each encyclopaedia.
>>
>> However, no thorough investigations have been made about the reliability of
>> the Finnish Wikipedia – until now.
>>
>> We first chose 134 articles in the Finnish Wikipedia, covering different
>> areas of life, to be studied. Then we asked an expert with a thorough
>> knowledge on the subject matter to evaluate each article. There were a total
>> of 96 people making assessments, most of them professors or other university
>> researchers. They work in eight different Finnish universities.
>>
>> The Finnish Wikipedia currently has nearly 340 000 articles, so the random
>> selection of 134 articles naturally does not give a complete picture of the
>> reliability of the encyclopedia. Our study is nevertheless the most
>> extensive individual investigation or report on the trustworthiness of
>> Wikipedia in the whole world, as far as we know, if the matter is measured
>> on the basis of the number of articles examined.
>>
>> “With this measure the study is unique in its scale”, says researcher Arto
>> Lanamäki of the University of Bergen in Norway. Not even the
>> English-language Wikipedia has undergone such an extensive single survey.
>>
>> The experts read the articles and scored them based on six different
>> indicators, which were lack of errors, coverage and balance, sourcing,
>> topicality, neutrality, and clarity. The articles were graded on a scale of
>> 1-5, in which 1 was the worst mark and 5 was the best.
>>
>> The entire result dataset was published as an open data, and is available
>> here.
>>
>> First the good news: The Finnish Wikipedia is largely error-free. The lack
>> of errors is the area in which Wikipedia clearly got its best score. For
>> instance, if someone uses Wikipedia to find the year in which Christopher
>> Columbus discovered the America, the information in the article is most
>> likely to be correct.
>>
>> No less than 70 per cent of the articles were judged to be good (4) or
>> excellent (5) with respect to lack of errors. According to the indicative
>> evaluation scale a four means that that the article has only “scattered
>> small errors, no big ones”.
>>
>> This is how we evaluated the Finnish language Wikipedia
>>
>> Journalists of Helsingin Sanomat first listed 150 topics from different
>> walks of life. Only after that did we check if any Wikipedia articles
>> actually existed on the topic (the vast majority did exist). 134 Wikipedia
>> articles were selected for the final evaluation.
>> To evaluate each article we chose an university-level researcher with
>> knowledge on the subject matter to be an evaluator. There were a total of 96
>> evaluators, of which 94 were researchers. On top of that, two experienced
>> sport journalists did evaluate 8 Wikipedia articles about sports.
>> Some of the evaluators assessed more than one article.
>> The evaluation was done via a web form.
>> The basic principle of the evaluation was: If an amateur would read the
>> article, would he or she get a truthful impression of the subject matter?
>> Each evaluator scored the article on the basis of six criteria: lack of
>> errors, coverage and balance, sourcing, topicality, neutrality, and clarity.
>> The scale of the points was 5=excellent, 4=good, 3=adequate, 2=tolerable,
>> 1=poor. The evaluators were also given the chance to comment the article
>> based on each criteria.
>> Most of the evaluations were made in early November. They were based on
>> copies of Wikipedia articles as they were on October 22, 2013.
>>
>> On the question of lack of errors the most common mark was four. What does
>> it mean in practice? According to the evaluators, factors such as these:
>>
>> “No misleading errors, but some imprecision.” (Higgs boson)
>>
>> “Quite proficient text, hardly any actual factual errors. However, odd
>> transliteration stands out.” (Icon)
>>
>> “Geological periods do not quite hit the mark, the factual errors are
>> insignificant.” (Climate change)
>>
>> However, the conclusion should not be drawn that Wikipedia can be trusted
>> naively. A total of 14 articles were graded as poor (1) or tolerable (2).
>>
>> “Already in the first sentence the definition of photosynthesis is odd, and
>> several mistakes were found.” (Photosynthesis)
>>
>> “The article is very uneven in quality, and at times quite propagandistic.”
>> (Syrian civil war)
>>
>> Many of the articles containing the most errors were about a complicated or
>> abstract phenomena, such as the eurozone debt crisis, evolution, or the
>> Syrian civil war. On the other hand, several articles dealing with equally
>> complex subjects got high marks for lack of errors. For instance, the
>> articles for the big bang and climate change were praised for their
>> precision.
>>
>> Although most of the articles were extolled for their accuracy, it can be
>> concluded that there is more variation in the quality of articles in Finnish
>> Wikipedia than in traditional encyclopedias. According to studies, the same
>> also applies to English Wikipedia: the best articles are brilliant, but many
>> are weak.
>>
>> How can a reader assess the quality of a Wikipedia article?
>>
>> A good way to examine the reliability of any particular article is to check
>> it’s factual sources. If the sources of information are listed at the end of
>> an entry or inside the text itself, the reader can become acquainted with
>> them and form an opinion of the credibility of the information on that
>> basis.
>>
>> According to our study there are serious shortcomings in the Finnish
>> Wikipedia specifically in the sourcing of articles. No less than 38 per cent
>> of the articles got a grade of weak (1) or tolerable (2). The evaluators had
>> harsh criticism for these articles.
>>
>> “Sources were not used at all and it shows.” (Internal combustion engine)
>>
>> “On the basis of the text it would seem that sources have mainly included TV
>> documentaries or children’s dinosaur books.” (Allosaurus)
>>
>> “Based primarily on a single disputed work. Sources from antiquity are
>> considered inadequate and hostile.” (Caligula)
>>
>> The last comment highlights a broader problem. The evaluators were critical
>> that many articles were based mostly or partly on one source. This sometimes
>> causes problems: the information from individual works can be selective, and
>> the interpretations biased.
>>
>> “The article seeks to be neutral. However, the points of emphasis of the
>> material that it was based on can clearly be seen.” (Protestant Reformation)
>>
>> Of Wikipedia’s own ideals, neutrality is one of the most important.
>> Wikipedia emphasises that the articles need to be written from a “neutral”
>> point of view.
>>
>> On the basis of our survey the Finnish Wikipedia has been written mainly
>> with a balanced approach, with 56 per cent of the articles getting good (4)
>> or excellent (5) marks. If some articles in the sample were slanted in their
>> statements or points of view, it usually was not attributable to the
>> writer’s deliberate partiality.
>>
>> “The article is positive in its attitude and aims at neutrality. The
>> problems in the content are connected with the source material and the
>> writer’s basic knowledge: it is clearly not a result of anything
>> deliberate.” (Middle Ages)
>>
>> Of the 134 articles that were surveyed, only six were found to have been
>> written with a clear bias, in the opinion of the reviewer. The articles were
>> connected with politics of the United States (Osama bin Laden’s death, the
>> US Democratic Party, the 9/11 terror attacks, Alan Greenspan) and events in
>> the Middle East (The civil war in Syria, the second intifada).
>>
>> “In conditions of war and with an intense conflict dominating, the
>> neutrality requirement for the article is difficult. Occasionally, when
>> reading this article it seems, however, that no actual attempt was made to
>> attain neutrality.” (Civil war in Syria)
>>
>> “A fairly flattering article for the Democrats. Other points of view would
>> also exist.” (US Democratic Party)
>>
>> It is interesting that the experts felt that the wrong kind of “neutrality”
>> and avoiding expressing points of view is also a problem. If the writer is
>> afraid to make interpretations of any kind, the article sometimes ends up
>> being superficial.
>>
>> “There are no interpretations and consequently no points of view either.”
>> (Kingdom of Mali)
>>
>> “Generally a text only lists things instead of pondering them and presenting
>> well-founded evaluations.” (Finnish composer Kaija Saariaho)
>>
>> “As the article seeks to be very objective, it starts strangely by
>> describing Pinochet as a ‘president’ – after all, he was one of the
>> best-known dictators of the 20th century.” (Augusto Pinochet)
>>
>> Sometimes individual facts in an article are correct, but the text fails to
>> mention relevant aspects of the topic, or mentions them too briefly. The
>> article might also ramble on excessively about insignificant details. In
>> such a situation the lay reader might get an inaccurate image of, for
>> instance, what the most important turning points of a country’s history or
>> what the most important achievements of a researcher might be.
>>
>> This is a quality factor which the evaluators measured by scoring each
>> article based on “coverage and balance”. It has an impact on how well
>> Wikipedia can convey an overall image of the matter to the reader.
>>
>> The experts’ marks on coverage and balance were divided almost equally
>> between good and bad. So at least half of the articles could have been more
>> comprehensive and balanced.
>>
>> “The article primarily tells about marriage based on Western – mainly Roman
>> law and the Christian tradition. Other cultures and religions have been left
>> completely outside the examination.” (Marriage)
>>
>> “The article puts far too much emphasis on personal history and even on
>> related insignificant details. The presentation of the main topic, the
>> scientific work, is far too short and superficial compared with the rest of
>> the material.” (Albert Einstein)
>>
>> Printed encyclopedias were often criticised as containing obsolete
>> information even when they were fresh off the press. Wikipedia has a better
>> chance to stay topical, since articles can be updated at any time.
>>
>> The evaluators gave Finnish Wikipedia fairly high marks for topicality, with
>> 43 per cent of the articles getting either excellent or good marks, but on
>> the other hand, 31 per cent were poor or just tolerable in terms of
>> freshness. When something big happens, breaking news are often updated into
>> Wikipedia, but the follow-up on events is poor.
>>
>> “All sources are old, which means that the content of the text has not been
>> updated after 2011 to any practical degree.” (Osama bin Laden’s death)
>>
>> On the other hand, articles can become obsolete even if the topic is a
>> mountain range that rose from inside the earth in prehistoric times, or a
>> notable person who died in the 1930s; research on these topics often still
>> goes on.
>>
>> “The biggest problem of the article is that it does not reflect the current
>> situation of international research.” (Middle Ages)
>>
>> “Within psychoanalysis, much has happened since Freud, and continues to
>> happen, and Freud’s ideas have been re-evaluated many times. There is almost
>> nothing about this in the article.” (Sigmund Freud)
>>
>> Table of results on the study of Finnish Wikipedia
>>   1: Lack of errors 2: Coverage and balance 3. Sourcing 4: Topicality 5:
>> Neutrality 6. Clarity
>> 1 (poor) 3 % 5 % 17 % 13 % 5 % 2 %
>> 2 (tolerable) 7 % 24 % 21 % 19 % 16 % 11 %
>> 3 (adequate) 19 % 41 % 34 % 25 % 22 % 31 %
>> 4 (good) 47 % 25 % 21 % 30 % 34 % 42 %
>> 5 (excellent) 23 % 5 % 7 % 13 % 22 % 13 %
>>
>> % = amount of articles inside the whole sample, which was 134 articles
>>
>> So what are we supposed to think about all of these results? Can information
>> from Finnish Wikipedia be trusted or not?
>>
>> There are two points of view on this matter.
>>
>> Those who have felt so far that everything that is in Wikipedia is true
>> would do well to re-examine their naiveté to a certain degree.
>>
>> Those who have felt that Wikipedia’s content is nothing but inaccurate and
>> biased pseudo-information, should ease up a little. Based on our survey,
>> this is not the case; as a source of information, Wikipedia is a better than
>> it’s reputation – which is not a particularly good one.
>>
>> Arto Lanamäki, who has studied Wikipedia at the University of Bergen, says
>> that people often take a very suspicious view on Wikipedia, even if they
>> themselves use it regularly for seeking information.
>>
>>  “In studies, the same article has been placed in the framework of the
>> Encyclopaedia Britannica and Wikipedia, and brought to different people for
>> evaluation. It is quite common for people to take a more suspicious view of
>> the article when it is presented in the framework of Wikipedia”, Lanamäki
>> says.
>>
>> On the other hand, there are some good reasons for the doubts: the quality
>> of Wikipedia fluctuates considerably. Seventy per cent of the articles got
>> good points for accuracy in our study. If the bar is lowered slightly, then
>> 90 per cent of the articles were at least “adequate” in the view of the
>> reviewers. That is a vast majority, but you might also ask yourself a
>> question: would you trust a printed encyclopedia if you knew that every
>> tenth article in it was inaccurate?
>>
>> Wikipedia’s undeniable strength, however, is that the information is updated
>> and upgraded all the time. If you paid good money in the early 2000s for a
>> set of books for your bookshelf, it is already obsolete in many respects.
>> Wikipedia’s massive popularity indicates that our view of factual
>> information is in flux. Rising alongside information confirmed by experts
>> and printed in dignified books, is peer-produced information that is
>> constantly spreading and accumulating on the internet.
>>
>> If you see that something written in Wikipedia makes no sense, do everyone a
>> favour and edit the article so that it is better. You don’t even need a
>> Wikipedia user account for it.
>>
>> Ask not how Wikipedia can help you – ask how you can help Wikipedia.
>>
>> (This article was originally published in Finnish in Helsingin Sanomat at
>> November 30th. The HS working group taking part in the drafting of the
>> survey included Riikka Haikarainen, Tuomas Kaseva, Niko Kettunen, Olavi
>> Koistinen, Veikko Lautsi, Siri Markula, Sami Simola and Timo Paukku.)
>>
>>
>>
>> Finnish Wikipedia has only a few hundred regular writers
>>
>> By Olavi Koistinen HS
>>
>> Wikipedia can be written or edited by anyone. About 70,000 people have done
>> one or more edits on the Finnish Wikipedia.
>>
>> However, most of the content comes from a much smaller group of active
>> contributors.
>>
>> About 200 people write or edit Finnish Wikipedia regularly, says Joonas
>> Lyytinen, a veteran Wikipedia contributor. He also says that there is an
>> even more active core group consisting of only 20–30 people who produce very
>> large amount of content and take care of many administrative tasks in
>> Finnish Wikipedia. Being part of that group, Lyytinen himself has written
>> about 2000 articles for Finnish version of Wikipedia.
>>
>> A typical Wikipedia author is a male university student. Often he is a
>> layman who enjoys writing Wikipedia articles because it is a nice way to
>> learn new things – by researching and writing about them.
>>
>> Researchers also write in Wikipedia, but less than laypeople do. At least
>> some of the researchers feel hesitant with respect to Wikipedia, says
>> University of Bergen researcher Arto Lanamäki, who has studied the Finnish
>> Wikipedia.
>>
>> The articles in Wikipedia are often the result of teamwork, and there is an
>> ongoing debate about the content in the discussion pages of the Wikipedia.
>> In this debate, a researcher thoroughly familiar with the subject does not
>> get any credibility over a layman just because of his or her position or
>> title – the best argument wins.
>>
>> Arguing with hobbyists can be stressful for many researchers. “Often a
>> writer has tried to explain his or her point of view by saying ‘don’t you
>> understand that I have a doctorate and I have researched this topic?’
>> However, this will not work with Wikipedia”, Lanamäki says.
>>
>> There is no advance censorship, but active users of Wikipedia monitor new
>> updates constantly to counter vandalism. Sometimes, for instance, middle
>> school students tend to sabotage articles.
>>
>> Finnish Wikipedia would benefit from having more writers. There are just 5
>> million Finnish speakers and only a few real experts in some fields of
>> knowledge. Wikipedia writers also suffer from a somewhat negative image
>> among occasional users of the service. This might even hinder some people
>> from participating in writing Wikipedia articles.
>>
>> “It has been observed in academic studies that an impediment to
>> participation in Wikipedia is that people do not want to be labelled
>> Wikipedia nerds”, Lanamäki says.
>>
>> Olavi Koistinen HS
>>
>>
>>
>> --
>> * I use this address for lists; send personal messages to phoebe.ayers <at>
>> gmail.com *
>>
>> _______________________________________________
>> Wiki-research-l mailing list
>> Wiki-research-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>>
>
>
>
> --
> * I use this address for lists; send personal messages to phoebe.ayers
> <at> gmail.com *
>
> _______________________________________________
> Wiki-research-l mailing list
> Wiki-research-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l