On Thu, May 8, 2014 at 2:41 AM, Risker <risker.wp(a)gmail.com> wrote:
I think perhaps there is a lack of research into the extent of research
already being done by independent, qualified third parties. Several
examples are provided in the references of the study you posted, Andreas.
For example, this one in the Journal of Oncology Practice[1] compares
specific Wikipedia articles for patient-oriented cancer information against
the professionally edited PDQ database. It appears that the two were
comparable in most areas, except for readability, where the PDQ database
was considered significantly more readable. Now, again, this is a small
study and it has not been reproduced; however, it took me minutes to find
more information on the very subject you're interested in, created by
completely independent bodies who have "no pony in the race". There did
seem to be a fair number of studies related to medical topics. Now if only
we could learn from them - especially on the readability point, which I
think really is a very serious issue. Wikipedia isn't really intended to
educate physicians about medical topics, it's intended to be a general
reference for non-specialists.
Very few people are going to make life-and-death decisions based on our
math or physics topic areas, but I'll lay odds that any study would find a
significant readability issue with both of them, as well.
Risker/Anne
[1]
http://jop.ascopubs.org/content/7/5/319.full
In the study you reference, Anne, reviewers spent all of 18 minutes on each
article. The readability analysis was done by automation.
I did review the link Phoebe posted earlier;
https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Medicine/Wikipedia_and_…
I found the number of recent studies assessing actual Wikipedia content in
this field there very scanty. The best seemed to be this February 2014
study in the European Journal of Gastroenterology & Hepatology:
http://www.ncbi.nlm.nih.gov/pubmed/24276492
The conclusion of that study, based on a review of 39 articles by three
assessors, was that
—o0o—
"Wikipedia is not a reliable source of information for medical students
searching for gastroenterology and hepatology articles. Several
limitations, deficiencies, and scientific errors have been identified in
the articles examined."
—o0o—
There was also this study, concluding that Wikipedia was "fairly reliable":
http://onlinelibrary.wiley.com/doi/10.1111/sdi.12059/full
But they say their reliability assessment was based on a simple count of
references, without reviewing the actual article content for accuracy!
—o0o—
Assessment of Reliability
The reliability of nephrology articles in Wikipedia was determined in two
ways: (i) mean number of references per article, and (ii) mean percentage
of “substantiated” references—which we defined as references corresponding
to works published in peer-reviewed journals or from texts with an
associated International Standard Book Number (ISBN).
—o0o—
As far as I am aware from discussions with members of Wiki Project Med
Foundation/WikiProject Medicine, all the studies that have been done to
date suffer from small sample size or other methodological limitations.
There is a real gap for a large-scale, well-designed study of a random
subset of Wikipedia's most-consulted medical articles.*
If Wikipedia wants to be serious about its mission, measuring article
quality is a must. Given how widely Wikipedia is used today, it is also a
question of social responsibility.
I imagine the reluctance I sense in this discussion is in part due to
people's fear that the results might be less than stellar. If so, that fear
is misplaced. There is no improvement without performance feedback. If the
results are indeed disappointing, the related publicity should lead to an
increased focus on improvement efforts, and indeed may encourage a greater
influx of better-qualified editors if such are needed.
The project is well into its second decade. It is mature and
well-established enough for such a test. It's a question of taking a
long-term view. Long-term improvement will be accelerated by honest,
knowledgeable feedback.
Studies could be repeated at annual intervals, to track progress and
measure improvement. I don't believe there is any other way of arriving at
a reliable reference source, which after all is what this entire effort
should be about.
* Some ongoing related discussions at WikiProject Medicine here:
https://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Medicine#This_conv…