On Thu, May 8, 2014 at 2:41 AM, Risker risker.wp@gmail.com wrote:
I think perhaps there is a lack of research into the extent of research already being done by independent, qualified third parties. Several examples are provided in the references of the study you posted, Andreas. For example, this one in the Journal of Oncology Practice[1] compares specific Wikipedia articles for patient-oriented cancer information against the professionally edited PDQ database. It appears that the two were comparable in most areas, except for readability, where the PDQ database was considered significantly more readable. Now, again, this is a small study and it has not been reproduced; however, it took me minutes to find more information on the very subject you're interested in, created by completely independent bodies who have "no pony in the race". There did seem to be a fair number of studies related to medical topics. Now if only we could learn from them - especially on the readability point, which I think really is a very serious issue. Wikipedia isn't really intended to educate physicians about medical topics, it's intended to be a general reference for non-specialists.
Very few people are going to make life-and-death decisions based on our math or physics topic areas, but I'll lay odds that any study would find a significant readability issue with both of them, as well.
Risker/Anne
In the study you reference, Anne, reviewers spent all of 18 minutes on each article. The readability analysis was done by automation.
I did review the link Phoebe posted earlier;
https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Medicine/Wikipedia_and_m...
I found the number of recent studies assessing actual Wikipedia content in this field there very scanty. The best seemed to be this February 2014 study in the European Journal of Gastroenterology & Hepatology:
http://www.ncbi.nlm.nih.gov/pubmed/24276492
The conclusion of that study, based on a review of 39 articles by three assessors, was that
—o0o—
"Wikipedia is not a reliable source of information for medical students searching for gastroenterology and hepatology articles. Several limitations, deficiencies, and scientific errors have been identified in the articles examined."
—o0o—
There was also this study, concluding that Wikipedia was "fairly reliable":
http://onlinelibrary.wiley.com/doi/10.1111/sdi.12059/full
But they say their reliability assessment was based on a simple count of references, without reviewing the actual article content for accuracy!
—o0o—
Assessment of Reliability
The reliability of nephrology articles in Wikipedia was determined in two ways: (i) mean number of references per article, and (ii) mean percentage of “substantiated” references—which we defined as references corresponding to works published in peer-reviewed journals or from texts with an associated International Standard Book Number (ISBN). —o0o—
As far as I am aware from discussions with members of Wiki Project Med Foundation/WikiProject Medicine, all the studies that have been done to date suffer from small sample size or other methodological limitations. There is a real gap for a large-scale, well-designed study of a random subset of Wikipedia's most-consulted medical articles.*
If Wikipedia wants to be serious about its mission, measuring article quality is a must. Given how widely Wikipedia is used today, it is also a question of social responsibility.
I imagine the reluctance I sense in this discussion is in part due to people's fear that the results might be less than stellar. If so, that fear is misplaced. There is no improvement without performance feedback. If the results are indeed disappointing, the related publicity should lead to an increased focus on improvement efforts, and indeed may encourage a greater influx of better-qualified editors if such are needed.
The project is well into its second decade. It is mature and well-established enough for such a test. It's a question of taking a long-term view. Long-term improvement will be accelerated by honest, knowledgeable feedback.
Studies could be repeated at annual intervals, to track progress and measure improvement. I don't believe there is any other way of arriving at a reliable reference source, which after all is what this entire effort should be about.
* Some ongoing related discussions at WikiProject Medicine here:
https://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Medicine#This_conve...