The journal article by Hasty et al published on May 1st 2014 basically took
ten Wikipedia articles and ten “researchers” (either medical students or
residents). Each Wikipedia article was then assessed by two of these
researchers to try to determine how many statements of fact they contained.
The first issue was that the number of statements of fact each reviewer
found sometimes differed by nearly 100%. They than took these individual
facts and the “researchers” compared them with the peer reviewed literature
as found on pubmed or the medical website Uptodate. They did not check to
see if the sources Wikipedia was using were high quality or were accurately
reflected. Additionally medical students and residents are hardly experts
in medical research.
No errors in Wikipedia are mentioned directly in the original journal
article. When I spoke with the lead author he declined to release the
underlying data for us at Wikipedia to correct the “errors” they had found
stating that he may 1) wish to publish more on the topic and 2) wished to
protect the researchers. So much for independent verifiability in science.
Hasty did make some claims to the popular press about errors on Wikipedia.
Some of the facts he mentioned however accurately reflected some of the
best available peer reviewed sources. For example he claimed that blood
pressure should only be checked twice to make the diagnosis of hypertension
and that when we state three times we are wrong. However look at the
National Institute for Health and Care Excellence (previous known as the
National Institute of Clinical Health / NICE) on page 7 in this document
http://www.nice.org.uk/nicemedia/pdf/CG18background.pdf It is thus a little
ironic that the Telegraph, a UK paper, repeated this incorrect statement
and the BBC covered the story so uncritically.
Wikipedia has strong recommendations for what counts as a suitable source.
We recommend the use of secondary sources published in well respected
journals from the last 3-5 years, position statements of national or
internationally recognized medical bodies or major textbooks. Is Wikipedia
a perfect source? No, but it is just as good as many and better than most
other sources out there. Or else why would the world be using it? Hasty's
work did not have a comparison group. Basically he invented a new method to
test the quality of medical content and then only applied this new method
to one source, Wikipedia. Without a comparator this single data point is
meaningless. I am curious what he would have found if he would have applied
this to a NICE guideline or emedicine?
We recently surveyed our top contributors and asked about their
backgrounds. What we found was that 52% have either a masters, PhD, or MD.
Another 33% have a BSc. About half are health care providers. 82% are male,
9% are female and 9% classified themselves as other or would rather not
say. This is very similar to results published by Nusa Faric in her
master's thesis. Additionally we are working with a number of organizations
including: the National Institute of Health, the Cochrane collaboration,
and the UCSF college of medicine among others to improve Wikipedia’s health
care content.
What Hasty did show was 1) the peer reviewed literature does not agree with
itself (ie different peer reviewed sources come to different conclusions
which is no surprise to anyone that has read much of it) 2) the peer review
process is sometimes flawed as he was able to publish a "peer reviewed"
article whose data does not support its conclusions. As someone who has
read a lot of the peer reviewed literature this is also not surprising.
--
James Heilman
MD, CCFP-EM, Wikipedian
The Wikipedia Open Textbook of Medicine
www.opentextbookofmedicine.com