On Wed, May 7, 2014 at 4:38 PM, Andreas Kolbe <jayen466(a)gmail.com> wrote:
On Thu, May 8, 2014 at 12:22 AM, phoebe ayers
<phoebe.wiki(a)gmail.com>
wrote:
On Wed, May 7, 2014 at 3:14 PM, Andreas Kolbe
<jayen466(a)gmail.com>
wrote:
> Anne, there are really well-established systems of scholarly peer
review.
There is
no need to reinvent the wheel, or add distractions such as
infoboxes and other bells and whistles.
And those peer review systems have lots and lots of problems as well as
upsides. Lots of people *are* trying to reinvent peer review, including
some very respected scientists.* As an academic science librarian, I can
attest to there being widespread and currently ongoing debates about how
to
review scientific knowledge, whether traditional
peer review is
sufficient,
and how to improve it. The current system for
scientific research is
often
opaque, messy, prone to failure and doesn't
always support innovation,
and
lots of smart people are thinking about it.
Erik: aha! I'd forgotten about those case studies, thanks!
Given that the post that started this thread referenced medical content,
are you telling me that you think it would be useless to have qualified
medical experts reviewing Wikipedia's medical content, because the process
would be "opaque, messy, prone to failure and doesn't always support
innovation"?
No, that is not what I am saying; and leaping to that conclusion seems like
a rather pointy and bad-faith approach, which makes it just that much more
of an effort to participate in this conversation -- if you want to have a
dialog with other people, please try to be more generous in your
assumptions.
What I was trying to say is that I don't think your implication that there
is already a well-designed solution that will fix all our problems is
correct -- both because it's difficult to apply peer review in this
context, and because peer review has plenty of problems itself. I think
blind-review quality studies can be useful, but I don't think they're a
panacea, anymore than scholarly peer review is itself a panacea for making
sure good scholarly work gets published.
Anyway, reviewer studies are one tool for assessing quality, but imho they
are mostly good for raising awareness of Wikipedia within a particular
field (thus possibly gaining new editors), and occasionally for correcting
the few articles that do get reviewed.
Article quality has lots of dimensions, including those that reviewers
might look for, and others that might not be apparent:
* factual accuracy -- that seems pretty straightforward, though of course
it's not always -- cf historical debates, new evidence coming to light,
etc.
* important facets of the topic being highlighted and appropriate coverage
-- also pretty straightforward, except when it's not: what if a new and
emerging theory isn't noted, or a historical one given short shrift? More
to the point for reviewers, what if *my* theory isn't highlighted?
* A good bibliography and references -- I think experts can particularly
weigh in on this, though standards vary widely across fields and articles
for what gets cited, and what's good/seminal/classic is of course never
easy to determine and is always under debate.
* clear writing -- sometimes we get accused of being too dry or pedantic,
when that's our house style. What to do with this?
* Accessibility -- depends entirely on who is reading it, doesn't it? are
our physics articles accessible to grad students? Usually. Accessible to
laypeople or 10th graders? Rarely.
* Answers readers' questions -- hard to know without something like article
feedback or another measuring mechanism. The questions of a new student are
rarely those of an expert. Using medicine as an example: does the article
on cancer answer the questions of doctors, or of newly-diagnosed patients
(who are likely to be reading it)? Or the patients' relatives and
caregivers? (Or none of the above?)
So yes, we should do reviewer studies to review for "objective" quality.
Also, if we're serious about seeing how our articles meet reader needs
[certainly one dimension of quality], we should also do reviewer studies
with lots of groups of reviewers (medical experts, high school students,
cancer patients!) And we should look at automated quality metrics, since
reviewing 31 million articles by hand does not necessarily scale. And, we
should look into ways to follow up on quality studies with things like
to-do lists generated from reviewers, getting people in societies and
universities engaged in editing based on the outcome of reviewing, etc. --
so that all of this work has the outcome of measurably improved quality.
Personally (not speaking for the WMF or other trustees here) I think the
best thing the WMF can do is provide a platform for this kind of work: yes,
we can (and do) fund research studies, but in line with our general mission
to provide the infrastructure for the projects to grow on, we can also help
build tools to make this work easier, so that groups like Wiki Project Med
etc can get studies done easily as well. And we (the community) should
develop a list of tools that those interested in doing this work need and
want -- and those tools could be developed anywhere, under the aegis of the
WMF or not.
(Off the top of my head, these could include: tools to pull a random/blind
sample from a category, perhaps across already-rated articles, that could
be replicated across topics to do multiple comparable reviewer studies.
Tools to consolidate editor-rating metrics from across languages; maybe
representing those ratings in Wikidata. A strong to-do list functionality,
and a strong category/quality rating intersection functionality, so that,
say, an oncologist interested in working on poor-quality cancer articles
could easily get to editing. Displaying all this data easily in the
projects, by article. etc. etc.)
-- phoebe