On Wed, May 7, 2014 at 4:38 PM, Andreas Kolbe jayen466@gmail.com wrote:
On Thu, May 8, 2014 at 12:22 AM, phoebe ayers phoebe.wiki@gmail.com wrote:
On Wed, May 7, 2014 at 3:14 PM, Andreas Kolbe jayen466@gmail.com
wrote:
Anne, there are really well-established systems of scholarly peer
review.
There is no need to reinvent the wheel, or add distractions such as infoboxes and other bells and whistles.
And those peer review systems have lots and lots of problems as well as upsides. Lots of people *are* trying to reinvent peer review, including some very respected scientists.* As an academic science librarian, I can attest to there being widespread and currently ongoing debates about how
to
review scientific knowledge, whether traditional peer review is
sufficient,
and how to improve it. The current system for scientific research is
often
opaque, messy, prone to failure and doesn't always support innovation,
and
lots of smart people are thinking about it.
Erik: aha! I'd forgotten about those case studies, thanks!
Given that the post that started this thread referenced medical content, are you telling me that you think it would be useless to have qualified medical experts reviewing Wikipedia's medical content, because the process would be "opaque, messy, prone to failure and doesn't always support innovation"?
No, that is not what I am saying; and leaping to that conclusion seems like a rather pointy and bad-faith approach, which makes it just that much more of an effort to participate in this conversation -- if you want to have a dialog with other people, please try to be more generous in your assumptions.
What I was trying to say is that I don't think your implication that there is already a well-designed solution that will fix all our problems is correct -- both because it's difficult to apply peer review in this context, and because peer review has plenty of problems itself. I think blind-review quality studies can be useful, but I don't think they're a panacea, anymore than scholarly peer review is itself a panacea for making sure good scholarly work gets published.
Anyway, reviewer studies are one tool for assessing quality, but imho they are mostly good for raising awareness of Wikipedia within a particular field (thus possibly gaining new editors), and occasionally for correcting the few articles that do get reviewed.
Article quality has lots of dimensions, including those that reviewers might look for, and others that might not be apparent:
* factual accuracy -- that seems pretty straightforward, though of course it's not always -- cf historical debates, new evidence coming to light, etc. * important facets of the topic being highlighted and appropriate coverage -- also pretty straightforward, except when it's not: what if a new and emerging theory isn't noted, or a historical one given short shrift? More to the point for reviewers, what if *my* theory isn't highlighted? * A good bibliography and references -- I think experts can particularly weigh in on this, though standards vary widely across fields and articles for what gets cited, and what's good/seminal/classic is of course never easy to determine and is always under debate. * clear writing -- sometimes we get accused of being too dry or pedantic, when that's our house style. What to do with this? * Accessibility -- depends entirely on who is reading it, doesn't it? are our physics articles accessible to grad students? Usually. Accessible to laypeople or 10th graders? Rarely. * Answers readers' questions -- hard to know without something like article feedback or another measuring mechanism. The questions of a new student are rarely those of an expert. Using medicine as an example: does the article on cancer answer the questions of doctors, or of newly-diagnosed patients (who are likely to be reading it)? Or the patients' relatives and caregivers? (Or none of the above?)
So yes, we should do reviewer studies to review for "objective" quality. Also, if we're serious about seeing how our articles meet reader needs [certainly one dimension of quality], we should also do reviewer studies with lots of groups of reviewers (medical experts, high school students, cancer patients!) And we should look at automated quality metrics, since reviewing 31 million articles by hand does not necessarily scale. And, we should look into ways to follow up on quality studies with things like to-do lists generated from reviewers, getting people in societies and universities engaged in editing based on the outcome of reviewing, etc. -- so that all of this work has the outcome of measurably improved quality.
Personally (not speaking for the WMF or other trustees here) I think the best thing the WMF can do is provide a platform for this kind of work: yes, we can (and do) fund research studies, but in line with our general mission to provide the infrastructure for the projects to grow on, we can also help build tools to make this work easier, so that groups like Wiki Project Med etc can get studies done easily as well. And we (the community) should develop a list of tools that those interested in doing this work need and want -- and those tools could be developed anywhere, under the aegis of the WMF or not.
(Off the top of my head, these could include: tools to pull a random/blind sample from a category, perhaps across already-rated articles, that could be replicated across topics to do multiple comparable reviewer studies. Tools to consolidate editor-rating metrics from across languages; maybe representing those ratings in Wikidata. A strong to-do list functionality, and a strong category/quality rating intersection functionality, so that, say, an oncologist interested in working on poor-quality cancer articles could easily get to editing. Displaying all this data easily in the projects, by article. etc. etc.)
-- phoebe