This is currently published on my user space on Wikinews at
as done in order to determine some of the issues that students are facing
with the review process at Wikinews in light of student complaints to their
instructor. I am hoping to do a follow up for it at some point in the
future to provide more detailed analysis on review processes on project,
but thought in the interim it might be of interest to both researchers and
people in education.
Wikinews Review Analysis
< User:LauraHale <http://en.wikinews.org/wiki/User:LauraHale>
One of the greatest challenges in attracting and retaining new contributors
to English Wikinews is the project's review process. This process requires
the journalist to have a symbiotic relationship with the reviewer as the
pair both work towards getting an article published. The review process
involves the submitted news article being checked for copyright,
newsworthiness, verifiability, neutral point of view and compliance with
the style guide. This process is compacted ideally into a window of
opportunity of no longer than 24 to 48 hours. It can be extremely
challenging for those not used to the style requirements, neutrality
requirements, verification requirements and above all doing this sort of
For those from Wikipedia, the review process on English Wikinews is close
in scope to English Wikipedia's Good Article
on a more compressed timetable. For those who have been to secondary school
or university, it mirrors a teacher giving you an assignment with your
grade being based on meeting the criteria stated on a rubric, and failure
to pass requires re-submission. If you do not work to the rubric, you just
do not pass and on English Wikinews, that means not getting published.
When you review regularly, you begin to notice certain patterns that
frequently occur in the review process. Most often, these appear to be a
failure to understand what is news or a failure to read and try to write to
the style guide specifications. Observational analysis only gets you so far
though when trying to determine a problem and how to develop solutions for
what appears to be a problem of articles not passing review or people being
discouraged from submitting.
From the period between January 1, 2013 and April 12,
2013, 203 failed
reviews were examined to determine which criteria were the biggest
stumbling block. The articles reviewed included published articles,
non-published articles moved to user space and deleted articles that were
not published. For each review, the primary author was assessed as either
accredited reporter, regular contributor with 10 or more published
articles, new reporter with 9 or fewer articles, or University of
Wollongong student. Of the 202 reviews examined, 104 were for articles by
new contributors, 47 by University of Wollongong students, 40 by regular
contributors and 11 by accredited reporters. Each review looked at also
noted if the article finally reached a published state. There were 110
different articles reviewed, of which 33 were published and 77 were not.
What were problems for articles did not pass review? Bearing in mind
articles can be marked not published for multiple reasons, 23 articles were
not passed for copyright reasons, 103 for newsworthiness issues, 70 for
verifiability issues, 43 for neutrality issues and 96 for style issues.
[image: English Wikinews Review
The different cohorts appear to have different sets of issues. Accredited
reporters did not have any problems with copyright or plagiarism. 57% of
University of Wollongong reviews and 51% of new reporters had not passing
reviews because of newsworthiness concerns. Accredited reporters had
problems with verifiability at 45% versus 26% for University of Wollongong
reporters. 25% of new contributors and 29% of University of Wollongong
reporters had problems with neutrality. 60% of University of Wollongong had
problems complying with the style guide compared to 33% of regular
Some of these patterns are explainable. There are times when it is
difficult to get reviewers, and an article may languish for 24 to 48 hours.
By that time, the article is no longer news or requires more and new
information in order to stay fresh. This does not explain all of it though.
Observational analysis suggests that at least two thirds of these articles
fail newsworthiness because of a lack of a clear focus on the news topic,
writing a news article as the topic itself is set to go stale, taking too
long to address problems with previous reviews or going through the review
process repeatedly to the point where by the time everything else gets
fixed, the story is no longer news.
In many cases, reviewers look at some things before others. Newsworthiness
and style guide compliance are two of the most easily visible problems.
They do not require looking at external links and doing intensive
examination of the text to look for more systemic, underlying problems. The
article does not state when an event happened or makes clear it happened 4
days ago? The article is written using lots of non-relative dating? There
are lots of external links inside the article? The article is clearly not
written in inverted pyramid style? The title lacks a verb? The first
paragraph does not answer who, what, when, how or why this is news? There
is no reason to look beyond newsworthiness and style as these obvious
problems need to be fixed before going forward. Copyright, NPOV and
verification can come later.
For accredited reporters, many of them are accredited because they do
original reporting. This often requires sending things to reviewers via
e-mail or posting extensive notes, pictures, audio, video on the talk page.
Verification is also often one of the last steps in the review process.
Thus, it makes sense that reporters with a track record of success are
likely to get caught up here.
Multiple problems identified on a review decreases the likelihood of
publication. Only 22 failed reviews were present on articles that
subsequently were published. This compares 74 reviews that identified
multiple problems where the article was never published. The only review
for an accredited reporter which identified multiple problems was
subsequently published. Regular contributors had 11 total reviews with
multiple problems, of which 7 of those reviews were done on articles that
were subsequently published. 14 reviews identifying multiple problems for
articles by new contributors went on to be published, with 45 reviews on
articles that were not published. All 24 of the UoW student articles with
reviews indicating multiple problems failed to reach a published state. The
percentages form an almost predictable slope based on experience for
chances of an article becoming published at 100% published for accredited
reporters reviewed with multiple issues, 63% for regulars, 23% for new
contributors, and 0% for University of Wollongong students.
On the other hand, there is no significant difference between articles with
only one issue identified being on an article subsequently getting
published. 81 reviews identifying only one problem were on articles that
were not subsequently published versus 24 reviews identifying only one
problem on articles that were subsequently published. This puts a
publishing rate at 22.8% for 1 problem reviews versus 22.9% for 2+ problems
reviews. There are differences in cohort performance when only one problem
is identified: 70% of accredited reporter reviews are on articles
subsequently published, 36% for regulars, 11% for new contributors and 4%
for University of Wollongong reporters. With the exception of the
University of Wollongong cohort, reviews that identify only one problem are
less likely to lead to an article eventually arriving at a published state.
With newsworthiness a major reason for all cohorts as a reason for a failed
review, there are distinct differences in the likelihood of this problem
being overcome based on cohort. Overall, 15% of all articles with
newsworthiness cited as a reason for an article not being published
subsequently becoming published. 40% of accredited reporter reviews
indicating this problem were on articles that eventually became published.
This rate is comparable to regular reporter reviews, with 38% of that
cohort becoming published after a failed review citing newsworthiness. New
reporters have an 11% rate of later publishing. 3% of University of
Wollongong reporters reviews indicating newsworthiness problems reach a
54% of the time when newsworthiness is a problem, a reviewer indicates some
other problem with the article. For articles with multiple problems
including newsworthiness, 93% of the time there is also a style problem,
46% of the time there is a verifiability problem , 44% of the time there is
a point of view problem and 4% of the time there is a copyright problem.
[image: English Wikinews Average
For articles that are not passed on their first attempt, there are
different continuing progress responses for each cohort. For accredited
reporters, they have one failed review before either submitting
successfully on their second attempt or before abandoning their work. This
suggests that accredited reporters are able to successfully respond to
feedback or understand when an article has systemic problems that will
result in it never being published. New and University of Wollongong
reporters are much more likely to continue to try to resubmit multiple
times, both successfully and unsuccessfully, than their regular reporter
counterparts. 4 new reporters out of 46 submitted their work 4 or more
times unsuccessfully. This contrasts to 3 out of 10 for regular reporters
and 4 out of 18 for University of Wollongong students : 8% to 30% to 22%.
High number of submissions for rereview are unlikely to lead to publication
of the article. Of the articles finally published, only one had failed at
review more than three times, United States deportation policies challenged
in Santa Clara County,
which had 6 failed reviews before being published. In this particular case,
the article was failed 3 times for copyright reasons, once for
newsworthiness, 3 times for verifiability, 4 times for neutrality and 2
times for style.
Accredited reporters were the most successful as a percentage of total
articles with failed initial reviews subsequently getting published, with 8
articles published after only 1 failed review. Regular reporters had 15
published articles out of 27 for a 55% success rate, new reporters had 9
published out of 57 for a 15% success rate, and University of Wollongong
students had 1 out of 21 for a success rate of 4%. As reporters become more
acclimatized, they are more likely to translate failed reviews into
successfully published articles.
This confirms observational bias that English Wikinews has a high barrier
of entry in terms of adapting to the local review process. It also confirms
that the feedback system for the review system works for established
contributors who have figured out the basics of preparing an article for a
published state. New reporters and University of Wollongong students have
problems that are similar, but new reporters are either more willing to
work through failure to accomplish a goal or more likely to find community
members who are willing to assist them getting the article over the line.
That the percentage of new and University of Wollongong students getting
not publish ready reviews for style suggests they are unfamiliar with the
style guide. How this can be addressed is difficult because it appears as
if they are not reading the style guide. One thought for increasing the
likelihood of getting an article published is to provide a form of
motivation that will encourage a reporter to keep with it until their work
—though not necessarily a specific article— is published. This may need to
be coupled with an improved feedback system, though how this would work
with a cohort of editors who are unmotivated to read existing materials
designed to increase their chances of getting published or interact with
contributors to seek advice in getting published, calls an over reliance on
an improved feedback system as a primary method to increase chances of
getting published into question.