This is currently published on my user space on Wikinews at http://en.wikinews.org/wiki/User:LauraHale/Wikinews_Review_Analysis and as done in order to determine some of the issues that students are facing with the review process at Wikinews in light of student complaints to their instructor. I am hoping to do a follow up for it at some point in the future to provide more detailed analysis on review processes on project, but thought in the interim it might be of interest to both researchers and people in education.
Wikinews Review Analysis < User:LauraHale http://en.wikinews.org/wiki/User:LauraHale http://en.wikinews.org/wiki/User:LauraHale/Wikinews_Review_Analysis#mw-navigationhttp://en.wikinews.org/wiki/User:LauraHale/Wikinews_Review_Analysis#p-search
One of the greatest challenges in attracting and retaining new contributors to English Wikinews is the project's review process. This process requires the journalist to have a symbiotic relationship with the reviewer as the pair both work towards getting an article published. The review process involves the submitted news article being checked for copyright, newsworthiness, verifiability, neutral point of view and compliance with the style guide. This process is compacted ideally into a window of opportunity of no longer than 24 to 48 hours. It can be extremely challenging for those not used to the style requirements, neutrality requirements, verification requirements and above all doing this sort of writing quickly.
For those from Wikipedia, the review process on English Wikinews is close in scope to English Wikipedia's Good Article criteriahttp://en.wikipedia.org/wiki/WP:GAN only on a more compressed timetable. For those who have been to secondary school or university, it mirrors a teacher giving you an assignment with your grade being based on meeting the criteria stated on a rubric, and failure to pass requires re-submission. If you do not work to the rubric, you just do not pass and on English Wikinews, that means not getting published.
When you review regularly, you begin to notice certain patterns that frequently occur in the review process. Most often, these appear to be a failure to understand what is news or a failure to read and try to write to the style guide specifications. Observational analysis only gets you so far though when trying to determine a problem and how to develop solutions for what appears to be a problem of articles not passing review or people being discouraged from submitting.
From the period between January 1, 2013 and April 12, 2013, 203 failed
reviews were examined to determine which criteria were the biggest stumbling block. The articles reviewed included published articles, non-published articles moved to user space and deleted articles that were not published. For each review, the primary author was assessed as either accredited reporter, regular contributor with 10 or more published articles, new reporter with 9 or fewer articles, or University of Wollongong student. Of the 202 reviews examined, 104 were for articles by new contributors, 47 by University of Wollongong students, 40 by regular contributors and 11 by accredited reporters. Each review looked at also noted if the article finally reached a published state. There were 110 different articles reviewed, of which 33 were published and 77 were not.
What were problems for articles did not pass review? Bearing in mind articles can be marked not published for multiple reasons, 23 articles were not passed for copyright reasons, 103 for newsworthiness issues, 70 for verifiability issues, 43 for neutrality issues and 96 for style issues. [image: English Wikinews Review Issues.png]http://en.wikinews.org/wiki/File:English_Wikinews_Review_Issues.png
The different cohorts appear to have different sets of issues. Accredited reporters did not have any problems with copyright or plagiarism. 57% of University of Wollongong reviews and 51% of new reporters had not passing reviews because of newsworthiness concerns. Accredited reporters had problems with verifiability at 45% versus 26% for University of Wollongong reporters. 25% of new contributors and 29% of University of Wollongong reporters had problems with neutrality. 60% of University of Wollongong had problems complying with the style guide compared to 33% of regular reporters.
Some of these patterns are explainable. There are times when it is difficult to get reviewers, and an article may languish for 24 to 48 hours. By that time, the article is no longer news or requires more and new information in order to stay fresh. This does not explain all of it though. Observational analysis suggests that at least two thirds of these articles fail newsworthiness because of a lack of a clear focus on the news topic, writing a news article as the topic itself is set to go stale, taking too long to address problems with previous reviews or going through the review process repeatedly to the point where by the time everything else gets fixed, the story is no longer news.
In many cases, reviewers look at some things before others. Newsworthiness and style guide compliance are two of the most easily visible problems. They do not require looking at external links and doing intensive examination of the text to look for more systemic, underlying problems. The article does not state when an event happened or makes clear it happened 4 days ago? The article is written using lots of non-relative dating? There are lots of external links inside the article? The article is clearly not written in inverted pyramid style? The title lacks a verb? The first paragraph does not answer who, what, when, how or why this is news? There is no reason to look beyond newsworthiness and style as these obvious problems need to be fixed before going forward. Copyright, NPOV and verification can come later.
For accredited reporters, many of them are accredited because they do original reporting. This often requires sending things to reviewers via e-mail or posting extensive notes, pictures, audio, video on the talk page. Verification is also often one of the last steps in the review process. Thus, it makes sense that reporters with a track record of success are likely to get caught up here.
Multiple problems identified on a review decreases the likelihood of publication. Only 22 failed reviews were present on articles that subsequently were published. This compares 74 reviews that identified multiple problems where the article was never published. The only review for an accredited reporter which identified multiple problems was subsequently published. Regular contributors had 11 total reviews with multiple problems, of which 7 of those reviews were done on articles that were subsequently published. 14 reviews identifying multiple problems for articles by new contributors went on to be published, with 45 reviews on articles that were not published. All 24 of the UoW student articles with reviews indicating multiple problems failed to reach a published state. The percentages form an almost predictable slope based on experience for chances of an article becoming published at 100% published for accredited reporters reviewed with multiple issues, 63% for regulars, 23% for new contributors, and 0% for University of Wollongong students.
On the other hand, there is no significant difference between articles with only one issue identified being on an article subsequently getting published. 81 reviews identifying only one problem were on articles that were not subsequently published versus 24 reviews identifying only one problem on articles that were subsequently published. This puts a publishing rate at 22.8% for 1 problem reviews versus 22.9% for 2+ problems reviews. There are differences in cohort performance when only one problem is identified: 70% of accredited reporter reviews are on articles subsequently published, 36% for regulars, 11% for new contributors and 4% for University of Wollongong reporters. With the exception of the University of Wollongong cohort, reviews that identify only one problem are less likely to lead to an article eventually arriving at a published state.
With newsworthiness a major reason for all cohorts as a reason for a failed review, there are distinct differences in the likelihood of this problem being overcome based on cohort. Overall, 15% of all articles with newsworthiness cited as a reason for an article not being published subsequently becoming published. 40% of accredited reporter reviews indicating this problem were on articles that eventually became published. This rate is comparable to regular reporter reviews, with 38% of that cohort becoming published after a failed review citing newsworthiness. New reporters have an 11% rate of later publishing. 3% of University of Wollongong reporters reviews indicating newsworthiness problems reach a published state.
54% of the time when newsworthiness is a problem, a reviewer indicates some other problem with the article. For articles with multiple problems including newsworthiness, 93% of the time there is also a style problem, 46% of the time there is a verifiability problem , 44% of the time there is a point of view problem and 4% of the time there is a copyright problem. [image: English Wikinews Average Submissions.png]http://en.wikinews.org/wiki/File:English_Wikinews_Average_Submissions.png
For articles that are not passed on their first attempt, there are different continuing progress responses for each cohort. For accredited reporters, they have one failed review before either submitting successfully on their second attempt or before abandoning their work. This suggests that accredited reporters are able to successfully respond to feedback or understand when an article has systemic problems that will result in it never being published. New and University of Wollongong reporters are much more likely to continue to try to resubmit multiple times, both successfully and unsuccessfully, than their regular reporter counterparts. 4 new reporters out of 46 submitted their work 4 or more times unsuccessfully. This contrasts to 3 out of 10 for regular reporters and 4 out of 18 for University of Wollongong students : 8% to 30% to 22%. High number of submissions for rereview are unlikely to lead to publication of the article. Of the articles finally published, only one had failed at review more than three times, United States deportation policies challenged in Santa Clara County, Californiahttp://en.wikinews.org/wiki/United_States_deportation_policies_challenged_in_Santa_Clara_County,_California, which had 6 failed reviews before being published. In this particular case, the article was failed 3 times for copyright reasons, once for newsworthiness, 3 times for verifiability, 4 times for neutrality and 2 times for style.
Accredited reporters were the most successful as a percentage of total articles with failed initial reviews subsequently getting published, with 8 articles published after only 1 failed review. Regular reporters had 15 published articles out of 27 for a 55% success rate, new reporters had 9 published out of 57 for a 15% success rate, and University of Wollongong students had 1 out of 21 for a success rate of 4%. As reporters become more acclimatized, they are more likely to translate failed reviews into successfully published articles.
This confirms observational bias that English Wikinews has a high barrier of entry in terms of adapting to the local review process. It also confirms that the feedback system for the review system works for established contributors who have figured out the basics of preparing an article for a published state. New reporters and University of Wollongong students have problems that are similar, but new reporters are either more willing to work through failure to accomplish a goal or more likely to find community members who are willing to assist them getting the article over the line. That the percentage of new and University of Wollongong students getting not publish ready reviews for style suggests they are unfamiliar with the style guide. How this can be addressed is difficult because it appears as if they are not reading the style guide. One thought for increasing the likelihood of getting an article published is to provide a form of motivation that will encourage a reporter to keep with it until their work —though not necessarily a specific article— is published. This may need to be coupled with an improved feedback system, though how this would work with a cohort of editors who are unmotivated to read existing materials designed to increase their chances of getting published or interact with contributors to seek advice in getting published, calls an over reliance on an improved feedback system as a primary method to increase chances of getting published into question.
This was an awesome read. Thank you for sharing it!
On Tue, Apr 16, 2013 at 4:35 PM, Laura Hale laura@fanhistory.com wrote:
This is currently published on my user space on Wikinews at http://en.wikinews.org/wiki/User:LauraHale/Wikinews_Review_Analysis and as done in order to determine some of the issues that students are facing with the review process at Wikinews in light of student complaints to their instructor. I am hoping to do a follow up for it at some point in the future to provide more detailed analysis on review processes on project, but thought in the interim it might be of interest to both researchers and people in education.
Wikinews Review Analysis < User:LauraHale http://en.wikinews.org/wiki/User:LauraHale http://en.wikinews.org/wiki/User:LauraHale/Wikinews_Review_Analysis#mw-navigationhttp://en.wikinews.org/wiki/User:LauraHale/Wikinews_Review_Analysis#p-search
One of the greatest challenges in attracting and retaining new contributors to English Wikinews is the project's review process. This process requires the journalist to have a symbiotic relationship with the reviewer as the pair both work towards getting an article published. The review process involves the submitted news article being checked for copyright, newsworthiness, verifiability, neutral point of view and compliance with the style guide. This process is compacted ideally into a window of opportunity of no longer than 24 to 48 hours. It can be extremely challenging for those not used to the style requirements, neutrality requirements, verification requirements and above all doing this sort of writing quickly.
For those from Wikipedia, the review process on English Wikinews is close in scope to English Wikipedia's Good Article criteriahttp://en.wikipedia.org/wiki/WP:GAN only on a more compressed timetable. For those who have been to secondary school or university, it mirrors a teacher giving you an assignment with your grade being based on meeting the criteria stated on a rubric, and failure to pass requires re-submission. If you do not work to the rubric, you just do not pass and on English Wikinews, that means not getting published.
When you review regularly, you begin to notice certain patterns that frequently occur in the review process. Most often, these appear to be a failure to understand what is news or a failure to read and try to write to the style guide specifications. Observational analysis only gets you so far though when trying to determine a problem and how to develop solutions for what appears to be a problem of articles not passing review or people being discouraged from submitting.
From the period between January 1, 2013 and April 12, 2013, 203 failed reviews were examined to determine which criteria were the biggest stumbling block. The articles reviewed included published articles, non-published articles moved to user space and deleted articles that were not published. For each review, the primary author was assessed as either accredited reporter, regular contributor with 10 or more published articles, new reporter with 9 or fewer articles, or University of Wollongong student. Of the 202 reviews examined, 104 were for articles by new contributors, 47 by University of Wollongong students, 40 by regular contributors and 11 by accredited reporters. Each review looked at also noted if the article finally reached a published state. There were 110 different articles reviewed, of which 33 were published and 77 were not.
What were problems for articles did not pass review? Bearing in mind articles can be marked not published for multiple reasons, 23 articles were not passed for copyright reasons, 103 for newsworthiness issues, 70 for verifiability issues, 43 for neutrality issues and 96 for style issues. [image: English Wikinews Review Issues.png]http://en.wikinews.org/wiki/File:English_Wikinews_Review_Issues.png
The different cohorts appear to have different sets of issues. Accredited reporters did not have any problems with copyright or plagiarism. 57% of University of Wollongong reviews and 51% of new reporters had not passing reviews because of newsworthiness concerns. Accredited reporters had problems with verifiability at 45% versus 26% for University of Wollongong reporters. 25% of new contributors and 29% of University of Wollongong reporters had problems with neutrality. 60% of University of Wollongong had problems complying with the style guide compared to 33% of regular reporters.
Some of these patterns are explainable. There are times when it is difficult to get reviewers, and an article may languish for 24 to 48 hours. By that time, the article is no longer news or requires more and new information in order to stay fresh. This does not explain all of it though. Observational analysis suggests that at least two thirds of these articles fail newsworthiness because of a lack of a clear focus on the news topic, writing a news article as the topic itself is set to go stale, taking too long to address problems with previous reviews or going through the review process repeatedly to the point where by the time everything else gets fixed, the story is no longer news.
In many cases, reviewers look at some things before others. Newsworthiness and style guide compliance are two of the most easily visible problems. They do not require looking at external links and doing intensive examination of the text to look for more systemic, underlying problems. The article does not state when an event happened or makes clear it happened 4 days ago? The article is written using lots of non-relative dating? There are lots of external links inside the article? The article is clearly not written in inverted pyramid style? The title lacks a verb? The first paragraph does not answer who, what, when, how or why this is news? There is no reason to look beyond newsworthiness and style as these obvious problems need to be fixed before going forward. Copyright, NPOV and verification can come later.
For accredited reporters, many of them are accredited because they do original reporting. This often requires sending things to reviewers via e-mail or posting extensive notes, pictures, audio, video on the talk page. Verification is also often one of the last steps in the review process. Thus, it makes sense that reporters with a track record of success are likely to get caught up here.
Multiple problems identified on a review decreases the likelihood of publication. Only 22 failed reviews were present on articles that subsequently were published. This compares 74 reviews that identified multiple problems where the article was never published. The only review for an accredited reporter which identified multiple problems was subsequently published. Regular contributors had 11 total reviews with multiple problems, of which 7 of those reviews were done on articles that were subsequently published. 14 reviews identifying multiple problems for articles by new contributors went on to be published, with 45 reviews on articles that were not published. All 24 of the UoW student articles with reviews indicating multiple problems failed to reach a published state. The percentages form an almost predictable slope based on experience for chances of an article becoming published at 100% published for accredited reporters reviewed with multiple issues, 63% for regulars, 23% for new contributors, and 0% for University of Wollongong students.
On the other hand, there is no significant difference between articles with only one issue identified being on an article subsequently getting published. 81 reviews identifying only one problem were on articles that were not subsequently published versus 24 reviews identifying only one problem on articles that were subsequently published. This puts a publishing rate at 22.8% for 1 problem reviews versus 22.9% for 2+ problems reviews. There are differences in cohort performance when only one problem is identified: 70% of accredited reporter reviews are on articles subsequently published, 36% for regulars, 11% for new contributors and 4% for University of Wollongong reporters. With the exception of the University of Wollongong cohort, reviews that identify only one problem are less likely to lead to an article eventually arriving at a published state.
With newsworthiness a major reason for all cohorts as a reason for a failed review, there are distinct differences in the likelihood of this problem being overcome based on cohort. Overall, 15% of all articles with newsworthiness cited as a reason for an article not being published subsequently becoming published. 40% of accredited reporter reviews indicating this problem were on articles that eventually became published. This rate is comparable to regular reporter reviews, with 38% of that cohort becoming published after a failed review citing newsworthiness. New reporters have an 11% rate of later publishing. 3% of University of Wollongong reporters reviews indicating newsworthiness problems reach a published state.
54% of the time when newsworthiness is a problem, a reviewer indicates some other problem with the article. For articles with multiple problems including newsworthiness, 93% of the time there is also a style problem, 46% of the time there is a verifiability problem , 44% of the time there is a point of view problem and 4% of the time there is a copyright problem. [image: English Wikinews Average Submissions.png]http://en.wikinews.org/wiki/File:English_Wikinews_Average_Submissions.png
For articles that are not passed on their first attempt, there are different continuing progress responses for each cohort. For accredited reporters, they have one failed review before either submitting successfully on their second attempt or before abandoning their work. This suggests that accredited reporters are able to successfully respond to feedback or understand when an article has systemic problems that will result in it never being published. New and University of Wollongong reporters are much more likely to continue to try to resubmit multiple times, both successfully and unsuccessfully, than their regular reporter counterparts. 4 new reporters out of 46 submitted their work 4 or more times unsuccessfully. This contrasts to 3 out of 10 for regular reporters and 4 out of 18 for University of Wollongong students : 8% to 30% to 22%. High number of submissions for rereview are unlikely to lead to publication of the article. Of the articles finally published, only one had failed at review more than three times, United States deportation policies challenged in Santa Clara County, Californiahttp://en.wikinews.org/wiki/United_States_deportation_policies_challenged_in_Santa_Clara_County,_California, which had 6 failed reviews before being published. In this particular case, the article was failed 3 times for copyright reasons, once for newsworthiness, 3 times for verifiability, 4 times for neutrality and 2 times for style.
Accredited reporters were the most successful as a percentage of total articles with failed initial reviews subsequently getting published, with 8 articles published after only 1 failed review. Regular reporters had 15 published articles out of 27 for a 55% success rate, new reporters had 9 published out of 57 for a 15% success rate, and University of Wollongong students had 1 out of 21 for a success rate of 4%. As reporters become more acclimatized, they are more likely to translate failed reviews into successfully published articles.
This confirms observational bias that English Wikinews has a high barrier of entry in terms of adapting to the local review process. It also confirms that the feedback system for the review system works for established contributors who have figured out the basics of preparing an article for a published state. New reporters and University of Wollongong students have problems that are similar, but new reporters are either more willing to work through failure to accomplish a goal or more likely to find community members who are willing to assist them getting the article over the line. That the percentage of new and University of Wollongong students getting not publish ready reviews for style suggests they are unfamiliar with the style guide. How this can be addressed is difficult because it appears as if they are not reading the style guide. One thought for increasing the likelihood of getting an article published is to provide a form of motivation that will encourage a reporter to keep with it until their work —though not necessarily a specific article— is published. This may need to be coupled with an improved feedback system, though how this would work with a cohort of editors who are unmotivated to read existing materials designed to increase their chances of getting published or interact with contributors to seek advice in getting published, calls an over reliance on an improved feedback system as a primary method to increase chances of getting published into question.
-- twitter: purplepopple blog: ozziesport.com
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
wiki-research-l@lists.wikimedia.org