Everyone,
As some of you may know, we launched an experimental Article Feedback feature as part of the Public Policy Initiative last week. The "Article Feedback Tool" enables readers to quickly assessthe sourcing, completeness, neutrality, and readability of a Wikipedia article on a five-point scale. It is currently deployed on about 300 articles [1] in the area of Public Policy on the English Wikipedia. More details may be found on the blog post [2] as well as the post on Foundation-l [3].
We've been capturing the ratings data and have some early analysis to share around the types of ratings users are providing. There are some interesting differences between anonymous and registered users:
http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Early_Dat...
The dump of the article-level data is also available [4] for those who are interested.
If anyone would like to be involved in the ongoing research and evaluation of this tool, please sign up on the Article Feedback Workgroup page. [5]
Howie
[1] http://en.wikipedia.org/wiki/Category:Article_Feedback_Pilot [2] http://blog.wikimedia.org/blog/2010/09/22/article-feedback-pilot-goes-live/ [3] http://lists.wikimedia.org/pipermail/foundation-l/2010-September/061056.html [4] https://spreadsheets.google.com/ccc?key=0Aikdcg5HdSKbdFRhdUN1Rm1iZzB5dUdMUlY... [5] http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Workgroup
This is really interesting Howie, thanks. I'm particularly fascinated to see the couple of trends emerging that you identified (although the sample size - quantitatively and chronologically is small). That we tend to be our own worst critics consistently rating our articles lower than non-logged-in users seems logical - we hold our work to a high standard. Also, that Wikimedians seem to have a greater consensus about what constitutes "good" according to four metrics. Your "to do" section seems pretty busy too, good luck with that! Especially to compare these ratings with the existing quality metrics of FA/GA/B/C/Start...
I wonder whether you have any plans to produce graphical representations of an article's rating over time? This would let us see not only how an article is rated *now* but how that rating has changed.
-Liam
wittylama.com/blog Peace, love & metadata
On 29 September 2010 23:50, Howie Fung hfung@wikimedia.org wrote:
Everyone,
As some of you may know, we launched an experimental Article Feedback feature as part of the Public Policy Initiative last week. The "Article Feedback Tool" enables readers to quickly assessthe sourcing, completeness, neutrality, and readability of a Wikipedia article on a five-point scale. It is currently deployed on about 300 articles [1] in the area of Public Policy on the English Wikipedia. More details may be found on the blog post [2] as well as the post on Foundation-l [3].
We've been capturing the ratings data and have some early analysis to share around the types of ratings users are providing. There are some interesting differences between anonymous and registered users:
http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Early_Dat...
The dump of the article-level data is also available [4] for those who are interested.
If anyone would like to be involved in the ongoing research and evaluation of this tool, please sign up on the Article Feedback Workgroup page. [5]
Howie
[1] http://en.wikipedia.org/wiki/Category:Article_Feedback_Pilot [2] http://blog.wikimedia.org/blog/2010/09/22/article-feedback-pilot-goes-live/ [3] http://lists.wikimedia.org/pipermail/foundation-l/2010-September/061056.html [4] https://spreadsheets.google.com/ccc?key=0Aikdcg5HdSKbdFRhdUN1Rm1iZzB5dUdMUlY... [5] http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Workgroup
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Yes, the to-do list is very long, but I do hope we can take the time to understand how users use the tool and how the reader feeback affects both reader engagement (e.g., getting readers to read more), reader conversation (e.g., providing an on-ramp into other forms of contribution) and editing (e.g., providing feedback for editors on which areas need attention).
The early data is definitely in line with your comment on holding our work to a high standard. If we continue to see a difference between the anon and registered distributions, we may want to offer the reader the ability to see all vs. registered only (similar to how Rotten Tomatoes has the [All Critics | Top Critics] filter).
We absolutely have plans to produce graphical representations of ratings over time. One of the main uses of the tool is to provide the reader perspective on whether/how much the public policy project improves the quality of articles. Amy Roth will probably be pushing this area of measurement forward, but if you have graphs you'd like to see, please comment on the wiki.
Howie
On 9/30/10 12:25 AM, Liam Wyatt wrote:
This is really interesting Howie, thanks. I'm particularly fascinated to see the couple of trends emerging that you identified (although the sample size - quantitatively and chronologically is small). That we tend to be our own worst critics consistently rating our articles lower than non-logged-in users seems logical - we hold our work to a high standard. Also, that Wikimedians seem to have a greater consensus about what constitutes "good" according to four metrics. Your "to do" section seems pretty busy too, good luck with that! Especially to compare these ratings with the existing quality metrics of FA/GA/B/C/Start...
I wonder whether you have any plans to produce graphical representations of an article's rating over time? This would let us see not only how an article is rated *now* but how that rating has changed.
-Liam
wittylama.com/blog http://wittylama.com/blog Peace, love & metadata
On 29 September 2010 23:50, Howie Fung <hfung@wikimedia.org mailto:hfung@wikimedia.org> wrote:
Everyone, As some of you may know, we launched an experimental Article Feedback feature as part of the Public Policy Initiative last week. The "Article Feedback Tool" enables readers to quickly assessthe sourcing, completeness, neutrality, and readability of a Wikipedia article on a five-point scale. It is currently deployed on about 300 articles [1] in the area of Public Policy on the English Wikipedia. More details may be found on the blog post [2] as well as the post on Foundation-l [3]. We've been capturing the ratings data and have some early analysis to share around the types of ratings users are providing. There are some interesting differences between anonymous and registered users: http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Early_Data The dump of the article-level data is also available [4] for those who are interested. If anyone would like to be involved in the ongoing research and evaluation of this tool, please sign up on the Article Feedback Workgroup page. [5] Howie [1] http://en.wikipedia.org/wiki/Category:Article_Feedback_Pilot [2] http://blog.wikimedia.org/blog/2010/09/22/article-feedback-pilot-goes-live/ [3] http://lists.wikimedia.org/pipermail/foundation-l/2010-September/061056.html [4] https://spreadsheets.google.com/ccc?key=0Aikdcg5HdSKbdFRhdUN1Rm1iZzB5dUdMUlY4YzAwNmc&hl=en#gid=0 <https://spreadsheets.google.com/ccc?key=0Aikdcg5HdSKbdFRhdUN1Rm1iZzB5dUdMUlY4YzAwNmc&hl=en#gid=0> [5] http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Workgroup _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Just out of interest, if the maximum value is 5, and (for example) the "neutral" mean is 3.6, how can the standard deviation be 1.82? Wouldn't that allow values up to 5.42?
If that's an effect of extreme skewing, maybe the median would be better suited to give a "common" value?
Cheers, Magnus
On Thu, Sep 30, 2010 at 12:50 AM, Howie Fung hfung@wikimedia.org wrote:
Everyone,
As some of you may know, we launched an experimental Article Feedback feature as part of the Public Policy Initiative last week. The "Article Feedback Tool" enables readers to quickly assessthe sourcing, completeness, neutrality, and readability of a Wikipedia article on a five-point scale. It is currently deployed on about 300 articles [1] in the area of Public Policy on the English Wikipedia. More details may be found on the blog post [2] as well as the post on Foundation-l [3].
We've been capturing the ratings data and have some early analysis to share around the types of ratings users are providing. There are some interesting differences between anonymous and registered users:
http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Early_Dat...
The dump of the article-level data is also available [4] for those who are interested.
If anyone would like to be involved in the ongoing research and evaluation of this tool, please sign up on the Article Feedback Workgroup page. [5]
Howie
[1] http://en.wikipedia.org/wiki/Category:Article_Feedback_Pilot [2] http://blog.wikimedia.org/blog/2010/09/22/article-feedback-pilot-goes-live/ [3] http://lists.wikimedia.org/pipermail/foundation-l/2010-September/061056.html [4] https://spreadsheets.google.com/ccc?key=0Aikdcg5HdSKbdFRhdUN1Rm1iZzB5dUdMUlY... [5] http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Workgroup
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
On Thu, 30 Sep 2010, Magnus Manske wrote:
Just out of interest, if the maximum value is 5, and (for example) the "neutral" mean is 3.6, how can the standard deviation be 1.82? Wouldn't that allow values up to 5.42?
If that's an effect of extreme skewing, maybe the median would be better suited to give a "common" value?
A data set [ 5, 5, 4, 3, 1 ] will have mean 3.6 and standard deviation 1.67. This is also above 5. There is skewness here. If you use the median (here 4) you wont get any decimals. I think the mean is fine.
cheers Finn
___________________________________________________________________
Finn Aarup Nielsen, DTU Informatics, Denmark Lundbeck Foundation Center for Integrated Molecular Brain Imaging http://www.imm.dtu.dk/~fn/ http://nru.dk/staff/fnielsen/ ___________________________________________________________________
Cheers, Magnus
On Thu, Sep 30, 2010 at 12:50 AM, Howie Fung hfung@wikimedia.org wrote:
Everyone,
As some of you may know, we launched an experimental Article Feedback feature as part of the Public Policy Initiative last week. The "Article Feedback Tool" enables readers to quickly assessthe sourcing, completeness, neutrality, and readability of a Wikipedia article on a five-point scale. It is currently deployed on about 300 articles [1] in the area of Public Policy on the English Wikipedia. More details may be found on the blog post [2] as well as the post on Foundation-l [3].
We've been capturing the ratings data and have some early analysis to share around the types of ratings users are providing. There are some interesting differences between anonymous and registered users:
http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Early_Dat...
The dump of the article-level data is also available [4] for those who are interested.
If anyone would like to be involved in the ongoing research and evaluation of this tool, please sign up on the Article Feedback Workgroup page. [5]
Howie
[1] http://en.wikipedia.org/wiki/Category:Article_Feedback_Pilot [2] http://blog.wikimedia.org/blog/2010/09/22/article-feedback-pilot-goes-live/ [3] http://lists.wikimedia.org/pipermail/foundation-l/2010-September/061056.html [4] https://spreadsheets.google.com/ccc?key=0Aikdcg5HdSKbdFRhdUN1Rm1iZzB5dUdMUlY... [5] http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Workgroup
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
In this case, looking at simply the standard deviations is actually a little misleading. I've posted the actual distributions to the page, which provides a different picture than the standard deviations would suggest. There is a definite skew in the distribution of the data, especially for anonymous users. Anonymous users are much more likely to give 5s while ratings from registered users appear to be more distributed.
http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Early_Dat...
Howie
On 9/30/10 1:54 AM, Finn Aarup Nielsen wrote:
On Thu, 30 Sep 2010, Magnus Manske wrote:
Just out of interest, if the maximum value is 5, and (for example) the "neutral" mean is 3.6, how can the standard deviation be 1.82? Wouldn't that allow values up to 5.42?
If that's an effect of extreme skewing, maybe the median would be better suited to give a "common" value?
A data set [ 5, 5, 4, 3, 1 ] will have mean 3.6 and standard deviation 1.67. This is also above 5. There is skewness here. If you use the median (here 4) you wont get any decimals. I think the mean is fine.
cheers Finn
Finn Aarup Nielsen, DTU Informatics, DenmarkLundbeck Foundation Center for Integrated Molecular Brain Imaging http://www.imm.dtu.dk/~fn/ http://nru.dk/staff/fnielsen/ ___________________________________________________________________
Cheers, Magnus
On Thu, Sep 30, 2010 at 12:50 AM, Howie Fung hfung@wikimedia.org wrote:
Everyone,
As some of you may know, we launched an experimental Article Feedback feature as part of the Public Policy Initiative last week. The "Article Feedback Tool" enables readers to quickly assessthe sourcing, completeness, neutrality, and readability of a Wikipedia article on a five-point scale. It is currently deployed on about 300 articles [1] in the area of Public Policy on the English Wikipedia. More details may be found on the blog post [2] as well as the post on Foundation-l [3].
We've been capturing the ratings data and have some early analysis to share around the types of ratings users are providing. There are some interesting differences between anonymous and registered users:
http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Early_Dat...
The dump of the article-level data is also available [4] for those who are interested.
If anyone would like to be involved in the ongoing research and evaluation of this tool, please sign up on the Article Feedback Workgroup page. [5]
Howie
[1] http://en.wikipedia.org/wiki/Category:Article_Feedback_Pilot [2] http://blog.wikimedia.org/blog/2010/09/22/article-feedback-pilot-goes-live/
[3] http://lists.wikimedia.org/pipermail/foundation-l/2010-September/061056.html
[4] https://spreadsheets.google.com/ccc?key=0Aikdcg5HdSKbdFRhdUN1Rm1iZzB5dUdMUlY...
[5] http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Workgroup
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
wiki-research-l@lists.wikimedia.org