On Wed, 26 Oct 2011 15:33:30 +0100, WereSpielChequers <werespielchequers(a)gmail.com> wrote: properly from actually pedia. from And I think this time showing it to the Research Committee prior to running the tests would be a good idea. Cheers Yaroslav

Hi WereSpielChequers, I worked on the data analysis for previous AFT versions and I believe I've already answered on a number of occasions your questions as to what we could test and what we couldn't in the previous phase, but I am happy to do this here and clarify what the research plans for the next version are. Subjective ratings We have definitely seen a lot of love/hate rating happen in the case of popular articles (e.g. Lady Gaga, Justin Bieber). Teasing apart ratings on the quality of the article and rater attitudes towards the topic of the article is pretty hard given the fact that an average enwiki article gets a very small number of ratings per day and articles that get a sufficient number of ratings tend to be attracting particularly opinionated or polarized visitors. To give you a measure of the problem: of the 3.7M articles in the main namespace of the English Wikipedia only 40 articles (0.001%) obtain 10 or more ratings per day. The vast majority of articles don't get any rating for several days or weeks or ever. FInding ways to increase the volume of ratings per article is one of the issues we're discussing in the context of v.5. The second problem is that we don't have enough observations on multiple ratings by the same user. Only 0.02% of unique raters rate more than one article and that means that on a single article basis we cannot easily filter out users who only rated a topic they love or hate and still have enough good data to process. This is unfortunate: the more rating data we can get per rater, the more we can identify gaming or rating biases and control them in public article feedback reports. Effects of AFT on participations I ran a number of pre/post analyses comparing editing activity before and after AFT was activated on a random sample of English Wikipedia articles, controlling for page views before and after the activation and found no statistically significant difference in the volume of edits. As I noted elsewhere the comparison between two random samples of articles is problematic because we cannot easily control for the multiple factors that affect editing activity in independent samples of articles so any result you may get out of this coarse analysis would be questionable. I agree that's a very important issue and the proper way to address it is by a/b testing different AFT interfaces (including no AFT widget whatsoever) for the same article and measuring the effects on edit activity for the same articles across different user groups: this is one of the plans we are considering for v.5 Another important limitation of AFT v.4 is that we only collected aggregate event counts for call to actions and we didn't mark edits or new accounts created via AFT, which means that we couldn't directly study the effects of AFT as an on-ramping tool for new editors (e.g. how many readers it is converting to registered users and what is the quality of edits generated via the AFT. i.e. how many users who create an account via AFT call to actions actually end up becoming editors? What is their survival compared to users who create an account in a standard way? And how many among the edits created via AFT are vandalism? How many are good faith tests that get reverted? These are all questions that we will be addressing as of v.5. We'll be still working on analyzing the current AFT data to support the design of v.5. In particular, we will be focusing on (1) correlations between consistent low ratings and poor quality or vandalism or the likelihood of an article to be nominated for deletion and (2) the relation between ratings and changes in other quality-related metrics on a per-article basis. I have also pitched the existing data to a number of external researchers interesting in article quality measurements and/or rating systems and I invite you to do the same. Hope this helps. I look forward to a more in-depth discussion during the office hours. Dario On Oct 26, 2011, at 7:33 AM, WereSpielChequers wrote: