Dear all,
I always thought that the role of the Rcom would be not only managing the incoming requests for research (in a broad sense), but also identifying what research results we would like to have in order to facilitate the future development of the WMF projects.
Recently, AFT has been deployed on every article in en.wp, which caused some controversy. Users are unsure on what the rankings actually mean (for instance, if the readers actually answer the questions they are asked, or instead give rankings based on their perception of the subject; whether it would be a good idea to invite the readers also to provide text input, and a number of other issues).
To me it looks like a good test case. It would be good for us to have some research results on the issue, and so far (from what I know) nobody volunteered to perform such research. Should WMF then indicate that they want these results? Probably at this stage we are not prepared to order it (I mean to pay for it), but it could be broadly advertised in certain places.
Just to make sure, what I asked is not one but two separate questions: 1) Do we (badly) want the research on AFT?
Even if the answer is no (not needed, too early, unrealistic etc), it makes sense in my opinion to discuss another question:
2) Is is appropriate and/or meaningful for us as Rcom to compile a priority list of needed research topics and try to advertise the necessity and urgency of research carried out on these topics?
Note that even if the answer to 1 is yes the answer to 2 can still be no (as a matter of principle).
Cheers Yaroslav
Hi Yaroslav,
Yes it was certainly my hope that one of the roles of RCom would be to identify issues in the community where independent research would be really useful.
As for the specific issue, I've raised concerns about AFT on more than one occasion, and I've had assurances that they will keep an eye on whether it is doing more harm than good. Though my preferred option of measuring the 100,000 test articles against a control sample of 100,000 and seeing which articles received more improvement doesn't seem to have happened.
AFT is an interesting test of several of what some of us believe to be cornerstones of the wiki; Firstly accountability, the theory that people with registered accounts edit positively in order to conserve a reputation that they acquire through having their edits logged to them. So it isn't just the practical concern that racist or homophobic rating of articles can't be dealt with as easily as blocking editors who make racist or homophobic edits, there is the theoretical possibility that anonymous ratings would give otherwise goodfaith editors a chance to vent their spleen against editors they've clashed with by giving poor ratings to their articles. Now this wouldn't make much difference if we were only displaying ratings where dozens of different IP addresses had rated the article, but for most of our articles hoping for a dozen ratings would be optimistic.
Secondly, and in my view far more seriously, if like me you buy the idea that the decline of "sofixit" and the rise of the templating culture in the last four years is a major cause of the community peaking and starting to decline, then such a major attempt to further shift people from improving articles to merely critiquing them is very unhelpful.
Thirdly and much more variably by language, we need to remember that for many of our editors a significant motive for editing Wikipedia is the opportunity to practice and improve a language they are learning. I suspect this will be a bigger issue for the English language wiki than for most others. Assuming that AFT will exacerbate the trend of shifting people from fixing things to critiquing them, then it would be worthwhile getting some research to see how much this demotivates those of our editors who are not native speakers. If you contribute something and someone else fixes typos and grammar you are liable to learn something, if they just rate it as poorly written you are left wondering why.
Of course it's possible that the extra editors this brings in will offset the damage, at least for a short period before people get blasé with the rating template and start to tune it out.
So this could be a great opportunity for an independent researcher to test and prove or disprove the criticisms.
WereSpielChequers
On 30 July 2011 08:01, Yaroslav M. Blanter putevod@mccme.ru wrote:
Dear all,
I always thought that the role of the Rcom would be not only managing the incoming requests for research (in a broad sense), but also identifying what research results we would like to have in order to facilitate the future development of the WMF projects.
Recently, AFT has been deployed on every article in en.wp, which caused some controversy. Users are unsure on what the rankings actually mean (for instance, if the readers actually answer the questions they are asked, or instead give rankings based on their perception of the subject; whether it would be a good idea to invite the readers also to provide text input, and a number of other issues).
To me it looks like a good test case. It would be good for us to have some research results on the issue, and so far (from what I know) nobody volunteered to perform such research. Should WMF then indicate that they want these results? Probably at this stage we are not prepared to order it (I mean to pay for it), but it could be broadly advertised in certain places.
Just to make sure, what I asked is not one but two separate questions:
- Do we (badly) want the research on AFT?
Even if the answer is no (not needed, too early, unrealistic etc), it makes sense in my opinion to discuss another question:
- Is is appropriate and/or meaningful for us as Rcom to compile a
priority list of needed research topics and try to advertise the necessity and urgency of research carried out on these topics?
Note that even if the answer to 1 is yes the answer to 2 can still be no (as a matter of principle).
Cheers Yaroslav
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
Yaroslav, WSC,
as you may know we have a bunch of analyses already in the pipeline regarding AFT with a focus on quality and engagement, including the one WSC suggested here and previously on mw.org. Previous results are extensively documented on the AFT research page [1] and I created a placeholder on Meta [2] (which actually still has to be filled out) to make it easier for the community and for researchers to learn more about what we are studying.
As for promoting this project among external researchers, the simple answer is: be bold :) I've previously advertised the availability of anonymized AFT data [3] (both as static dumps and real time data on the toolserver) on wiki-research-l and I circulated the announcement among my own research contacts. You should feel free to do the same: the more brains we have to crunch our data the better. There is no need of a formal RCom approval and I think the purpose of the committee is precisely to be proactive and connect with the research community as we see fit. If you have any question or have research contacts who would like to know more about the data, do not hesitate to point them to me.
Dario
[1] http://www.mediawiki.org/wiki/Article_feedback/Research [2] http://meta.wikimedia.org/wiki/Research:Article_feedback [3] http://www.mediawiki.org/wiki/Article_feedback/Data
On Jul 30, 2011, at 2:42 PM, WereSpielChequers wrote:
Hi Yaroslav,
Yes it was certainly my hope that one of the roles of RCom would be to identify issues in the community where independent research would be really useful.
As for the specific issue, I've raised concerns about AFT on more than one occasion, and I've had assurances that they will keep an eye on whether it is doing more harm than good. Though my preferred option of measuring the 100,000 test articles against a control sample of 100,000 and seeing which articles received more improvement doesn't seem to have happened.
AFT is an interesting test of several of what some of us believe to be cornerstones of the wiki; Firstly accountability, the theory that people with registered accounts edit positively in order to conserve a reputation that they acquire through having their edits logged to them. So it isn't just the practical concern that racist or homophobic rating of articles can't be dealt with as easily as blocking editors who make racist or homophobic edits, there is the theoretical possibility that anonymous ratings would give otherwise goodfaith editors a chance to vent their spleen against editors they've clashed with by giving poor ratings to their articles. Now this wouldn't make much difference if we were only displaying ratings where dozens of different IP addresses had rated the article, but for most of our articles hoping for a dozen ratings would be optimistic.
Secondly, and in my view far more seriously, if like me you buy the idea that the decline of "sofixit" and the rise of the templating culture in the last four years is a major cause of the community peaking and starting to decline, then such a major attempt to further shift people from improving articles to merely critiquing them is very unhelpful.
Thirdly and much more variably by language, we need to remember that for many of our editors a significant motive for editing Wikipedia is the opportunity to practice and improve a language they are learning. I suspect this will be a bigger issue for the English language wiki than for most others. Assuming that AFT will exacerbate the trend of shifting people from fixing things to critiquing them, then it would be worthwhile getting some research to see how much this demotivates those of our editors who are not native speakers. If you contribute something and someone else fixes typos and grammar you are liable to learn something, if they just rate it as poorly written you are left wondering why.
Of course it's possible that the extra editors this brings in will offset the damage, at least for a short period before people get blasé with the rating template and start to tune it out.
So this could be a great opportunity for an independent researcher to test and prove or disprove the criticisms.
WereSpielChequers
On 30 July 2011 08:01, Yaroslav M. Blanter putevod@mccme.ru wrote:
Dear all,
I always thought that the role of the Rcom would be not only managing the incoming requests for research (in a broad sense), but also identifying what research results we would like to have in order to facilitate the future development of the WMF projects.
Recently, AFT has been deployed on every article in en.wp, which caused some controversy. Users are unsure on what the rankings actually mean (for instance, if the readers actually answer the questions they are asked, or instead give rankings based on their perception of the subject; whether it would be a good idea to invite the readers also to provide text input, and a number of other issues).
To me it looks like a good test case. It would be good for us to have some research results on the issue, and so far (from what I know) nobody volunteered to perform such research. Should WMF then indicate that they want these results? Probably at this stage we are not prepared to order it (I mean to pay for it), but it could be broadly advertised in certain places.
Just to make sure, what I asked is not one but two separate questions:
- Do we (badly) want the research on AFT?
Even if the answer is no (not needed, too early, unrealistic etc), it makes sense in my opinion to discuss another question:
- Is is appropriate and/or meaningful for us as Rcom to compile a
priority list of needed research topics and try to advertise the necessity and urgency of research carried out on these topics?
Note that even if the answer to 1 is yes the answer to 2 can still be no (as a matter of principle).
Cheers Yaroslav
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l
RCom-l mailing list RCom-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l