Given all the bots, both in terms of vandals and in repulsing them, I am not sure the following question even makes sense: but how can we characterize the ratio of productive to unproductive contribution/edits on the English Wikipedia? Has this changed over time? I do have figures from the literature on percentages (and their deltas) for administrator activity, policy edits, time to revert vandalism, etc. The only data point I can find is a single one: the "Bush article had 28,000 revisions, one-third were reverts and, conceivably, another third vandalism" (Spinellis, Louridas 2008).
On 11/06/2009 12:20 PM, Joseph Reagle wrote:
Given all the bots, both in terms of vandals and in repulsing them, I am not sure the following question even makes sense: but how can we characterize the ratio of productive to unproductive contribution/edits on the English Wikipedia? Has this changed over time? I do have figures from the literature on percentages (and their deltas) for administrator activity, policy edits, time to revert vandalism, etc. The only data point I can find is a single one: the "Bush article had 28,000 revisions, one-third were reverts and, conceivably, another third vandalism" (Spinellis, Louridas 2008).
Forgive me for the self-promotion, but I wonder if our paper:
Priedhorsky et al., "Creating, Destroying, and Restoring Value in Wikipedia", GROUP 2007.
might be helpful. We did some analysis on the impact of damage (i.e., vandalism and other damaging edits) and that code could certainly be revived, modernized, and extended. I can put you in touch with our Wikipedia team if you are interested in pursuing that or collaborating.
Reid
Joseph, as someone who doesn't follow every discussion in this thread, I am curious as to how you are defining "productive"? I'm thinking right now that the addition of concepts and ideas cannot always be reliably measured in terms of word count, for example. So how does one measure productivity? Also, what is the "product," of productivity? Is it text? Is it greater understanding?
Said Hamideh President and Co-Founder Mindbounce.com
On Fri, Nov 6, 2009 at 12:26 PM, Reid Priedhorsky reid@umn.edu wrote:
On 11/06/2009 12:20 PM, Joseph Reagle wrote:
Given all the bots, both in terms of vandals and in repulsing them, I am not sure the following question even makes sense: but how can we characterize the ratio of productive to unproductive contribution/edits on the English Wikipedia? Has this changed over time? I do have figures from the literature on percentages (and their deltas) for administrator activity, policy edits, time to revert vandalism, etc. The only data point I can find is a single one: the "Bush article had 28,000 revisions, one-third were reverts and, conceivably, another third vandalism" (Spinellis, Louridas 2008).
Forgive me for the self-promotion, but I wonder if our paper:
Priedhorsky et al., "Creating, Destroying, and Restoring Value in Wikipedia", GROUP 2007.
might be helpful. We did some analysis on the impact of damage (i.e., vandalism and other damaging edits) and that code could certainly be revived, modernized, and extended. I can put you in touch with our Wikipedia team if you are interested in pursuing that or collaborating.
Reid
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
On Friday 06 November 2009, Said Hamideh wrote:
Joseph, as someone who doesn't follow every discussion in this thread, I am curious as to how you are defining "productive"?
Yes, it requires some synthetic variable, of which there are a few in the quant literature. (For example, Priedhorsky et al. defined a damaged article view (DAV).) However, it could be something as simple as what Spinellis and Louridas (2008) imply: a reversion is evidence of an unproductive contribution (ignoring revert wars for the moment), and so what percentage of all edits to WP are reverts? Has this changed over time.
I skipped over the part where you started by talking about bots. Sorry I was just thinking about the beautiful intellectual productivity that arises from revert wars between two human beings, at least for me.
On Fri, Nov 6, 2009 at 12:52 PM, Joseph Reagle reagle@mit.edu wrote:
On Friday 06 November 2009, Said Hamideh wrote:
Joseph, as someone who doesn't follow every discussion in this thread, I
am
curious as to how you are defining "productive"?
Yes, it requires some synthetic variable, of which there are a few in the quant literature. (For example, Priedhorsky et al. defined a damaged article view (DAV).) However, it could be something as simple as what Spinellis and Louridas (2008) imply: a reversion is evidence of an unproductive contribution (ignoring revert wars for the moment), and so what percentage of all edits to WP are reverts? Has this changed over time.
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
If you run WikiTrust on a dump, and you compute the "statistics", this gives you, for each revision, a score between -1 and +1, where:
- -1 = perfectly reverted - 0 = heavily modified - +1 = perfectly preserved
It also tells you how big the change was. So, if you wanted to run this analysis on a dump, it would be failry trivial and fast (say, half a day for the Italian wikipedia? One day for the German one?). We would be happy to provide the details.
Of course, this would just give you a measure of how much of each contribution is kept. It would not tell you much about the intellectual contribution -- your adding a date for an event, will be judges as much as a bot adding a new category to the revision (both 1-word changes).
But at least, it could be a quick start? And more could be built on that?
Luca
On Fri, Nov 6, 2009 at 10:59 AM, Said Hamideh said.hamideh@mindbounce.comwrote:
I skipped over the part where you started by talking about bots. Sorry I was just thinking about the beautiful intellectual productivity that arises from revert wars between two human beings, at least for me.
On Fri, Nov 6, 2009 at 12:52 PM, Joseph Reagle reagle@mit.edu wrote:
On Friday 06 November 2009, Said Hamideh wrote:
Joseph, as someone who doesn't follow every discussion in this thread, I
am
curious as to how you are defining "productive"?
Yes, it requires some synthetic variable, of which there are a few in the quant literature. (For example, Priedhorsky et al. defined a damaged article view (DAV).) However, it could be something as simple as what Spinellis and Louridas (2008) imply: a reversion is evidence of an unproductive contribution (ignoring revert wars for the moment), and so what percentage of all edits to WP are reverts? Has this changed over time.
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
I forgot to say: the WikiTrust analysis is not based on just comparing consecutive revisions. It compares each revision with many subsequent ones, computing an average of how much the revision is kept.
I had put a short writeup on this at http://www.wikitrust.net/computing-wiki-statistics
Luca
On Fri, Nov 6, 2009 at 11:21 AM, Luca de Alfaro luca@dealfaro.org wrote:
If you run WikiTrust on a dump, and you compute the "statistics", this gives you, for each revision, a score between -1 and +1, where:
- -1 = perfectly reverted
- 0 = heavily modified
- +1 = perfectly preserved
It also tells you how big the change was. So, if you wanted to run this analysis on a dump, it would be failry trivial and fast (say, half a day for the Italian wikipedia? One day for the German one?). We would be happy to provide the details.
Of course, this would just give you a measure of how much of each contribution is kept. It would not tell you much about the intellectual contribution -- your adding a date for an event, will be judges as much as a bot adding a new category to the revision (both 1-word changes).
But at least, it could be a quick start? And more could be built on that?
Luca
On Fri, Nov 6, 2009 at 10:59 AM, Said Hamideh <said.hamideh@mindbounce.com
wrote:
I skipped over the part where you started by talking about bots. Sorry I was just thinking about the beautiful intellectual productivity that arises from revert wars between two human beings, at least for me.
On Fri, Nov 6, 2009 at 12:52 PM, Joseph Reagle reagle@mit.edu wrote:
On Friday 06 November 2009, Said Hamideh wrote:
Joseph, as someone who doesn't follow every discussion in this thread,
I am
curious as to how you are defining "productive"?
Yes, it requires some synthetic variable, of which there are a few in the quant literature. (For example, Priedhorsky et al. defined a damaged article view (DAV).) However, it could be something as simple as what Spinellis and Louridas (2008) imply: a reversion is evidence of an unproductive contribution (ignoring revert wars for the moment), and so what percentage of all edits to WP are reverts? Has this changed over time.
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
On Friday 06 November 2009, Luca de Alfaro wrote:
Of course, this would just give you a measure of how much of each contribution is kept. It would not tell you much about the intellectual contribution -- your adding a date for an event, will be judges as much as a bot adding a new category to the revision (both 1-word changes).
I don't think I have the expertise or computational resources to run such an analysis, but if it's convenient, I'd be interested in some results. Basically, I'm just wondering how much of the stuff that comes in is at least in some way useful.
We could give you these results for the Italian, Polish, German, or Portuguese Wikipedias. For the English ones, you need to wait a little bit longer (we need to do one more run, probably a month). The results are all in a standard format, so once you write a script to look at one, you can look at them all. Which ones are of interest to you? We should actually put them on a web page.
Luca
On Fri, Nov 6, 2009 at 11:46 AM, Joseph Reagle reagle@mit.edu wrote:
On Friday 06 November 2009, Luca de Alfaro wrote:
Of course, this would just give you a measure of how much of each contribution is kept. It would not tell you much about the intellectual contribution -- your adding a date for an event, will be judges as much
as a
bot adding a new category to the revision (both 1-word changes).
I don't think I have the expertise or computational resources to run such an analysis, but if it's convenient, I'd be interested in some results. Basically, I'm just wondering how much of the stuff that comes in is at least in some way useful.
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Should we make these results available in a web page, where people could download them? Would there be wider interest?
Luca
---------- Forwarded message ----------
We could give you these results for the Italian, Polish, German, or Portuguese Wikipedias. For the English ones, you need to wait a little bit longer (we need to do one more run, probably a month). The results are all in a standard format, so once you write a script to look at one, you can look at them all. Which ones are of interest to you? We should actually put them on a web page.
Luca
On Fri, Nov 6, 2009 at 11:46 AM, Joseph Reagle reagle@mit.edu wrote:
On Friday 06 November 2009, Luca de Alfaro wrote:
Of course, this would just give you a measure of how much of each contribution is kept. It would not tell you much about the intellectual contribution -- your adding a date for an event, will be judges as much
as a
bot adding a new category to the revision (both 1-word changes).
I don't think I have the expertise or computational resources to run such an analysis, but if it's convenient, I'd be interested in some results. Basically, I'm just wondering how much of the stuff that comes in is at least in some way useful.
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Should we make these results available in a web page, where people could download them? Would there be wider interest?
yes please!
Dario
-- Dario Taraborelli
Research Fellow Centre for Research in Social Simulation University of Surrey, Guildford GU2 7XH United Kingdom http://cress.soc.surrey.ac.uk http://nitens.org/taraborelli
Ok, I will try to have this up by Monday. Luca
On Fri, Nov 6, 2009 at 12:15 PM, Dario Taraborelli < dario.taraborelli@gmail.com> wrote:
Should we make these results available in a web page, where people could download them? Would there be wider interest?
yes please!
Dario
-- Dario Taraborelli
Research Fellow Centre for Research in Social Simulation University of Surrey, Guildford GU2 7XH United Kingdom http://cress.soc.surrey.ac.uk http://nitens.org/taraborelli
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Luca de Alfaro wrote:
Should we make these results available in a web page, where people could download them? Would there be wider interest?
Public datasets for further analysis are always nice :)
On Friday 06 November 2009, Reid Priedhorsky wrote:
Forgive me for the self-promotion, but I wonder if our paper:
Not at all -- that's why I asked! :)
Priedhorsky et al., "Creating, Destroying, and Restoring Value in Wikipedia", GROUP 2007.
In my notes I have that you estimated that there was a .0037 probability of seeing a "damaged" page, and this is close to my question, but not quite, since I'm asking about the activity that goes into the page rather than its state at any given moment in time. But feel free to correct me!
This isn't a research question for me, I'm just curious and it tends to come up when people ask me about WP.
Joseph Reagle wrote:
Given all the bots, both in terms of vandals and in repulsing them, I am not sure the following question even makes sense: but how can we characterize the ratio of productive to unproductive contribution/edits on the English Wikipedia? Has this changed over time? I do have figures from the literature on percentages (and their deltas) for administrator activity, policy edits, time to revert vandalism, etc. The only data point I can find is a single one: the "Bush article had 28,000 revisions, one-third were reverts and, conceivably, another third vandalism" (Spinellis, Louridas 2008).
There are some things touching on this in my recent WikiSym paper[1] that you may wish to consider. Specifically, we define a metric for measuring discarded work that is a bit more subtle than just reverts, cast in terms of whether or not the community of editors around an article has "accepted" a revision (and a revert itself can be accepted). While not the primary result, we include charts of the proportion of revisions which are accepted. The metric should be applicable for a more rigorous study of unproductive vs. productive work.
- Michael
1. http://www.wikisym.org/ws2009/procfiles/p104-ekstrand.pdf
On Friday 06 November 2009, Michael Ekstrand wrote:
While not the primary result, we include charts of the proportion of revisions which are accepted. The metric should be applicable for a more rigorous study of unproductive vs. productive work.
Since figure 15 seems to have a lower bound at 0.82, would it be safe to a further that at least 82% of contributions are productive (i.e., accepted by the community/k-acceptance)?
Joseph Reagle wrote:
On Friday 06 November 2009, Michael Ekstrand wrote:
While not the primary result, we include charts of the proportion of revisions which are accepted. The metric should be applicable for a more rigorous study of unproductive vs. productive work.
Since figure 15 seems to have a lower bound at 0.82, would it be safe to a further that at least 82% of contributions are productive (i.e., accepted by the community/k-acceptance)?
Short answer: probably, but that's based on a number of assumptions that may or may not hold.
As always, the devil is in the details, but your statement does seem to be likely to be true assuming that editors will continue to accept articles at roughly the same rate as they did up to January 2008. We do not know right now how acceptance behavior changes with time, though, so that assumption may or may not be true. Also, there are cases where which k-acceptance will incorrectly classify. Some of the false positives are discussed in the paper. It is also possible for an edit to be reverted and then re-made in a possibly altered form later, in which case it will be detected as rejected. In using k-acceptance as a metric for productivity, this gets more complicated as indirect influence may be a factor as well (e.g. when a rejected edit inspires a later accepted edit; there is an example of this in [Krip07]).
The other confounding factor is edits to very-low-traffic articles. Edits languishing in stub articles that no one edits don't have an opportunity to be accepted by this metric (since acceptance depends on other editors editing the article without reverting the change in question). Those edits factor into the "undefined" classification in our metric; I do not know off-hand how many revisions are affected by this, or what the net impact of this effect is.
It should be possible to resolve these with a careful analysis of the edge cases (investigating the impact of stub and other short-history articles and possibly doing human-coding of other cases to see how often k-acceptance misses what you're looking for in "productivity"). So I think that this metric can be a starting point for what you're getting at, but more work is needed to validate the assumptions in order to make statements like that with confidence. Fortunately, most of the adjustments made by resolving these assumptions should increase the percentage of edits classified as productive.
- Michael
Let me attempt to broaden and slightly critique the version of productivity this thread has been considering so far. It seems somewhat limiting to discuss productivity solely in terms of what content makes it into an article. For one, this assumes that articles are somehow independent -- text that is reverted from one article will not reappear in another article. For some articles this is an ok assumption, while for others its way off. We've observed editors who push their viewpoint across a wide set of thematically related articles.
This way of considering productivity also does not account for how seemingly unproductive edits may impact the editing process (meta impact if you will). As one example, a controversial edit that is eventually reverted may generate insightful discussion on the talk page. This discussion can impact future edits (and not just on the same article). To be more concrete, highly controversial pages (e.g. Jesus) maintain oodles of archives of discussions (many of which originated with controversial edits). Newcomers are socialized into the practices surrounding an article by being pointed to these discussions whenever they bring up topics that have been extensively discussed in the past. I would consider edits that generate such discussions to be productive.
A reverted edit on an article page may be productive in other ways as well. For example, it may spark discussion that leads to edits to policy pages. And I'm completely at a loss at how to evaluate productivity of edits in the WP namespace. Word count may be an ok proxy for completeness of an article on a topic, but it is clearly the wrong productivity metric for things like policy pages.
thanks, ivan.
Joseph Reagle wrote:
Given all the bots, both in terms of vandals and in repulsing them, I am not sure the following question even makes sense: but how can we characterize the ratio of productive to unproductive contribution/edits on the English Wikipedia? Has this changed over time? I do have figures from the literature on percentages (and their deltas) for administrator activity, policy edits, time to revert vandalism, etc. The only data point I can find is a single one: the "Bush article had 28,000 revisions, one-third were reverts and, conceivably, another third vandalism" (Spinellis, Louridas 2008).
While the definition and discussion of "(un)productivity" are fascinating, what about a simpler question:
How many editors are active?
A major issue is to decide on a time frame. Do we mean activity within 1 hour? 1 day? 1 week? 1 month?
It should be possible to develop this little tool to provide information on this metrics: http://en.wikipedia.org/wiki/User_talk:Piotrus/Archive_30#RC_active_users It would be nice if we had a tool that would be running this on a constant basis (and preferably log past results in a downloadable form); it seems like a nice addition to all those wikirages, dashboards and such :)
Or those with appropriate capabilities could just run a database dump analysis for historical patterns (I am sure it was done in the past, but the advantage of a real time monitoring tool versus results in published work (usually older than a year) should be pretty obvious).
Some results from the time I run this tool in July are reported here http://en.wikipedia.org/wiki/Wikipedia:EDITORS#Demographics
But it would be nice to have better numbers.
Activity, of course, is a larger subset of productivity, as it includes vandals - but it should also be easier to measure. And perhaps by analyzing a sample of active editors and establishing a number of obvious vandals*, we can find out the numbers of productive editors, using the simple arithmetic of
active - vandals = productive
*regarding counting only "obvious vandalism" for vandalism: as others pointed out here, anything more complex that obvious vandalism may in fact be productive. Of course, this is erring on the side of productivity (just like discarding all revert edits is erring of the side of unproductivity - WP:BRD and so on...).
--- El dom, 8/11/09, Piotr Konieczny piokon@post.pl escribió:
De: Piotr Konieczny piokon@post.pl Asunto: Re: [Wiki-research-l] How many contributions are unproductive? Para: "Research into Wikimedia content and communities" wiki-research-l@lists.wikimedia.org Fecha: domingo, 8 de noviembre, 2009 20:39
While the definition and discussion of "(un)productivity" are fascinating, what about a simpler question:
How many editors are active?
A major issue is to decide on a time frame. Do we mean activity within 1 hour? 1 day? 1 week? 1 month?
Or those with appropriate capabilities could just run a database dump analysis for historical patterns (I am sure it was done in the past, but the advantage of a real time monitoring tool versus results in published work (usually older than a year) should be pretty obvious).
Some results from the time I run this tool in July are reported here http://en.wikipedia.org/wiki/Wikipedia:EDITORS#Demographics
But it would be nice to have better numbers.
My apologies for self-promotion.
On my dissertation you can find a survival analysis for the top 10 language versions answering that question with a formal statistical approach.
http://libresoft.es/Members/jfelipe/thesis-wkp-quantanalysis
For updated numbers, we are waiting to publish new work but, in the mean time, suffice to say that all Wikipedias are losing contributors rapidly. For example, EN Wikipedia is losing "active editors" at a rate of about 15.000 contributors per month (I presented the graphs on the last WikiSym for the first time).
All the same, in my opinion these are 2 very different questions. This one has to do with community size and editorial effort. Joseph's question is related to the "acceptance" of contributions.
Best, F --
-- Piotr Konieczny
"The problem about Wikipedia is, that it just works in reality, not in theory."
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
On Fri, Nov 6, 2009 at 10:20 AM, Joseph Reagle reagle@mit.edu wrote:
Given all the bots, both in terms of vandals and in repulsing them, I am not sure the following question even makes sense: but how can we characterize the ratio of productive to unproductive contribution/edits on the English Wikipedia? Has this changed over time? I do have figures from the literature on percentages (and their deltas) for administrator activity, policy edits, time to revert vandalism, etc. The only data point I can find is a single one: the "Bush article had 28,000 revisions, one-third were reverts and, conceivably, another third vandalism" (Spinellis, Louridas 2008).
It doesn't answer your question that I can see, but there is http://en.wikipedia.org/wiki/Wikipedia:WikiProject_Vandalism_studies :)
There's a few factors making this complicated:
* simple vandalism/revert interactions define the vast majority of reverts (rather than disputes over content, etc); so is the "revert" itself productive or unproductive? People who do RC patrol certainly think they are doing productive work, but nothing is added to an article, so there's nothing to measure in any conventional sense. * as time and technology progresses, bots and [[Special:AbuseFilter]] are doing far more of the simple reversion work that used to be done by hand pre-2006 or so; so a measurement based on bot edits won't give you any sort of accurate ratio over time. But it could give you an idea of the current state of affairs. * more philosophically: how much intellectual content has to be added to make an edit productive? Is a bot that just makes minor wiki-formatting changes doing productive work? * Low-edit-history, long-tail articles are problematic, as Michael says; 'acceptance' is more like benign neglect for these articles. Indeed, (again anecdotally) the majority of articles created just languish over the long-term; you might have years between someone placing a cleanup notice and someone else doing anything about it. Should these cleanup edits, if they remove text and start over, be counted as very slow reverts? (see the 2006-era articles in any category in [[Category:Wikipedia backlog]] for some hint of this issue).
-- phoebe
wiki-research-l@lists.wikimedia.org