Hi,
Is there any research about the effect of vandalism in wiki content pages on readers, experienced editors, and new and potential editors?
And of abuse in discussion pages and edit summaries on experienced editors and new and potential editors?
Intuitively and anecdotally one could think of the following: 1. Vandalism in content pages (articles) wastes editors' and patrollers' time. This (probably) doesn't require proof (or does it?). But some people say it also causes some experienced editors to burn out and leave. Is there any data about it, beyond intuition?
2. Does vandalism *measurably* affect the perception of the wikis' reliability? (This may be wildly different in different languages and wikis.)
3. Abusive language on discussion pages and edit summaries affects editors, and may cause them to reduce their editing, to stop editing about certain topics, or to leave the wiki entirely. Is this effect measurable? How does it differ for various groups by gender, age, religion, country, professional and educational background, seniority at the wiki, etc.?
Thanks! :)
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
Hi Amir,
This is one of those areas of research where we really need the annual editor survey. I think it ran once after the 2009/10 Strategy process, and I don't know if the best questions got included.
But the best time to ask editors what prompted them to start editing has to be fairly soon after they started as memories fade. I once went back to my early edits and the edit I remembered starting me editing barely made it into my first 50.
There is a longstanding theory that a lot of new editors start or started to fix some vandalism that they saw, and that this group went into steep decline a decade ago with the rise of Cluebot and other antivandalism tools that work faster than a newbie could. But without an annual survey to ask editors what prompted them to edit you are going to struggle to research this. Of course you could look at the early logged in edits of active/prolific wikipedians, but if it is true that many/most Wikipedians start with some IP edits, the earliest edits of many Wikipedians won't be available.
Abuse one assumes has a differential effect on the targets of abuse, disproportionately women, gays and ethnic minorities. But I'd be inclined to look at stuff targeted at their user and usertalkpages rather than talkpages and edit summaries, though an email survey of former editors would be useful.
My suspicion is that when we revert, block and maybe even revdel or oversight abuse we assume that fixes the problem, and if we want to tackle abuse we need more edit filters to prevent such abuse from going live.
WSC
On Sat, 16 Jan 2021 at 15:16, Amir E. Aharoni amir.aharoni@mail.huji.ac.il wrote:
Hi,
Is there any research about the effect of vandalism in wiki content pages on readers, experienced editors, and new and potential editors?
And of abuse in discussion pages and edit summaries on experienced editors and new and potential editors?
Intuitively and anecdotally one could think of the following:
- Vandalism in content pages (articles) wastes editors' and patrollers'
time. This (probably) doesn't require proof (or does it?). But some people say it also causes some experienced editors to burn out and leave. Is there any data about it, beyond intuition?
- Does vandalism *measurably* affect the perception of the wikis'
reliability? (This may be wildly different in different languages and wikis.)
- Abusive language on discussion pages and edit summaries affects editors,
and may cause them to reduce their editing, to stop editing about certain topics, or to leave the wiki entirely. Is this effect measurable? How does it differ for various groups by gender, age, religion, country, professional and educational background, seniority at the wiki, etc.?
Thanks! :)
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
To WSC's point about the difficulty of detecting such behavior or surveying at a point in which it would still be salient, I'd add that in general we have a large gap in our knowledge about why people choose to stop editing because almost all of our survey mechanisms depend on existing logged-in usage of the wikis. This is a challenge with many other websites too but it's generally easier to find and survey who, for instance, has left Facebook (example http://socialmedia.soc.northwestern.edu/wp-content/uploads/2013/05/CHI2013-FBLL.pdf) by collecting a random sample of people than it is to find and survey someone who was a former editor of Wikipedia. There were surveys that did ask about major barriers to editing (which presumably contribute to burnout) such as the 2012 survey: https://upload.wikimedia.org/wikipedia/commons/8/81/Editor_Survey_2012_-_Wik... (see the editor survey category https://meta.wikimedia.org/wiki/Category:Editor_surveys if you're looking for others)
Some things that come to mind though:
- I suspect very few readers see vandalism in their daily browsing (as a very frequent, long-term reader of English Wikipedia, I have trouble recalling encountering any clear vandalism in the course of normal reading). That said, I do suspect that most people have seen plenty of stories of outlandish vandalism to Wikipedia -- some legitimate but many more about vandalism that literally lasted minutes -- that may lead to lower trust. Whether or not lower trust in Wikipedia leads to lower readership is a separate question though. Jonathan Morgan ran some recent surveys on reader trust and what factors affected it that might be relevant: https://meta.wikimedia.org/wiki/Research:The_role_of_citations_in_how_reader... - Specifically in the context of harassment and gender equity: - Harassment as barrier: https://meta.wikimedia.org/wiki/Gender_equity_report_2018/Barriers_to_equity - Edit summaries in particular as harassment: https://www.elizabethwhittaker.net/wmf-internship (more details https://www.mediawiki.org/wiki/Wikimedia_Research/Showcase#July_2019 ) - Annual Community Insights Reports often have a section on this -- e.g., https://meta.wikimedia.org/wiki/Community_Insights/Community_Insights_2020_R... - 2015 Harassment Survey: https://meta.wikimedia.org/wiki/Research:Harassment_survey_2015 - The body of work around barriers to newcomers might have some good insights too -- e.g., https://www-users.cs.umn.edu/~halfaker/publications/The_Rise_and_Decline/
On Sun, Jan 17, 2021 at 5:44 AM WereSpielChequers < werespielchequers@gmail.com> wrote:
Hi Amir,
This is one of those areas of research where we really need the annual editor survey. I think it ran once after the 2009/10 Strategy process, and I don't know if the best questions got included.
But the best time to ask editors what prompted them to start editing has to be fairly soon after they started as memories fade. I once went back to my early edits and the edit I remembered starting me editing barely made it into my first 50.
There is a longstanding theory that a lot of new editors start or started to fix some vandalism that they saw, and that this group went into steep decline a decade ago with the rise of Cluebot and other antivandalism tools that work faster than a newbie could. But without an annual survey to ask editors what prompted them to edit you are going to struggle to research this. Of course you could look at the early logged in edits of active/prolific wikipedians, but if it is true that many/most Wikipedians start with some IP edits, the earliest edits of many Wikipedians won't be available.
Abuse one assumes has a differential effect on the targets of abuse, disproportionately women, gays and ethnic minorities. But I'd be inclined to look at stuff targeted at their user and usertalkpages rather than talkpages and edit summaries, though an email survey of former editors would be useful.
My suspicion is that when we revert, block and maybe even revdel or oversight abuse we assume that fixes the problem, and if we want to tackle abuse we need more edit filters to prevent such abuse from going live.
WSC
On Sat, 16 Jan 2021 at 15:16, Amir E. Aharoni < amir.aharoni@mail.huji.ac.il> wrote:
Hi,
Is there any research about the effect of vandalism in wiki content pages on readers, experienced editors, and new and potential editors?
And of abuse in discussion pages and edit summaries on experienced
editors
and new and potential editors?
Intuitively and anecdotally one could think of the following:
- Vandalism in content pages (articles) wastes editors' and patrollers'
time. This (probably) doesn't require proof (or does it?). But some
people
say it also causes some experienced editors to burn out and leave. Is
there
any data about it, beyond intuition?
- Does vandalism *measurably* affect the perception of the wikis'
reliability? (This may be wildly different in different languages and wikis.)
- Abusive language on discussion pages and edit summaries affects
editors,
and may cause them to reduce their editing, to stop editing about certain topics, or to leave the wiki entirely. Is this effect measurable? How
does
it differ for various groups by gender, age, religion, country, professional and educational background, seniority at the wiki, etc.?
Thanks! :)
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
See page 7 of Priedhorsky, R., Chen, J., Lam, S. T. K., Panciera, K., Terveen, L., & Riedl, J. (2007, November). Creating, destroying, and restoring value in Wikipedia. In *Proceedings of the 2007 international ACM conference on Supporting group work* (pp. 259-268). http://reidster.net/pubs/group282-priedhorsky.pdf
They discuss the probability of a page view of Wikipedia containing vandalism rising over time. I wanted to replicate this analysis and extend it past 2007 but I never got the chance. I think the methodology is really interesting though.
It doesn't directly answer the question but it does get at the *impact* of vandalism.
On Mon, Jan 18, 2021 at 12:13 PM Isaac Johnson isaac@wikimedia.org wrote:
To WSC's point about the difficulty of detecting such behavior or surveying at a point in which it would still be salient, I'd add that in general we have a large gap in our knowledge about why people choose to stop editing because almost all of our survey mechanisms depend on existing logged-in usage of the wikis. This is a challenge with many other websites too but it's generally easier to find and survey who, for instance, has left Facebook (example < http://socialmedia.soc.northwestern.edu/wp-content/uploads/2013/05/CHI2013-F...
)
by collecting a random sample of people than it is to find and survey someone who was a former editor of Wikipedia. There were surveys that did ask about major barriers to editing (which presumably contribute to burnout) such as the 2012 survey:
https://upload.wikimedia.org/wikipedia/commons/8/81/Editor_Survey_2012_-_Wik... (see the editor survey category https://meta.wikimedia.org/wiki/Category:Editor_surveys if you're looking for others)
Some things that come to mind though:
- I suspect very few readers see vandalism in their daily browsing (as a
very frequent, long-term reader of English Wikipedia, I have trouble recalling encountering any clear vandalism in the course of normal reading). That said, I do suspect that most people have seen plenty of stories of outlandish vandalism to Wikipedia -- some legitimate but many more about vandalism that literally lasted minutes -- that may lead to lower trust. Whether or not lower trust in Wikipedia leads to lower readership is a separate question though. Jonathan Morgan ran some recent surveys on reader trust and what factors affected it that might be relevant:
https://meta.wikimedia.org/wiki/Research:The_role_of_citations_in_how_reader...
- Specifically in the context of harassment and gender equity:
- Harassment as barrier:
https://meta.wikimedia.org/wiki/Gender_equity_report_2018/Barriers_to_equity - Edit summaries in particular as harassment: https://www.elizabethwhittaker.net/wmf-internship (more details < https://www.mediawiki.org/wiki/Wikimedia_Research/Showcase#July_2019%3E ) - Annual Community Insights Reports often have a section on this -- e.g.,
https://meta.wikimedia.org/wiki/Community_Insights/Community_Insights_2020_R... - 2015 Harassment Survey: https://meta.wikimedia.org/wiki/Research:Harassment_survey_2015
- The body of work around barriers to newcomers might have some good
insights too -- e.g.,
https://www-users.cs.umn.edu/~halfaker/publications/The_Rise_and_Decline/
On Sun, Jan 17, 2021 at 5:44 AM WereSpielChequers < werespielchequers@gmail.com> wrote:
Hi Amir,
This is one of those areas of research where we really need the annual editor survey. I think it ran once after the 2009/10 Strategy process,
and
I don't know if the best questions got included.
But the best time to ask editors what prompted them to start editing
has
to be fairly soon after they started as memories fade. I once went back
to
my early edits and the edit I remembered starting me editing barely made
it
into my first 50.
There is a longstanding theory that a lot of new editors start or started to fix some vandalism that they saw, and that this group went into steep decline a decade ago with the rise of Cluebot and other antivandalism
tools
that work faster than a newbie could. But without an annual survey to ask editors what prompted them to edit you are going to struggle to research this. Of course you could look at the early logged in edits of active/prolific wikipedians, but if it is true that many/most Wikipedians start with some IP edits, the earliest edits of many Wikipedians won't be available.
Abuse one assumes has a differential effect on the targets of abuse, disproportionately women, gays and ethnic minorities. But I'd be inclined to look at stuff targeted at their user and usertalkpages rather than talkpages and edit summaries, though an email survey of former editors would be useful.
My suspicion is that when we revert, block and maybe even revdel or oversight abuse we assume that fixes the problem, and if we want to
tackle
abuse we need more edit filters to prevent such abuse from going live.
WSC
On Sat, 16 Jan 2021 at 15:16, Amir E. Aharoni < amir.aharoni@mail.huji.ac.il> wrote:
Hi,
Is there any research about the effect of vandalism in wiki content
pages
on readers, experienced editors, and new and potential editors?
And of abuse in discussion pages and edit summaries on experienced
editors
and new and potential editors?
Intuitively and anecdotally one could think of the following:
- Vandalism in content pages (articles) wastes editors' and
patrollers'
time. This (probably) doesn't require proof (or does it?). But some
people
say it also causes some experienced editors to burn out and leave. Is
there
any data about it, beyond intuition?
- Does vandalism *measurably* affect the perception of the wikis'
reliability? (This may be wildly different in different languages and wikis.)
- Abusive language on discussion pages and edit summaries affects
editors,
and may cause them to reduce their editing, to stop editing about
certain
topics, or to leave the wiki entirely. Is this effect measurable? How
does
it differ for various groups by gender, age, religion, country, professional and educational background, seniority at the wiki, etc.?
Thanks! :)
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Isaac Johnson (he/him/his) -- Research Scientist -- Wikimedia Foundation _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Hi Aaron,
That was an interesting read and a bit of a time capsule. 2002-2006 is a bit before I started editing Wikipedia. Before many of the tools such as huggle that give vandalfighters such an advantage over vandals, I think before the era of bot reversion of vandalism when vandalism had to be reverted by humans rather than computers, and certainly before the edit filters that prevent much, possibly most vandalism from even being saved. It also seems to predate the whole panoply of page protection that stops vandals even editing many common vandalism targets (they do say that every single article is available for anyone to edit).
It would be interesting to see a study now when recent changes patrollers boast of the times they have got to some vandalism faster than Cluebot.
I know there were predictions in the early years that eventually the tidal wave of vandalism would overwhelm the defenders of the wiki, that study seems to have been part of that. I wonder if anyone in 2004 predicted that we would get to the current situation where adolescent vandalism has turned out to be so predictable that dealing with it has been mostly automated and now we are more worried about spam than vandalism.
WSC
On Mon, 18 Jan 2021 at 23:52, Aaron Halfaker aaron.halfaker@gmail.com wrote:
See page 7 of Priedhorsky, R., Chen, J., Lam, S. T. K., Panciera, K., Terveen, L., & Riedl, J. (2007, November). Creating, destroying, and restoring value in Wikipedia. In *Proceedings of the 2007 international ACM conference on Supporting group work* (pp. 259-268). http://reidster.net/pubs/group282-priedhorsky.pdf
They discuss the probability of a page view of Wikipedia containing vandalism rising over time. I wanted to replicate this analysis and extend it past 2007 but I never got the chance. I think the methodology is really interesting though.
It doesn't directly answer the question but it does get at the *impact* of vandalism.
On Mon, Jan 18, 2021 at 12:13 PM Isaac Johnson isaac@wikimedia.org wrote:
To WSC's point about the difficulty of detecting such behavior or
surveying
at a point in which it would still be salient, I'd add that in general we have a large gap in our knowledge about why people choose to stop editing because almost all of our survey mechanisms depend on existing logged-in usage of the wikis. This is a challenge with many other websites too but it's generally easier to find and survey who, for instance, has left Facebook (example <
http://socialmedia.soc.northwestern.edu/wp-content/uploads/2013/05/CHI2013-F...
)
by collecting a random sample of people than it is to find and survey someone who was a former editor of Wikipedia. There were surveys that did ask about major barriers to editing (which presumably contribute to burnout) such as the 2012 survey:
https://upload.wikimedia.org/wikipedia/commons/8/81/Editor_Survey_2012_-_Wik...
(see the editor survey category https://meta.wikimedia.org/wiki/Category:Editor_surveys if you're looking for others)
Some things that come to mind though:
- I suspect very few readers see vandalism in their daily browsing
(as a
very frequent, long-term reader of English Wikipedia, I have trouble recalling encountering any clear vandalism in the course of normal reading). That said, I do suspect that most people have seen plenty of stories of outlandish vandalism to Wikipedia -- some legitimate but
many
more about vandalism that literally lasted minutes -- that may lead to lower trust. Whether or not lower trust in Wikipedia leads to lower readership is a separate question though. Jonathan Morgan ran some recent surveys on reader trust and what factors affected it that might be relevant:
https://meta.wikimedia.org/wiki/Research:The_role_of_citations_in_how_reader...
- Specifically in the context of harassment and gender equity:
- Harassment as barrier:
https://meta.wikimedia.org/wiki/Gender_equity_report_2018/Barriers_to_equity
- Edit summaries in particular as harassment: https://www.elizabethwhittaker.net/wmf-internship (more details <https://www.mediawiki.org/wiki/Wikimedia_Research/Showcase#July_2019%3E ) - Annual Community Insights Reports often have a section on this -- e.g.,
https://meta.wikimedia.org/wiki/Community_Insights/Community_Insights_2020_R...
- 2015 Harassment Survey: https://meta.wikimedia.org/wiki/Research:Harassment_survey_2015
- The body of work around barriers to newcomers might have some good
insights too -- e.g.,
https://www-users.cs.umn.edu/~halfaker/publications/The_Rise_and_Decline/
On Sun, Jan 17, 2021 at 5:44 AM WereSpielChequers < werespielchequers@gmail.com> wrote:
Hi Amir,
This is one of those areas of research where we really need the annual editor survey. I think it ran once after the 2009/10 Strategy process,
and
I don't know if the best questions got included.
But the best time to ask editors what prompted them to start editing
has
to be fairly soon after they started as memories fade. I once went back
to
my early edits and the edit I remembered starting me editing barely
made
it
into my first 50.
There is a longstanding theory that a lot of new editors start or
started
to fix some vandalism that they saw, and that this group went into
steep
decline a decade ago with the rise of Cluebot and other antivandalism
tools
that work faster than a newbie could. But without an annual survey to
ask
editors what prompted them to edit you are going to struggle to
research
this. Of course you could look at the early logged in edits of active/prolific wikipedians, but if it is true that many/most
Wikipedians
start with some IP edits, the earliest edits of many Wikipedians won't
be
available.
Abuse one assumes has a differential effect on the targets of abuse, disproportionately women, gays and ethnic minorities. But I'd be
inclined
to look at stuff targeted at their user and usertalkpages rather than talkpages and edit summaries, though an email survey of former editors would be useful.
My suspicion is that when we revert, block and maybe even revdel or oversight abuse we assume that fixes the problem, and if we want to
tackle
abuse we need more edit filters to prevent such abuse from going live.
WSC
On Sat, 16 Jan 2021 at 15:16, Amir E. Aharoni < amir.aharoni@mail.huji.ac.il> wrote:
Hi,
Is there any research about the effect of vandalism in wiki content
pages
on readers, experienced editors, and new and potential editors?
And of abuse in discussion pages and edit summaries on experienced
editors
and new and potential editors?
Intuitively and anecdotally one could think of the following:
- Vandalism in content pages (articles) wastes editors' and
patrollers'
time. This (probably) doesn't require proof (or does it?). But some
people
say it also causes some experienced editors to burn out and leave. Is
there
any data about it, beyond intuition?
- Does vandalism *measurably* affect the perception of the wikis'
reliability? (This may be wildly different in different languages and wikis.)
- Abusive language on discussion pages and edit summaries affects
editors,
and may cause them to reduce their editing, to stop editing about
certain
topics, or to leave the wiki entirely. Is this effect measurable? How
does
it differ for various groups by gender, age, religion, country, professional and educational background, seniority at the wiki, etc.?
Thanks! :)
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Isaac Johnson (he/him/his) -- Research Scientist -- Wikimedia Foundation _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
+1 WSC. When I thought about replicating it, I expected to see a dramatic decline in the impact of vandalism with the advent of counter-vandalism tools and abuse filter.
It would be interesting to see that on a cross-wiki basis as different wikis employ different strategies (or seemingly none at all) for counter-vandalism over time.
On Tue, Jan 19, 2021 at 10:58 AM WereSpielChequers < werespielchequers@gmail.com> wrote:
Hi Aaron,
That was an interesting read and a bit of a time capsule. 2002-2006 is a bit before I started editing Wikipedia. Before many of the tools such as huggle that give vandalfighters such an advantage over vandals, I think before the era of bot reversion of vandalism when vandalism had to be reverted by humans rather than computers, and certainly before the edit filters that prevent much, possibly most vandalism from even being saved. It also seems to predate the whole panoply of page protection that stops vandals even editing many common vandalism targets (they do say that every single article is available for anyone to edit).
It would be interesting to see a study now when recent changes patrollers boast of the times they have got to some vandalism faster than Cluebot.
I know there were predictions in the early years that eventually the tidal wave of vandalism would overwhelm the defenders of the wiki, that study seems to have been part of that. I wonder if anyone in 2004 predicted that we would get to the current situation where adolescent vandalism has turned out to be so predictable that dealing with it has been mostly automated and now we are more worried about spam than vandalism.
WSC
On Mon, 18 Jan 2021 at 23:52, Aaron Halfaker aaron.halfaker@gmail.com wrote:
See page 7 of Priedhorsky, R., Chen, J., Lam, S. T. K., Panciera, K., Terveen, L., & Riedl, J. (2007, November). Creating, destroying, and restoring value in Wikipedia. In *Proceedings of the 2007 international
ACM
conference on Supporting group work* (pp. 259-268). http://reidster.net/pubs/group282-priedhorsky.pdf
They discuss the probability of a page view of Wikipedia containing vandalism rising over time. I wanted to replicate this analysis and
extend
it past 2007 but I never got the chance. I think the methodology is
really
interesting though.
It doesn't directly answer the question but it does get at the *impact*
of
vandalism.
On Mon, Jan 18, 2021 at 12:13 PM Isaac Johnson isaac@wikimedia.org wrote:
To WSC's point about the difficulty of detecting such behavior or
surveying
at a point in which it would still be salient, I'd add that in general
we
have a large gap in our knowledge about why people choose to stop
editing
because almost all of our survey mechanisms depend on existing
logged-in
usage of the wikis. This is a challenge with many other websites too
but
it's generally easier to find and survey who, for instance, has left Facebook (example <
http://socialmedia.soc.northwestern.edu/wp-content/uploads/2013/05/CHI2013-F...
)
by collecting a random sample of people than it is to find and survey someone who was a former editor of Wikipedia. There were surveys that
did
ask about major barriers to editing (which presumably contribute to burnout) such as the 2012 survey:
https://upload.wikimedia.org/wikipedia/commons/8/81/Editor_Survey_2012_-_Wik...
(see the editor survey category https://meta.wikimedia.org/wiki/Category:Editor_surveys if you're looking for others)
Some things that come to mind though:
- I suspect very few readers see vandalism in their daily browsing
(as a
very frequent, long-term reader of English Wikipedia, I have trouble recalling encountering any clear vandalism in the course of normal reading). That said, I do suspect that most people have seen plenty
of
stories of outlandish vandalism to Wikipedia -- some legitimate but
many
more about vandalism that literally lasted minutes -- that may lead
to
lower trust. Whether or not lower trust in Wikipedia leads to lower readership is a separate question though. Jonathan Morgan ran some recent surveys on reader trust and what factors affected it that might be relevant:
https://meta.wikimedia.org/wiki/Research:The_role_of_citations_in_how_reader...
- Specifically in the context of harassment and gender equity:
- Harassment as barrier:
https://meta.wikimedia.org/wiki/Gender_equity_report_2018/Barriers_to_equity
- Edit summaries in particular as harassment: https://www.elizabethwhittaker.net/wmf-internship (more details <https://www.mediawiki.org/wiki/Wikimedia_Research/Showcase#July_2019%3E ) - Annual Community Insights Reports often have a section on this
--
e.g.,https://meta.wikimedia.org/wiki/Community_Insights/Community_Insights_2020_R...
- 2015 Harassment Survey: https://meta.wikimedia.org/wiki/Research:Harassment_survey_2015
- The body of work around barriers to newcomers might have some good
insights too -- e.g.,
https://www-users.cs.umn.edu/~halfaker/publications/The_Rise_and_Decline/
On Sun, Jan 17, 2021 at 5:44 AM WereSpielChequers < werespielchequers@gmail.com> wrote:
Hi Amir,
This is one of those areas of research where we really need the
annual
editor survey. I think it ran once after the 2009/10 Strategy
process,
and
I don't know if the best questions got included.
But the best time to ask editors what prompted them to start
editing
has
to be fairly soon after they started as memories fade. I once went
back
to
my early edits and the edit I remembered starting me editing barely
made
it
into my first 50.
There is a longstanding theory that a lot of new editors start or
started
to fix some vandalism that they saw, and that this group went into
steep
decline a decade ago with the rise of Cluebot and other antivandalism
tools
that work faster than a newbie could. But without an annual survey to
ask
editors what prompted them to edit you are going to struggle to
research
this. Of course you could look at the early logged in edits of active/prolific wikipedians, but if it is true that many/most
Wikipedians
start with some IP edits, the earliest edits of many Wikipedians
won't
be
available.
Abuse one assumes has a differential effect on the targets of abuse, disproportionately women, gays and ethnic minorities. But I'd be
inclined
to look at stuff targeted at their user and usertalkpages rather than talkpages and edit summaries, though an email survey of former
editors
would be useful.
My suspicion is that when we revert, block and maybe even revdel or oversight abuse we assume that fixes the problem, and if we want to
tackle
abuse we need more edit filters to prevent such abuse from going
live.
WSC
On Sat, 16 Jan 2021 at 15:16, Amir E. Aharoni < amir.aharoni@mail.huji.ac.il> wrote:
Hi,
Is there any research about the effect of vandalism in wiki content
pages
on readers, experienced editors, and new and potential editors?
And of abuse in discussion pages and edit summaries on experienced
editors
and new and potential editors?
Intuitively and anecdotally one could think of the following:
- Vandalism in content pages (articles) wastes editors' and
patrollers'
time. This (probably) doesn't require proof (or does it?). But some
people
say it also causes some experienced editors to burn out and leave.
Is
there
any data about it, beyond intuition?
- Does vandalism *measurably* affect the perception of the wikis'
reliability? (This may be wildly different in different languages
and
wikis.)
- Abusive language on discussion pages and edit summaries affects
editors,
and may cause them to reduce their editing, to stop editing about
certain
topics, or to leave the wiki entirely. Is this effect measurable?
How
does
it differ for various groups by gender, age, religion, country, professional and educational background, seniority at the wiki,
etc.?
Thanks! :)
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Isaac Johnson (he/him/his) -- Research Scientist -- Wikimedia
Foundation
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Hi Aaron,
I would be very interested in that. In particular re flagged revisions as used on the German language Wikipedia (DE) and I think some other wikis. DE has been one of the shrinking communities, which could of course be something unconnected if it is more related to the ratio of tablet to PC users (Wikipedia being near uneditable on the mobile platform). If the Portuguese and potentially other wikes are going to drop IP editing then that also is likely to have an effect on vandalism that would be worthwhile researching.
WSC
On Tue, 19 Jan 2021 at 19:43, Aaron Halfaker aaron.halfaker@gmail.com wrote:
+1 WSC. When I thought about replicating it, I expected to see a dramatic decline in the impact of vandalism with the advent of counter-vandalism tools and abuse filter.
It would be interesting to see that on a cross-wiki basis as different wikis employ different strategies (or seemingly none at all) for counter-vandalism over time.
On Tue, Jan 19, 2021 at 10:58 AM WereSpielChequers < werespielchequers@gmail.com> wrote:
Hi Aaron,
That was an interesting read and a bit of a time capsule. 2002-2006 is a bit before I started editing Wikipedia. Before many of the tools such as huggle that give vandalfighters such an advantage over vandals, I think before the era of bot reversion of vandalism when vandalism had to be reverted by humans rather than computers, and certainly before the edit filters that prevent much, possibly most vandalism from even being saved. It also seems to predate the whole panoply of page protection that stops vandals even editing many common vandalism targets (they do say that
every
single article is available for anyone to edit).
It would be interesting to see a study now when recent changes patrollers boast of the times they have got to some vandalism faster than Cluebot.
I know there were predictions in the early years that eventually the
tidal
wave of vandalism would overwhelm the defenders of the wiki, that study seems to have been part of that. I wonder if anyone in 2004 predicted
that
we would get to the current situation where adolescent vandalism has
turned
out to be so predictable that dealing with it has been mostly automated
and
now we are more worried about spam than vandalism.
WSC
On Mon, 18 Jan 2021 at 23:52, Aaron Halfaker aaron.halfaker@gmail.com wrote:
See page 7 of Priedhorsky, R., Chen, J., Lam, S. T. K., Panciera, K., Terveen, L., & Riedl, J. (2007, November). Creating, destroying, and restoring value in Wikipedia. In *Proceedings of the 2007 international
ACM
conference on Supporting group work* (pp. 259-268). http://reidster.net/pubs/group282-priedhorsky.pdf
They discuss the probability of a page view of Wikipedia containing vandalism rising over time. I wanted to replicate this analysis and
extend
it past 2007 but I never got the chance. I think the methodology is
really
interesting though.
It doesn't directly answer the question but it does get at the *impact*
of
vandalism.
On Mon, Jan 18, 2021 at 12:13 PM Isaac Johnson isaac@wikimedia.org wrote:
To WSC's point about the difficulty of detecting such behavior or
surveying
at a point in which it would still be salient, I'd add that in
general
we
have a large gap in our knowledge about why people choose to stop
editing
because almost all of our survey mechanisms depend on existing
logged-in
usage of the wikis. This is a challenge with many other websites too
but
it's generally easier to find and survey who, for instance, has left Facebook (example <
http://socialmedia.soc.northwestern.edu/wp-content/uploads/2013/05/CHI2013-F...
)
by collecting a random sample of people than it is to find and survey someone who was a former editor of Wikipedia. There were surveys that
did
ask about major barriers to editing (which presumably contribute to burnout) such as the 2012 survey:
https://upload.wikimedia.org/wikipedia/commons/8/81/Editor_Survey_2012_-_Wik...
(see the editor survey category https://meta.wikimedia.org/wiki/Category:Editor_surveys if you're looking for others)
Some things that come to mind though:
- I suspect very few readers see vandalism in their daily browsing
(as a
very frequent, long-term reader of English Wikipedia, I have
trouble
recalling encountering any clear vandalism in the course of normal reading). That said, I do suspect that most people have seen
plenty
of
stories of outlandish vandalism to Wikipedia -- some legitimate
but
many
more about vandalism that literally lasted minutes -- that may
lead
to
lower trust. Whether or not lower trust in Wikipedia leads to
lower
readership is a separate question though. Jonathan Morgan ran some recent surveys on reader trust and what factors affected it that might be relevant:
https://meta.wikimedia.org/wiki/Research:The_role_of_citations_in_how_reader...
- Specifically in the context of harassment and gender equity:
- Harassment as barrier:
https://meta.wikimedia.org/wiki/Gender_equity_report_2018/Barriers_to_equity
- Edit summaries in particular as harassment: https://www.elizabethwhittaker.net/wmf-internship (moredetails
<https://www.mediawiki.org/wiki/Wikimedia_Research/Showcase#July_2019
) - Annual Community Insights Reports often have a section onthis
--
e.g.,https://meta.wikimedia.org/wiki/Community_Insights/Community_Insights_2020_R...
- 2015 Harassment Survey:https://meta.wikimedia.org/wiki/Research:Harassment_survey_2015
- The body of work around barriers to newcomers might have some
good
insights too -- e.g.,
https://www-users.cs.umn.edu/~halfaker/publications/The_Rise_and_Decline/
On Sun, Jan 17, 2021 at 5:44 AM WereSpielChequers < werespielchequers@gmail.com> wrote:
Hi Amir,
This is one of those areas of research where we really need the
annual
editor survey. I think it ran once after the 2009/10 Strategy
process,
and
I don't know if the best questions got included.
But the best time to ask editors what prompted them to start
editing
has
to be fairly soon after they started as memories fade. I once went
back
to
my early edits and the edit I remembered starting me editing barely
made
it
into my first 50.
There is a longstanding theory that a lot of new editors start or
started
to fix some vandalism that they saw, and that this group went into
steep
decline a decade ago with the rise of Cluebot and other
antivandalism
tools
that work faster than a newbie could. But without an annual survey
to
ask
editors what prompted them to edit you are going to struggle to
research
this. Of course you could look at the early logged in edits of active/prolific wikipedians, but if it is true that many/most
Wikipedians
start with some IP edits, the earliest edits of many Wikipedians
won't
be
available.
Abuse one assumes has a differential effect on the targets of
abuse,
disproportionately women, gays and ethnic minorities. But I'd be
inclined
to look at stuff targeted at their user and usertalkpages rather
than
talkpages and edit summaries, though an email survey of former
editors
would be useful.
My suspicion is that when we revert, block and maybe even revdel or oversight abuse we assume that fixes the problem, and if we want to
tackle
abuse we need more edit filters to prevent such abuse from going
live.
WSC
On Sat, 16 Jan 2021 at 15:16, Amir E. Aharoni < amir.aharoni@mail.huji.ac.il> wrote:
Hi,
Is there any research about the effect of vandalism in wiki
content
pages
on readers, experienced editors, and new and potential editors?
And of abuse in discussion pages and edit summaries on
experienced
editors
and new and potential editors?
Intuitively and anecdotally one could think of the following:
- Vandalism in content pages (articles) wastes editors' and
patrollers'
time. This (probably) doesn't require proof (or does it?). But
some
people
say it also causes some experienced editors to burn out and
leave.
Is
there
any data about it, beyond intuition?
- Does vandalism *measurably* affect the perception of the
wikis'
reliability? (This may be wildly different in different languages
and
wikis.)
- Abusive language on discussion pages and edit summaries
affects
editors,
and may cause them to reduce their editing, to stop editing about
certain
topics, or to leave the wiki entirely. Is this effect measurable?
How
does
it differ for various groups by gender, age, religion, country, professional and educational background, seniority at the wiki,
etc.?
Thanks! :)
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Isaac Johnson (he/him/his) -- Research Scientist -- Wikimedia
Foundation
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
I'm sure there has been a survey of former editors done using the email this user function (as I remember it one of the more common responses was I haven't left yet). However this would not be a great way to survey re harassment as harassed people are more likely to close an email account or disable the email future.
As for how many readers saw vandalism in the era before edit filters etc, it didn't need to be many readers who saw it, and many of those to remove it for this to be an important way to recruit editors. We have such a huge imbalance between readers and editors that even if only 1% of readers saw vandalism and only 1% of those fixed it, that would still be an extra hundred editors for every million readers.
WSC
On Mon, 18 Jan 2021 at 20:13, Isaac Johnson isaac@wikimedia.org wrote:
To WSC's point about the difficulty of detecting such behavior or surveying at a point in which it would still be salient, I'd add that in general we have a large gap in our knowledge about why people choose to stop editing because almost all of our survey mechanisms depend on existing logged-in usage of the wikis. This is a challenge with many other websites too but it's generally easier to find and survey who, for instance, has left Facebook (example < http://socialmedia.soc.northwestern.edu/wp-content/uploads/2013/05/CHI2013-F...
)
by collecting a random sample of people than it is to find and survey someone who was a former editor of Wikipedia. There were surveys that did ask about major barriers to editing (which presumably contribute to burnout) such as the 2012 survey:
https://upload.wikimedia.org/wikipedia/commons/8/81/Editor_Survey_2012_-_Wik... (see the editor survey category https://meta.wikimedia.org/wiki/Category:Editor_surveys if you're looking for others)
Some things that come to mind though:
- I suspect very few readers see vandalism in their daily browsing (as a
very frequent, long-term reader of English Wikipedia, I have trouble recalling encountering any clear vandalism in the course of normal reading). That said, I do suspect that most people have seen plenty of stories of outlandish vandalism to Wikipedia -- some legitimate but many more about vandalism that literally lasted minutes -- that may lead to lower trust. Whether or not lower trust in Wikipedia leads to lower readership is a separate question though. Jonathan Morgan ran some recent surveys on reader trust and what factors affected it that might be relevant:
https://meta.wikimedia.org/wiki/Research:The_role_of_citations_in_how_reader...
- Specifically in the context of harassment and gender equity:
- Harassment as barrier:
https://meta.wikimedia.org/wiki/Gender_equity_report_2018/Barriers_to_equity - Edit summaries in particular as harassment: https://www.elizabethwhittaker.net/wmf-internship (more details < https://www.mediawiki.org/wiki/Wikimedia_Research/Showcase#July_2019%3E ) - Annual Community Insights Reports often have a section on this -- e.g.,
https://meta.wikimedia.org/wiki/Community_Insights/Community_Insights_2020_R... - 2015 Harassment Survey: https://meta.wikimedia.org/wiki/Research:Harassment_survey_2015
- The body of work around barriers to newcomers might have some good
insights too -- e.g.,
https://www-users.cs.umn.edu/~halfaker/publications/The_Rise_and_Decline/
On Sun, Jan 17, 2021 at 5:44 AM WereSpielChequers < werespielchequers@gmail.com> wrote:
Hi Amir,
This is one of those areas of research where we really need the annual editor survey. I think it ran once after the 2009/10 Strategy process,
and
I don't know if the best questions got included.
But the best time to ask editors what prompted them to start editing
has
to be fairly soon after they started as memories fade. I once went back
to
my early edits and the edit I remembered starting me editing barely made
it
into my first 50.
There is a longstanding theory that a lot of new editors start or started to fix some vandalism that they saw, and that this group went into steep decline a decade ago with the rise of Cluebot and other antivandalism
tools
that work faster than a newbie could. But without an annual survey to ask editors what prompted them to edit you are going to struggle to research this. Of course you could look at the early logged in edits of active/prolific wikipedians, but if it is true that many/most Wikipedians start with some IP edits, the earliest edits of many Wikipedians won't be available.
Abuse one assumes has a differential effect on the targets of abuse, disproportionately women, gays and ethnic minorities. But I'd be inclined to look at stuff targeted at their user and usertalkpages rather than talkpages and edit summaries, though an email survey of former editors would be useful.
My suspicion is that when we revert, block and maybe even revdel or oversight abuse we assume that fixes the problem, and if we want to
tackle
abuse we need more edit filters to prevent such abuse from going live.
WSC
On Sat, 16 Jan 2021 at 15:16, Amir E. Aharoni < amir.aharoni@mail.huji.ac.il> wrote:
Hi,
Is there any research about the effect of vandalism in wiki content
pages
on readers, experienced editors, and new and potential editors?
And of abuse in discussion pages and edit summaries on experienced
editors
and new and potential editors?
Intuitively and anecdotally one could think of the following:
- Vandalism in content pages (articles) wastes editors' and
patrollers'
time. This (probably) doesn't require proof (or does it?). But some
people
say it also causes some experienced editors to burn out and leave. Is
there
any data about it, beyond intuition?
- Does vandalism *measurably* affect the perception of the wikis'
reliability? (This may be wildly different in different languages and wikis.)
- Abusive language on discussion pages and edit summaries affects
editors,
and may cause them to reduce their editing, to stop editing about
certain
topics, or to leave the wiki entirely. Is this effect measurable? How
does
it differ for various groups by gender, age, religion, country, professional and educational background, seniority at the wiki, etc.?
Thanks! :)
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Isaac Johnson (he/him/his) -- Research Scientist -- Wikimedia Foundation _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
wiki-research-l@lists.wikimedia.org