Hi all ,
two questions, maybe someone can help:
1. I was trying to compile a complete list of all bots that were ever (potentially) active on the English Wikipedia so that one can identify bot accounts in the dumps. Below are all the lists (including historic bots) that I could find [1]. Out of those overlapping lists, I extracted 2795 unique bot names (some seem to be just names for bot approval request pages). Going through the historic edit data (no current redirects), 1377 user names were actually in that list. Does anyone know if that should cover (almost) all ever active bots, or is there even a better list/method? I would like to avoid using unreliable regular expressions. (Similar question for other language editions)
2. I counted bot edits per half year in en.wikipedia and saw a major decrease between in the first half of 2013 from ~ 3 M to ~1M edits per half year between January and July 2013, which seems to be in line with official stats [2]. This is likely not news, so can someone enlighten me regarding what brought about that sharp decline of bot edits?
Cheers, Fabian
[1] https://en.wikipedia.org/wiki/Wikipedia:List_of_bots_by_number_of_edits https://en.wikipedia.org/wiki/Wikipedia:Bots/Status/inactive_bots_1 https://en.wikipedia.org/wiki/Wikipedia:Bots/Status/inactive_bots_2 https://en.wikipedia.org/wiki/Wikipedia:List_of_Wikipedians_by_number_of_edi... https://en.wikipedia.org/w/api.php?action=query&list=allusers&augrou... https://en.wikipedia.org/w/api.php?action=query&list=categorymembers&... https://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/Approved (+ contents of all archive pages) https://stats.wikimedia.org/EN/TablesWikipediaEN.htm#bots
[2] https://stats.wikimedia.org/EN/TablesWikipediaEN.htm#editor_activity_levels
— Dr. Fabian Flöck Researcher Computational Social Science department GESIS - Leibniz Institute for the Social Sciences Unter Sachsenhausen 6-8, 50667 Cologne, Germany Tel: + 49 (0) 221-47694-208 fabian.floeck@gesis.org
www.gesis.org www.facebook.com/gesis.org
On Tue, Oct 11, 2016 at 11:15 AM, Flöck, Fabian Fabian.Floeck@gesis.org wrote:
Hi all ,
two questions, maybe someone can help:
I was trying to compile a complete list of all bots that were ever (potentially) active on the English Wikipedia so that one can identify bot accounts in the dumps. Below are all the lists (including historic bots) that I could find [1]. Out of those overlapping lists, I extracted 2795 unique bot names (some seem to be just names for bot approval request pages). Going through the historic edit data (no current redirects), 1377 user names were actually in that list. Does anyone know if that should cover (almost) all ever active bots, or is there even a better list/method? I would like to avoid using unreliable regular expressions. (Similar question for other language editions)
I counted bot edits per half year in en.wikipedia and saw a major decrease between in the first half of 2013 from ~ 3 M to ~1M edits per half year between January and July 2013, which seems to be in line with official stats [2]. This is likely not news, so can someone enlighten me regarding what brought about that sharp decline of bot edits?
This is probably caused by Wikidata making it unnecessary to run bots to synchronize language links between language versions.
Cheers Lydia
Wikistats knows about 8017 bot usernames according to https://dumps.wikimedia.org/other/pagecounts-ez/wikistats/csv_wp_main.zip (cut -f2 -d, StatisticsBots.csv | sort -u | wc -l ). Given active editors tend to complain a lot if they get counted as bots, a comprehensive list should probably be a superset of that one.
Flöck, Fabian, 11/10/2016 11:15:
This is likely not news, so can someone enlighten me regarding what brought about that sharp decline of bot edits?
The migration of interwiki links to Wikidata, which is very visible in https://stats.wikimedia.org/EN/PlotsPngEditHistoryTop.htm .
There was also some statistic by WMF on whether active users had "migrated" to Wikidata from other projects, but I can't quickly find it now; maybe it was around the time of http://infodisiac.com/blog/2014/03/wikimedia-editor-trends-broken-down-by-pr... .
Nemo
Hi Fabian,
We recently did the same exercise for this paper: Even Good Bots Fight https://arxiv.org/abs/1609.04285. Have a look at the data collection, where we explained how we made a list of all bots. Also re edit statistics, see Fig S1.
Happy to compare the lists and share data.
Best, Taha
On Tue, Oct 11, 2016 at 11:08 AM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
Wikistats knows about 8017 bot usernames according to https://dumps.wikimedia.org/other/pagecounts-ez/wikistats/csv_wp_main.zip (cut -f2 -d, StatisticsBots.csv | sort -u | wc -l ). Given active editors tend to complain a lot if they get counted as bots, a comprehensive list should probably be a superset of that one.
Flöck, Fabian, 11/10/2016 11:15:
This is likely not news, so can someone enlighten me regarding what brought about that sharp decline of bot edits?
The migration of interwiki links to Wikidata, which is very visible in https://stats.wikimedia.org/EN/PlotsPngEditHistoryTop.htm .
There was also some statistic by WMF on whether active users had "migrated" to Wikidata from other projects, but I can't quickly find it now; maybe it was around the time of http://infodisiac.com/blog/201 4/03/wikimedia-editor-trends-broken-down-by-project/ .
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
nice, thanks all :)
fabian
On 11.10.2016, at 12:25, Taha Yasseri <taha.yaseri@gmail.commailto:taha.yaseri@gmail.com> wrote:
Hi Fabian,
We recently did the same exercise for this paper: Even Good Bots Fight https://arxiv.org/abs/1609.04285 . Have a look at the data collection, where we explained how we made a list of all bots. Also re edit statistics, see Fig S1.
Happy to compare the lists and share data.
Best, Taha
On Tue, Oct 11, 2016 at 11:08 AM, Federico Leva (Nemo) <nemowiki@gmail.commailto:nemowiki@gmail.com> wrote: Wikistats knows about 8017 bot usernames according to https://dumps.wikimedia.org/other/pagecounts-ez/wikistats/csv_wp_main.zip (cut -f2 -d, StatisticsBots.csv | sort -u | wc -l ). Given active editors tend to complain a lot if they get counted as bots, a comprehensive list should probably be a superset of that one.
Flöck, Fabian, 11/10/2016 11:15: This is likely not news, so can someone enlighten me regarding what brought about that sharp decline of bot edits?
The migration of interwiki links to Wikidata, which is very visible in https://stats.wikimedia.org/EN/PlotsPngEditHistoryTop.htm .
There was also some statistic by WMF on whether active users had "migrated" to Wikidata from other projects, but I can't quickly find it now; maybe it was around the time of http://infodisiac.com/blog/2014/03/wikimedia-editor-trends-broken-down-by-pr... .
Nemo
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.orgmailto:Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- ==New Paper== Editorial: At the Crossroads: Lessons and Challenges in Computational Social Sciencehttp://journal.frontiersin.org/article/10.3389/fphy.2016.00037/full Borge-Holthoefer J, Moreno Y and Yasseri T Front. Phys. 4:37 (2016). =============
Dr Taha Yasseri http://www.oii.ox.ac.uk/people/yasseri/http://www.oii.ox.ac.uk/people/yasseri Research Fellow in Computational Social Science, Oxford Internet Institute, Research Fellow in Humanities and Social Sciences, Wolfson College, University of Oxford, and Faculty Fellow, Alan Turing Institute for Data Science.
Tel. +44-1865-287229 1 St. Giles Oxford OX1 3JS UK
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.orgmailto:Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Gruß, Fabian
— Dr. Fabian Flöck Researcher Computational Social Science department GESIS - Leibniz Institute for the Social Sciences Unter Sachsenhausen 6-8, 50667 Cologne, Germany Tel: + 49 (0) 221-47694-208 fabian.floeck@gesis.orgmailto:fabian.floeck@gesis.org
www.gesis.org www.facebook.com/gesis.org
On the English Wikipedia you can start with the current bots which should all be in https://en.wikipedia.org/wiki/Category:All_Wikipedia_bots
There are also former bots in the category https://en.wikipedia.org/wiki/Wikipedia:List_of_Wikipedians_by_number_of_edi... but we are unlikely to spot them and add them to that list unless they did enough edits to make the list of 10,000 most active Wikipedians after they were deflagged.
Good to hear some figures about the drop in bot editing because of the intrawiki links. I've recent taken over maintaining one of our more venerable stats on Wikipedia https://en.wikipedia.org/wiki/User:Katalaveno/TBE, and I was looking for a figure on the drop in editing due to the intrawiki change.
Anecdotally I was told that the Wikidata community was half existing wikimedians moving to a new project and half new recruits to the community. But I don't remember seeing detailed stats on that, it might make an interesting Phd project for someone.. As with so many other spinoffs both within the community to Wikitonary, various languages of Wikipedia and of course Wikimedia Commons and to fansites in Wikia of course there will be some loss to the community they were spunoff from.
The other, bigger and harder change to quantify is the amount of vandalfighting, bot and manual that moved to the edit filters between 2009 and 2014. Because it was a gradual process as filters were tested and refined it mainly looks like a general decline in editing.
On 11 October 2016 at 11:25, Taha Yasseri taha.yaseri@gmail.com wrote:
Hi Fabian,
We recently did the same exercise for this paper: Even Good Bots Fight https://arxiv.org/abs/1609.04285. Have a look at the data collection, where we explained how we made a list of all bots. Also re edit statistics, see Fig S1.
Happy to compare the lists and share data.
Best, Taha
On Tue, Oct 11, 2016 at 11:08 AM, Federico Leva (Nemo) <nemowiki@gmail.com
wrote:
Wikistats knows about 8017 bot usernames according to https://dumps.wikimedia.org/other/pagecounts-ez/wikistats/csv_wp_main.zip (cut -f2 -d, StatisticsBots.csv | sort -u | wc -l ). Given active editors tend to complain a lot if they get counted as bots, a comprehensive list should probably be a superset of that one.
Flöck, Fabian, 11/10/2016 11:15:
This is likely not news, so can someone enlighten me regarding what brought about that sharp decline of bot edits?
The migration of interwiki links to Wikidata, which is very visible in https://stats.wikimedia.org/EN/PlotsPngEditHistoryTop.htm .
There was also some statistic by WMF on whether active users had "migrated" to Wikidata from other projects, but I can't quickly find it now; maybe it was around the time of http://infodisiac.com/blog/201 4/03/wikimedia-editor-trends-broken-down-by-project/ .
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- ==New Paper== Editorial: At the Crossroads: Lessons and Challenges in Computational Social Science http://journal.frontiersin.org/article/10.3389/fphy.2016.00037/full Borge-Holthoefer J, Moreno Y and Yasseri T
*Front. Phys*. 4:37 (2016).
Dr Taha Yasseri http://www.oii.ox.ac.uk/people/yasseri/ Research Fellow in Computational Social Science, Oxford Internet Institute, Research Fellow in Humanities and Social Sciences, Wolfson College, University of Oxford, and Faculty Fellow, Alan Turing Institute for Data Science.
Tel. +44-1865-287229 1 St. Giles Oxford OX1 3JS UK
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Hello Fabian,
For the French project (2015-10-12) : https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Bot/Liste This page is updated by a bot : Bub's wikibot[1] (perhaps the code would interest you).
An other list : https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Bot#Liste_des_bots_sur_la_Wikip...
If you need translation, ask me. French wikipedians' words are sometimes a bit odd.
Yours sincerely,
Erwan Joud PhD Student - Université de Bretagne Occidentale Laboratoire ICI EA2652 Supervisors : Pr Marine Le Gall-Ely & Dr Nicolas Jullien
Quoting Taha Yasseri taha.yaseri@gmail.com:
Hi Fabian, We recently did the same exercise for this paper: Even Good Bots Fight [2]. Have a look at the data collection, where we explained how we made a list of all bots. Also re edit statistics, see Fig S1. Happy to compare the lists and share data. Best, Taha
On Tue, Oct 11, 2016 at 11:08 AM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
Wikistats knows about 8017 bot usernames according to
https://dumps.wikimedia.org/other/pagecounts-ez/wikistats/csv_wp_main.zip (cut -f2 -d, StatisticsBots.csv | sort -u | wc -l ). Given active editors tend to complain a lot if they get counted as bots, a comprehensive list should probably be a superset of that
one.
Flöck, Fabian, 11/10/2016 11:15:
This is likely not news, so can someone enlighten me regarding what brought about that sharp decline of bot edits?
The migration of interwiki links to Wikidata, which is very visible in https://stats.wikimedia.org/EN/PlotsPngEditHistoryTop.htm .
There was also some statistic by WMF on whether active users had "migrated" to Wikidata from other projects, but I can't quickly find it now; maybe it was around the time of
http://infodisiac.com/blog/2014/03/wikimedia-editor-trends-broken-down-by-pr...
.
Nemo
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
==New Paper== Editorial: At the Crossroads: Lessons and Challenges in Computational Social Science[3] Borge-Holthoefer J, Moreno Y and Yasseri T /Front. Phys/. 4:37 (2016).
============= Dr Taha Yasseri http://www.oii.ox.ac.uk/people/yasseri/ Research Fellow in Computational SocialScience, Oxford Internet Institute, Research Fellow in Humanities and Social Sciences, Wolfson College, University of Oxford, and Faculty Fellow, Alan Turing Institute for Data Science. Tel. +44-1865-287229 1 St. Giles
Oxford OX1 3JS UK
Links: ------ [1] https://fr.wikipedia.org/wiki/Utilisateur:Bub%27s_wikibot [2] https://arxiv.org/abs/1609.04285 [3] http://journal.frontiersin.org/article/10.3389/fphy.2016.00037/full
wiki-research-l@lists.wikimedia.org