Identifying bots and bot edit decline

List overview All Threads
Download

newer

older

Feature Requests: Suggeted new...

Monthly (predicted) article...

Flöck, Fabian

11 Oct 2016 11 Oct '16

9:15 a.m.

Hi all , two questions, maybe someone can help: 1. I was trying to compile a complete list of all bots that were ever (potentially) active on the English Wikipedia so that one can identify bot accounts in the dumps. Below are all the lists (including historic bots) that I could find [1]. Out of those overlapping lists, I extracted 2795 unique bot names (some seem to be just names for bot approval request pages). Going through the historic edit data (no current redirects), 1377 user names were actually in that list. Does anyone know if that should cover (almost) all ever active bots, or is there even a better list/method? I would like to avoid using unreliable regular expressions. (Similar question for other language editions) 2. I counted bot edits per half year in en.wikipedia and saw a major decrease between in the first half of 2013 from ~ 3 M to ~1M edits per half year between January and July 2013, which seems to be in line with official stats [2]. This is likely not news, so can someone enlighten me regarding what brought about that sharp decline of bot edits? Cheers, Fabian [1] https://en.wikipedia.org/wiki/Wikipedia:List_of_bots_by_number_of_edits https://en.wikipedia.org/wiki/Wikipedia:Bots/Status/inactive_bots_1 https://en.wikipedia.org/wiki/Wikipedia:Bots/Status/inactive_bots_2 https://en.wikipedia.org/wiki/Wikipedia:List_of_Wikipedians_by_number_of_ed… https://en.wikipedia.org/w/api.php?action=query&list=allusers&augro… https://en.wikipedia.org/w/api.php?action=query&list=categorymembers&am… https://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/Approved (+ contents of all archive pages) https://stats.wikimedia.org/EN/TablesWikipediaEN.htm#bots [2] https://stats.wikimedia.org/EN/TablesWikipediaEN.htm#editor_activity_levels — Dr. Fabian Flöck Researcher Computational Social Science department GESIS - Leibniz Institute for the Social Sciences Unter Sachsenhausen 6-8, 50667 Cologne, Germany Tel: + 49 (0) 221-47694-208 fabian.floeck(a)gesis.org www.gesis.org www.facebook.com/gesis.org

Show replies by thread

Lydia Pintscher

11 Oct 11 Oct

10:05 a.m.

On Tue, Oct 11, 2016 at 11:15 AM, Flöck, Fabian <Fabian.Floeck(a)gesis.org> wrote:

...

This is probably caused by Wikidata making it unnecessary to run bots to synchronize language links between language versions. Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

Federico Leva (Nemo)

10:08 a.m.

New subject: [Analytics] Identifying bots and bot edit decline

Wikistats knows about 8017 bot usernames according to https://dumps.wikimedia.org/other/pagecounts-ez/wikistats/csv_wp_main.zip (cut -f2 -d, StatisticsBots.csv | sort -u | wc -l ). Given active editors tend to complain a lot if they get counted as bots, a comprehensive list should probably be a superset of that one. Flöck, Fabian, 11/10/2016 11:15:

...

This is likely not news, so can someone enlighten me regarding what brought about that sharp decline of bot edits?

Taha Yasseri

10:25 a.m.

New subject: [Analytics] Identifying bots and bot edit decline

Hi Fabian, We recently did the same exercise for this paper: Even Good Bots Fight <https://arxiv.org/abs/1609.04285>. Have a look at the data collection, where we explained how we made a list of all bots. Also re edit statistics, see Fig S1. Happy to compare the lists and share data. Best, Taha On Tue, Oct 11, 2016 at 11:08 AM, Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote:

...

This is likely not news, so can someone enlighten me regarding what brought about that sharp decline of bot edits?

The migration of interwiki links to Wikidata, which is very visible in https://stats.wikimedia.org/EN/PlotsPngEditHistoryTop.htm . There was also some statistic by WMF on whether active users had "migrated" to Wikidata from other projects, but I can't quickly find it now; maybe it was around the time of http://infodisiac.com/blog/201 4/03/wikimedia-editor-trends-broken-down-by-project/ . Nemo _______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

-- ==New Paper== Editorial: At the Crossroads: Lessons and Challenges in Computational Social Science <http://journal.frontiersin.org/article/10.3389/fphy.2016.00037/full> Borge-Holthoefer J, Moreno Y and Yasseri T *Front. Phys*. 4:37 (2016). ============= Dr Taha Yasseri http://www.oii.ox.ac.uk/people/yasseri/ Research Fellow in Computational Social Science, Oxford Internet Institute, Research Fellow in Humanities and Social Sciences, Wolfson College, University of Oxford, and Faculty Fellow, Alan Turing Institute for Data Science. Tel. +44-1865-287229 1 St. Giles Oxford OX1 3JS UK

Flöck, Fabian

11:54 a.m.

New subject: [Analytics] Identifying bots and bot edit decline

nice, thanks all :) fabian On 11.10.2016, at 12:25, Taha Yasseri <taha.yaseri@gmail.com<mailto:taha.yaseri@gmail.com>> wrote: Hi Fabian, We recently did the same exercise for this paper: Even Good Bots Fight <https://arxiv.org/abs/1609.04285> . Have a look at the data collection, where we explained how we made a list of all bots. Also re edit statistics, see Fig S1. Happy to compare the lists and share data. Best, Taha On Tue, Oct 11, 2016 at 11:08 AM, Federico Leva (Nemo) <nemowiki@gmail.com<mailto:nemowiki@gmail.com>> wrote: Wikistats knows about 8017 bot usernames according to https://dumps.wikimedia.org/other/pagecounts-ez/wikistats/csv_wp_main.zip (cut -f2 -d, StatisticsBots.csv | sort -u | wc -l ). Given active editors tend to complain a lot if they get counted as bots, a comprehensive list should probably be a superset of that one. Flöck, Fabian, 11/10/2016 11:15: This is likely not news, so can someone enlighten me regarding what brought about that sharp decline of bot edits? The migration of interwiki links to Wikidata, which is very visible in https://stats.wikimedia.org/EN/PlotsPngEditHistoryTop.htm . There was also some statistic by WMF on whether active users had "migrated" to Wikidata from other projects, but I can't quickly find it now; maybe it was around the time of http://infodisiac.com/blog/2014/03/wikimedia-editor-trends-broken-down-by-p… . Nemo _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org<mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l -- ==New Paper== Editorial: At the Crossroads: Lessons and Challenges in Computational Social Science<http://journal.frontiersin.org/article/10.3389/fphy.2016.00037/f… Borge-Holthoefer J, Moreno Y and Yasseri T Front. Phys. 4:37 (2016). ============= Dr Taha Yasseri http://www.oii.ox.ac.uk/people/yasseri/<http://www.oii.ox.ac.uk/people/y… Research Fellow in Computational Social Science, Oxford Internet Institute, Research Fellow in Humanities and Social Sciences, Wolfson College, University of Oxford, and Faculty Fellow, Alan Turing Institute for Data Science. Tel. +44-1865-287229 1 St. Giles Oxford OX1 3JS UK _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org<mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l Gruß, Fabian — Dr. Fabian Flöck Researcher Computational Social Science department GESIS - Leibniz Institute for the Social Sciences Unter Sachsenhausen 6-8, 50667 Cologne, Germany Tel: + 49 (0) 221-47694-208 fabian.floeck@gesis.org<mailto:fabian.floeck@gesis.org> www.gesis.org www.facebook.com/gesis.org

WereSpielChequers

1:10 p.m.

New subject: [Analytics] Identifying bots and bot edit decline

On the English Wikipedia you can start with the current bots which should all be in https://en.wikipedia.org/wiki/Category:All_Wikipedia_bots There are also former bots in the category https://en.wikipedia.org/wiki/Wikipedia:List_of_Wikipedians_by_number_of_ed… but we are unlikely to spot them and add them to that list unless they did enough edits to make the list of 10,000 most active Wikipedians after they were deflagged. Good to hear some figures about the drop in bot editing because of the intrawiki links. I've recent taken over maintaining one of our more venerable stats on Wikipedia <https://en.wikipedia.org/wiki/User:Katalaveno/TBE>, and I was looking for a figure on the drop in editing due to the intrawiki change. Anecdotally I was told that the Wikidata community was half existing wikimedians moving to a new project and half new recruits to the community. But I don't remember seeing detailed stats on that, it might make an interesting Phd project for someone.. As with so many other spinoffs both within the community to Wikitonary, various languages of Wikipedia and of course Wikimedia Commons and to fansites in Wikia of course there will be some loss to the community they were spunoff from. The other, bigger and harder change to quantify is the amount of vandalfighting, bot and manual that moved to the edit filters between 2009 and 2014. Because it was a gradual process as filters were tested and refined it mainly looks like a general decline in editing. On 11 October 2016 at 11:25, Taha Yasseri <taha.yaseri(a)gmail.com> wrote:

...

wrote:

This is likely not news, so can someone enlighten me regarding what brought about that sharp decline of bot edits?

Erwan Joud

12 Oct 12 Oct

1 p.m.

New subject: [Analytics] Identifying bots and bot edit decline

Hello Fabian, For the French project (2015-10-12) : https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Bot/Liste This page is updated by a bot : Bub's wikibot[1] (perhaps the code would interest you). An other list : https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Bot#Liste_des_bots_sur_la_Wiki… If you need translation, ask me. French wikipedians' words are sometimes a bit odd. Yours sincerely, Erwan Joud PhD Student - Université de Bretagne Occidentale Laboratoire ICI EA2652 Supervisors : Pr Marine Le Gall-Ely & Dr Nicolas Jullien Quoting Taha Yasseri <taha.yaseri(a)gmail.com>om>:

...

Hi Fabian, We recently did the same exercise for this paper: Even Good Bots Fight [2]. Have a look at the data collection, where we explained how we made a list of all bots. Also re edit statistics, see Fig S1. Happy to compare the lists and share data. Best, Taha On Tue, Oct 11, 2016 at 11:08 AM, Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote: > Wikistats knows about 8017 bot usernames according to >

https://dumps.wikimedia.org/other/pagecounts-ez/wikistats/csv_wp_main.zip (cut -f2 -d, StatisticsBots.csv | sort -u | wc -l ). Given active editors tend to complain a lot if they get counted as bots, a comprehensive list should probably be a superset of that

...

> one. > > Flöck, Fabian, 11/10/2016 11:15: > >> This is likely not news, so can someone enlighten me regarding what >> brought about that sharp decline of bot edits? > > The migration of interwiki links to Wikidata, which is very visible in > https://stats.wikimedia.org/EN/PlotsPngEditHistoryTop.htm . > > There was also some statistic by WMF on whether active users had > "migrated" to Wikidata from other projects, but I can't quickly find it > now; maybe it was around the time of >

http://infodisiac.com/blog/2014/03/wikimedia-editor-trends-broken-down-by-p…

...

. Nemo _______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

-- ==New Paper== Editorial: At the Crossroads: Lessons and Challenges in Computational Social Science[3] Borge-Holthoefer J, Moreno Y and Yasseri T /Front. Phys/. 4:37 (2016). ============= Dr Taha Yasseri http://www.oii.ox.ac.uk/people/yasseri/ Research Fellow in Computational Social Science, Oxford Internet Institute, Research Fellow in Humanities and Social Sciences, Wolfson College, University of Oxford, and Faculty Fellow, Alan Turing Institute for Data Science. Tel. +44-1865-287229 1 St. Giles Oxford OX1 3JS UK

Links: ------ [1] https://fr.wikipedia.org/wiki/Utilisateur:Bub%27s_wikibot [2] https://arxiv.org/abs/1609.04285 [3] http://journal.frontiersin.org/article/10.3389/fphy.2016.00037/full

2770

days inactive

2771

days old

wiki-research-l@lists.wikimedia.org

Manage subscription

6 comments

6 participants

tags (0)

participants (6)

Erwan Joud
Federico Leva (Nemo)
Flöck, Fabian
Lydia Pintscher
Taha Yasseri
WereSpielChequers