[Labs-l] Some using a Python framework is relentlessly hammering Harvard sites, resulting an IP range ban.

[Tim Landscheidt]

"Maximilian Doerr" <maximilian.doerr at gmail.com> wrote:

> […]

> Consequently, InternetArchiveBot cannot ascertain if the site is alive or
> not because of this ban, and while I am working on a solution for these
> cases, it's really best if the bot can be able to make these decisions on
> its own rather than deferring to a whitelist of sorts.

Ceterum censeo: Bots checking external web links should be
merged, and ideally made a production-grade thingy by WMF.
(MediaWiki extension for reuse by other sites?)  In the
past, interwiki bots were en vogue which meant that for each
wiki there were a number of them doing the exact same thing,
and now this pattern has shifted to link checking bots.

To apply [[Cunningham's Law]]: From the looks of it, Inter-
netArchiveBot seems to be the most useful bot checking ex-
ternal web links, and I think it is most productive to aim
efforts in that direction.

Tim