[Labs-l] Some using a Python framework is relentlessly hammering Harvard sites, resulting an IP range ban.

[Maximilian Doerr]

Would the user who is querying the Harvard sites for planet data, that is
carrying the UA "weblinkchecker Pywikibot/3.0-dev (g7171) requests/2.2.1
Python/2.7.6.final.0", please stop, or severely throttle the GET requests.
It's making 168 requests to that site a minute, and consequently they banned
labs from accessing it, according to the IT department there, who kindly
shared with me the access log.

 

I would imagine it's also not being very friendly with the bandwidth usage
of Labs itself.

 

Consequently, InternetArchiveBot cannot ascertain if the site is alive or
not because of this ban, and while I am working on a solution for these
cases, it's really best if the bot can be able to make these decisions on
its own rather than deferring to a whitelist of sorts.

 

Cyberpower678

English Wikipedia Account Creation Team

Mailing List Moderator

Global User Renamer

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20161203/4a18d769/attachment.html>