https://bugzilla.wikimedia.org/show_bug.cgi?id=69204
Bug ID: 69204 Summary: weblinkchecker.py uses a fake user-agent Product: Pywikibot Version: core (2.0) Hardware: All OS: All Status: NEW Severity: major Priority: Unprioritized Component: network Assignee: Pywikipedia-bugs@lists.wikimedia.org Reporter: jayvdb@gmail.com Web browser: --- Mobile Platform: ---
weblinkchecker.py (core) contains this comment:
# we fake being Firefox because some webservers block unknown # clients, e.g. https://images.google.de/images?q=Albit gives a 403 # when using the PyWikipediaBot user agent. 'User-agent': 'Mozilla/5.0 (X11; U; Linux i686; de; rv:1.8) Gecko/20051128 SUSE/1.5-0.1 Firefox/1.5',
Which was added to core in Jan 2007: https://www.mediawiki.org/wiki/Special:Code/pywikipedia/3165
Faking the user-agent should be an option, default disabled, or only used for servers known to be problematic.