[Labs-l] 403 Error on web server when requesting too much at one time.

[Bryan White]

1)  What happened about 3-4 weeks ago to speed everything up?  I used to
click something on the web server and 3 seconds later it would pop up.  Now
it is almost instantaneous.  Programs on the queue are running an order of
magnitude faster.  Thank you for whatever you did.

2)  Unfortunately this has brought about a problem.  If I bombard the web
server with quick requests, I get  "ERROR 403: Forbidden".
   a) This was first noticed in the  WPCleaner program
([[:en:WP:CLEANER]]).  The program would make many calls right after
another to the web server, but after the fourth or fifth call, the error
message would pop up.  I originally thought WPCleaner couldn't handle the
web server being so fast.  So, I put in a 1 second delay into the cgi
program and everything worked fine.
   b) However, WPCleaner was not the problem.  If a shell script is run
with 20 calls to the web server, it too would receive the error message.
   c) The cgi program was a Perl program making calls to the database.
 Wondering if that was a problem, I tried a shell script that retrieved a
very small html page.  It would run longer, but it still ended up with the
error.

So, what is going on and can it be fixed?


Location with delay: (change id # to get a different report and do about 20
calls)
wget
http://tools.wmflabs.org/checkwiki/cgi-bin/checkwiki_bots.cgi?limit=1000&id=002&project=enwiki&action=list
wget
http://tools.wmflabs.org/checkwiki/cgi-bin/checkwiki_bots.cgi?limit=1000&id=003&project=enwiki&action=list

Location with no delay
wget
http://tools.wmflabs.org/checkwiki/cgi-bin/ack.cgi?limit=1000&id=002&project=enwiki&action=list
wget
http://tools.wmflabs.org/checkwiki/cgi-bin/ack.cgi?limit=1000&id=002&project=enwiki&action=list

Location of small html program: (do about 20 calls)
wget http://tools.wmflabs.org/abot/index.html
wget http://tools.wmflabs.org/abot/index.html

Bryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/labs-l/attachments/20130903/89d2fae2/attachment.html>