"Scot Wilcoxon" scot@wilcoxon.org wrote:
The way I have my bots avoid irritating the servers is to back off very quickly, and slowly speed back up. Starting with a minimum of 5 seconds between updates... If there is no reply after 30 seconds, the bot sleeps for 60 seconds before retrying. Another failure increases the sleep time to 120 seconds. A successful update decreases the sleep time by 10 seconds, until the minimum 5 seconds is reached. So a slow period can slow the bot to several minutes between attempts, and it then takes many minutes to return to normal.
On Thu, 2011-03-03 at 23:44 -0500, richardcavell@mail.com wrote:
Hi all and TIA.
I usually run my bot with maxlag = 5 seconds.
When it attempts a communication with Wikipedia, if it receives Error:maxlag or HTTP 503 , it retries up to 5 times with 10 seconds between each try. If it still can't get through, it aborts. At the time I'm writing this, about 3pm Friday afternoon in Australia, my bot is routinely aborting due to 5 maxlag or 503s in a row.
The algorithm I've used in working on pywikipediabot is, upon receiving a maxlag error, to have the bot sleep for min(5, lag / 2) seconds, where 'lag' is the lag reported in the error message. If there is a long replication lag, then it is going to take longer for the database slaves to catch up than with a short lag, so there is no point in rechecking every 5 seconds. Aborting after 5 maxlag errors is purely a design choice; if you don't mind having your bot wait minutes or hours for the lag to subside, you can do that too.
Russ