Hello,
in the last days Toolserver experienced outages of web pages which were caused by too many
queries from only a few hosts.
They are using OSM images and - please dont ask me why - single IPs tend to query about
40-50 pictures per second - for minutes or hours, peaks can be worse.
At some points our web server give up then.
Yes sorry ;). I can proudly say that only today about 11.7 millions web queries were
answered somehow.
I tried to mitigate the problem of "too many requests per IP" via blocking but
it is not an option.
One problem is that users of at least one portal then complain and another is that the IP
addresses seem random - coming even from dial up ranges.
There might be something badly wrong with cache-control headers for the images (or
probably we can tweak at that point) or - I dont know what it could be.
To make the long story short - I rate limited the OSM tile delivery to 40 images per
second per IP - allowed burst is 55.
Users will then get a 503 error if the rate exceeds until it decreases - but delivery isnt
stopped completely.
It seems to work since I have some notices which IPs were throtteled and these are IPs
that have heavy usage.
I used this here to throttle:
http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
I dont want to have this option configured forever - I rather hope we can do something
about caching or give the pictures they need to the projects themselves (I doubt we have
to deliver hill shading pictures for everyone - this is Toolserver)
If anyone has an idea what to do / questions - please let me know.
Cheers
Marlen/nosy