Kai, thank you for your prompt and informative response.
On Thu, 3 Oct 2013, Kai Krueger wrote:
do you have any more details of which tile layers are getting hit? Is it
low or high zoom tiles? What referes / user-agents do they come from? Is
it the tiles that get served through mod_tile, the hillshading tiles or
the tiles for the wiki mini atlas?
I can send you some example log lines of the throttled IPs:
2013/10/04 08:11:30 [error] 28822#0: *53658093 limiting requests, excess: 55.240 by zone
"hikebike", client: 18.104.22.168, server: toolserver.org
, request: "GET
/tiles/hikebike/15/17169/11177.png HTTP/1.1", host: "toolserver.org",
2013/10/04 08:11:02 [error] 28822#0: *53650597 limiting requests, excess: 55.120 by zone
"hikebike", client: 22.214.171.124, server: toolserver.org
, request: "GET
osm/3/4/0.png HTTP/1.1", host: "c.www.toolserver.org", referrer:
If you want to get more logs I'd send them to you in private.
It seems to relate especially hikebike/cmarq tools.
Too high load from individual clients has been an issue on many other
tileservers as well. Mostly it comes from various mobile apps, that
offer their users to download large areas (e.g. Germany) for offline
use. These areas then cover potentially millions of tiles, that the
clients then try and download as fast as the connection allows.
For that reason, the tileservers on osm.org
have a significant list of
user-agents that they block completely and in addition they also have an
automatic rate limiting per IP. There is also a specific tile usage
policy ( https://wiki.openstreetmap.org/wiki/Tile_usage_policy
gouverns how you are allowed to technically access the tile servers
(once you have it downloaded, the use is freely gouverned by the
Other tileservers like the opencyclemap, equally have restrictions and
mod_tile (the apache module used to deliver tiles) has a number of
features available to limit traffic. mod_tile also has a complex system
to try and ensure maximum cachability of tiles while still ensuring
up-to-dateness. This system can furthermore be tuned either towards
fresshness or cacheability as needed.
My impression was so far this has never been an issue with the
toolserver and I wasn't aware of any explicit policies of how the
toolserver tiles are allowed to be accessed, so I never activated any of
the limiting features. But if it is becoming an issue we can see how
best to compat the issue.
stated they use the cache-control header which is sometimes not set reasonably
probably as far as I tried to see - i had a look at these hikebike URL delivered from
The will look at the problem closer on their side so I expect some more details in the
I could set cache control headers in the nginx which acts as load balancer for TS for
tiles where it makes sense.
Do you have any advices on this? Which tiles dont change for what time about?
At least on the munin graphs for ptolemy, I don't see much increased
load. But if it is the hillshading tiles, or the WMA tiles, those don't
get served through ptolemy as far as I am aware.
Seems these tiles are delivered via ortelius/wolfsbane. Unfortunatelly the high load times
lead to munin not graphing anymore so I dont have any accutal data but the error
logs/loads via console.
> I dont want to have this option configured forever - I rather hope we
> can do something about caching or give the pictures they need to the
> projects themselves (I doubt we have to deliver hill shading pictures
> for everyone - this is Toolserver)