Hello,
Kai, thank you for your prompt and informative response.
On Thu, 3 Oct 2013, Kai Krueger wrote:
do you have any more details of which tile layers are getting hit? Is it low or high zoom tiles? What referes / user-agents do they come from? Is it the tiles that get served through mod_tile, the hillshading tiles or the tiles for the wiki mini atlas?
I can send you some example log lines of the throttled IPs:
2013/10/04 08:11:30 [error] 28822#0: *53658093 limiting requests, excess: 55.240 by zone "hikebike", client: 213.73.96.44, server: toolserver.org, request: "GET /tiles/hikebike/15/17169/11177.png HTTP/1.1", host: "toolserver.org", referrer: "http://www.gpsies.com/map.do?fileId=gbcojbhrdfqlglbc"
2013/10/04 08:11:02 [error] 28822#0: *53650597 limiting requests, excess: 55.120 by zone "hikebike", client: 85.0.37.63, server: toolserver.org, request: "GET /tiles/ osm/3/4/0.png HTTP/1.1", host: "c.www.toolserver.org", referrer: "http://toolserver.org/~kolossos/openlayers/embed.html?layer=mapnik&bbox=... 5214844,39.778011016595514,43.614232047851566&marker=43.583333,39.733333
If you want to get more logs I'd send them to you in private.
It seems to relate especially hikebike/cmarq tools.
Too high load from individual clients has been an issue on many other tileservers as well. Mostly it comes from various mobile apps, that offer their users to download large areas (e.g. Germany) for offline use. These areas then cover potentially millions of tiles, that the clients then try and download as fast as the connection allows.
Sounds plausible.
For that reason, the tileservers on osm.org have a significant list of user-agents that they block completely and in addition they also have an automatic rate limiting per IP. There is also a specific tile usage policy ( https://wiki.openstreetmap.org/wiki/Tile_usage_policy ) that gouverns how you are allowed to technically access the tile servers (once you have it downloaded, the use is freely gouverned by the CC-BY-SA licence)
Other tileservers like the opencyclemap, equally have restrictions and mod_tile (the apache module used to deliver tiles) has a number of features available to limit traffic. mod_tile also has a complex system to try and ensure maximum cachability of tiles while still ensuring up-to-dateness. This system can furthermore be tuned either towards fresshness or cacheability as needed.
My impression was so far this has never been an issue with the toolserver and I wasn't aware of any explicit policies of how the toolserver tiles are allowed to be accessed, so I never activated any of the limiting features. But if it is becoming an issue we can see how best to compat the issue.
gpsies.com stated they use the cache-control header which is sometimes not set reasonably probably as far as I tried to see - i had a look at these hikebike URL delivered from cmarq. The will look at the problem closer on their side so I expect some more details in the next days.
I could set cache control headers in the nginx which acts as load balancer for TS for tiles where it makes sense. Do you have any advices on this? Which tiles dont change for what time about?
At least on the munin graphs for ptolemy, I don't see much increased load. But if it is the hillshading tiles, or the WMA tiles, those don't get served through ptolemy as far as I am aware.
Seems these tiles are delivered via ortelius/wolfsbane. Unfortunatelly the high load times lead to munin not graphing anymore so I dont have any accutal data but the error logs/loads via console.
I dont want to have this option configured forever - I rather hope we can do something about caching or give the pictures they need to the projects themselves (I doubt we have to deliver hill shading pictures for everyone - this is Toolserver)
Cheers nosy