This script is included inside geohack, so it's no wonder if a lot of
requests coming with this referrer from different IP's. But if a lot of
requests are coming only from one IP than perhaps a kid want to create
the next big search engine after Google. So it's correct to throttle
them. Perhaps we should also check our robot.txt.
is a special topic, as they have many users they can have a
positive effect for OSM. They still use cached hillshading generated by
us, but there will be no updates necessary in the next time.
The tiles from hikebike-styles can change each minute, and it depends on
the user which delay is acceptable. For a mapper the hikbike map is the
only chance to see own modifications on hiking routes, so a delay of 10
minutes is the maximum. For a normal user a delay of a week or a month
should be no problem.
Greetings Tim alias Kolossos
Am 04.10.2013 23:39, schrieb Marlen Caemmerer:
Hello,
Kai, thank you for your prompt and informative response.
On Thu, 3 Oct 2013, Kai Krueger wrote:
do you have any more details of which tile layers are getting hit? Is it
low or high zoom tiles? What referes / user-agents do they come from? Is
it the tiles that get served through mod_tile, the hillshading tiles or
the tiles for the wiki mini atlas?
I can send you some example log lines of the throttled IPs:
2013/10/04 08:11:30 [error] 28822#0: *53658093 limiting requests,
excess: 55.240 by zone "hikebike", client: 213.73.96.44, server:
toolserver.org, request: "GET /tiles/hikebike/15/17169/11177.png
HTTP/1.1", host: "toolserver.org", referrer:
"http://www.gpsies.com/map.do?fileId=gbcojbhrdfqlglbc"
2013/10/04 08:11:02 [error] 28822#0: *53650597 limiting requests,
excess: 55.120 by zone "hikebike", client: 85.0.37.63, server:
toolserver.org, request: "GET /tiles/
osm/3/4/0.png HTTP/1.1", host: "c.www.toolserver.org", referrer:
"http://toolserver.org/~kolossos/openlayers/embed.html?layer=mapnik&bbox=39.68865498340449,43.5524339
5214844,39.778011016595514,43.614232047851566&marker=43.583333,39.733333
If you want to get more logs I'd send them to you in private.
It seems to relate especially hikebike/cmarq tools.
Too high load from individual clients has been an issue on many other
tileservers as well. Mostly it comes from various mobile apps, that
offer their users to download large areas (e.g. Germany) for offline
use. These areas then cover potentially millions of tiles, that the
clients then try and download as fast as the connection allows.
Sounds plausible.
For that reason, the tileservers on
osm.org have a significant list of
user-agents that they block completely and in addition they also have an
automatic rate limiting per IP. There is also a specific tile usage
policy (
https://wiki.openstreetmap.org/wiki/Tile_usage_policy ) that
gouverns how you are allowed to technically access the tile servers
(once you have it downloaded, the use is freely gouverned by the
CC-BY-SA licence)
Other tileservers like the opencyclemap, equally have restrictions and
mod_tile (the apache module used to deliver tiles) has a number of
features available to limit traffic. mod_tile also has a complex system
to try and ensure maximum cachability of tiles while still ensuring
up-to-dateness. This system can furthermore be tuned either towards
fresshness or cacheability as needed.
My impression was so far this has never been an issue with the
toolserver and I wasn't aware of any explicit policies of how the
toolserver tiles are allowed to be accessed, so I never activated any of
the limiting features. But if it is becoming an issue we can see how
best to compat the issue.
gpsies.com stated they use the cache-control header which is sometimes
not set reasonably probably as far as I tried to see - i had a look at
these hikebike URL delivered from cmarq.
The will look at the problem closer on their side so I expect some more
details in the next days.
I could set cache control headers in the nginx which acts as load
balancer for TS for tiles where it makes sense. Do you have any advices
on this? Which tiles dont change for what time about?
At least on the munin graphs for ptolemy, I don't see much increased
load. But if it is the hillshading tiles, or the WMA tiles, those don't
get served through ptolemy as far as I am aware.
Seems these tiles are delivered via ortelius/wolfsbane. Unfortunatelly
the high load times lead to munin not graphing anymore so I dont have
any accutal data but the error logs/loads via console.
>
> I dont want to have this option configured forever - I rather hope we
> can do something about caching or give the pictures they need to the
> projects themselves (I doubt we have to deliver hill shading pictures
> for everyone - this is Toolserver)
>
Cheers
nosy
_______________________________________________
Toolserver-l mailing list (Toolserver-l(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette