Colin Marquardt schrieb:
2009/11/25 Peter Körner osm-lists@mazdermind.de:
Atm. the load on cassini is around 20 and the diff imports are fighting to stay up. I think it's cmarq's hillshade-generator takes the most resources on cassini. top reports ~57.1% IO waiting so I think there's a lot of disk going on.
This must have gotten worse as generate_tiles goes to lower zoom levels. I can kill it and maybe run with a single thread or even some artificial slowdown tonight. Right now it's three threads, which wasn't a problem so far.
I don't think that this is necessary atm. I just wanted to talk about this upcoming issue. Can you estimate how long it will keep running with the current parameters? Getting some days out of sync is not that big problem (we're not expiring tiles currently anyway).
I tried to do a "du -hs /mnt/user-store/osm_hillshading" but ir ran loooong until I canceled it.
Well, I generate hillshading for the land mass of Europe (plus the UK) right now, so there isn't such a lot of empty sea tiles.
Okay, cool.
It would indeed be nice to generate metatiles instead of the single PNGs though.
Yes, it would. Is there a tool to pack the existing tiles into metatiles?
I'm also not rendering down to zoom 18. Pre-rendering those hillshading tiles (up to a certain zoom level at least) in general is not bad IMO since they are very much static (they don't contain any changing information), so will be useful for many maps for the next several years.
Yes I see that, too, but rendering on the fly and then caching the results is a compromise between disk space and cpu power.
Dynamically creating these tile (and then never expiring them) would also be possible if renderd or mapnik are extended, but right now, only generate_tiles can do the necessary postprocessing.
Both problems together create a real huge number of files in a single directory which is not good for most filesystems. To make things worse, /mnt/user-store is connected to cassini via NFS, so there's another bottleneck.
We could determine what the maximum sensible number of files is in a single directory and from that see up to which zoom level we could support
It's not the absolute "max number" I'm worried about but the decreasing performance when working with a lot of files in a directory (and maybe the inode usage). This could both be solved by using metatiles.
Peter