2009/11/25 Peter Körner <osm-lists(a)mazdermind.de>de>:
Atm. the load on cassini is around 20 and the diff
imports are fighting
to stay up. I think it's cmarq's hillshade-generator takes the most
resources on cassini. top reports ~57.1% IO waiting so I think there's a
lot of disk going on.
This must have gotten worse as generate_tiles goes to lower zoom
levels. I can kill it and maybe run with a single thread or even some
artificial slowdown tonight. Right now it's three threads, which
wasn't a problem so far.
I tried to do a "du -hs
/mnt/user-store/osm_hillshading" but ir ran
loooong until I canceled it.
What I think is going on is that the generate_tiles script creates a
real huge number of single pngs. Unlike mod_tile it is not combining the
small tiles into larger chunks (mod_tile packs 8x8 tiles into a
metatile). Also unless mod_tile it renders each and every tile in each
and every zoomlevel, while mod_tile only renders tiles that are really
shown (e.g. it leaves out most of the atlantic at zoomlevel 18, as
nobody would view the empty sea at this zoomlevel.
Well, I generate hillshading for the land mass of Europe (plus the UK)
right now, so there isn't such a lot of empty sea tiles. It would
indeed be nice to generate metatiles instead of the single PNGs
though.
I'm also not rendering down to zoom 18. Pre-rendering those
hillshading tiles (up to a certain zoom level at least) in general is
not bad IMO since they are very much static (they don't contain any
changing information), so will be useful for many maps for the next
several years.
Dynamically creating these tile (and then never expiring them) would
also be possible if renderd or mapnik are extended, but right now,
only generate_tiles can do the necessary postprocessing.
Both problems together create a real huge number of
files in a single
directory which is not good for most filesystems. To make things worse,
/mnt/user-store is connected to cassini via NFS, so there's another
bottleneck.
We could determine what the maximum sensible number of files is in a
single directory and from that see up to which zoom level we could
support (whether pre-rendered or dynamically created). Maybe mod_tile
could even be extended to handle metatiles with e.g. 16x16?
An alternative would be a special file system, but I suppose that's
rather undesirable...
Cheers
Colin