Colin Marquardt wrote:
2009/11/25 Peter Körner
<osm-lists(a)mazdermind.de>de>:
Atm. the load on cassini is around 20 and the
diff imports are fighting
to stay up. I think it's cmarq's hillshade-generator takes the most
resources on cassini. top reports ~57.1% IO waiting so I think there's a
lot of disk going on.
Load is now back to ~4 without me changing anything. Maybe it was some
interaction with other processes?
Just looking at the munin graphs, (i.e. not looking at more detailed
stats), my guess would be that it is probably not the hillshading , but
rather the standard mod_tile /renderd setup that is causing the load
spikes. Those load spikes correlate quite strongly with renderd queue
lengths and rendering times. Currently I think renderd is set to use 8
threads. So if there is a render queue, all 8 threads will try to run,
each connecting to a separate postgres process. Therefore, there are
about 16 processes/threads that try to run and thus depending on how
much of the wait time gets accounted to load, a load of 20 doesn't sound
all that unreasonable.
I think the problem is that the current setup is too slow to render low
zooms, with single metatiles taking partly over a minute to render. This
is therefore clearly not feasible to render on the fly. However, with
the 200+ styles of cassini, the fraction of low zoom tiles to high zoom
tiles is skewed in the wrong direction. Once you get down to zoom level
12 or so, it starts just about being fast enough to render on the fly.
I think it is therefore necessary to pre-render all tiles to zoom level
of probably at least 9 - 10. One can use render_list to pre-render in
the background through renderd, thus playing nicely with the mod_tiles
on-the-fly rendering. It will probably also be necessary to not expire
the low zoom tiles as frequently to not need to rerender those as much.
The OSM mapnik server basically doesn't expire lowzoom tiles at all,
other than after a full DB reimport. Given that probably most of the
name localisation still happens on large features like country / major
city names, the low zoom tiles probably change much more though. So
perhaps the the lowzoom tiles can be bunched up and only expired once a
day and then immediately pre-rendered again. render_old can probably
handle this. It goes through all of the previously rendered metatiles
and checks if they are marked dirty. If yes, it submits them for
background rendering.
It might still make sense though to convert the hillshading tiles to
metatiles though. It might be that the program convert_meta allows to
convert individual pngs into metatiles. But I haven't used convert_meta
so I don't know if it actually does that.
Kai
Cheers
Colin
_______________________________________________
Maps-l mailing list
Maps-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/maps-l