I used the last rainy weekends to write a little python-tool[1] to bring
the rendering processes for each day in a systematically order.
This ordering by geometrical position should reduce our load on
harddiscs by reducing random-requests, better usage of disk-cache and
preventing out-of-memory (during low-level rendering in different areas
of the world at the same time).
The tools goes through all styles, from z=9 to z=15. It checks all
metatile-files and looks to access-time and modify-time of each tile.
So the tile must have for re-rendering:
*watched in the last 64 days
*and must be older than a year (expiring process makes tiles much older).
*in an specified area of the earth
This specified area depends on zoomlevel. z=15 will be renderd to 50% so
it switch day by day from northern to southern hemissphere. z=14 will be
renderd to 25% each day (NW,SW,NE,SE). this going further until z=9
where we render 1.56% and completed the world after 64 days. The system
is so that the areas of lower zoomlevels are included in areas of higher
zoomlevels. Lower zoomlevels cost us more and changes a harder to see,
so I think an update-interval of 2 month for z=9 is acceptable.
At the end, I have a list of tiles that need a rerendering, I order this
tiles by z-curve[2] and export it in an format that should be usable for
tirex-batch (example: map=default z=15 x=5072 y=11208).
This batch-process should starting at beginning of each night (lower
server load). z=16-18 would be updated like now on request, because we
have no problems there.
The scanning process of the meta-tiles needs on ptolemy nearly 1 hour.
Perhaps somebody can check my code. I know that the tiles are not on
ptolemy, so my scan is running over the network to willow (afaik), not
optimal, but it seems acceptable.
In the moment we render 10.000 tiles in nearly 8hours[3], afterwards we
should be much faster.
How can we bring the tool to live-system?
Some ideas to optimize the tool/rendering process?
Greetings Kolossos
[1] public-sourcecode:
http://toolserver.org/~osm/batchjob-creator/dir-search_source.php
place of tool: /home/project/o/s/m/osm/tools/batchjob-creator/
[
2]http://en.wikipedia.org/wiki/Z-order_%28curve%29
[3]tail -n -10000 tirex/var/log/jobs.log|less