I used the last rainy weekends to write a little python-tool[1] to bring the rendering processes for each day in a systematically order.
This ordering by geometrical position should reduce our load on harddiscs by reducing random-requests, better usage of disk-cache and preventing out-of-memory (during low-level rendering in different areas of the world at the same time).
The tools goes through all styles, from z=9 to z=15. It checks all metatile-files and looks to access-time and modify-time of each tile.
So the tile must have for re-rendering: *watched in the last 64 days *and must be older than a year (expiring process makes tiles much older). *in an specified area of the earth
This specified area depends on zoomlevel. z=15 will be renderd to 50% so it switch day by day from northern to southern hemissphere. z=14 will be renderd to 25% each day (NW,SW,NE,SE). this going further until z=9 where we render 1.56% and completed the world after 64 days. The system is so that the areas of lower zoomlevels are included in areas of higher zoomlevels. Lower zoomlevels cost us more and changes a harder to see, so I think an update-interval of 2 month for z=9 is acceptable.
At the end, I have a list of tiles that need a rerendering, I order this tiles by z-curve[2] and export it in an format that should be usable for tirex-batch (example: map=default z=15 x=5072 y=11208). This batch-process should starting at beginning of each night (lower server load). z=16-18 would be updated like now on request, because we have no problems there.
The scanning process of the meta-tiles needs on ptolemy nearly 1 hour. Perhaps somebody can check my code. I know that the tiles are not on ptolemy, so my scan is running over the network to willow (afaik), not optimal, but it seems acceptable.
In the moment we render 10.000 tiles in nearly 8hours[3], afterwards we should be much faster.
How can we bring the tool to live-system? Some ideas to optimize the tool/rendering process?
Greetings Kolossos
[1] public-sourcecode: http://toolserver.org/~osm/batchjob-creator/dir-search_source.php place of tool: /home/project/o/s/m/osm/tools/batchjob-creator/
[2]http://en.wikipedia.org/wiki/Z-order_%28curve%29
[3]tail -n -10000 tirex/var/log/jobs.log|less