tool for systematically rendering - Maps-l

6 Aug 2011


      I used the last rainy weekends to write a little python-tool[1] to bring 
the rendering processes for each day in a systematically order.
This ordering by geometrical position should reduce our load on 
harddiscs by reducing random-requests, better usage of disk-cache and 
preventing out-of-memory (during low-level rendering in different areas 
of the world at the same time).
The tools goes through all styles, from z=9 to z=15. It checks all 
metatile-files and looks to access-time and modify-time of each tile.
So the tile must have for re-rendering:
*watched in the last 64 days
*and must be older than a year (expiring process makes tiles much older).
*in an specified area of the earth
This specified area depends on zoomlevel. z=15 will be renderd to 50% so 
it switch day by day from northern to southern hemissphere. z=14 will be 
renderd to 25% each day (NW,SW,NE,SE). this going further until z=9 
where we render 1.56% and completed the world after 64 days. The system 
is so that the areas of lower zoomlevels are included in areas of higher 
zoomlevels. Lower zoomlevels cost us more and changes a harder to see, 
so I think an update-interval of 2 month for z=9 is acceptable.
At the end, I have a list of tiles that need a rerendering, I order this 
tiles by z-curve[2] and export it in an format that should be usable for 
tirex-batch (example: map=default z=15 x=5072 y=11208).
This batch-process should starting at beginning of each night (lower 
server load). z=16-18 would be updated like now on request, because we 
have no problems there.
The scanning process of the meta-tiles needs on ptolemy nearly 1 hour.
Perhaps somebody can check my code. I know that the tiles are not on 
ptolemy, so my scan is running over the network to willow (afaik), not 
optimal, but it seems acceptable.
In the moment we render 10.000 tiles in nearly 8hours[3], afterwards we 
should be much faster.
How can we bring the tool to live-system?
Some ideas to optimize the tool/rendering process?
Greetings Kolossos
[1] public-sourcecode:
http://toolserver.org/~osm/batchjob-creator/dir-search_source.php
place of tool: /home/project/o/s/m/osm/tools/batchjob-creator/
[2]http://en.wikipedia.org/wiki/Z-order_%28curve%29
[3]tail -n -10000  tirex/var/log/jobs.log|less