On 09/08/2011 09:18 AM, Tim Alder wrote:
Hello, we haved now reached with the re-rendering the point z-curve=0.25. So we are now rendering the emptiness of north norway. But this has no visible effect on the rendering speed. The rendering performance correlate very good with the queue length. I found now the parameter -num in tirex-batch to limit the queue length. I will restart the process for tiles with z>0.25 and limit the queue to a value of around 5.000.
I am not sure that the length of the queue is really to blame for the slowdown. The good correlation is probably more simple because both the increase of the queue length and the slow down of rendering are exponential functions with time.
Renderd (rather than tirex), also limits the queue length (in that case to 1000) metatiles. It's original reason to do this was as far as I know, due to the fact that queueing and dequeueing were O(n) algorithms (Simple linear list). So with ever growing lists, this time could have become significant. By now it is using a better algorithm and this limitation should no longer be a problem. Although I don't know what algorithm tirex actually uses for its queues, I suspect it isn't using a simple linear list. So this shouldn't be a problem.
The other reason why renderd limits its queue is because it uses a simple fifo (first in first out) rendering strategy. A lot of tiles are likely to get requested for rerender and then not viewed in a very long time or ever again. Other tiles however get viewed frequently. With a fifo strategy without limits, the frequently viewed tile render requests get stuck behind all those renderings that will not be viewed again. With a limit, entering the queue becomes probabilistic. If a tile gets viewed multiple times its chance of entering the queue will increase, giving somewhat of an effect of a priority queue for frequently viewed tiles. As tirex has a proper priority queue, the length of the queue doesn't matter as much.
In neither of the cases should this really affect rendering speed.
I still suspect it has more to do with that the priority queue of tirex starts loosing the systematicity over time and thus slows it down. In this case limiting the queue would probably help, because you don't you constantly inject the tiles fresh, with less opportunity to loose the systematic ordering.
"vacuum full" is now running more than 1 day. If it's not done tomorrow, I believe I will stop it. Ok?
If those error messages are indeed coming from database corruption, then we need to fix the database. Given that the alternatives to a "vacuum full" (assuming it works, which it may not given the potential corruption of the database), basically are a full reimport of the database, which last time took longer than a few days, I think it would be good to give vacuum full a chance for a little longer.
Greetings Kolossos
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l