On 09/08/2011 09:18 AM, Tim Alder wrote:
Hello,
we haved now reached with the re-rendering the point z-curve=0.25.
So we are now rendering the emptiness of north norway. But this has no
visible effect on the rendering speed. The rendering performance
correlate very good with the queue length. I found now the parameter
-num in tirex-batch to limit the queue length. I will restart the
process for tiles with z>0.25 and limit the queue to a value of around
5.000.
I am not sure that the length of the queue is really to blame for the
slowdown. The good correlation is probably more simple because both the
increase of the queue length and the slow down of rendering are
exponential functions with time.
Renderd (rather than tirex), also limits the queue length (in that case
to 1000) metatiles. It's original reason to do this was as far as I
know, due to the fact that queueing and dequeueing were O(n) algorithms
(Simple linear list). So with ever growing lists, this time could have
become significant. By now it is using a better algorithm and this
limitation should no longer be a problem. Although I don't know what
algorithm tirex actually uses for its queues, I suspect it isn't using a
simple linear list. So this shouldn't be a problem.
The other reason why renderd limits its queue is because it uses a
simple fifo (first in first out) rendering strategy. A lot of tiles are
likely to get requested for rerender and then not viewed in a very long
time or ever again. Other tiles however get viewed frequently. With a
fifo strategy without limits, the frequently viewed tile render requests
get stuck behind all those renderings that will not be viewed again.
With a limit, entering the queue becomes probabilistic. If a tile gets
viewed multiple times its chance of entering the queue will increase,
giving somewhat of an effect of a priority queue for frequently viewed
tiles. As tirex has a proper priority queue, the length of the queue
doesn't matter as much.
In neither of the cases should this really affect rendering speed.
I still suspect it has more to do with that the priority queue of tirex
starts loosing the systematicity over time and thus slows it down. In
this case limiting the queue would probably help, because you don't you
constantly inject the tiles fresh, with less opportunity to loose the
systematic ordering.
"vacuum full" is now running more than 1 day. If it's not done tomorrow,
I believe I will stop it. Ok?
If those error messages are indeed coming from database corruption, then
we need to fix the database. Given that the alternatives to a "vacuum
full" (assuming it works, which it may not given the potential
corruption of the database), basically are a full reimport of the
database, which last time took longer than a few days, I think it would
be good to give vacuum full a chance for a little longer.
Greetings Kolossos
_______________________________________________
Maps-l mailing list
Maps-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/maps-l