On 07/25/2011 02:46 PM, Tim Alder wrote:
It's possible to read the reason why expiring was
broken in:
https://jira.toolserver.org/browse/TS-971 (march of this year)
The reason why we don't restart expiring was performance problems, we
had a lot of queries that running longer than 20 minutes for
low-zoomlevels renderings (time-out) and the Queue seems not stable. (We
were too slow.)
I don't know what the parameters for the expiry process were, but the
low zoom levels shouldn't be automatically expired. Z12 or Z11 seem like
a good place to cut off the automatic expiry. This way, the long running
low-zoom tiles don't matter. These can simply periodically be added via
tirex-batch to the background rendering queue.
Does it matter if the rendering queue is not stable? It will eventually
asymptote. At the very latest when all tiles are in the queue. If a long
queue itself causes performance problems (which I don't think it does)
then one could set a maximum queue size (like renderd does) after which
requests get dropped. This way more frequently visited tiles have a hire
chance of making it into the queue, thus adding a certain amount of
"priority queue" towards the most frequently visited tiles.
It will still result in a faster rerendering schedule than no expiry at all.
Now (with the clustering) it seems better and the Queue is going done
today. Tiles on z=9 needs very long (>600s). But it seems acceptable.
It does seem to be getting through the queue in a reasonably rapid pace,
which perhaps indicates that the expiry won't be to bad.
With the tile storage now on a separate set of disks, the actual
touching of meta-tiles for expiry will also no longer interfere with db
access for the rendering process.
I presume you clustered on the geometry index? (I am asking as it is not
entirely clear from
https://jira.toolserver.org/browse/TS-1122).
However, there still seems to be a factor of 5 - 10 times more disk
operations per rendered tile (ignoring non tirex related db ops) than on
osm's tile server and I suspect figuring out why is key to solving the
performance issues.
---
We have now indexes for geometry,hstore and osm-id. If somebody needs
index for name or ref -> please give me a signal.
It seems that we can restart the db updating tomorrow.
If it is not too much trouble, it might be good to update to the latest
osm2pgsql. I added a potential optimization a week or so ago, that may
help performance in the case of slow disks and low node cache hit ratio
(which the update process often is).
May I also suggest to not do a minutely update. If the tiles are only
expired once every 8 month or so, it makes little sense to keep the db
minutely up to date. Hourly, or perhaps even only daily, would probably
help performance both with the expiry process, the rendering, as well as
the db update process.
Kai
----
So or so I will try to optimize the rendering strategy from zoom-level
9-15. see talk-de:
http://lists.openstreetmap.org/pipermail/talk-de/2011-July/087862.html
Greetings Tim alias Kolossos
Am 25.07.2011 17:20, schrieb Kay Drangmeister:
Hi River,
Am 25.07.2011, 16:54 Uhr, schrieb River Tarnell<river.tarnell(a)wikimedia.de>de>:
Is there a particular reason why expiry was
disabled? The load seems
fine at the moment, even though quite a few tiles are being rendered.
I don't know either who disabled it or why it has been disabled.
Probably it had been deactivated in favour of another process...
Kay
_______________________________________________
Maps-l mailing list
Maps-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/maps-l
_______________________________________________
Maps-l mailing list
Maps-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/maps-l