Hi,
Looking at the current ptolemy queue size
http://toolserver.org/~mazder/tirex-status/?short=1&extended=0&refre...
and
http://munin.toolserver.org/OSM/ptolemy/tirex_status_queued_requests.html
It seems like tile expiry has been turned on again, as on the one hand, tiles _are_ rendered correctly but on the other hand, the queue is being filled continuously, currently at a rate slightly higher than the rendering capacity.
Kind regards, Kay
Hi
Seems I can partly answer my own question:
Am 24.07.2011 20:08, schrieb Kay Drangmeister:
http://munin.toolserver.org/OSM/ptolemy/tirex_status_queued_requests.html It seems like tile expiry has been turned on again
See https://jira.toolserver.org/browse/TS-1112 Is my assumption right that tile expiry is _not_ active after the re-render of everything?
Ciao, Kay
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Kay Drangmeister:
Is my assumption right that tile expiry is _not_ active after the re-render of everything?
AFAIK, tile expiry was disabled before, and I haven't enabled it, so once a few more tiles have been rendered, the render load will drop back to nothing again.
Is there a particular reason why expiry was disabled? The load seems fine at the moment, even though quite a few tiles are being rendered.
- river.
Hi River,
Am 25.07.2011, 16:54 Uhr, schrieb River Tarnell river.tarnell@wikimedia.de:
Is there a particular reason why expiry was disabled? The load seems fine at the moment, even though quite a few tiles are being rendered.
I don't know either who disabled it or why it has been disabled. Probably it had been deactivated in favour of another process...
Kay
It's possible to read the reason why expiring was broken in: https://jira.toolserver.org/browse/TS-971 (march of this year)
The reason why we don't restart expiring was performance problems, we had a lot of queries that running longer than 20 minutes for low-zoomlevels renderings (time-out) and the Queue seems not stable. (We were too slow.)
Now (with the clustering) it seems better and the Queue is going done today. Tiles on z=9 needs very long (>600s). But it seems acceptable. --- We have now indexes for geometry,hstore and osm-id. If somebody needs index for name or ref -> please give me a signal.
It seems that we can restart the db updating tomorrow. ---- So or so I will try to optimize the rendering strategy from zoom-level 9-15. see talk-de: http://lists.openstreetmap.org/pipermail/talk-de/2011-July/087862.html
Greetings Tim alias Kolossos
Am 25.07.2011 17:20, schrieb Kay Drangmeister:
Hi River,
Am 25.07.2011, 16:54 Uhr, schrieb River Tarnellriver.tarnell@wikimedia.de:
Is there a particular reason why expiry was disabled? The load seems fine at the moment, even though quite a few tiles are being rendered.
I don't know either who disabled it or why it has been disabled. Probably it had been deactivated in favour of another process...
Kay
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
On 07/25/2011 02:46 PM, Tim Alder wrote:
It's possible to read the reason why expiring was broken in: https://jira.toolserver.org/browse/TS-971 (march of this year)
The reason why we don't restart expiring was performance problems, we had a lot of queries that running longer than 20 minutes for low-zoomlevels renderings (time-out) and the Queue seems not stable. (We were too slow.)
I don't know what the parameters for the expiry process were, but the low zoom levels shouldn't be automatically expired. Z12 or Z11 seem like a good place to cut off the automatic expiry. This way, the long running low-zoom tiles don't matter. These can simply periodically be added via tirex-batch to the background rendering queue.
Does it matter if the rendering queue is not stable? It will eventually asymptote. At the very latest when all tiles are in the queue. If a long queue itself causes performance problems (which I don't think it does) then one could set a maximum queue size (like renderd does) after which requests get dropped. This way more frequently visited tiles have a hire chance of making it into the queue, thus adding a certain amount of "priority queue" towards the most frequently visited tiles.
It will still result in a faster rerendering schedule than no expiry at all.
Now (with the clustering) it seems better and the Queue is going done today. Tiles on z=9 needs very long (>600s). But it seems acceptable.
It does seem to be getting through the queue in a reasonably rapid pace, which perhaps indicates that the expiry won't be to bad.
With the tile storage now on a separate set of disks, the actual touching of meta-tiles for expiry will also no longer interfere with db access for the rendering process.
I presume you clustered on the geometry index? (I am asking as it is not entirely clear from https://jira.toolserver.org/browse/TS-1122).
However, there still seems to be a factor of 5 - 10 times more disk operations per rendered tile (ignoring non tirex related db ops) than on osm's tile server and I suspect figuring out why is key to solving the performance issues.
We have now indexes for geometry,hstore and osm-id. If somebody needs index for name or ref -> please give me a signal.
It seems that we can restart the db updating tomorrow.
If it is not too much trouble, it might be good to update to the latest osm2pgsql. I added a potential optimization a week or so ago, that may help performance in the case of slow disks and low node cache hit ratio (which the update process often is).
May I also suggest to not do a minutely update. If the tiles are only expired once every 8 month or so, it makes little sense to keep the db minutely up to date. Hourly, or perhaps even only daily, would probably help performance both with the expiry process, the rendering, as well as the db update process.
Kai
So or so I will try to optimize the rendering strategy from zoom-level 9-15. see talk-de: http://lists.openstreetmap.org/pipermail/talk-de/2011-July/087862.html
Greetings Tim alias Kolossos
Am 25.07.2011 17:20, schrieb Kay Drangmeister:
Hi River,
Am 25.07.2011, 16:54 Uhr, schrieb River Tarnellriver.tarnell@wikimedia.de:
Is there a particular reason why expiry was disabled? The load seems fine at the moment, even though quite a few tiles are being rendered.
I don't know either who disabled it or why it has been disabled. Probably it had been deactivated in favour of another process...
Kay
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Am 26.07.2011 03:48, schrieb Kai Krueger:
May I also suggest to not do a minutely update. If the tiles are only expired once every 8 month or so
I really would like to see changes instantly (as most mappers do), especially when developing or modifying styles.
I think the long tile expiry holds only for the low zoom tiles, high zoom tiles are expired instantly, if I'm not mistaken.
Kind regards, Kay