Hello everyone,
Over the past month, the server has been busy trying to re-render all of the tiles to try to ensure more up-to date maps. This process is more or less complete now and all of the map tiles should then be at least from the 1st of December 2011.
As one month old tiles is still not overly satisfying, I am now trying to re-activate diff based tile expiry again.
The full expiry of all styles remains infeasible for the moment and so I am trying to introduce expiry slowly to see how far it can be pushed.
As a start, I have activated tile expiry only for the highest zoom levels (z15 - z18) and then only for the styles that currently are embedded in one of the wikipedias.
As far as I can tell these are the following styles:
osm-no-labels osm-labels-de osm-labels-ca osm-labels-bg osm-labels-da osm-labels-el osm-labels-es osm-labels-fa osm-labels-fr osm-labels-it osm-labels-lv osm-labels-nl osm-labels-no osm-labels-pl osm-labels-ru germany default hikebike
If anyone knows of a language that I have forgotten, please let me know and I will include it as well.
So far, it looks like the server can cope with this, although I have only activated it earlier today. Should it remain this way, I will try and expand the zoom levels.
As it is critical to reduce the number of low zoom tiles rendered, as those still kill performance, lower zoom tiles will probably not be expired as frequently. E.g. perhaps Z14 - Z11 on a daily basis, below that even only in the background.
We will have to see, though, how performance behaves over time.
Happy new year,
Kai
Hi all,
Happy new year!
Am 01.01.2012, 23:58 Uhr, schrieb Kai Krueger kakrueger@gmail.com:
Over the past month, the server has been busy trying to re-render all of the tiles to try to ensure more up-to date maps. This process is more or less complete now and all of the map tiles should then be at least from the 1st of December 2011.
While it does not matter much, here's a still old one: http://a.www.toolserver.org/tiles/parking-bw/17/69173/44572.png/status So I hope it's just not finished yet, but not broken :-)
As one month old tiles is still not overly satisfying, I am now trying to re-activate diff based tile expiry again.
Off-topic question: is space also a rare resource? I.e. is it needed to delete old tiles (e.g. ones that have been rendered once but are never requested again since)?
I have activated tile expiry [...] osm-labels-lv osm-labels-nl osm-labels-no osm-labels-pl osm-labels-ru germany default hikebike
Please excuse my ignorance, but what is "default"? Is it the same as "osm"?
Cheers, Kay
Hi,
On 01/02/2012 12:33 PM, Kay Drangmeister wrote:
Hi all,
Happy new year!
Am 01.01.2012, 23:58 Uhr, schrieb Kai Krueger kakrueger@gmail.com:
Over the past month, the server has been busy trying to re-render all of the tiles to try to ensure more up-to date maps. This process is more or less complete now and all of the map tiles should then be at least from the 1st of December 2011.
While it does not matter much, here's a still old one: http://a.www.toolserver.org/tiles/parking-bw/17/69173/44572.png/status So I hope it's just not finished yet, but not broken :-)
A bit of both I think... ;-). It seems it is less finished than I originally thought. It is now already on the third pass through with render_old, which should render all of the old tiles.
However, Tirex still needs to be regularly restarted due to memory leaks, and in the past to tweak some settings. In the time it needs to restart, the requests get dropped.
There also still seem to be occasionally issues with socket errors, such that some requests again get dropped.
By the looks more tiles got missed on the first and second pass than I had expected, so we'll see how long the third pass still takes.
As one month old tiles is still not overly satisfying, I am now trying to re-activate diff based tile expiry again.
Off-topic question: is space also a rare resource? I.e. is it needed to delete old tiles (e.g. ones that have been rendered once but are never requested again since)?
Yes, space is an issue.
The tiles are on a large 5Tb NFS array that is shared between all toolserver users. Exactly how much can be used for osm tiles is a little flexible I guess, but of cause one needs to be considerate with other users and in total only about 150Gb of space was left. Therefore DaB has started to delete old tiles yesterday to free up some space. ( https://jira.toolserver.org/browse/MNT-1168 )
I let a du run a couple of days ago (before the deleting of old tiles) and the result was the following:
377G /tiles/ 165G /tiles/default 134G /tiles/hikebike 31G /tiles/osm-no-labels 9.4G /tiles/bw-mapnik 6.5G /tiles/germany 5.6G /tiles/parking-bw 5.4G /tiles/bw-noicons 3.5G /tiles/osm-labels-de 3.3G /tiles/osm-locale-de 1.9G /tiles/osm-labels-ru 1.9G /tiles/osm-labels-fr 1.2G /tiles/osm-locale-hsb 909M /tiles/osm-labels-pl 868M /tiles/osm-labels-it 835M /tiles/osm-labels-en 728M /tiles/osm-labels-es 722M /tiles/osm-locale-zh 621M /tiles/bicycle 586M /tiles/osm-labels-nl 391M /tiles/shape-names 347M /tiles/osm-labels-da 307M /tiles/lighting 272M /tiles/powermap 248M /tiles/osm-labels-no 185M /tiles/osm-labels-fa 173M /tiles/osm-labels-ca 99M /tiles/parktrans 92M /tiles/osm-labels-ro 86M /tiles/mtb-overlay 83M /tiles/bicycle_network 74M /tiles/osm-labels-he 67M /tiles/osm-labels-sv 66M /tiles/osm-labels-et 61M /tiles/osm-labels-gsw 56M /tiles/osm-labels-zh 52M /tiles/surveillance 52M /tiles/osm-labels-lv
Other styles are less than 50Mb each currently.
I have activated tile expiry [...] osm-labels-lv osm-labels-nl osm-labels-no osm-labels-pl osm-labels-ru germany default hikebike
Please excuse my ignorance, but what is "default"? Is it the same as "osm"?
Yes, that is the name of the "default" mapnik style used on osm.org. It is just called default for legacy reasons really.
Kai
Cheers, Kay
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Hello, please have in mind that we take the osm-labels-lang styles in wikipedia only for zoom 0-6. For the more interesting zoom-levels we take only default style for performance reasons. The only exeption is the small hsb.wp ("Obersorbisch").
People from ru.wp ask me to support russian but I did affraid that it will be too much load for us. But now is russian map so popular that it is perhaps usefull, we should use pre-rendering for it.
|Kay: Please excuse my ignorance, but what is "default"? Is it the same as |"osm"? Yes.
Greetings Kolossos
Zitat von Kai Krueger kakrueger@gmail.com:
Hello everyone,
Over the past month, the server has been busy trying to re-render all of the tiles to try to ensure more up-to date maps. This process is more or less complete now and all of the map tiles should then be at least from the 1st of December 2011.
As one month old tiles is still not overly satisfying, I am now trying to re-activate diff based tile expiry again.
The full expiry of all styles remains infeasible for the moment and so I am trying to introduce expiry slowly to see how far it can be pushed.
As a start, I have activated tile expiry only for the highest zoom levels (z15 - z18) and then only for the styles that currently are embedded in one of the wikipedias.
As far as I can tell these are the following styles:
osm-no-labels osm-labels-de osm-labels-ca osm-labels-bg osm-labels-da osm-labels-el osm-labels-es osm-labels-fa osm-labels-fr osm-labels-it osm-labels-lv osm-labels-nl osm-labels-no osm-labels-pl osm-labels-ru germany default hikebike
If anyone knows of a language that I have forgotten, please let me know and I will include it as well.
So far, it looks like the server can cope with this, although I have only activated it earlier today. Should it remain this way, I will try and expand the zoom levels.
As it is critical to reduce the number of low zoom tiles rendered, as those still kill performance, lower zoom tiles will probably not be expired as frequently. E.g. perhaps Z14 - Z11 on a daily basis, below that even only in the background.
We will have to see, though, how performance behaves over time.
Happy new year,
Kai
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Hi,
Am 02.01.2012, 13:55 Uhr, schrieb Kai Krueger kakrueger@gmail.com:
I let a du run a couple of days ago (before the deleting of old tiles) and the result was the following:
377G /tiles/ 165G /tiles/default
Please excuse my ignorance, but what is "default"? Is it the same as "osm"?
Yes, that is the name of the "default" mapnik style used on osm.org. It is just called default for legacy reasons really.
Obvious follow-up question given "default" is the most space and presumably also the most time consuming style: why are we duplicating the original osm tiles in the first place? As a backup, or for keeping load off the osm.org server?
Kind regards, Kay
On Mon, Jan 2, 2012 at 9:06 PM, Kay Drangmeister kay@drangmeister.net wrote:
Obvious follow-up question given "default" is the most space and presumably also the most time consuming style: why are we duplicating the original osm tiles in the first place? As a backup, or for keeping load off the osm.org server?
The OSM tile server cannot handle the amount of traffic that Wikipedia would generate, so yes, it's to keep load off of the OSM server.
On 01/02/2012 01:57 PM, Tim Alder wrote:
Hello, please have in mind that we take the osm-labels-lang styles in wikipedia only for zoom 0-6. For the more interesting zoom-levels we take only default style for performance reasons. The only exeption is the small hsb.wp ("Obersorbisch").
Interesting.
At least according to my local proxy logs, it loads the language tiles in the german wikipedia at z15 as well.
I am not sure if it only displays the default style, but it is loading the other ones, too.
It first loads tiles for "osm-no-labels" then "osm-labels-de" and finally "default". I have seen that for all of the different coordinate types, some of which load at z11, some at z13 and some at z15.
People from ru.wp ask me to support russian but I did affraid that it will be too much load for us. But now is russian map so popular that it is perhaps usefull, we should use pre-rendering for it.
Is the load really that much? The maps displayed in wikipedia are fairly small and (spatially) localized around where the articles are. So that is probably only a small subset of tiles that would be needed for a general purpose slippy map.
Perhaps it would be good to analyze the apache logs and check the referrers and user-agent to see how much traffic really comes from which wikipedia, and how much comes from other sites like hikebikemap.de or from e.g. smartphone apps using the toolserver as a source for tiles.
Kai
|Kay: Please excuse my ignorance, but what is "default"? Is it the same as |"osm"? Yes.
Greetings Kolossos
Zitat von Kai Krueger kakrueger@gmail.com:
Hello everyone,
Over the past month, the server has been busy trying to re-render all of the tiles to try to ensure more up-to date maps. This process is more or less complete now and all of the map tiles should then be at least from the 1st of December 2011.
As one month old tiles is still not overly satisfying, I am now trying to re-activate diff based tile expiry again.
The full expiry of all styles remains infeasible for the moment and so I am trying to introduce expiry slowly to see how far it can be pushed.
As a start, I have activated tile expiry only for the highest zoom levels (z15 - z18) and then only for the styles that currently are embedded in one of the wikipedias.
As far as I can tell these are the following styles:
osm-no-labels osm-labels-de osm-labels-ca osm-labels-bg osm-labels-da osm-labels-el osm-labels-es osm-labels-fa osm-labels-fr osm-labels-it osm-labels-lv osm-labels-nl osm-labels-no osm-labels-pl osm-labels-ru germany default hikebike
If anyone knows of a language that I have forgotten, please let me know and I will include it as well.
So far, it looks like the server can cope with this, although I have only activated it earlier today. Should it remain this way, I will try and expand the zoom levels.
As it is critical to reduce the number of low zoom tiles rendered, as those still kill performance, lower zoom tiles will probably not be expired as frequently. E.g. perhaps Z14 - Z11 on a daily basis, below that even only in the background.
We will have to see, though, how performance behaves over time.
Happy new year,
Kai
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Am 02.01.2012 17:14, schrieb Kai Krueger: ...
Interesting.
At least according to my local proxy logs, it loads the language tiles in the german wikipedia at z15 as well.
It first loads tiles for "osm-no-labels" then "osm-labels-de" and finally "default". I have seen that for all of the different coordinate types, some of which load at z11, some at z13 and some at z15.
On which page you look: *http://toolserver.org/~osm/locale/de.html or *http://de.wikipedia.org/wiki/Hilfe:OpenStreetMap ?
First page contains off-course language tiles, but is not a promoted page. The second page (included in Wikipedia) works correctly for me with default-tiles if I look in my network-traffic in Firebug.
Greetings Kolossos
On 01/02/2012 07:16 PM, Tim Alder wrote:
Am 02.01.2012 17:14, schrieb Kai Krueger: ...
Interesting.
At least according to my local proxy logs, it loads the language tiles in the german wikipedia at z15 as well.
It first loads tiles for "osm-no-labels" then "osm-labels-de" and finally "default". I have seen that for all of the different coordinate types, some of which load at z11, some at z13 and some at z15.
On which page you look: *http://toolserver.org/~osm/locale/de.html or *http://de.wikipedia.org/wiki/Hilfe:OpenStreetMap ?
First page contains off-course language tiles, but is not a promoted page. The second page (included in Wikipedia) works correctly for me with default-tiles if I look in my network-traffic in Firebug.
I went to e.g. http://de.wikipedia.org/wiki/Statue_of_Liberty as a non-logged in user and clicked on "Karte" in the top right corner.
According to my local squid logs, it loads all three styles (although if you repeat it, it will be cached in firefox and not load again).
Kai
Greetings Kolossos
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Yes, you have right. The problem is only at the beginning, so I and nobody else see it before.
So there is a bug but I can't find it. I experiment this evening with the "isVisible" and "isBaseLayer"-Parameter of the layers in the init-function but it doesn't help.
So perhaps there is a problem with the toolserver-class or I have tomatoes on my eyes.
Other Websites doesn't show this effect: http://openlayers-buch.de/beispiele/chapter-06/rudimentary_hello_osm.html
Can somebody help? Thanks Kolossos
Am 02.01.2012 19:20, schrieb Kai Krueger:
On 01/02/2012 07:16 PM, Tim Alder wrote:
Am 02.01.2012 17:14, schrieb Kai Krueger: ...
Interesting.
At least according to my local proxy logs, it loads the language tiles in the german wikipedia at z15 as well.
It first loads tiles for "osm-no-labels" then "osm-labels-de" and finally "default". I have seen that for all of the different coordinate types, some of which load at z11, some at z13 and some at z15.
On which page you look: *http://toolserver.org/~osm/locale/de.html or *http://de.wikipedia.org/wiki/Hilfe:OpenStreetMap ?
First page contains off-course language tiles, but is not a promoted page. The second page (included in Wikipedia) works correctly for me with default-tiles if I look in my network-traffic in Firebug.
I went to e.g. http://de.wikipedia.org/wiki/Statue_of_Liberty as a non-logged in user and clicked on "Karte" in the top right corner.
According to my local squid logs, it loads all three styles (although if you repeat it, it will be cached in firefox and not load again).
Kai
Greetings Kolossos
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
On 01/02/2012 01:57 PM, Tim Alder wrote:
Hello, please have in mind that we take the osm-labels-lang styles in wikipedia only for zoom 0-6. For the more interesting zoom-levels we take only default style for performance reasons. The only exeption is the small hsb.wp ("Obersorbisch").
If the language labels are only used for zoom 0-6, then there is not much point in expiring them at z15 - 18. I might as well disable expiry and move over to a background re-render for everything but the default style.
As zoom 0 - 10 tiles can take many minutes per tile, those zoom levels would always be redered in the background anyway.
So far the server seems to be able to cope reasonably well with expiry of z15 - z18. I will start expanding expiry further to try and get up to expiring z11, which is the lowest zoom that is shown by default in wikipedia.
People from ru.wp ask me to support russian but I did affraid that it will be too much load for us. But now is russian map so popular that it is perhaps usefull, we should use pre-rendering for it.
We should probably test it to see how it goes. Who would be the people on the russian wikipedia to talk to, to see what they need? Also, should any changes take effect, who can tel us if everything is working out and if the changes are positively received?
Kai
|Kay: Please excuse my ignorance, but what is "default"? Is it the same as |"osm"? Yes.
Greetings Kolossos
Zitat von Kai Krueger kakrueger@gmail.com:
Hello everyone,
Over the past month, the server has been busy trying to re-render all of the tiles to try to ensure more up-to date maps. This process is more or less complete now and all of the map tiles should then be at least from the 1st of December 2011.
As one month old tiles is still not overly satisfying, I am now trying to re-activate diff based tile expiry again.
The full expiry of all styles remains infeasible for the moment and so I am trying to introduce expiry slowly to see how far it can be pushed.
As a start, I have activated tile expiry only for the highest zoom levels (z15 - z18) and then only for the styles that currently are embedded in one of the wikipedias.
As far as I can tell these are the following styles:
osm-no-labels osm-labels-de osm-labels-ca osm-labels-bg osm-labels-da osm-labels-el osm-labels-es osm-labels-fa osm-labels-fr osm-labels-it osm-labels-lv osm-labels-nl osm-labels-no osm-labels-pl osm-labels-ru germany default hikebike
If anyone knows of a language that I have forgotten, please let me know and I will include it as well.
So far, it looks like the server can cope with this, although I have only activated it earlier today. Should it remain this way, I will try and expand the zoom levels.
As it is critical to reduce the number of low zoom tiles rendered, as those still kill performance, lower zoom tiles will probably not be expired as frequently. E.g. perhaps Z14 - Z11 on a daily basis, below that even only in the background.
We will have to see, though, how performance behaves over time.
Happy new year,
Kai
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l