On Thu, Apr 15, 2010 at 12:26, Dmitry Granovsky dima.granovsky@gmail.com wrote:
Hello, Ævar!
Some time ago you set up a server (http://cassini.toolserver.org) providing Mapnik tiles in a number of languages, which is really great for anyone dealing with name localization in OSM, like me. As far as I can see, it's been a long time since the last update. Therefore I was wondering if any further updates are planned. If you just don't have sufficient time, maybe you could hand the server management to someone else?
Maps-l is the best venue to discuss stuff like this. CCing it.
Those servers need a lot of love. People other than me have been doing some work on it recently.
We're very interested in getting more people involved. Are you interested in helping in any way?
Hi, Ævar and all,
Those servers need a lot of love. People other than me have been doing some work on it recently. We're very interested in getting more people involved. Are you interested in helping in any way?
I am. However, what I can do depends on what kind of help you need. Say, my *nix skills are far from excellent.
I'm now subscribed to maps-l@, so I can follow the discussion there.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Ævar Arnfjörð Bjarmason:
Those servers need a lot of love. People other than me have been doing some work on it recently.
I've been distracted by other things the last couple of weeks, but I plan to do some working on getting renderd running properly next week. We may need some additional programming to make it feasible to run it with a lot of styles (e.g. loading styles on demand instead of at startup).
- river.
River Tarnell schrieb:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Ævar Arnfjörð Bjarmason:
Those servers need a lot of love. People other than me have been doing some work on it recently.
I've been distracted by other things the last couple of weeks, but I plan to do some working on getting renderd running properly next week. We may need some additional programming to make it feasible to run it with a lot of styles (e.g. loading styles on demand instead of at startup).
I'd suggest taking a look at tirex. On the devserver-list we're talking about how to configure it with various styles, each having differend min- and max zoomlevels etc.
In adition I don't think that the toolserver should host the 250+ language styles - this is in my eyes the job of the live cluster (ortelius & cassini).
Peter
It would be useful to document the format of requests for map tiles or static maps somewhere (so what the base url is, the parameters, and which values they should have). If the server side stuff is in a decent working state I can include the OSM stuff again in the upcoming 0.6 release of Maps, else it'll be out. -- Jeroen De Dauw * http://blog.bn2vs.com * http://wiki.bn2vs.com Don't panic. Don't be evil. 70 72 6F 67 72 61 6D 6D 69 6E 67 20 34 20 6C 69 66 65! --
On 15 April 2010 15:44, Peter Körner osm-lists@mazdermind.de wrote:
River Tarnell schrieb:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Ævar Arnfjörð Bjarmason:
Those servers need a lot of love. People other than me have been doing some work on it recently.
I've been distracted by other things the last couple of weeks, but I plan to do some working on getting renderd running properly next week. We may need some additional programming to make it feasible to run it with a lot of styles (e.g. loading styles on demand instead of at startup).
I'd suggest taking a look at tirex. On the devserver-list we're talking about how to configure it with various styles, each having differend min- and max zoomlevels etc.
In adition I don't think that the toolserver should host the 250+ language styles - this is in my eyes the job of the live cluster (ortelius & cassini).
Peter
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
On 04/15/2010 02:44 PM, Peter Körner wrote:
River Tarnell schrieb:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Ævar Arnfjörð Bjarmason:
Those servers need a lot of love. People other than me have been doing some work on it recently.
I've been distracted by other things the last couple of weeks, but I plan to do some working on getting renderd running properly next week. We may need some additional programming to make it feasible to run it with a lot of styles (e.g. loading styles on demand instead of at startup).
I'd suggest taking a look at tirex.
May I ask what the practical advantages to using tirex over renderd are? It sounds like it has a nicer architecture for when you whant to extend it and do more complicated things. But so far I am not sure any of those extensions have been implemented or for the moment are necessary for the toolserver setup. So given that tirex still seems a bit experimental, it might not be a bad idea to experiment with it on dev.osm.de for the moment and continue to use renderd on ptolemy for the moment. One thing that does look quite interesting is tirex's ability to automatically restart a crashed render, but then it shouldn't really crash in the first place and I haven't heard of any stability problems with renderd on tile.osm.org yet.
On the devserver-list we're talking
about how to configure it with various styles, each having differend min- and max zoomlevels etc.
The min and max zoomlevels also need to be specified in mod_tile, and I think that might actually be hardcoded in the source. So having a flexible renderd alone might not help.
In adition I don't think that the toolserver should host the 250+ language styles - this is in my eyes the job of the live cluster (ortelius& cassini).
Are there any plans on setting up the live cluster? It does in some way sound like those services shouldn't necessary be on the toolserver, but then it makes sense to test it on the toolserver first and to me (not really knowing anything about the wikipedia side of things) it appears as if WMA and geohack are still running on the toolserver too, even though they have been "in production" for a while.
Kai
Peter
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Kai Krueger schrieb:
On 04/15/2010 02:44 PM, Peter Körner wrote:
I'd suggest taking a look at tirex.
May I ask what the practical advantages to using tirex over renderd are?
I'm not sure if there are practical reasons but tirex seems beeing under active development while i'm not sure about this in renderd. It sounded to me like the next logical step.
So given that tirex still seems a bit experimental, it might not be a bad idea to experiment with it on dev.osm.de for the moment and continue to use renderd on ptolemy for the moment.
Okay, let's do so. I'm 50:50 about tirex & renderd as I just don't know enough about them.
In adition I don't think that the toolserver should host the 250+ language styles - this is in my eyes the job of the live cluster (ortelius& cassini).
Are there any plans on setting up the live cluster? It does in some way
Give me some access and I'll see what I can do.
sound like those services shouldn't necessary be on the toolserver, but then it makes sense to test it on the toolserver first and to me (not really knowing anything about the wikipedia side of things) it appears as if WMA and geohack are still running on the toolserver too, even though they have been "in production" for a while.
How about getting up the map rendering for the languages of the 10 (or maybe 25) biggest Wikipedias. This would not create as much load on the toolserver as 250 styles and we can test most multilingual rendering things.
That would be: en, de, fr, it, pt, nl, ru, pl, es, ja
Regarding the use of the hstore: osm2pgsql HEAD currently can do an hstore import, but it can only import all tags into one hstore and additionally some tags into their own columns. Our testings points out, that tag-lookups are getting really slow for often used tags.
So I would sugest a small osm2pgsql stylesheet with the most used tags (amenity, highway and such), especially those that are often uses for lookups. Columns that are used most the time for value-lookups (addr:*, name, name:*, wikipedia:*) sould be imported into the hstore.
I'd like to do this as well as I'd like to help out in getting the live cluster up.
Peter
On 04/16/2010 07:42 AM, Peter Körner wrote:
Kai Krueger schrieb:
On 04/15/2010 02:44 PM, Peter Körner wrote:
I'd suggest taking a look at tirex.
May I ask what the practical advantages to using tirex over renderd are?
I'm not sure if there are practical reasons but tirex seems beeing under active development while i'm not sure about this in renderd. It sounded to me like the next logical step.
It depends a bit on how you define active development. As far as I can tell, when ever there were any specific needs, it got extended and developed. At the moment things (at least on tile.osm.org) seem to be running quite smoothly though, so there isn't much need for further development from that point.
One of the things I think people have discussed as a nice to have feature of renderd was to be able to do more sophisticated rendering a la topOSM with layers of contours and hillshading and compositing it directly in renderd which might be easier to do with tirex, but I think that also hasn't been implemented yet.
So given that tirex still seems a bit experimental, it might not be a bad idea to experiment with it on dev.osm.de for the moment and continue to use renderd on ptolemy for the moment.
Okay, let's do so. I'm 50:50 about tirex & renderd as I just don't know enough about them.
I guess it would be interesting to hear from Jon Burgess (main person behind renderd) and Frederik Ramm / Jochen Topf (main developers behind Tirex) how they see the future of either of them and their relation.
In adition I don't think that the toolserver should host the 250+ language styles - this is in my eyes the job of the live cluster (ortelius& cassini).
Are there any plans on setting up the live cluster? It does in some way
Give me some access and I'll see what I can do.
sound like those services shouldn't necessary be on the toolserver, but then it makes sense to test it on the toolserver first and to me (not really knowing anything about the wikipedia side of things) it appears as if WMA and geohack are still running on the toolserver too, even though they have been "in production" for a while.
How about getting up the map rendering for the languages of the 10 (or maybe 25) biggest Wikipedias. This would not create as much load on the toolserver as 250 styles and we can test most multilingual rendering things.
That would be: en, de, fr, it, pt, nl, ru, pl, es, ja
That would sound like a reasonable compromise to me. It might in addition be good to include some more of the different scripts too like for example Arabic, Chinese, one of the Indian scripts, Greek and Hebrew.
Regarding the use of the hstore: osm2pgsql HEAD currently can do an hstore import, but it can only import all tags into one hstore and additionally some tags into their own columns. Our testings points out, that tag-lookups are getting really slow for often used tags.
The hstore features also sounds quite experimental still and that it might need more development, but as it fits well with the large number of extra columns with all the name: and wikipedia: tags and the more general nature of the toolserver setup it seems like it would be worth giving it a try and help iron out any potential problems with it.
So I would sugest a small osm2pgsql stylesheet with the most used tags (amenity, highway and such), especially those that are often uses for lookups. Columns that are used most the time for value-lookups (addr:*, name, name:*, wikipedia:*) sould be imported into the hstore.
Perhaps it can just use the keys that are in the standard osm2pgsql (plus perhaps the 10 most common name: tags) as columns and everything else in the hstore.
Do you know how performance compares between hstore and separate columns if your where clause doesn't touch the hstore, but the select pulls in data from there? The need for multiple accesses per data row might still slow things down, but that might not be a problem for features that are used less often.
Kai
I'd like to do this as well as I'd like to help out in getting the live cluster up.
Peter
Kai Krueger schrieb:
Perhaps it can just use the keys that are in the standard osm2pgsql (plus perhaps the 10 most common name: tags) as columns and everything else in the hstore.
I would only keep the columns from the standard osm2pgsql style that are used for lookups. Eg. width, operator, ele can be dropped. but for the sake of simplicity we should just use the default osm2pgsql style so that we can keep the main mapnik-osm.
Do you know how performance compares between hstore and separate columns if your where clause doesn't touch the hstore, but the select pulls in data from there? The need for multiple accesses per data row might still slow things down, but that might not be a problem for features that are used less often.
I don't think this has a big impact. But as on the devderver we have a hstore-only import without extra columns I don't have timings for that.
Peter
On 04/15/2010 02:38 PM, River Tarnell wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Ævar Arnfjörð Bjarmason:
Those servers need a lot of love. People other than me have been doing some work on it recently.
I've been distracted by other things the last couple of weeks, but I plan to do some working on getting renderd running properly next week. We may need some additional programming to make it feasible to run it with a lot of styles (e.g. loading styles on demand instead of at startup).
It appears as if renderd/Mapnik uses up about 5 - 6 Mb of Ram per instance. Part of the problem may be that renderd uses a new instance per style per rendering thread. So if you have 250 styles and 8 rendering threads you end up needing about 250*8*5.5Mb ~ 11 Gb of Ram, which sounds roughly like the numbers given for cassini.
If Mapniks Map Object could be reused for all of the rendering threads, that would reduce memory by a factor of 8. It would still be a lot, but at least get back into the feasible range. But someone more familiar with Mapniks internals would have to comment on that.
Otherwise implementing a load on demand with a time based expiry for map styles might not be a bad idea, given that most of the 250 odd styles will likely very seldom be used and sounds like it wouldn't be too hard to implement.
Kai
- river.
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (HP-UX)
iEYEARECAAYFAkvHFuIACgkQIXd7fCuc5vJWyQCfcoW6A81hZYUAFmnNKTLXOWoH ccUAnA59GN6GxIfUHNdlGnY9LHGykK/j =YzWW -----END PGP SIGNATURE-----
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
On Apr 15, 2010, at 2:02 PM, Kai Krueger wrote:
On 04/15/2010 02:38 PM, River Tarnell wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Ævar Arnfjörð Bjarmason:
Those servers need a lot of love. People other than me have been doing some work on it recently.
I've been distracted by other things the last couple of weeks, but I plan to do some working on getting renderd running properly next week. We may need some additional programming to make it feasible to run it with a lot of styles (e.g. loading styles on demand instead of at startup).
It appears as if renderd/Mapnik uses up about 5 - 6 Mb of Ram per instance.
Note, that a minor memory leak in XML loading was fixed in the Mapnik 0.7.0 release:
http://trac.mapnik.org/ticket/473 http://trac.mapnik.org/changeset/1506
Part of the problem may be that renderd uses a new instance per style per rendering thread.
Yes, this certainly uses more memory, but it is an intended feature AFAIK.
The mapnik.Map object is not (by design) intended to shared between threads.
So if you have 250 styles and 8 rendering threads you end up needing about 250*8*5.5Mb ~ 11 Gb of Ram, which sounds roughly like the numbers given for cassini.
Yes, I can see this being a problem with that many styles! :)
If Mapniks Map Object could be reused for all of the rendering threads, that would reduce memory by a factor of 8. It would still be a lot, but at least get back into the feasible range. But someone more familiar with Mapniks internals would have to comment on that.
In my experience with python-based servers (deployed with mod_wsgi), it works fine sharing a global map object if deployed in multiprocess mode (rather than multithreaded).
I'd be interested to hear from Jon Burgess or Artem about alternative approaches for threaded apps in C.
Kai, could you post this question to mapnik-devel? Or, if you are not subscribed, I can - just let me know.
Otherwise implementing a load on demand with a time based expiry for map styles might not be a bad idea, given that most of the 250 odd styles will likely very seldom be used and sounds like it wouldn't be too hard to implement.
I think that's great idea.
Dane
Kai
- river.
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (HP-UX)
iEYEARECAAYFAkvHFuIACgkQIXd7fCuc5vJWyQCfcoW6A81hZYUAFmnNKTLXOWoH ccUAnA59GN6GxIfUHNdlGnY9LHGykK/j =YzWW -----END PGP SIGNATURE-----
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
On 04/16/2010 12:41 AM, Dane Springmeyer wrote:
On Apr 15, 2010, at 2:02 PM, Kai Krueger wrote:
On 04/15/2010 02:38 PM, River Tarnell wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Ævar Arnfjörð Bjarmason:
Those servers need a lot of love. People other than me have been doing some work on it recently.
I've been distracted by other things the last couple of weeks, but I plan to do some working on getting renderd running properly next week. We may need some additional programming to make it feasible to run it with a lot of styles (e.g. loading styles on demand instead of at startup).
It appears as if renderd/Mapnik uses up about 5 - 6 Mb of Ram per instance.
Note, that a minor memory leak in XML loading was fixed in the Mapnik 0.7.0 release:
http://trac.mapnik.org/ticket/473 http://trac.mapnik.org/changeset/1506
Part of the problem may be that renderd uses a new instance per style per rendering thread.
Yes, this certainly uses more memory, but it is an intended feature AFAIK.
The mapnik.Map object is not (by design) intended to shared between threads.
So if you have 250 styles and 8 rendering threads you end up needing about 250*8*5.5Mb ~ 11 Gb of Ram, which sounds roughly like the numbers given for cassini.
Yes, I can see this being a problem with that many styles! :)
If Mapniks Map Object could be reused for all of the rendering threads, that would reduce memory by a factor of 8. It would still be a lot, but at least get back into the feasible range. But someone more familiar with Mapniks internals would have to comment on that.
In my experience with python-based servers (deployed with mod_wsgi), it works fine sharing a global map object if deployed in multiprocess mode (rather than multithreaded).
I wonder if tirex-renderd might be able to help in that case. It uses a multiprocess approach I think. On the other hand, I think it forks before loading any of the mapnik styles, so it can't use any of the copy on write "tricks" during fork to help reduce memory usage. (Was that the way you were thinking to use a global map object across multiple processes?) It would also then not go to well with a potential loading of styles on demand.
I'd be interested to hear from Jon Burgess or Artem about alternative approaches for threaded apps in C.
Kai, could you post this question to mapnik-devel? Or, if you are not subscribed, I can - just let me know.
I am not subscribed to mapnik-devel. So it would be nice if you could send the question to mapnik's list.
Kai
Otherwise implementing a load on demand with a time based expiry for map styles might not be a bad idea, given that most of the 250 odd styles will likely very seldom be used and sounds like it wouldn't be too hard to implement.
I think that's great idea.
Dane
Kai
- river.
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (HP-UX)
iEYEARECAAYFAkvHFuIACgkQIXd7fCuc5vJWyQCfcoW6A81hZYUAFmnNKTLXOWoH ccUAnA59GN6GxIfUHNdlGnY9LHGykK/j =YzWW -----END PGP SIGNATURE-----
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Dane Springmeyer wrote:
Part of the problem may be that renderd uses a new instance
per style per rendering thread.
Yes, this certainly uses more memory, but it is an intended feature AFAIK.
The mapnik.Map object is not (by design) intended to shared between threads.
If renderd is able to only read the map after initialization (ie. work with a const mapnik::Map&) there should be little problem with sharing between threads.