On 04/16/2010 12:41 AM, Dane Springmeyer wrote:
On Apr 15, 2010, at 2:02 PM, Kai Krueger wrote:
On 04/15/2010 02:38 PM, River Tarnell wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Ævar Arnfjörð Bjarmason:
Those servers need a lot of love. People other than me have been doing some work on it recently.
I've been distracted by other things the last couple of weeks, but I plan to do some working on getting renderd running properly next week. We may need some additional programming to make it feasible to run it with a lot of styles (e.g. loading styles on demand instead of at startup).
It appears as if renderd/Mapnik uses up about 5 - 6 Mb of Ram per instance.
Note, that a minor memory leak in XML loading was fixed in the Mapnik 0.7.0 release:
http://trac.mapnik.org/ticket/473 http://trac.mapnik.org/changeset/1506
Part of the problem may be that renderd uses a new instance per style per rendering thread.
Yes, this certainly uses more memory, but it is an intended feature AFAIK.
The mapnik.Map object is not (by design) intended to shared between threads.
So if you have 250 styles and 8 rendering threads you end up needing about 250*8*5.5Mb ~ 11 Gb of Ram, which sounds roughly like the numbers given for cassini.
Yes, I can see this being a problem with that many styles! :)
If Mapniks Map Object could be reused for all of the rendering threads, that would reduce memory by a factor of 8. It would still be a lot, but at least get back into the feasible range. But someone more familiar with Mapniks internals would have to comment on that.
In my experience with python-based servers (deployed with mod_wsgi), it works fine sharing a global map object if deployed in multiprocess mode (rather than multithreaded).
I wonder if tirex-renderd might be able to help in that case. It uses a multiprocess approach I think. On the other hand, I think it forks before loading any of the mapnik styles, so it can't use any of the copy on write "tricks" during fork to help reduce memory usage. (Was that the way you were thinking to use a global map object across multiple processes?) It would also then not go to well with a potential loading of styles on demand.
I'd be interested to hear from Jon Burgess or Artem about alternative approaches for threaded apps in C.
Kai, could you post this question to mapnik-devel? Or, if you are not subscribed, I can - just let me know.
I am not subscribed to mapnik-devel. So it would be nice if you could send the question to mapnik's list.
Kai
Otherwise implementing a load on demand with a time based expiry for map styles might not be a bad idea, given that most of the 250 odd styles will likely very seldom be used and sounds like it wouldn't be too hard to implement.
I think that's great idea.
Dane
Kai
- river.
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (HP-UX)
iEYEARECAAYFAkvHFuIACgkQIXd7fCuc5vJWyQCfcoW6A81hZYUAFmnNKTLXOWoH ccUAnA59GN6GxIfUHNdlGnY9LHGykK/j =YzWW -----END PGP SIGNATURE-----
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l