[Maps-l] Concerning cassini.toolserver.org

[Kai Krueger]

On 04/16/2010 12:41 AM, Dane Springmeyer wrote:
>
> On Apr 15, 2010, at 2:02 PM, Kai Krueger wrote:
>
>> On 04/15/2010 02:38 PM, River Tarnell wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> Ævar Arnfjörð Bjarmason:
>>>> Those servers need a lot of love. People other than me have been
>>>> doing
>>>> some work on it recently.
>>>
>>> I've been distracted by other things the last couple of weeks, but I
>>> plan to do some working on getting renderd running properly next
>>> week.
>>> We may need some additional programming to make it feasible to run it
>>> with a lot of styles (e.g. loading styles on demand instead of at
>>> startup).
>>
>> It appears as if renderd/Mapnik uses up about 5 - 6 Mb of Ram per
>> instance.
>
> Note, that a minor memory leak in XML loading was fixed in the Mapnik
> 0.7.0 release:
>
> http://trac.mapnik.org/ticket/473
> http://trac.mapnik.org/changeset/1506
>
>> Part of the problem may be that renderd uses a new instance
>> per style per rendering thread.
>
> Yes, this certainly uses more memory, but it is an intended feature
> AFAIK.
>
> The mapnik.Map object is not (by design) intended to shared between
> threads.
>
>> So if you have 250 styles and 8
>> rendering threads you end up needing about 250*8*5.5Mb ~ 11 Gb of Ram,
>> which sounds  roughly like the numbers given for cassini.
>>
>
> Yes, I can see this being a problem with that many styles! :)
>
>> If Mapniks Map Object could be reused for all of the rendering
>> threads,
>> that would reduce memory by a factor of 8. It would still be a lot,
>> but
>> at least get back into the feasible range. But someone more familiar
>> with Mapniks internals would have to comment on that.
>
> In my experience with python-based servers (deployed with mod_wsgi),
> it works fine sharing a global map object if deployed in multiprocess
> mode (rather than multithreaded).

I wonder if tirex-renderd might be able to help in that case. It uses a 
multiprocess approach I think. On the other hand, I think it forks 
before loading any of the mapnik styles, so it can't use any of the copy 
on write "tricks" during fork to help reduce memory usage. (Was that the 
way you were thinking to use a global map object across multiple 
processes?) It would also then not go to well with a potential loading 
of styles on demand.

>
> I'd be interested to hear from Jon Burgess or Artem about alternative
> approaches for threaded apps in C.
>
> Kai, could you post this question to mapnik-devel? Or, if you are not
> subscribed, I can - just let me know.

I am not subscribed to mapnik-devel. So it would be nice if you could 
send the question to mapnik's list.

Kai

>
>>
>> Otherwise implementing a load on demand with a time based expiry for
>> map
>> styles might not be a bad idea, given that most of the 250 odd styles
>> will likely very seldom be used and sounds like it wouldn't be too
>> hard
>> to implement.
>
> I think that's great idea.
>
> Dane
>
>
>>
>> Kai
>>
>>>
>>> 	- river.
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v1.4.10 (HP-UX)
>>>
>>> iEYEARECAAYFAkvHFuIACgkQIXd7fCuc5vJWyQCfcoW6A81hZYUAFmnNKTLXOWoH
>>> ccUAnA59GN6GxIfUHNdlGnY9LHGykK/j
>>> =YzWW
>>> -----END PGP SIGNATURE-----
>>>
>>> _______________________________________________
>>> Maps-l mailing list
>>> Maps-l at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/maps-l
>>
>>
>> _______________________________________________
>> Maps-l mailing list
>> Maps-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/maps-l
>
>
> _______________________________________________
> Maps-l mailing list
> Maps-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/maps-l