[Maps-l] Server admin procedures wrt ptolemy and ortelius

[Kai Krueger]

Hi,

I would like to ask the slightly higher level question of what the 
overall plans are to what services are running on what machines? So far 
I understand it as Ptolemy will contain a postgis database and the 
mapnik rendering will be on Ortelius?

http://meta.wikimedia.org/wiki/Maps_server_setup_tasks shows some ideas, 
but it seems to have more questions than answers about the architecture 
and the different parts of software that are supposed to run on these 
machines.



> 
> As soon as we confirm that we do not run out of space by removing two
> drives from RAID-10 I would definitely go for reinstall on a separate
> RAID#1 pair (taken out of the current RAID-10 if we have nothing small
> available).

I am not sure what exactly you want to put on the disks, but if I am not 
mistaken, the postgis rendering database is about 100 Gb in size. In 
addition OSM currently has about 350Gb of tiles for its single mapnik 
layer. The full main OSM I think is on the order of about 1 Tb, but a 
good portion of that would presumably not be relevant to wikipedia, i.e. 
the GPX points and history and isn't currently available anyway. Only 
the current OSM data is probably more on the order of 200Gb, but someone 
else can probably provide more accurate data if it is needed. On the 
other hand, is there any need for the OSM database on the production 
servers at the moment? Are there any services yet that use that data, or 
should that be something for the toolserver?

> 
...

> 
> Spread (the tool I am thinking of) requires basically either broadcast
> or multicast. The choice is yours :) Should (a) this model prove as
> workable and (b) we will quickly find out we need to start to grow a
> farm of rendering servers (hopefully not) - you might very well decide
> that WMF might need to carry mcast traffic for example across Atlantic.
> For now, we are just our little family of few boxes in The Netherlands.

May I ask what your plans are for a potential rendering farm is and how 
multicast / broadcast come into this? Renderd, the rendering backend 
behind mod_tile does have some ability to scale across multiple servers 
if needed, which may or may not be useful here, although currently 
untested as OSM has so far only used one server. (Currently a 8 core + 
hyperthreading machine). But in case it doesn't fit, it can probably be 
extended to do so.

> 
> Before we do that, I'd like to check whether how we can max out what we
> have. And I'd like to know, for example, do I need more smaller machines
> or just one big? And what exactly are our storage requirements (thinking
> about i18n-zed tiles for example)? I think we should be prepared for
> a higher demand than OSM currently has - that's where my concerns come
> from.

Which parts of the software stack do you think will need most work to 
scale up? My guess would be that the static rendering, i.e. the 
non-slippy map, non-tile rendering of the maps still needs some 
thoughts. Currently the static part of the SlippyMap extension just 
calls directly into the rendering stack and renders the same image every 
time. The "export scripts" do add some caching headers, so one can put a 
cache in front of it to not rerender it every time, but I would guess 
this needs some form of render side queueing too to deal with load spikes.

I'd like to avoid unnecessary duplication of infrastructure where
> we could have just more power. Maps in many ways different than casual
> PHP/Mediawiki bot stuff run on Toolserver - we have much more power to
> control the environment (like putting users' rendering requests at
> the lowest priority).
> 
> To sum up:
> 
> (1) we will be working on architecture with the goal to make cassini
> work as optimal as possible for the project
> (2) as soon as we find out how much PostgreSQL space we need, I would
> ask you to reinstall ptolemy for us
> (3) at least multicast group would be fine for now
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Maps-l mailing list
> Maps-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/maps-l