On Mon, Sep 14, 2009 at 7:02 PM, Aude aude.wiki@gmail.com wrote:

On Mon, Sep 14, 2009 at 2:57 PM, Aude aude.wiki@gmail.com wrote:

...

I haven't had so much free time lately, but am comfortable with helping set up and configure PostgreSQL and PostGIS and get the OSM database import going.  Though, there are many more tasks to do that I have time to do.

Partitioning the servers in an optimal manner for us is something I'm not as comfortable with.  I could figure it out if necessary, but Saper is probably better capable of that.  One thing I do know is that the database logs should be on separate partition from the database itself, and both kept separate from the OS.

Here are a list of tasks:

http://meta.wikimedia.org/wiki/Maps_server_setup_tasks

On Cassini, work is needed on PostgreSQL configuration.

https://jira.toolserver.org/browse/TS-302

https://jira.toolserver.org/browse/TS-303

-Kate

Another thing that might help is if we had documentation all in one place. Some of it is on the toolserver wiki, some on meta, some on the mediawiki wiki, along with items in jira and bugzilla, and various supporting info on the OSM wiki

Is there one best place to consolidate the documentation and notes?

It's not really a problem that they're on separate wikis as long as things are clearly separated. E.g. the stuff on mediawiki.org has to do with the extension and the toolserver wiki to do with the servers. and meta with the project at large.

What really needs doing though is to organize all these pages. Just getting a list of all of them would be a good start, they're all over the place.

On Mon, Sep 14, 2009 at 9:04 PM, Ævar Arnfjörð Bjarmason avarab@gmail.com wrote:

...

https://jira.toolserver.org/browse/TS-302

I'll take care of this one.

Okey done.

But what do we actually need PostGIS for on Cassini now that we have ptolemy? Just to give users PostGIS access?

But what do we actually need PostGIS for on Cassini now that we have ptolemy? Just to give users PostGIS access?

From the Tool-Authors point we would need read-access to both, the osm-db mirror and the postgis db, no matter if they reside on cassini or ptolemy.

Despite that: having our own postgis on cassini would allow tool authors to play with different mapnik stylesheets and/or completely new rendering technologies. That's what the toolserver is for, right?

Peter

Ævar Arnfjörð Bjarmason wrote:

...

I will be upgrading the driver tommorow CET from 2449 to 2463.

(shall we keep server logs on wiki or use some fancy irc stuff?)

I don't care. But I think the IRC format is more conductive to actually being used. Since it's easier.

I haven't used the logging bot in #wikimedia-tech though. I don't know what to use without spamming logs for regular operations.

Hi,

Just a few notes from me regarding server administration of ptolemy and ortelius...

Please keep in mind that ptolemy and ortelius are meant to be WMF production boxes. That means they're (also) managed by the Wikimedia Ops team. I think that for the near future we're happy to let you play with the boxes and experiment with what OSM/integration software/architecture works best. But eventually, when these maps are integrated into our core web sites, the servers and software will need to be managed by WMF as well as you guys. Especially since you volunteers might lose interest in the long run... :)

That means:

- Please work with us; keep us informed - Put documentation in our documentation wiki, http://wikitech.wikimedia.org. If you need access, please contact me and I'll get you set up. - Logging of server actions can be done on #wikimedia-tech using the log bot. Just use "!log <message>" in the channel, it will work. Put the server-name in the line. - If you have any problems/issues/needs related to managing the servers in general (RAID controller/driver issues?), as opposed to OSM software specific things, then certainly ask us! Chances are we've already solved it or have a certain way of doing things, and there is no need for you to reinvent the wheel. :)

Ptolemy and ortelius are, in the long run, *not* meant to be used by toolserver users. Those boxes are explicitly separate. You can't run a production database when users are running all kinds of inefficient and uncoordinated queries on it. :) For now it doesn't matter, but keep this in mind.

Cassini is a toolserver, and managed by Wikimedia Germany. They do things differently than WMF, coordinate with them to see what works there.

Thanks!

Mark Bergsma wrote:

Ævar Arnfjörð Bjarmason wrote: Please keep in mind that ptolemy and ortelius are meant to be WMF production boxes. That means they're (also) managed by the Wikimedia Ops team. I think that for the near future we're happy to let you play with the boxes and experiment with what OSM/integration software/architecture works best. But eventually, when these maps are integrated into our core web sites, the servers and software will need to be managed by WMF as well as you guys. Especially since you volunteers might lose interest in the long run... :)

I think being "production" is very good - we will be on monitoring from the very beginning :-)

That means:

• Please work with us; keep us informed

So far the only update for ptolemy: - LOM firmware update by river - Sun STK RAID INT firmware update from 5.2-0 (15825) to 5.2-0 (16732) - Tool to manage STK RAID INT installed in /usr/sbin/arcconf (Version 6.10 (B17551) from Intel website). I think that Adaptec's version (Version 6.10 (B18359)) is a bit more informative, what are you using on other servers?

- Installed as dependencies for arcconf: libstdc++5 gcc-3.3-base libgcc1

• Put documentation in our documentation wiki,

http://wikitech.wikimedia.org. If you need access, please contact me and I'll get you set up.

Can you create accounts for Aude and myself (Saper)? Is Ævar there as well?

Looking at http://wikitech.wikimedia.org/view/Platform-specific_documentation how far is Sun Fire X4250 different?

• Logging of server actions can be done on #wikimedia-tech using the log

bot. Just use "!log <message>" in the channel, it will work. Put the server-name in the line.

Cool, thanks.

• If you have any problems/issues/needs related to managing the servers

in general (RAID controller/driver issues?), as opposed to OSM software specific things, then certainly ask us! Chances are we've already solved it or have a certain way of doing things, and there is no need for you to reinvent the wheel. :)

Yes, here are my questions:

(1) It has been reported that RAID controller has serious stability problems (causes kernel abends). I think this should be fixed in the new firmware OR the new driver, see below.

(2) What are the kernel upgrade procedures on the WMF servers?

(3) What are the OS upgrade procedure on the WMF servers?

(4) /home/saper/raid/linux_x86_x64_driver_v1.1.5-2463 contains Linux driver version 2463 for Sun STK RAID INT that we probably should be running. I can do that given (2) above :)

(5) I asked on #ts-admins about the management console access, that would be beneficial to perfom changes to kernel and partitioning, see the next points what we need to be done from there.

(6) I think we should reconfigure RAID - for now, I would like to put the current filesystem on a single RAID1 pair of drives. It's root, so I think this shouldn't be done from the running system. I think we can disband the current RAID 10 setup for now, we will be testing one or two possible RAID setups for Postgres as soon as we have space.

(7) I'd love to have OS repartitioned - small /, large /usr, mid-large /var, small /tmp in a traditional UNIX way. All of this on a RAID 1 volume created in step #6

(8) It would be nice to have different OS (FreeBSD or Solaris) but I understand that probably you'd like to have a uniform setup accross WMF and I think I can live with it. Would be nice to have information re #3 though if we stick to Ubuntu.

I think I could do (4)...(7) myself given access to the management console and with some possibility to have some netboot/CD-boot from there. This leads me to:

(9) I've seen this: http://wikitech.wikimedia.org/view/Automated_installation Do you have some kind of minimal netboot/recovery system to be invoked from LOM to do stuff like total repartitioning?

Ptolemy and ortelius are, in the long run, *not* meant to be used by toolserver users. Those boxes are explicitly separate. You can't run a production database when users are running all kinds of inefficient and uncoordinated queries on it. :) For now it doesn't matter, but keep this in mind.

Cassini is a toolserver, and managed by Wikimedia Germany. They do things differently than WMF, coordinate with them to see what works there.

I hope that we can have a joint on project on maps and use resources efficiently. For example, we might not have space for the full OSM database anywhere else then on ptolemy. However, I think we can find a way to provide production-level stability and stay within our resource base. Besides, I have no objections to having exactly the same production/monitoring features on cassini as well.

Uff, that's all from me for now :)

Hi Marcin,

Marcin Cieslak wrote:

...

• Put documentation in our documentation wiki,

http://wikitech.wikimedia.org. If you need access, please contact me and I'll get you set up.

Can you create accounts for Aude and myself (Saper)? Is Ævar there as well?

Yes, I will get the details to you.

Yes, here are my questions:

(1) It has been reported that RAID controller has serious stability problems (causes kernel abends). I think this should be fixed in the new firmware OR the new driver, see below.

(2) What are the kernel upgrade procedures on the WMF servers? (3) What are the OS upgrade procedure on the WMF servers?

<snip>

I like your enthusiasm in managing these systems. :-) However, especially for the long run I think it would be easiest if you let us (WMF operations) deal with these system level things, and you concentrate on the OSM-specific software setup. Otherwise these systems will naturally diverge from our other servers, and for these production systems that's something we'll need to avoid.

What we normally do:

We upgrade the OS and kernels on need (features/stability), or when serious (remote) security issues are identified. We tend to upgrade to the latest Ubuntu kernel and use that unless we experience problems in practice. In this case there is indeed an issue with that RAID controller, so we'll upgrade it. We try to stick with Ubuntu Hardy as long as we can for miscellaneous servers. If you feel you really need newer versions for OSM setup, then let us know and we can upgrade it.

Partitioning: we know that it's traditional to separate /usr /var etc, but we have found that this usually has very little use in practice, and is more often a nuisance. These days we put everything in one large enough / and only split off data partitions on servers where it matters. Of course your databases should be running off a special partition, but for the rest there is probably no real need. If you think otherwise and have good arguments, we can surely change it, of course. We do tend to use LVM for everything non-root in those cases.

The same holds for the RAID setup: on our databases and big storage systems, most often we just run it off the same big RAID-10 array. It's more convenient and flexible and if well-configured the rest of the OS is not hitting that array much at all. If you feel there is a need, we can of course change it - but we'll need to reinstall the OS. A different RAID level would be totally fine as well of course - this is very much dependent on your needs. I picked RAID-10 as neither Aevar nor Katie knew what was necessary, and RAID-10 tends to be the best choice for databases and high performance I/O systems.

Serial console/LOM access cannot easily be handed out, but should also not be necessary usually. In the unlikely event that the system becomes unmanageable in-band, just contact us directly (ask on #wikimedia-tech for example) and we'll restore it quickly.

I hope that we can have a joint on project on maps and use resources efficiently. For example, we might not have space for the full OSM database anywhere else then on ptolemy. However, I think we can find a way to provide production-level stability and stay within our resource base. Besides, I have no objections to having exactly the same production/monitoring features on cassini as well.

I really want to stress that these systems need to be *separate*, they cannot be used together at all. Ideally there is no traffic between those servers at all, except in the form of cassini generating visitor traffic like the rest of the Internet. Cassini is meant for playing around where lots of people have access, the other two are (in the end) really meant for production use with limited access. Stable operation is simply not possible when arbitrary users can do arbitrary things on a system, and that's why we intended these systems to be very isolated from the start. Cassini is also managed by WMDE / Toolserver, ptolemy and ortelius are Wikimedia Foundation managed. So I'm afraid that we really cannot use those servers in one resource pool... If those separate clusters do not have enough resources/space to do what we need, I think we should look into buying more hardware. That is really not impossible. :)

Thanks for sharing your ideas. :)

Hi,

I would like to ask the slightly higher level question of what the overall plans are to what services are running on what machines? So far I understand it as Ptolemy will contain a postgis database and the mapnik rendering will be on Ortelius?

http://meta.wikimedia.org/wiki/Maps_server_setup_tasks shows some ideas, but it seems to have more questions than answers about the architecture and the different parts of software that are supposed to run on these machines.

As soon as we confirm that we do not run out of space by removing two drives from RAID-10 I would definitely go for reinstall on a separate RAID#1 pair (taken out of the current RAID-10 if we have nothing small available).

I am not sure what exactly you want to put on the disks, but if I am not mistaken, the postgis rendering database is about 100 Gb in size. In addition OSM currently has about 350Gb of tiles for its single mapnik layer. The full main OSM I think is on the order of about 1 Tb, but a good portion of that would presumably not be relevant to wikipedia, i.e. the GPX points and history and isn't currently available anyway. Only the current OSM data is probably more on the order of 200Gb, but someone else can probably provide more accurate data if it is needed. On the other hand, is there any need for the OSM database on the production servers at the moment? Are there any services yet that use that data, or should that be something for the toolserver?

...

Spread (the tool I am thinking of) requires basically either broadcast or multicast. The choice is yours :) Should (a) this model prove as workable and (b) we will quickly find out we need to start to grow a farm of rendering servers (hopefully not) - you might very well decide that WMF might need to carry mcast traffic for example across Atlantic. For now, we are just our little family of few boxes in The Netherlands.

May I ask what your plans are for a potential rendering farm is and how multicast / broadcast come into this? Renderd, the rendering backend behind mod_tile does have some ability to scale across multiple servers if needed, which may or may not be useful here, although currently untested as OSM has so far only used one server. (Currently a 8 core + hyperthreading machine). But in case it doesn't fit, it can probably be extended to do so.

Before we do that, I'd like to check whether how we can max out what we have. And I'd like to know, for example, do I need more smaller machines or just one big? And what exactly are our storage requirements (thinking about i18n-zed tiles for example)? I think we should be prepared for a higher demand than OSM currently has - that's where my concerns come from.

Which parts of the software stack do you think will need most work to scale up? My guess would be that the static rendering, i.e. the non-slippy map, non-tile rendering of the maps still needs some thoughts. Currently the static part of the SlippyMap extension just calls directly into the rendering stack and renders the same image every time. The "export scripts" do add some caching headers, so one can put a cache in front of it to not rerender it every time, but I would guess this needs some form of render side queueing too to deal with load spikes.

I'd like to avoid unnecessary duplication of infrastructure where

we could have just more power. Maps in many ways different than casual PHP/Mediawiki bot stuff run on Toolserver - we have much more power to control the environment (like putting users' rendering requests at the lowest priority).

To sum up:

(1) we will be working on architecture with the goal to make cassini work as optimal as possible for the project (2) as soon as we find out how much PostgreSQL space we need, I would ask you to reinstall ptolemy for us (3) at least multicast group would be fine for now

---

---

Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l

Ah, I forgot one things.

I'd like to see how IP failover may work for us in practice (I am thinking between easy switch between ptolemy and cassini in case of some PostgreSQL problems).

(10) Do you mind allocating us one more virtual IPv4 address that will be shared between cassini and ptolemy (for now)? This leads up to:

(11) I would like to use generate multicast traffic between OSM servers. I think it would be good to have few IP addresses from the 239.192.0.0/14 range allocated.