[Labs-l] Wikimedia Labs Software Stack

[Andreas Trawoeger]

Hi Ryan!

The ultimate goal is to combine three geo data related projects that are
currently supported by Wikimedia Germany and are in different stages of
development:
- WIWOSM (Wikipedia where in OSM) [0]
- Multilingual Wikipedia Maps [1]
- OpenGeoServer.org

Kolossos from WIWOSM made a rough sketch how such a combination could look
like:
https://toolserver.org/~kolossos/openlayers/kml-on-ol-json3.php?lang=de&title=Donau&layers=00000B0FTTTT

WIWOSM & Multilingual Maps runs fine using the current toolserver
infrastructure. But with OpenGeoServer.org I'm currently doing a lot of low
level stuff like tile cache performance optimization that would be
difficult to do on toolserver.org. The next step on my side is be to get a
full Landsat [2] layer with about 3TB of data up and running and see how to
scale it well across multiple servers.

Thanks for the documentation and I will take a look at saltstack too.

cu andreas

[0] https://wiki.openstreetmap.org/wiki/WIWOSM
[1] https://wiki.openstreetmap.org/wiki/Multilingual_maps_Wikipedia_project
[2] https://en.wikipedia.org/wiki/Landsat


2012/10/4 Ryan Lane <rlane32 at gmail.com>

> On Thu, Oct 4, 2012 at 4:13 AM, Andreas Trawoeger
> <atrawog at kartenwerkstatt.at> wrote:
> > Hi everybody!
> >
>
> Hi!
>
> > I'm currently running a project called opengeoserver.org that collects
> and
> > redistributes freely available Open Geodata (at the moment I'm at 1.3 TB
> of
> > data) . The project is currently receiving some funding from Wikimedia
> > Germany and is a likely candidate to be migrated to the Wikimedia Labs
> > sometime in the near future.
> >
>
> Do you mean that your test and development will move to Labs, or that
> your production site will? The former is ok, the latter is not. Labs
> is not meant for production services. If the production service is to
> be moved, then we should be getting it going in a test and development
> mode in Labs, then move it to production.
>
> > But being a sysadmin that neither wants to replace his fine tuned XFS
> > storage with Gluster nor his Fabric [0] deployment scripts with Puppet
> > without a lot of testing. I would actually prefer to set up my own
> OpenStack
> > instance on my existing systems, test different configurations options
> and
> > then migrate the services to Wikimedia Labs.
> >
>
> That's a decent amount of data. We have about 40TB of gluster space
> available in the pmtpa cluster and will have another 40+TB in eqiad.
> It'll be slower than direct XFS access, but we don't really have a
> means of providing that much space in a direct access mode.
>
> We'll be using saltstack for deployment soon, which works in a very
> fabric-like way. It's a little more flexible than fabric, in fact.
> Puppet really shouldn't be used for deployment, just configuration and
> dependency management.
>
> > It's easy to figure out that Wikimedia Labs is using Ubuntu, OpenStack &
> > Gluster, but I couldn't find any detailed information about what versions
> > are currently used and how the current hardware & storage configuration
> > actually looks like.
> >
>
> We have some slightly outdated docs here:
>
> http://wikitech.wikimedia.org/view/OpenStack
>
> - Ryan
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/labs-l/attachments/20121005/09467005/attachment.html>