Hi all,

as previously announced, we've been evaluating a "clustering solution" for use as an alternative to GridEngine for toollabs

https://lists.wikimedia.org/pipermail/wikitech-l/2015-August/082853.html

Our goal is also to find a suitable, modern, stable tool to run not only toollabs webservices, but also - on a longer term - to find a modern, easier, more convenient way to run our microservices in production: a clusterized environment that will allow us to enhance single service availalbility and also to apply easier scaling of applications, reducing further the friction surface and the direct ops involvement in the day-to-day setup and deployment of services.

Our evaluation of the available solutions is ongoing, and while we're mostly done filling up an "evaluation spreadsheet" (https://docs.google.com/spreadsheets/d/1YkVsd8Y5wBn9fvwVQmp9Sf8K9DZCqmyJ-ew-...), we would welcome and we encourage further involvement/suggestions. You can provide these easily on the tracking ticket for the evaluation, https://phabricator.wikimedia.org/T106475

We received some interesting feedback already, and we look forward incorporating more!

We are considering two solutions - mesospheres' Marathon (which is based on Mesos) - https://mesosphere.github.io/marathon/ and Google's Kubernetes https://kubernetes.io.

Now let us summarize a bit our findings so far: MESOS/MARATHON:

Pros: - Mesos is stable and battle tested, although Marathon is quite young and mostly used in mesosphere's commercial offering - Supports overcommitting resources (which is important in toollabs, probably less so in production) - Has a nice, clean API and is fully distributed with no potential SPOFs - Chronos is another framework that can run on mesos and is a great distributed cron

Cons: - Multitenancy story is non-existent, it was not designed to be a public PaaS offering. This is an issue even in production if we want to grant independence to single teams. - Container support seems experimental at best.(but getting better in newer versions) - Adoption of Marathon seems little and the community is not very lively. - Discovery/scaling logic is somewhat limited

KUBERNETES

Pros: - The design seems to be very well thought out, based off of experiences running Google's internal Borg system (see http://research.google.com/pubs/pub43438.html for details of Google's Borg clustering system). - A pretty refined security model is already implemented, so that single users/teams could be given access to individual namespaces and act independently - The community is very lively, and adoption is gaining momentum: kubernetes is the default way to deploy apps on Google Compute Engine, it's used by Red Hat for its own cloud solution (and they contribute patches to it), it has a clear roadmap to overcome most of its limitations - Container support is native and it's tecnology-agnostic, allowing (for now) Docker and Rkt containers to be used - The API is quite nice - Documentation is decently complete - Google engineers are actively supporting us in evaluating its usage Cons: - The master node is not highly available, although our cluster survived a pretty serious outage in labs that froze the master and wiped out one worker - No overcommitting allowed, it will be possible to mimic it with QoS (coming in the next version) - The ability to schedule one-off jobs is offered, but there is no distributed cron facility - In general it's a younger project with some outstanding bugs

As you can see there are pretty big pros/cons for both these technologies, due to the fact they are still quite "not boring" - although one could argue that mesos and chronos at least have entered their "boring" stage. Our spreadsheet slightly favours Kubernetes at the moment, but that might change drastically, if we evaluate that some limitations are absolute showstoppers for us.

In the remainder of this week and the next few ones, we will keep stress testing both our test installations to find out "surprises" and bugs.

Let us know what you think - or reach out to us if you want to help in this evaluation process. We will keep you posted!

Cheers,

Giuseppe & Yuvi