FYI

---------- Forwarded message ---------- From: Yuvi Panda yuvipanda@gmail.com Date: Thu, Aug 20, 2015 at 4:45 PM Subject: Evaluation of opt-in alternatives to Grid Engine on Tool Labs ('clustering solution') To: Wikimedia Labs labs-l@lists.wikimedia.org

Hello!

One of the experimental goals for this quarter for labs' team is to make available an new, more modern gridengine alternative just for webservices on Tool Labs. We are starting to evaluate which systems we should use - this is tracked at https://phabricator.wikimedia.org/T106475. The (still incomplete) evaluation spreadsheet is at https://docs.google.com/spreadsheets/d/1YkVsd8Y5wBn9fvwVQmp9Sf8K9DZCqmyJ-ew-...

We are evaluationg Kubernetes and Mesos/Marathon as alternatives. GridEngine is also being scored along with them, so if we find that it wins we'll abandon the experiment and continue using GridEngine only. Do provide comments on the phab ticket and follow along :)

== WHY? ==

Because our current webservices setup is a pile of hacks on top of GridEngine, causing... interesting problems due to the complexity involved.

GridEngine doesn't support a lot of features that people using more modern systems take for granted - like containerization + isolation, a nice API, continuous deploy, autoscaling.... Having an alternative to play with allows us to build newer, better featured and more robust systems.

== OMG, WILL I HAVE TO CHANGE THE WAY MY CODE WORKS NOW?! ==

Nope. For now this is just an alternative - when completed, you will be able to run your webservice on this cluster by something like:

webservice --provider=<something> start

And nothing else will change - everything else should still be compatible. We'll eventually provide more features on the new setup, but there will be no forced migration of any sort. If the alternative becomes the default at any point, we'll ensure that things that worked before continue working without any extra effort from the Tool Author's part.

-- Yuvi Panda T http://yuvi.in/blog