[Labs-admin] Ways to go forward with Kubernetes

[Yuvi Panda]

Hello!

The original plan with k8s was to provide a 'no changes seen'
migration pathway from GridEngine to K8S for webservices and jobs. We
already did part 1 of this (webservice backend) with reasonable
success, but the 'no changes seen' part bumped into a problem recently
when we found that building a container with *all* the packages we put
in exec_environment is untenable (is > 20G, and totally crashes docker
daemon). Given that, and the fact that I might not be around for too
long, here are the ways in which we can go forward:

== 1. Finish the webservice migration to k8s completely ==

We would start by defaulting new tools to k8s, then slowly flip things
over one webservice type at a time.

Pros:

1. Gridengine use lessens
2. We can get rid of all our custom webproxy related code, which
should simplify everything - it would mean we're now running a
'normal' gridengine setup

Cons:
1. Since we don't have an omnibus container, we need to figure out A
Solution(tm) for some users (PHP script shelling out to Python to call
a Java JAR etc), if/when we encounter them.
2. There are at least a couple of tools that call jsub from their
webservices, and that won't work here if we don't have a kubernetes
backend for jsub
3. This won't be truly 'no changes seen' from a UX POV - versions of
software will likely change, since we don't have the omnibus
container. This will cause change fatigue when we move to a PaaS.

== 2. Add a kubernetes backend to jsub ==

Add a --backend=kubernetes to jsub/jstart. This would let people use
k8s to submit jobs / crons / tasks.

Pros:

1. Gridengine use lessens
2. Easy way to move off gridengine for people 'stuck up'
3. Allows continuing function of webservice tools that call jsub

Cons:

1. Stuck up people might continue to be stuck up
2. Same as cons (1) and (3) from webservice mgiration option

== 3. Write up docs on how to use k8s directly for job submission ==

kubectl is super nice and fairly friendly, and has good docs upstream.
We can just write up really nice documentation on how people can just
use that directly, and help some people switch.

Pros:

1. Not much code work
2. Purely upstream supported solution

Cons:

1. Requires a fair amount of change
2. The omnibus container problem exists / persists
3. change averse users will be change averse

== 4. Go full on on evaluating / setting up PaaS ==

Tools is a PaaS, and we should be running a PaaS. There are several
that are running on k8s, and they are all a bit more mature now than
when we started (that was the reason we didn't just go with a PaaS in
the beginning). We'll just set one up, and move people over slowly

Pros:
1. Have a real upstream!
2. Brings tools to the early 2010s!

Cons:
1. Lots of change!
2. Big project, and Yuvi will definitely not be around to do much in
it. Might be a Pro.
3. Change averse people will be change averse.

Whatever we end up doing will be a combination of these 4 I guess, and
it's a question of prioritization. Thoughts?
-- 
Yuvi Panda T
http://yuvi.in/blog