[Labs-admin] Ways to go forward with Kubernetes

[Andrew Bogott]

Is there not the option of fixing docker so that it can handle big 
containers?  Is "running complex software with many dependencies" truly 
outside the scope of what docker seeks to accomplish?

On 12/2/16 12:09 PM, Yuvi Panda wrote:
> Hello!
>
> The original plan with k8s was to provide a 'no changes seen'
> migration pathway from GridEngine to K8S for webservices and jobs. We
> already did part 1 of this (webservice backend) with reasonable
> success, but the 'no changes seen' part bumped into a problem recently
> when we found that building a container with *all* the packages we put
> in exec_environment is untenable (is > 20G, and totally crashes docker
> daemon). Given that, and the fact that I might not be around for too
> long, here are the ways in which we can go forward:
>
> == 1. Finish the webservice migration to k8s completely ==
>
> We would start by defaulting new tools to k8s, then slowly flip things
> over one webservice type at a time.
>
> Pros:
>
> 1. Gridengine use lessens
> 2. We can get rid of all our custom webproxy related code, which
> should simplify everything - it would mean we're now running a
> 'normal' gridengine setup
>
> Cons:
> 1. Since we don't have an omnibus container, we need to figure out A
> Solution(tm) for some users (PHP script shelling out to Python to call
> a Java JAR etc), if/when we encounter them.
> 2. There are at least a couple of tools that call jsub from their
> webservices, and that won't work here if we don't have a kubernetes
> backend for jsub
> 3. This won't be truly 'no changes seen' from a UX POV - versions of
> software will likely change, since we don't have the omnibus
> container. This will cause change fatigue when we move to a PaaS.
>
> == 2. Add a kubernetes backend to jsub ==
>
> Add a --backend=kubernetes to jsub/jstart. This would let people use
> k8s to submit jobs / crons / tasks.
>
> Pros:
>
> 1. Gridengine use lessens
> 2. Easy way to move off gridengine for people 'stuck up'
> 3. Allows continuing function of webservice tools that call jsub
>
> Cons:
>
> 1. Stuck up people might continue to be stuck up
> 2. Same as cons (1) and (3) from webservice mgiration option
>
> == 3. Write up docs on how to use k8s directly for job submission ==
>
> kubectl is super nice and fairly friendly, and has good docs upstream.
> We can just write up really nice documentation on how people can just
> use that directly, and help some people switch.
>
> Pros:
>
> 1. Not much code work
> 2. Purely upstream supported solution
>
> Cons:
>
> 1. Requires a fair amount of change
> 2. The omnibus container problem exists / persists
> 3. change averse users will be change averse
>
> == 4. Go full on on evaluating / setting up PaaS ==
>
> Tools is a PaaS, and we should be running a PaaS. There are several
> that are running on k8s, and they are all a bit more mature now than
> when we started (that was the reason we didn't just go with a PaaS in
> the beginning). We'll just set one up, and move people over slowly
>
> Pros:
> 1. Have a real upstream!
> 2. Brings tools to the early 2010s!
>
> Cons:
> 1. Lots of change!
> 2. Big project, and Yuvi will definitely not be around to do much in
> it. Might be a Pro.
> 3. Change averse people will be change averse.
>
> Whatever we end up doing will be a combination of these 4 I guess, and
> it's a question of prioritization. Thoughts?