[Labs-admin] Ways to go forward with Kubernetes

[Yuvi Panda]

I think even if we do, a single container that's 20G+ will always
cause issues. Doing more investigation that way is also a possible
option :)

On Sat, Dec 3, 2016 at 2:48 AM, Andrew Bogott <abogott at wikimedia.org> wrote:
> Is there not the option of fixing docker so that it can handle big
> containers?  Is "running complex software with many dependencies" truly
> outside the scope of what docker seeks to accomplish?
>
>
> On 12/2/16 12:09 PM, Yuvi Panda wrote:
>>
>> Hello!
>>
>> The original plan with k8s was to provide a 'no changes seen'
>> migration pathway from GridEngine to K8S for webservices and jobs. We
>> already did part 1 of this (webservice backend) with reasonable
>> success, but the 'no changes seen' part bumped into a problem recently
>> when we found that building a container with *all* the packages we put
>> in exec_environment is untenable (is > 20G, and totally crashes docker
>> daemon). Given that, and the fact that I might not be around for too
>> long, here are the ways in which we can go forward:
>>
>> == 1. Finish the webservice migration to k8s completely ==
>>
>> We would start by defaulting new tools to k8s, then slowly flip things
>> over one webservice type at a time.
>>
>> Pros:
>>
>> 1. Gridengine use lessens
>> 2. We can get rid of all our custom webproxy related code, which
>> should simplify everything - it would mean we're now running a
>> 'normal' gridengine setup
>>
>> Cons:
>> 1. Since we don't have an omnibus container, we need to figure out A
>> Solution(tm) for some users (PHP script shelling out to Python to call
>> a Java JAR etc), if/when we encounter them.
>> 2. There are at least a couple of tools that call jsub from their
>> webservices, and that won't work here if we don't have a kubernetes
>> backend for jsub
>> 3. This won't be truly 'no changes seen' from a UX POV - versions of
>> software will likely change, since we don't have the omnibus
>> container. This will cause change fatigue when we move to a PaaS.
>>
>> == 2. Add a kubernetes backend to jsub ==
>>
>> Add a --backend=kubernetes to jsub/jstart. This would let people use
>> k8s to submit jobs / crons / tasks.
>>
>> Pros:
>>
>> 1. Gridengine use lessens
>> 2. Easy way to move off gridengine for people 'stuck up'
>> 3. Allows continuing function of webservice tools that call jsub
>>
>> Cons:
>>
>> 1. Stuck up people might continue to be stuck up
>> 2. Same as cons (1) and (3) from webservice mgiration option
>>
>> == 3. Write up docs on how to use k8s directly for job submission ==
>>
>> kubectl is super nice and fairly friendly, and has good docs upstream.
>> We can just write up really nice documentation on how people can just
>> use that directly, and help some people switch.
>>
>> Pros:
>>
>> 1. Not much code work
>> 2. Purely upstream supported solution
>>
>> Cons:
>>
>> 1. Requires a fair amount of change
>> 2. The omnibus container problem exists / persists
>> 3. change averse users will be change averse
>>
>> == 4. Go full on on evaluating / setting up PaaS ==
>>
>> Tools is a PaaS, and we should be running a PaaS. There are several
>> that are running on k8s, and they are all a bit more mature now than
>> when we started (that was the reason we didn't just go with a PaaS in
>> the beginning). We'll just set one up, and move people over slowly
>>
>> Pros:
>> 1. Have a real upstream!
>> 2. Brings tools to the early 2010s!
>>
>> Cons:
>> 1. Lots of change!
>> 2. Big project, and Yuvi will definitely not be around to do much in
>> it. Might be a Pro.
>> 3. Change averse people will be change averse.
>>
>> Whatever we end up doing will be a combination of these 4 I guess, and
>> it's a question of prioritization. Thoughts?
>
>
>



-- 
Yuvi Panda T
http://yuvi.in/blog