[Labs-admin] Ways to go forward with Kubernetes

[Yuvi Panda]

On Sat, Dec 3, 2016 at 6:04 AM, Bryan Davis <bd808 at wikimedia.org> wrote:
> What would the main challenges be to allowing new images to be derived
> from our base images by defining a method for a tool account to
> specify a Dockerfile? We wouldn't need to go all the way to allowing
> arbitrary Dockerfiles from anywhere on the net, but could instead
> enforce that they had to derive from one of our base images. The tool
> maintainers could then pick and choose packages to install via apt
> from the repos that we already control. If we wanted to restrict it
> only to apt installs it could even be a "requirements.txt" style file
> that was checked for an applied rather than a full Dockerfile.

1. Mass rebuild jobs whenever we need to upgrade the base container
(for security or otherwise)
2. We'll be kinda re-inventing the wheel, since a lot of PaaS' already
have something like this.
3. Arbitrary Dockerfile building is not the easiest to secure (since
access to the docker socket makes you equivalent to root on a box).
This is why a lot of PaaS' offer a non-Dockerfile mode of building
things (deis does this, I think). When I was talking to Deis people a
year and something back, they were building Docker images inside a
qemu instance. This is mitigated by the 'requirements.txt' style setup
tho.
4. We'll need to massively scale our docker registry setup (probably
needs swift)

I think (2) is the most important one - if we want to do this, we
should just adopt a PaaS and not build it ourselves.

>> 2. There are at least a couple of tools that call jsub from their
>> webservices, and that won't work here if we don't have a kubernetes
>> backend for jsub
>
> Won't a jsub backend have the exact same mega-container requirement
> problem? At the very least the webservice embedded jsub calls would
> need to be changed to specify the container variant to spawn. It seems
> likely that we'd also want them to change to specify the k8s backend
> explicitly.
>
>> 3. This won't be truly 'no changes seen' from a UX POV - versions of
>> software will likely change, since we don't have the omnibus
>> container. This will cause change fatigue when we move to a PaaS.
>
> This was my big worry when I asked Yuvi to kick off this discussion.
> There were a few comments in the last survey that basically said
> "every time I learn something it becomes deprecated" or "I only have
> to touch my tools because you keep changing the platform underneath
> me". If we are really going to try and move away from homegrown PaaS
> hacks and adopt OpenShift/Deis/Whatever (and I think we should), I'm
> not sure I see the need to first move everyone from OGE to our k8s
> prototype and then immediately after tell them they need to move to
> the PaaS.

Yeah, if we can't really do a 'no-ux change' situation, the original
timeline kinda goes out the window. I guess I should've done this
evaluation (build the trusty container) back then, and not just now.
My bad.


> My biggest concern here is that if we choose an opinionated PaaS such
> as OpenShift this direct interaction with kubectl may be blocked in
> the future and we are back in the two migrations instead of one
> scenario. It was almost trivially easy to move Stashbot to direct
> kubectl thanks to all of the work that Yuvi did for webservice, but we
> still have some kinks to work out with logging and restarts for things
> that can't run parallel processes.

While I agree with the logging concern, any PaaS that doesn't let you
expose kubectl to users gets a pretty bag rep in my book. It'll be a
leaky abstraction - you'll need to know both k8s and the PaaS bits to
use it, but don't have the ability to access the k8s API? I don't know
if that's how OpenShift works, but if that does that's a major
negative IMO. PAWS can't exist in such a scenario, and neither can a
lot of other use cases that require direct access to the k8s API.

>> == 4. Go full on on evaluating / setting up PaaS ==
>>
>> Tools is a PaaS, and we should be running a PaaS. There are several
>> that are running on k8s, and they are all a bit more mature now than
>> when we started (that was the reason we didn't just go with a PaaS in
>> the beginning). We'll just set one up, and move people over slowly
>>
>> Pros:
>> 1. Have a real upstream!
>> 2. Brings tools to the early 2010s!
>>
>> Cons:
>> 1. Lots of change!
>> 2. Big project, and Yuvi will definitely not be around to do much in
>> it. Might be a Pro.
>> 3. Change averse people will be change averse.
>
> Con #1 is both a pro and con in my opinion, but yes we will be moving
> all of the cheese and will also have to invest in tooling,
> documentation, and hands-on support to make the change as easy as we
> can for the largest percentage of tools we can.
>
> There is literally nothing we can do for con #3 short of freezing
> everything. At some point we will have to move forward and there will
> be some users and tools who chose not to invest in the changes needed.
> I don't want to aggravate that problem with unnecessary intermediate
> steps, but we can't hold back everything for the sake of a few.

I agree.

>> Whatever we end up doing will be a combination of these 4 I guess, and
>> it's a question of prioritization. Thoughts?
>
> The timelines and roadmaps for killing OGE come from before any of my
> direct involvement. Is EOL of Ubuntu 14.04 the main driver or is it
> adoption of Jessie due to Puppet changes that we will be holding
> production back on or is it something else entirely? Assuming we could
> select a PaaS by the end of Q3 (March) and be ready to start doing
> major evangelization at the May hackathon how much time would we have
> left before we had to shutdown OGE?

Here were the reasons for it:

1. None of the PaaS solutions were viable at that time (K8S itself had
only recently reached 1.0)
2.  EOL of Ubuntu 14.04 wasn't even a concern at that point, and I
don't think it is now. Pure red herring.
3. It was mostly that the amount of GridEngine knowledge is dwindling,
and GridEngine is a PITA to maintain. From the original discussions,
this was entirely focused on making the life of the admins easier
(this started right after a massive GridEngine bdb corruption scare,
IIRC) and any user benefits were collateral only. The big user
benefits were supposed to come from the PaaS and using kubectl
directly, not from just k8s.
4. A PaaS is a pretty big undertaking, given the resources the team
had at that time. Practically there was only 80% of me working on
this, and almost all PaaS require at least object storage (which we
don't have, and is a large conversation by itself) - this isn't
counting the user education needed. This situation is a little bit
better now, ofcourse :)

I'll try to dig up some of the documentation we wrote about this
decision back then - might help understand the thought processes, see
which conditions still hold true and which don't.

-- 
Yuvi Panda T
http://yuvi.in/blog