[QA] [Ops] security patches handling

[Stas Malyshev]

Hi!

>     1. We had broken code which is not in gerrit or actually anywhere in
>     repos running in production.
> 
> 
> Half-true. The code is not in gerrit but it *is* committed locally to the
> deployment branches on the repos. They're always rebased on top
> so they're immediately visible in the git log.

This is probably where my low familiarity with deployment details is
showing - when I tried to look where the code comes from, code on
terbium was different from gerrit, but I didn't find any git repo I
could look at, so I'm still not sure what "locally" means in this context.

> Yes, we did know there were problems applying these patches. This current
> set of security patches have been particularly problematic in rebasing each
> week for the train. As discussed on IRC last night, the problem comes in
> trying to sort out the merge conflicts. Most of the time we get it right and
> nobody notices, but the context here wasn't great and we made a mistake.
> 
> The *.failed was just a lack of cleanup after resolving the conflicts...the
> patches *were* applied.

I wonder if we need to somehow make it more explicit (and known to
whoever changed the code in question) that there were security patches
that didn't apply cleanly. Sometimes it can be hard to apply old patch
to refactored code without understanding how it changed, which may not
be obvious by just looking at small piece of the code.

> Had we not gotten a fix in place yesterday, I would've halted the train for
> wider deployment.

Good to know :)

> To help the situation, I think a few things are necessary:
> - We (me, mostly) need to *drastically* cut down the time it takes to get
> security releases public. Carrying non-master fixes in production should

Agree.

> - Using merges/shared git history between deployment branches instead
> of patchfiles would probably simplify a lot of this, needs further thinking
> through though

I think if we could have private/restricted access branches instead of
plain files that might be more helpful. Is it feasible?

-- 
Stas Malyshev
smalyshev at wikimedia.org