[QA] Fwd: security patches handling

[Stas Malyshev]

Hi!

Yesterday we've had search broken on mediawiki.org for some time, which
is seemingly caused by a security patch not being applied correctly to
production code. I am not sure what chain of events led to it (and
please feel free to correct me if I misunderstood anything) but it makes
me a bit worried, for the following reasons:

1. We had broken code which is not in gerrit or actually anywhere in
repos running in production.

2. As far as I know, none of people knowing the actual code that was
broken knew there were problems with applying the patch (even though it
has .failed as filename in filesystem, so I wonder what happened there)
or that this particular piece of code was substantially different in
production than in repos. Of course we've figured it out but it was not
immediately known AFAIK.

3. We knew about the problem only after stumbling on it by chance on
mediawiki.org. It could as well be less visible but much more critical
and would be deployed further by production train before we noticed it.

4. We have out-of-repo code living on our production long term, while
code underneath it changes, and developers of the code being patched are
not aware of this going on. These changes may as well have invalidated
the security patch instead of producing visible error and we wouldn't know.

I understand the challenges with providing and applying security patches
for code that is both public and production under heavy use.  But the
failure scenario we've had is rather worrying for me and I wonder if we
could do better in handling security patches to lower the probability of
this happening in the future.

I don't know enough about security patches handling process to propose
anything, but I hope that people who do would help with this, so I'd
like to initiate the discussion about it.

I also avoided specific details in case the discussion is publicly
archived, but please ping me if you want to know them.

Thanks,
-- 
Stas Malyshev
smalyshev at wikimedia.org