On Fri, Mar 7, 2014 at 1:54 PM, Tyler Romeo tylerromeo@gmail.com wrote:
On Fri, Mar 7, 2014 at 4:49 PM, Matthew Flaschen <mflaschen@wikimedia.org
wrote:
Yes, it does. Unless the entire branch has a serious problem (500s or major caching problems, etc.), we don't generally switch the entire
branch
back.
That means the only option is fix or revert a commit. The general rule
is
to do changes in master before cherry-picking to the branch.
What you're saying is that the software development process for MediaWiki is so tightly coupled with the operations deployment process, that development has to be held up because of problems in operations. That's a problem.
With all due respect; hell, yes, development comes in second to operational stability.
This is not disrespecting development, which is extremely important by any measure. But we're running a top-10 worldwide website, a key worldwide information resource for humanity as a whole. We cannot cripple development to try and maximize stability, but stability has to be priority 1. Any large website's teams will have the same attitude.
I've had operational outages reach the top of everyone's news source/feed/newspaper/broadcast. This is an exceptionally unpleasant experience.
It's true that an enlightened balanced approach understands that existing instability that is baked in to prod must be developed out, in many cases, and that too many roadblocks in development will therefore be counterproductive. But too fast is lethal; too slow builds up technical debt, but at a comprehensible rate. If you aren't sure what just happened, the technical debt can be retired at everyone's patience and leisure, and with some feature slip.