On Fri, Jun 4, 2010 at 3:12 PM, Platonides Platonides@gmail.com wrote:
Rob Lanphier wrote:
A full test pass with all of the different configurations isn't going to
be
possible, so some help with testing the different configurations would be wonderful. We'll have a fast fallback plan in place should we
accidentally
break the other wikis, but obviously it'd be better to get it right the first time.
A bit off topic but, what makes it impossible? It shouldn't be /that/ hard. Fixing it may be useful for future deployments.
Well, it's possible, but it would delay the deployment. One plan we discussed on #wikimedia-tech was to deploy the FlaggedRevs_alpha branch to en.wikipedia.org, rather than deploying the FlaggedRevs trunk. FlaggedRevs_alpha is what is currently running on flaggedrevs.labs.wikimedia
A couple of different ways of going about it: 1. Deploy FlaggedRevs, but then drop back to FlaggedRevs_alpha if there's obvious breakage on one of the other wikis we can't fix quickly 2. Deploy FlaggedRevs_alpha from the start
Right now, we really don't want to delay deployment, since delays beget delays. Thoughts on a preferred strategy?
Rob
2010/6/5 Rob Lanphier robla@wikimedia.org:
A couple of different ways of going about it:
- Deploy FlaggedRevs, but then drop back to FlaggedRevs_alpha if there's
obvious breakage on one of the other wikis we can't fix quickly 2. Deploy FlaggedRevs_alpha from the start
Right now, we really don't want to delay deployment, since delays beget delays. Thoughts on a preferred strategy?
I don't think strategy #1 makes any sense at all. What I'd do is: * Update FlaggedRevs_alpha to the desired state and check it on the labs wikis * Switch wikis from FlaggedRevs to FlaggedRevs_alpha * The old FlaggedRevs dir now sits around not being used. Leave it like this for a while so we can easily switch wikis back to it * After some time, update FlaggedRevs to be identical to FlaggedRevs_alpha, and switch the wikis back, so we don't drag this _alpha thing along forever.
The rationale behind this is that FlaggedRevs is a known-good state more so than FlaggedRevs_alpha.
Roan Kattouw (Catrope)
On Fri, Jun 4, 2010 at 3:49 PM, Roan Kattouw roan.kattouw@gmail.com wrote:
I don't think [deploying the new FlaggedRevs to all wikis] makes any sense at all. What I'd do is:
- Update FlaggedRevs_alpha to the desired state and check it on the labs
wikis
- Switch wikis from FlaggedRevs to FlaggedRevs_alpha
- The old FlaggedRevs dir now sits around not being used. Leave it
like this for a while so we can easily switch wikis back to it
- After some time, update FlaggedRevs to be identical to
FlaggedRevs_alpha, and switch the wikis back, so we don't drag this _alpha thing along forever.
The rationale behind this is that FlaggedRevs is a known-good state more so than FlaggedRevs_alpha.
Well phooey. We discussed this option today in our status update, and based on what Aaron says, this might not be an option. I didn't fully understand the issue, so my apologies if I get this wrong, but I'll give it a shot anyway.
What Aaron says is that the use of FlaggedRevs_alpha on flaggedrevs.labs.wikimedia.org involves some more hacks than merely using a different branch. There are also some configuration changes that disable the message cache and possibly some other stuff. Without the hacks in place, there's a race condition that can cause different messages to show up randomly.
So, now we've got two different options: 1. If it's even possible, figure out how to work around the message cache race condition 2. Try to deploy to all production wikis at once next week.
Neither of those gives me warm fuzzies right now, but #2 seems to be our new plan of record. Thoughts?
Rob
2010/6/8 Rob Lanphier robla@wikimedia.org:
What Aaron says is that the use of FlaggedRevs_alpha on flaggedrevs.labs.wikimedia.org involves some more hacks than merely using a different branch. There are also some configuration changes that disable the message cache and possibly some other stuff. Without the hacks in place, there's a race condition that can cause different messages to show up randomly.
Ah, crap, totally forgot about the message cache issues.
So, now we've got two different options:
- If it's even possible, figure out how to work around the message cache
race condition 2. Try to deploy to all production wikis at once next week.
Neither of those gives me warm fuzzies right now, but #2 seems to be our new plan of record. Thoughts?
Yeah, #2 is probably easiest. The message cache currently can't be coerced into storing two different values for the same message, to be used on different wikis, AFAIK.
Roan Kattouw (Catrope)
On Mon, Jun 7, 2010 at 3:15 PM, Roan Kattouw roan.kattouw@gmail.com wrote:
2010/6/8 Rob Lanphier robla@wikimedia.org:
So, now we've got two different options [for deploying FlaggedRevs]:
- If it's even possible, figure out how to work around the message
cache
race condition [and deploy FlaggedRevs_alpha only to en.wikipedia.org] 2. Try to deploy to all production wikis at once next week.
Neither of those gives me warm fuzzies right now, but #2 seems to be our
new
plan of record. Thoughts?
Yeah, #2 is probably easiest. The message cache currently can't be coerced into storing two different values for the same message, to be used on different wikis, AFAIK.
One idea I'd like to float: we might be able to reconcile the messages between the two branches, and using the "Mediawiki:" namespace to override the messages we need to override (pretty much what we're already doing). I have no idea just how awful that would be, and there may be other risks I'm not accounting for, but I'm throwing that idea out there just in case it might work.
We really don't want to delay the launch at this point, but we also don't want to break any of the other production wikis in the process of rolling this out to en.wikipedia.org, so any ideas you all have on risk mitigation/minimization would be greatly appreciated.
Thanks! Rob
2010/6/8 Rob Lanphier robla@robla.net:
One idea I'd like to float: we might be able to reconcile the messages between the two branches, and using the "Mediawiki:" namespace to override the messages we need to override (pretty much what we're already doing). I have no idea just how awful that would be, and there may be other risks I'm not accounting for, but I'm throwing that idea out there just in case it might work.
For a one-wiki rollout, this would work. However, it's probably more complex than just updating the software across the board and crossing our fingers.
We really don't want to delay the launch at this point, but we also don't want to break any of the other production wikis in the process of rolling this out to en.wikipedia.org, so any ideas you all have on risk mitigation/minimization would be greatly appreciated.
IMO you should ascertain with some degree of confidence that your updated code won't break vanilla FR setups (using a labs wiki or something), then just roll it out.
Roan Kattouw (Catrope)
wikitech-l@lists.wikimedia.org