Extrapolating from our experience from two-week deploy cycles and what bugs we find at each stage, would moving to a one-week deploy cycle substantially increase the number of users exposed to really bad bugs? For instance, if we're currently finding and fixing about 2 deployment blocker bugs per cycle after the mediawiki.org deploy but before the non-Wikipedias deploy, that data might imply that we should structure a 1-week cycle differently -- perhaps with three stages rather than two.

One other plan that was discussed in February, if I recall, was something like this: week0, Thursday: Deploy wmf1 to test and mw.org week1, Monday: Deploy wmf1 to first-round wikis week1, Thursday: Deploy wmf1 to remaining wikis and wmf2 to test and mw.org. week2, Monday: Deploy wmf2 to first-round wikis week2, Thursday: Deploy wmf2 to remaining wikis and wmf3 to test and mw.org. etc. That has the advantage of preserving the separation of test+mw.org from the more user-focused wikis, as Sumana mentioned.

I'm pretty sure this email will have a mid-air collision with Robla and his writeup of the proposal you outlined, Brad.

Also, the majority of bugs that are in the Highest/Immediate priority level (from my gut assessment, I don't have the data here) are found after a deploy to non-WP projects.

> Backing that impression up with Bugzilla data: > <tl;dr>: That's hard.

Andre, thanks for trying to run the query on this. Yeah, we split, rename, reprioritize, and otherwise mess with our Bugzilla tickets so often that it'd probably be a huge travail to thoroughly research the question I asked, and that would delay this decision beyond this week. So I'm fine with going on people's rough estimates and impressions instead of going the "ALL DATA ALL THE TIME" route.

On Mon, May 6, 2013 at 7:20 PM, MZMcBride &lt;z(a)mzmcbride.com&gt; wrote: make A lot of this could potentially be addressed in a consistent manner across wikis if we applied the alpha->beta->prod (or just beta->prod for starters) channel model that's used on the Wikimedia mobile sites. Then features (whether in core or extensions) could be flagged for alpha or beta readiness, and users would only get them if they've decided to opt into either of those channels. We could still flip the switch from beta->prod, but that decision could be decoupled from the weekly deployment cycle. This would likely be done for features & changes which have significant user-facing impact, and where segregation into "on" and "off" modes is possible (not always the case). We may want to consider at least putting some such scaffolding for beta->prod desktop modes into place before shifting to weekly deployments, although if that holds up this change significantly, I'd be in favor of making the shift first and then iterating. Right now we have lots of individual "experimental" prefs, some dark launch URL parameters (&useNew=1 for the account creation forms etc.), some changes that are announced widely but then rolled out immediately (section edit link change), etc. What would be the disadvantage of having a single "I'd like the latest and greatest changes once they come in" preference for our users? The main disadvantage I see is that we'd need to temporarily retain two codepaths for significant user-facing changes, potentially increasing code complexity a fair bit, but perhaps reducing post-launch cost in return. And we'd need to consider more carefully if/when to make the beta/prod switch -- not necessarily a bad thing. ;-) Have there been any negative experiences with this model on the mobile sites? Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

We may want to consider at least putting some such scaffolding for beta->prod desktop modes into place before shifting to weekly deployments, although if that holds up this change significantly, I'd be in favor of making the shift first and then iterating. Right now we have lots of individual "experimental" prefs, some dark launch URL parameters (&useNew=1 for the account creation forms etc.), some changes that are announced widely but then rolled out immediately (section edit link change), etc. What would be the disadvantage of having a single "I'd like the latest and greatest changes once they come in" preference for our users? The main disadvantage I see is that we'd need to temporarily retain two codepaths for significant user-facing changes, potentially increasing code complexity a fair bit, but perhaps reducing post-launch cost in return. And we'd need to consider more carefully if/when to make the beta/prod switch -- not necessarily a bad thing. ;-) Have there been any negative experiences with this model on the mobile sites?

I think this is awesome for features ... but if we're putting work into this, I would love even more to have a clustered a+b production environment, such that 10% of folks are put on the new release (cluster a) and then it gets pushed over to cluster b. Then we can also test performance in a real world environment, and breakages only happen for 10% (PS the 10% number was pulled out of thin air).

I was interpreting Leslie as saying 10% cross-cut throughout. I.E. 10% of German Wiktionary, 10% of Japanese Wikinews, 10% of English Wikipedia, 10% of everything. That would mean we would really get wide testing before it rolled out to the 90%.

I think this is awesome for features ... but if we're putting work into this, I would love even more to have a clustered a+b production environment, such that 10% of folks are put on the new release (cluster a) and then it gets pushed over to cluster b. Then we can also test performance in a real world environment, and breakages only happen for 10% (PS the 10% number was pulled out of thin air). The opt-in beta is much too limited, as well as being inapplicable to the vast majority of our traffic (logged in users are such a small percentage) to make proper comparisons. You could also see the impact of features on usage for average users.