Hi everyone, There is a long time between production release on Android. The reason

this is because we have featured merged into master that are sound

engineering standpoint, but aren't quite ready for release yet. These features often block production releases. As product owner, pushing out regular bug fix builds would make me very happy! But there is a requirement that we are able to not push out

that are merged but not ready for production yet. This can probably me managed by feature flags. Dan Duvall (on cc) from Release Engineering will consult to help

Bernd figure out what their process should be for maximum

Thanks! Dan -- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org

https://lists.wikimedia.org/mailman/listinfo/mobile-l

Feature flags would help, but they also add an extra development investment to make sure all features are engineered in a way that a flag can shut them off without other bad things happening (not necessarily a bad thing, but require more effort). Another route to go is to manage this in git using the branches. There are several methodologies for this, and your branching strategy will depend mostly on how the team wants to operate. Pretty much all of them boil down to NOT merging features into master that are not going to be deployed after the current sprint. On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc <tfinc(a)wikimedia.org> wrote: > Excited to see this. In order for this to be successful we'll want to > developer a health dashboard for the app and set alarming for it > whenever we dip above/below certain thresholds. One of the challenges > that you face when releasing often is that the amount of attention you > keep during small iterative releases. It's easy to keep very focused > for 2-3 releases but after that attention can drift to just the major > releases. And while it's great to read reviews and find out a > subjective metric of how we're doing we need to get in front of it > with objective metrics. > > Thus having an app health dashboard showcasing: search, readership, > editing, etc can easily show you if you've had any regressions. These > would not only be useful for small bug fix releases but would also > help validate our major product releases. > > I'll leave it with you guys to define what metrics are necessary to > define a healthy app. > > --tomasz > > On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry <dgarry(a)wikimedia.org> > wrote: > > Hi everyone, > > > > There is a long time between production release on Android. The > reason for > > this is because we have featured merged into master that are sound > from an > > engineering standpoint, but aren't quite ready for release yet. These > > features often block production releases. > > > > As product owner, pushing out regular bug fix builds would make me > very > > happy! But there is a requirement that we are able to not push out > features > > that are merged but not ready for production yet. This can probably > me > > managed by feature flags. > > > > Dan Duvall (on cc) from Release Engineering will consult to help > Dmitry and > > Bernd figure out what their process should be for maximum > effectiveness. > > > > Thanks! > > > > Dan > > > > -- > > Dan Garry > > Associate Product Manager, Mobile Apps > > Wikimedia Foundation > > > > _______________________________________________ > > Mobile-l mailing list > > Mobile-l(a)lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > > > _______________________________________________ > Mobile-l mailing list > Mobile-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > -- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

As luck would have it, a very good tech podcast I subscribe to recently discussed this subject. It is a pretty good listen and discusses trade offs for both feature flags and branching. http://edgecasesshow.com/123-whats-the-deal-with-nsinteger.html (Topic is covered in the 2nd half of the show) On Monday, March 9, 2015, Dan Duvall &lt;dduvall(a)wikimedia.org&gt; wrote: > One very simple model that I think works well with distributed > software (and should play somewhat nice with Gerrit) is to branch per minor > release (assuming somewhat semantic versioning). In this model, master can > continue to serve as an integration branch, bug fixes are developed against > the release branch and merged following each patch release, and features > are developed against master per usual. With Gerrit in the mix, you're > essentially left with a two-stage merge for bug fixes which can be a bit of > extra work to get them merged into master, but at least the pipeline is > greased for getting them released. > > I can't say whether this would fit your team's workflow and, as Corey > mentioned, there are many different branching models to consider, each with > its own focus and drawbacks.[1][2] The one I've outlined above is geared > more for stability and maintenance but can probably be tweaked for more > frequent releases of features as well. I'm happy to brainstorm further. > > [1] > http://blog.codinghorror.com/software-branching-and-parallel-universes/ > [2] https://msdn.microsoft.com/en-us/library/bb668955.aspx > > > On Mon, Mar 9, 2015 at 2:30 PM, Corey Floyd &lt;cfloyd(a)wikimedia.org&gt; > wrote: > >> Feature flags would help, but they also add an extra development >> investment to make sure all features are engineered in a way that a flag >> can shut them off without other bad things happening (not necessarily a bad >> thing, but require more effort). >> >> Another route to go is to manage this in git using the branches. >> There are several methodologies for this, and your branching strategy will >> depend mostly on how the team wants to operate. Pretty much all of them >> boil down to NOT merging features into master that are not going to be >> deployed after the current sprint. >> >> >> >> >> On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc &lt;tfinc(a)wikimedia.org&gt; >> wrote: >> >>> Excited to see this. In order for this to be successful we'll want to >>> developer a health dashboard for the app and set alarming for it >>> whenever we dip above/below certain thresholds. One of the challenges >>> that you face when releasing often is that the amount of attention >>> you >>> keep during small iterative releases. It's easy to keep very focused >>> for 2-3 releases but after that attention can drift to just the major >>> releases. And while it's great to read reviews and find out a >>> subjective metric of how we're doing we need to get in front of it >>> with objective metrics. >>> >>> Thus having an app health dashboard showcasing: search, readership, >>> editing, etc can easily show you if you've had any regressions. These >>> would not only be useful for small bug fix releases but would also >>> help validate our major product releases. >>> >>> I'll leave it with you guys to define what metrics are necessary to >>> define a healthy app. >>> >>> --tomasz >>> >>> On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry &lt;dgarry(a)wikimedia.org&gt; >>> wrote: >>> > Hi everyone, >>> > >>> > There is a long time between production release on Android. The >>> reason for >>> > this is because we have featured merged into master that are sound >>> from an >>> > engineering standpoint, but aren't quite ready for release yet. >>> These >>> > features often block production releases. >>> > >>> > As product owner, pushing out regular bug fix builds would make me >>> very >>> > happy! But there is a requirement that we are able to not push out >>> features >>> > that are merged but not ready for production yet. This can >>> probably me >>> > managed by feature flags. >>> > >>> > Dan Duvall (on cc) from Release Engineering will consult to help >>> Dmitry and >>> > Bernd figure out what their process should be for maximum >>> effectiveness. >>> > >>> > Thanks! >>> > >>> > Dan >>> > >>> > -- >>> > Dan Garry >>> > Associate Product Manager, Mobile Apps >>> > Wikimedia Foundation >>> > >>> > _______________________________________________ >>> > Mobile-l mailing list >>> > Mobile-l(a)lists.wikimedia.org >>> > https://lists.wikimedia.org/mailman/listinfo/mobile-l >>> > >>> >>> _______________________________________________ >>> Mobile-l mailing list >>> Mobile-l(a)lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >>> >> >> >> >> -- >> Corey Floyd >> Software Engineer >> Mobile Apps / iOS >> Wikimedia Foundation >> >> _______________________________________________ >> Mobile-l mailing list >> Mobile-l(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> >> > > > -- > Dan Duvall > Automation Engineer > Wikimedia Foundation <http://wikimediafoundation.org> > -- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

I have not listened to the podcast yet, but my opinion is similar to what Brian described. So, +1 for feature flags. Bernd On Wed, Mar 11, 2015 at 1:40 PM, Brian Gerstle &lt;bgerstle(a)wikimedia.org&gt; wrote: > Listened to it this morning, gotta say I think I'm on "Team Flag," if > it's done well. IMO we should try one (or both) and see how it goes. > Here's my take on each approach: > > Branches are cheap to implement, but come at the a potentially high > cost if you don't continually rebase and try to keep the work scope as > small as possible. At a previous job we used git submodules as pseudo > feature branches. There were common problems w/ dependencies between > branches and between branches and the main repo, which as the hosts mention > are often pushed later in the process. > > Flags are more expensive to implement up front, but allow for truly > continuous integration and delivery—as well as the potential for gradual > rollout. IMO it could also lead to better architected code since feature > flags require you to codify the boundaries between the "platform" and the > features (and between the features themselves). You also need to limit > global state and have good test coverage (which we should do anyway) in > order to keep undesired side-effects to a minimum. My previous job also > switched to this model and was able to improve their release cadence and > sync between features (IIRC, don't have any concrete evidence to back it up > unfortunately). > > > On Tue, Mar 10, 2015 at 11:16 PM, Corey Floyd &lt;cfloyd(a)wikimedia.org&gt; > wrote: > >> As luck would have it, a very good tech podcast I subscribe >> to recently discussed this subject. It is a pretty good listen and >> discusses trade offs for both feature flags and branching. >> >> >> http://edgecasesshow.com/123-whats-the-deal-with-nsinteger.html >> >> (Topic is covered in the 2nd half of the show) >> >> >> On Monday, March 9, 2015, Dan Duvall &lt;dduvall(a)wikimedia.org&gt; wrote: >> >>> One very simple model that I think works well with distributed >>> software (and should play somewhat nice with Gerrit) is to branch per minor >>> release (assuming somewhat semantic versioning). In this model, master can >>> continue to serve as an integration branch, bug fixes are developed against >>> the release branch and merged following each patch release, and features >>> are developed against master per usual. With Gerrit in the mix, you're >>> essentially left with a two-stage merge for bug fixes which can be a bit of >>> extra work to get them merged into master, but at least the pipeline is >>> greased for getting them released. >>> >>> I can't say whether this would fit your team's workflow and, as >>> Corey mentioned, there are many different branching models to consider, >>> each with its own focus and drawbacks.[1][2] The one I've outlined above is >>> geared more for stability and maintenance but can probably be tweaked for >>> more frequent releases of features as well. I'm happy to brainstorm further. >>> >>> [1] >>> http://blog.codinghorror.com/software-branching-and-parallel-universes/ >>> [2] https://msdn.microsoft.com/en-us/library/bb668955.aspx >>> >>> >>> On Mon, Mar 9, 2015 at 2:30 PM, Corey Floyd &lt;cfloyd(a)wikimedia.org&gt; >>> wrote: >>> >>>> Feature flags would help, but they also add an extra development >>>> investment to make sure all features are engineered in a way that a flag >>>> can shut them off without other bad things happening (not necessarily a bad >>>> thing, but require more effort). >>>> >>>> Another route to go is to manage this in git using the branches. >>>> There are several methodologies for this, and your branching strategy will >>>> depend mostly on how the team wants to operate. Pretty much all of them >>>> boil down to NOT merging features into master that are not going to be >>>> deployed after the current sprint. >>>> >>>> >>>> >>>> >>>> On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc &lt;tfinc(a)wikimedia.org&gt; >>>> wrote: >>>> >>>>> Excited to see this. In order for this to be successful we'll want >>>>> to >>>>> developer a health dashboard for the app and set alarming for it >>>>> whenever we dip above/below certain thresholds. One of the >>>>> challenges >>>>> that you face when releasing often is that the amount of attention >>>>> you >>>>> keep during small iterative releases. It's easy to keep very >>>>> focused >>>>> for 2-3 releases but after that attention can drift to just the >>>>> major >>>>> releases. And while it's great to read reviews and find out a >>>>> subjective metric of how we're doing we need to get in front of it >>>>> with objective metrics. >>>>> >>>>> Thus having an app health dashboard showcasing: search, readership, >>>>> editing, etc can easily show you if you've had any regressions. >>>>> These >>>>> would not only be useful for small bug fix releases but would also >>>>> help validate our major product releases. >>>>> >>>>> I'll leave it with you guys to define what metrics are necessary to >>>>> define a healthy app. >>>>> >>>>> --tomasz >>>>> >>>>> On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry &lt;dgarry(a)wikimedia.org&gt; >>>>> wrote: >>>>> > Hi everyone, >>>>> > >>>>> > There is a long time between production release on Android. The >>>>> reason for >>>>> > this is because we have featured merged into master that are >>>>> sound from an >>>>> > engineering standpoint, but aren't quite ready for release yet. >>>>> These >>>>> > features often block production releases. >>>>> > >>>>> > As product owner, pushing out regular bug fix builds would make >>>>> me very >>>>> > happy! But there is a requirement that we are able to not push >>>>> out features >>>>> > that are merged but not ready for production yet. This can >>>>> probably me >>>>> > managed by feature flags. >>>>> > >>>>> > Dan Duvall (on cc) from Release Engineering will consult to help >>>>> Dmitry and >>>>> > Bernd figure out what their process should be for maximum >>>>> effectiveness. >>>>> > >>>>> > Thanks! >>>>> > >>>>> > Dan >>>>> > >>>>> > -- >>>>> > Dan Garry >>>>> > Associate Product Manager, Mobile Apps >>>>> > Wikimedia Foundation >>>>> > >>>>> > _______________________________________________ >>>>> > Mobile-l mailing list >>>>> > Mobile-l(a)lists.wikimedia.org >>>>> > https://lists.wikimedia.org/mailman/listinfo/mobile-l >>>>> > >>>>> >>>>> _______________________________________________ >>>>> Mobile-l mailing list >>>>> Mobile-l(a)lists.wikimedia.org >>>>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >>>>> >>>> >>>> >>>> >>>> -- >>>> Corey Floyd >>>> Software Engineer >>>> Mobile Apps / iOS >>>> Wikimedia Foundation >>>> >>>> _______________________________________________ >>>> Mobile-l mailing list >>>> Mobile-l(a)lists.wikimedia.org >>>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >>>> >>>> >>> >>> >>> -- >>> Dan Duvall >>> Automation Engineer >>> Wikimedia Foundation <http://wikimediafoundation.org> >>> >> >> >> -- >> Corey Floyd >> Software Engineer >> Mobile Apps / iOS >> Wikimedia Foundation >> >> >> _______________________________________________ >> Mobile-l mailing list >> Mobile-l(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> >> > > > -- > EN Wikipedia user page: > https://en.wikipedia.org/wiki/User:Brian.gerstle > IRC: bgerstle > > _______________________________________________ > Mobile-l mailing list > Mobile-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > >

Yeah - after listening to the podcast I was on the fence about it too… I think they bring up good points on the tradeoffs of both (even the hosts were split on their preference at the end). I was initially attracted towards the git methodology as it gets these business decisions out of the code base - but I also think that having these flags that work at runtime (from a debug menu) is really valuable for testing purposes. One thing I am not convinced on is that there will be "less work at the end" when integrating features into a release with the feature flag methodology - I feel like that could still be a problem if you are not merging/rebasing. They discussed this in the podcast almost as an aside - but it is entirely possible to develop conflicting features using feature flags if developers do not coordinate (I feel this is really an orthogonal issue to enabling features). Still - I think the feature flag solution has promise - but I am way more interested in it if we do it as run time option to get the testing/QA benefits. On Wed, Mar 11, 2015 at 4:08 PM, Bernd Sitzmann &lt;bernd(a)wikimedia.org&gt; wrote: > I have not listened to the podcast yet, but my opinion is similar to > what Brian described. > So, +1 for feature flags. > > Bernd > > On Wed, Mar 11, 2015 at 1:40 PM, Brian Gerstle &lt;bgerstle(a)wikimedia.org > > wrote: > >> Listened to it this morning, gotta say I think I'm on "Team Flag," if >> it's done well. IMO we should try one (or both) and see how it goes. >> Here's my take on each approach: >> >> Branches are cheap to implement, but come at the a potentially high >> cost if you don't continually rebase and try to keep the work scope as >> small as possible. At a previous job we used git submodules as pseudo >> feature branches. There were common problems w/ dependencies between >> branches and between branches and the main repo, which as the hosts mention >> are often pushed later in the process. >> >> Flags are more expensive to implement up front, but allow for truly >> continuous integration and delivery—as well as the potential for gradual >> rollout. IMO it could also lead to better architected code since feature >> flags require you to codify the boundaries between the "platform" and the >> features (and between the features themselves). You also need to limit >> global state and have good test coverage (which we should do anyway) in >> order to keep undesired side-effects to a minimum. My previous job also >> switched to this model and was able to improve their release cadence and >> sync between features (IIRC, don't have any concrete evidence to back it up >> unfortunately). >> >> >> On Tue, Mar 10, 2015 at 11:16 PM, Corey Floyd &lt;cfloyd(a)wikimedia.org&gt; >> wrote: >> >>> As luck would have it, a very good tech podcast I subscribe >>> to recently discussed this subject. It is a pretty good listen and >>> discusses trade offs for both feature flags and branching. >>> >>> >>> http://edgecasesshow.com/123-whats-the-deal-with-nsinteger.html >>> >>> (Topic is covered in the 2nd half of the show) >>> >>> >>> On Monday, March 9, 2015, Dan Duvall &lt;dduvall(a)wikimedia.org&gt; wrote: >>> >>>> One very simple model that I think works well with distributed >>>> software (and should play somewhat nice with Gerrit) is to branch per minor >>>> release (assuming somewhat semantic versioning). In this model, master can >>>> continue to serve as an integration branch, bug fixes are developed against >>>> the release branch and merged following each patch release, and features >>>> are developed against master per usual. With Gerrit in the mix, you're >>>> essentially left with a two-stage merge for bug fixes which can be a bit of >>>> extra work to get them merged into master, but at least the pipeline is >>>> greased for getting them released. >>>> >>>> I can't say whether this would fit your team's workflow and, as >>>> Corey mentioned, there are many different branching models to consider, >>>> each with its own focus and drawbacks.[1][2] The one I've outlined above is >>>> geared more for stability and maintenance but can probably be tweaked for >>>> more frequent releases of features as well. I'm happy to brainstorm further. >>>> >>>> [1] >>>> http://blog.codinghorror.com/software-branching-and-parallel-universes/ >>>> [2] https://msdn.microsoft.com/en-us/library/bb668955.aspx >>>> >>>> >>>> On Mon, Mar 9, 2015 at 2:30 PM, Corey Floyd &lt;cfloyd(a)wikimedia.org&gt; >>>> wrote: >>>> >>>>> Feature flags would help, but they also add an extra development >>>>> investment to make sure all features are engineered in a way that a flag >>>>> can shut them off without other bad things happening (not necessarily a bad >>>>> thing, but require more effort). >>>>> >>>>> Another route to go is to manage this in git using the branches. >>>>> There are several methodologies for this, and your branching strategy will >>>>> depend mostly on how the team wants to operate. Pretty much all of them >>>>> boil down to NOT merging features into master that are not going to be >>>>> deployed after the current sprint. >>>>> >>>>> >>>>> >>>>> >>>>> On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc &lt;tfinc(a)wikimedia.org&gt; >>>>> wrote: >>>>> >>>>>> Excited to see this. In order for this to be successful we'll >>>>>> want to >>>>>> developer a health dashboard for the app and set alarming for it >>>>>> whenever we dip above/below certain thresholds. One of the >>>>>> challenges >>>>>> that you face when releasing often is that the amount of >>>>>> attention you >>>>>> keep during small iterative releases. It's easy to keep very >>>>>> focused >>>>>> for 2-3 releases but after that attention can drift to just the >>>>>> major >>>>>> releases. And while it's great to read reviews and find out a >>>>>> subjective metric of how we're doing we need to get in front of it >>>>>> with objective metrics. >>>>>> >>>>>> Thus having an app health dashboard showcasing: search, >>>>>> readership, >>>>>> editing, etc can easily show you if you've had any regressions. >>>>>> These >>>>>> would not only be useful for small bug fix releases but would also >>>>>> help validate our major product releases. >>>>>> >>>>>> I'll leave it with you guys to define what metrics are necessary >>>>>> to >>>>>> define a healthy app. >>>>>> >>>>>> --tomasz >>>>>> >>>>>> On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry &lt;dgarry(a)wikimedia.org&gt; >>>>>> wrote: >>>>>> > Hi everyone, >>>>>> > >>>>>> > There is a long time between production release on Android. The >>>>>> reason for >>>>>> > this is because we have featured merged into master that are >>>>>> sound from an >>>>>> > engineering standpoint, but aren't quite ready for release yet. >>>>>> These >>>>>> > features often block production releases. >>>>>> > >>>>>> > As product owner, pushing out regular bug fix builds would make >>>>>> me very >>>>>> > happy! But there is a requirement that we are able to not push >>>>>> out features >>>>>> > that are merged but not ready for production yet. This can >>>>>> probably me >>>>>> > managed by feature flags. >>>>>> > >>>>>> > Dan Duvall (on cc) from Release Engineering will consult to >>>>>> help Dmitry and >>>>>> > Bernd figure out what their process should be for maximum >>>>>> effectiveness. >>>>>> > >>>>>> > Thanks! >>>>>> > >>>>>> > Dan >>>>>> > >>>>>> > -- >>>>>> > Dan Garry >>>>>> > Associate Product Manager, Mobile Apps >>>>>> > Wikimedia Foundation >>>>>> > >>>>>> > _______________________________________________ >>>>>> > Mobile-l mailing list >>>>>> > Mobile-l(a)lists.wikimedia.org >>>>>> > https://lists.wikimedia.org/mailman/listinfo/mobile-l >>>>>> > >>>>>> >>>>>> _______________________________________________ >>>>>> Mobile-l mailing list >>>>>> Mobile-l(a)lists.wikimedia.org >>>>>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Corey Floyd >>>>> Software Engineer >>>>> Mobile Apps / iOS >>>>> Wikimedia Foundation >>>>> >>>>> _______________________________________________ >>>>> Mobile-l mailing list >>>>> Mobile-l(a)lists.wikimedia.org >>>>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >>>>> >>>>> >>>> >>>> >>>> -- >>>> Dan Duvall >>>> Automation Engineer >>>> Wikimedia Foundation <http://wikimediafoundation.org> >>>> >>> >>> >>> -- >>> Corey Floyd >>> Software Engineer >>> Mobile Apps / iOS >>> Wikimedia Foundation >>> >>> >>> _______________________________________________ >>> Mobile-l mailing list >>> Mobile-l(a)lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >>> >>> >> >> >> -- >> EN Wikipedia user page: >> https://en.wikipedia.org/wiki/User:Brian.gerstle >> IRC: bgerstle >> >> _______________________________________________ >> Mobile-l mailing list >> Mobile-l(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> >> > _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l