Hi everyone,
There is a long time between production release on Android. The reason for this is because we have featured merged into master that are sound from an engineering standpoint, but aren't quite ready for release yet. These features often block production releases.
As product owner, pushing out regular bug fix builds would make me very happy! But there is a requirement that we are able to not push out features that are merged but not ready for production yet. This can probably me managed by feature flags.
Dan Duvall (on cc) from Release Engineering will consult to help Dmitry and Bernd figure out what their process should be for maximum effectiveness.
Thanks!
Dan
Excited to see this. In order for this to be successful we'll want to developer a health dashboard for the app and set alarming for it whenever we dip above/below certain thresholds. One of the challenges that you face when releasing often is that the amount of attention you keep during small iterative releases. It's easy to keep very focused for 2-3 releases but after that attention can drift to just the major releases. And while it's great to read reviews and find out a subjective metric of how we're doing we need to get in front of it with objective metrics.
Thus having an app health dashboard showcasing: search, readership, editing, etc can easily show you if you've had any regressions. These would not only be useful for small bug fix releases but would also help validate our major product releases.
I'll leave it with you guys to define what metrics are necessary to define a healthy app.
--tomasz
On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry dgarry@wikimedia.org wrote:
Hi everyone,
There is a long time between production release on Android. The reason for this is because we have featured merged into master that are sound from an engineering standpoint, but aren't quite ready for release yet. These features often block production releases.
As product owner, pushing out regular bug fix builds would make me very happy! But there is a requirement that we are able to not push out features that are merged but not ready for production yet. This can probably me managed by feature flags.
Dan Duvall (on cc) from Release Engineering will consult to help Dmitry and Bernd figure out what their process should be for maximum effectiveness.
Thanks!
Dan
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Feature flags would help, but they also add an extra development investment to make sure all features are engineered in a way that a flag can shut them off without other bad things happening (not necessarily a bad thing, but require more effort).
Another route to go is to manage this in git using the branches. There are several methodologies for this, and your branching strategy will depend mostly on how the team wants to operate. Pretty much all of them boil down to NOT merging features into master that are not going to be deployed after the current sprint.
On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc tfinc@wikimedia.org wrote:
Excited to see this. In order for this to be successful we'll want to developer a health dashboard for the app and set alarming for it whenever we dip above/below certain thresholds. One of the challenges that you face when releasing often is that the amount of attention you keep during small iterative releases. It's easy to keep very focused for 2-3 releases but after that attention can drift to just the major releases. And while it's great to read reviews and find out a subjective metric of how we're doing we need to get in front of it with objective metrics.
Thus having an app health dashboard showcasing: search, readership, editing, etc can easily show you if you've had any regressions. These would not only be useful for small bug fix releases but would also help validate our major product releases.
I'll leave it with you guys to define what metrics are necessary to define a healthy app.
--tomasz
On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry dgarry@wikimedia.org wrote:
Hi everyone,
There is a long time between production release on Android. The reason
for
this is because we have featured merged into master that are sound from
an
engineering standpoint, but aren't quite ready for release yet. These features often block production releases.
As product owner, pushing out regular bug fix builds would make me very happy! But there is a requirement that we are able to not push out
features
that are merged but not ready for production yet. This can probably me managed by feature flags.
Dan Duvall (on cc) from Release Engineering will consult to help Dmitry
and
Bernd figure out what their process should be for maximum effectiveness.
Thanks!
Dan
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
One very simple model that I think works well with distributed software (and should play somewhat nice with Gerrit) is to branch per minor release (assuming somewhat semantic versioning). In this model, master can continue to serve as an integration branch, bug fixes are developed against the release branch and merged following each patch release, and features are developed against master per usual. With Gerrit in the mix, you're essentially left with a two-stage merge for bug fixes which can be a bit of extra work to get them merged into master, but at least the pipeline is greased for getting them released.
I can't say whether this would fit your team's workflow and, as Corey mentioned, there are many different branching models to consider, each with its own focus and drawbacks.[1][2] The one I've outlined above is geared more for stability and maintenance but can probably be tweaked for more frequent releases of features as well. I'm happy to brainstorm further.
[1] http://blog.codinghorror.com/software-branching-and-parallel-universes/ [2] https://msdn.microsoft.com/en-us/library/bb668955.aspx
On Mon, Mar 9, 2015 at 2:30 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Feature flags would help, but they also add an extra development investment to make sure all features are engineered in a way that a flag can shut them off without other bad things happening (not necessarily a bad thing, but require more effort).
Another route to go is to manage this in git using the branches. There are several methodologies for this, and your branching strategy will depend mostly on how the team wants to operate. Pretty much all of them boil down to NOT merging features into master that are not going to be deployed after the current sprint.
On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc tfinc@wikimedia.org wrote:
Excited to see this. In order for this to be successful we'll want to developer a health dashboard for the app and set alarming for it whenever we dip above/below certain thresholds. One of the challenges that you face when releasing often is that the amount of attention you keep during small iterative releases. It's easy to keep very focused for 2-3 releases but after that attention can drift to just the major releases. And while it's great to read reviews and find out a subjective metric of how we're doing we need to get in front of it with objective metrics.
Thus having an app health dashboard showcasing: search, readership, editing, etc can easily show you if you've had any regressions. These would not only be useful for small bug fix releases but would also help validate our major product releases.
I'll leave it with you guys to define what metrics are necessary to define a healthy app.
--tomasz
On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry dgarry@wikimedia.org wrote:
Hi everyone,
There is a long time between production release on Android. The reason
for
this is because we have featured merged into master that are sound from
an
engineering standpoint, but aren't quite ready for release yet. These features often block production releases.
As product owner, pushing out regular bug fix builds would make me very happy! But there is a requirement that we are able to not push out
features
that are merged but not ready for production yet. This can probably me managed by feature flags.
Dan Duvall (on cc) from Release Engineering will consult to help Dmitry
and
Bernd figure out what their process should be for maximum effectiveness.
Thanks!
Dan
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
As luck would have it, a very good tech podcast I subscribe to recently discussed this subject. It is a pretty good listen and discusses trade offs for both feature flags and branching.
http://edgecasesshow.com/123-whats-the-deal-with-nsinteger.html
(Topic is covered in the 2nd half of the show)
On Monday, March 9, 2015, Dan Duvall dduvall@wikimedia.org wrote:
One very simple model that I think works well with distributed software (and should play somewhat nice with Gerrit) is to branch per minor release (assuming somewhat semantic versioning). In this model, master can continue to serve as an integration branch, bug fixes are developed against the release branch and merged following each patch release, and features are developed against master per usual. With Gerrit in the mix, you're essentially left with a two-stage merge for bug fixes which can be a bit of extra work to get them merged into master, but at least the pipeline is greased for getting them released.
I can't say whether this would fit your team's workflow and, as Corey mentioned, there are many different branching models to consider, each with its own focus and drawbacks.[1][2] The one I've outlined above is geared more for stability and maintenance but can probably be tweaked for more frequent releases of features as well. I'm happy to brainstorm further.
[1] http://blog.codinghorror.com/software-branching-and-parallel-universes/ [2] https://msdn.microsoft.com/en-us/library/bb668955.aspx
On Mon, Mar 9, 2015 at 2:30 PM, Corey Floyd <cfloyd@wikimedia.org javascript:_e(%7B%7D,'cvml','cfloyd@wikimedia.org');> wrote:
Feature flags would help, but they also add an extra development investment to make sure all features are engineered in a way that a flag can shut them off without other bad things happening (not necessarily a bad thing, but require more effort).
Another route to go is to manage this in git using the branches. There are several methodologies for this, and your branching strategy will depend mostly on how the team wants to operate. Pretty much all of them boil down to NOT merging features into master that are not going to be deployed after the current sprint.
On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc <tfinc@wikimedia.org javascript:_e(%7B%7D,'cvml','tfinc@wikimedia.org');> wrote:
Excited to see this. In order for this to be successful we'll want to developer a health dashboard for the app and set alarming for it whenever we dip above/below certain thresholds. One of the challenges that you face when releasing often is that the amount of attention you keep during small iterative releases. It's easy to keep very focused for 2-3 releases but after that attention can drift to just the major releases. And while it's great to read reviews and find out a subjective metric of how we're doing we need to get in front of it with objective metrics.
Thus having an app health dashboard showcasing: search, readership, editing, etc can easily show you if you've had any regressions. These would not only be useful for small bug fix releases but would also help validate our major product releases.
I'll leave it with you guys to define what metrics are necessary to define a healthy app.
--tomasz
On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry <dgarry@wikimedia.org javascript:_e(%7B%7D,'cvml','dgarry@wikimedia.org');> wrote:
Hi everyone,
There is a long time between production release on Android. The reason
for
this is because we have featured merged into master that are sound
from an
engineering standpoint, but aren't quite ready for release yet. These features often block production releases.
As product owner, pushing out regular bug fix builds would make me very happy! But there is a requirement that we are able to not push out
features
that are merged but not ready for production yet. This can probably me managed by feature flags.
Dan Duvall (on cc) from Release Engineering will consult to help
Dmitry and
Bernd figure out what their process should be for maximum
effectiveness.
Thanks!
Dan
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org
javascript:_e(%7B%7D,'cvml','Mobile-l@lists.wikimedia.org');
Mobile-l mailing list Mobile-l@lists.wikimedia.org javascript:_e(%7B%7D,'cvml','Mobile-l@lists.wikimedia.org'); https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org javascript:_e(%7B%7D,'cvml','Mobile-l@lists.wikimedia.org'); https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Duvall Automation Engineer Wikimedia Foundation http://wikimediafoundation.org
Listened to it this morning, gotta say I think I'm on "Team Flag," if it's done well. IMO we should try one (or both) and see how it goes. Here's my take on each approach:
Branches are cheap to implement, but come at the a potentially high cost if you don't continually rebase and try to keep the work scope as small as possible. At a previous job we used git submodules as pseudo feature branches. There were common problems w/ dependencies between branches and between branches and the main repo, which as the hosts mention are often pushed later in the process.
Flags are more expensive to implement up front, but allow for truly continuous integration and delivery—as well as the potential for gradual rollout. IMO it could also lead to better architected code since feature flags require you to codify the boundaries between the "platform" and the features (and between the features themselves). You also need to limit global state and have good test coverage (which we should do anyway) in order to keep undesired side-effects to a minimum. My previous job also switched to this model and was able to improve their release cadence and sync between features (IIRC, don't have any concrete evidence to back it up unfortunately).
On Tue, Mar 10, 2015 at 11:16 PM, Corey Floyd cfloyd@wikimedia.org wrote:
As luck would have it, a very good tech podcast I subscribe to recently discussed this subject. It is a pretty good listen and discusses trade offs for both feature flags and branching.
http://edgecasesshow.com/123-whats-the-deal-with-nsinteger.html
(Topic is covered in the 2nd half of the show)
On Monday, March 9, 2015, Dan Duvall dduvall@wikimedia.org wrote:
One very simple model that I think works well with distributed software (and should play somewhat nice with Gerrit) is to branch per minor release (assuming somewhat semantic versioning). In this model, master can continue to serve as an integration branch, bug fixes are developed against the release branch and merged following each patch release, and features are developed against master per usual. With Gerrit in the mix, you're essentially left with a two-stage merge for bug fixes which can be a bit of extra work to get them merged into master, but at least the pipeline is greased for getting them released.
I can't say whether this would fit your team's workflow and, as Corey mentioned, there are many different branching models to consider, each with its own focus and drawbacks.[1][2] The one I've outlined above is geared more for stability and maintenance but can probably be tweaked for more frequent releases of features as well. I'm happy to brainstorm further.
[1] http://blog.codinghorror.com/software-branching-and-parallel-universes/ [2] https://msdn.microsoft.com/en-us/library/bb668955.aspx
On Mon, Mar 9, 2015 at 2:30 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Feature flags would help, but they also add an extra development investment to make sure all features are engineered in a way that a flag can shut them off without other bad things happening (not necessarily a bad thing, but require more effort).
Another route to go is to manage this in git using the branches. There are several methodologies for this, and your branching strategy will depend mostly on how the team wants to operate. Pretty much all of them boil down to NOT merging features into master that are not going to be deployed after the current sprint.
On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc tfinc@wikimedia.org wrote:
Excited to see this. In order for this to be successful we'll want to developer a health dashboard for the app and set alarming for it whenever we dip above/below certain thresholds. One of the challenges that you face when releasing often is that the amount of attention you keep during small iterative releases. It's easy to keep very focused for 2-3 releases but after that attention can drift to just the major releases. And while it's great to read reviews and find out a subjective metric of how we're doing we need to get in front of it with objective metrics.
Thus having an app health dashboard showcasing: search, readership, editing, etc can easily show you if you've had any regressions. These would not only be useful for small bug fix releases but would also help validate our major product releases.
I'll leave it with you guys to define what metrics are necessary to define a healthy app.
--tomasz
On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry dgarry@wikimedia.org wrote:
Hi everyone,
There is a long time between production release on Android. The
reason for
this is because we have featured merged into master that are sound
from an
engineering standpoint, but aren't quite ready for release yet. These features often block production releases.
As product owner, pushing out regular bug fix builds would make me
very
happy! But there is a requirement that we are able to not push out
features
that are merged but not ready for production yet. This can probably me managed by feature flags.
Dan Duvall (on cc) from Release Engineering will consult to help
Dmitry and
Bernd figure out what their process should be for maximum
effectiveness.
Thanks!
Dan
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Duvall Automation Engineer Wikimedia Foundation http://wikimediafoundation.org
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
I have not listened to the podcast yet, but my opinion is similar to what Brian described. So, +1 for feature flags.
Bernd
On Wed, Mar 11, 2015 at 1:40 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
Listened to it this morning, gotta say I think I'm on "Team Flag," if it's done well. IMO we should try one (or both) and see how it goes. Here's my take on each approach:
Branches are cheap to implement, but come at the a potentially high cost if you don't continually rebase and try to keep the work scope as small as possible. At a previous job we used git submodules as pseudo feature branches. There were common problems w/ dependencies between branches and between branches and the main repo, which as the hosts mention are often pushed later in the process.
Flags are more expensive to implement up front, but allow for truly continuous integration and delivery—as well as the potential for gradual rollout. IMO it could also lead to better architected code since feature flags require you to codify the boundaries between the "platform" and the features (and between the features themselves). You also need to limit global state and have good test coverage (which we should do anyway) in order to keep undesired side-effects to a minimum. My previous job also switched to this model and was able to improve their release cadence and sync between features (IIRC, don't have any concrete evidence to back it up unfortunately).
On Tue, Mar 10, 2015 at 11:16 PM, Corey Floyd cfloyd@wikimedia.org wrote:
As luck would have it, a very good tech podcast I subscribe to recently discussed this subject. It is a pretty good listen and discusses trade offs for both feature flags and branching.
http://edgecasesshow.com/123-whats-the-deal-with-nsinteger.html
(Topic is covered in the 2nd half of the show)
On Monday, March 9, 2015, Dan Duvall dduvall@wikimedia.org wrote:
One very simple model that I think works well with distributed software (and should play somewhat nice with Gerrit) is to branch per minor release (assuming somewhat semantic versioning). In this model, master can continue to serve as an integration branch, bug fixes are developed against the release branch and merged following each patch release, and features are developed against master per usual. With Gerrit in the mix, you're essentially left with a two-stage merge for bug fixes which can be a bit of extra work to get them merged into master, but at least the pipeline is greased for getting them released.
I can't say whether this would fit your team's workflow and, as Corey mentioned, there are many different branching models to consider, each with its own focus and drawbacks.[1][2] The one I've outlined above is geared more for stability and maintenance but can probably be tweaked for more frequent releases of features as well. I'm happy to brainstorm further.
[1] http://blog.codinghorror.com/software-branching-and-parallel-universes/ [2] https://msdn.microsoft.com/en-us/library/bb668955.aspx
On Mon, Mar 9, 2015 at 2:30 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Feature flags would help, but they also add an extra development investment to make sure all features are engineered in a way that a flag can shut them off without other bad things happening (not necessarily a bad thing, but require more effort).
Another route to go is to manage this in git using the branches. There are several methodologies for this, and your branching strategy will depend mostly on how the team wants to operate. Pretty much all of them boil down to NOT merging features into master that are not going to be deployed after the current sprint.
On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc tfinc@wikimedia.org wrote:
Excited to see this. In order for this to be successful we'll want to developer a health dashboard for the app and set alarming for it whenever we dip above/below certain thresholds. One of the challenges that you face when releasing often is that the amount of attention you keep during small iterative releases. It's easy to keep very focused for 2-3 releases but after that attention can drift to just the major releases. And while it's great to read reviews and find out a subjective metric of how we're doing we need to get in front of it with objective metrics.
Thus having an app health dashboard showcasing: search, readership, editing, etc can easily show you if you've had any regressions. These would not only be useful for small bug fix releases but would also help validate our major product releases.
I'll leave it with you guys to define what metrics are necessary to define a healthy app.
--tomasz
On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry dgarry@wikimedia.org wrote:
Hi everyone,
There is a long time between production release on Android. The
reason for
this is because we have featured merged into master that are sound
from an
engineering standpoint, but aren't quite ready for release yet. These features often block production releases.
As product owner, pushing out regular bug fix builds would make me
very
happy! But there is a requirement that we are able to not push out
features
that are merged but not ready for production yet. This can probably
me
managed by feature flags.
Dan Duvall (on cc) from Release Engineering will consult to help
Dmitry and
Bernd figure out what their process should be for maximum
effectiveness.
Thanks!
Dan
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Duvall Automation Engineer Wikimedia Foundation http://wikimediafoundation.org
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Yeah - after listening to the podcast I was on the fence about it too… I think they bring up good points on the tradeoffs of both (even the hosts were split on their preference at the end).
I was initially attracted towards the git methodology as it gets these business decisions out of the code base - but I also think that having these flags that work at runtime (from a debug menu) is really valuable for testing purposes.
One thing I am not convinced on is that there will be "less work at the end" when integrating features into a release with the feature flag methodology - I feel like that could still be a problem if you are not merging/rebasing. They discussed this in the podcast almost as an aside - but it is entirely possible to develop conflicting features using feature flags if developers do not coordinate (I feel this is really an orthogonal issue to enabling features).
Still - I think the feature flag solution has promise - but I am way more interested in it if we do it as run time option to get the testing/QA benefits.
On Wed, Mar 11, 2015 at 4:08 PM, Bernd Sitzmann bernd@wikimedia.org wrote:
I have not listened to the podcast yet, but my opinion is similar to what Brian described. So, +1 for feature flags.
Bernd
On Wed, Mar 11, 2015 at 1:40 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
Listened to it this morning, gotta say I think I'm on "Team Flag," if it's done well. IMO we should try one (or both) and see how it goes. Here's my take on each approach:
Branches are cheap to implement, but come at the a potentially high cost if you don't continually rebase and try to keep the work scope as small as possible. At a previous job we used git submodules as pseudo feature branches. There were common problems w/ dependencies between branches and between branches and the main repo, which as the hosts mention are often pushed later in the process.
Flags are more expensive to implement up front, but allow for truly continuous integration and delivery—as well as the potential for gradual rollout. IMO it could also lead to better architected code since feature flags require you to codify the boundaries between the "platform" and the features (and between the features themselves). You also need to limit global state and have good test coverage (which we should do anyway) in order to keep undesired side-effects to a minimum. My previous job also switched to this model and was able to improve their release cadence and sync between features (IIRC, don't have any concrete evidence to back it up unfortunately).
On Tue, Mar 10, 2015 at 11:16 PM, Corey Floyd cfloyd@wikimedia.org wrote:
As luck would have it, a very good tech podcast I subscribe to recently discussed this subject. It is a pretty good listen and discusses trade offs for both feature flags and branching.
http://edgecasesshow.com/123-whats-the-deal-with-nsinteger.html
(Topic is covered in the 2nd half of the show)
On Monday, March 9, 2015, Dan Duvall dduvall@wikimedia.org wrote:
One very simple model that I think works well with distributed software (and should play somewhat nice with Gerrit) is to branch per minor release (assuming somewhat semantic versioning). In this model, master can continue to serve as an integration branch, bug fixes are developed against the release branch and merged following each patch release, and features are developed against master per usual. With Gerrit in the mix, you're essentially left with a two-stage merge for bug fixes which can be a bit of extra work to get them merged into master, but at least the pipeline is greased for getting them released.
I can't say whether this would fit your team's workflow and, as Corey mentioned, there are many different branching models to consider, each with its own focus and drawbacks.[1][2] The one I've outlined above is geared more for stability and maintenance but can probably be tweaked for more frequent releases of features as well. I'm happy to brainstorm further.
[1] http://blog.codinghorror.com/software-branching-and-parallel-universes/ [2] https://msdn.microsoft.com/en-us/library/bb668955.aspx
On Mon, Mar 9, 2015 at 2:30 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Feature flags would help, but they also add an extra development investment to make sure all features are engineered in a way that a flag can shut them off without other bad things happening (not necessarily a bad thing, but require more effort).
Another route to go is to manage this in git using the branches. There are several methodologies for this, and your branching strategy will depend mostly on how the team wants to operate. Pretty much all of them boil down to NOT merging features into master that are not going to be deployed after the current sprint.
On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc tfinc@wikimedia.org wrote:
Excited to see this. In order for this to be successful we'll want to developer a health dashboard for the app and set alarming for it whenever we dip above/below certain thresholds. One of the challenges that you face when releasing often is that the amount of attention you keep during small iterative releases. It's easy to keep very focused for 2-3 releases but after that attention can drift to just the major releases. And while it's great to read reviews and find out a subjective metric of how we're doing we need to get in front of it with objective metrics.
Thus having an app health dashboard showcasing: search, readership, editing, etc can easily show you if you've had any regressions. These would not only be useful for small bug fix releases but would also help validate our major product releases.
I'll leave it with you guys to define what metrics are necessary to define a healthy app.
--tomasz
On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry dgarry@wikimedia.org wrote: > Hi everyone, > > There is a long time between production release on Android. The reason for > this is because we have featured merged into master that are sound from an > engineering standpoint, but aren't quite ready for release yet. These > features often block production releases. > > As product owner, pushing out regular bug fix builds would make me very > happy! But there is a requirement that we are able to not push out features > that are merged but not ready for production yet. This can probably me > managed by feature flags. > > Dan Duvall (on cc) from Release Engineering will consult to help Dmitry and > Bernd figure out what their process should be for maximum effectiveness. > > Thanks! > > Dan > > -- > Dan Garry > Associate Product Manager, Mobile Apps > Wikimedia Foundation > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l >
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Duvall Automation Engineer Wikimedia Foundation http://wikimediafoundation.org
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Yeah, I'll make sure to listen to the podcast, but generally I'm on the side of feature flags.
On Wed, Mar 11, 2015 at 4:58 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Yeah - after listening to the podcast I was on the fence about it too… I think they bring up good points on the tradeoffs of both (even the hosts were split on their preference at the end).
I was initially attracted towards the git methodology as it gets these business decisions out of the code base - but I also think that having these flags that work at runtime (from a debug menu) is really valuable for testing purposes.
One thing I am not convinced on is that there will be "less work at the end" when integrating features into a release with the feature flag methodology - I feel like that could still be a problem if you are not merging/rebasing. They discussed this in the podcast almost as an aside - but it is entirely possible to develop conflicting features using feature flags if developers do not coordinate (I feel this is really an orthogonal issue to enabling features).
Still - I think the feature flag solution has promise - but I am way more interested in it if we do it as run time option to get the testing/QA benefits.
On Wed, Mar 11, 2015 at 4:08 PM, Bernd Sitzmann bernd@wikimedia.org wrote:
I have not listened to the podcast yet, but my opinion is similar to what Brian described. So, +1 for feature flags.
Bernd
On Wed, Mar 11, 2015 at 1:40 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
Listened to it this morning, gotta say I think I'm on "Team Flag," if it's done well. IMO we should try one (or both) and see how it goes. Here's my take on each approach:
Branches are cheap to implement, but come at the a potentially high cost if you don't continually rebase and try to keep the work scope as small as possible. At a previous job we used git submodules as pseudo feature branches. There were common problems w/ dependencies between branches and between branches and the main repo, which as the hosts mention are often pushed later in the process.
Flags are more expensive to implement up front, but allow for truly continuous integration and delivery—as well as the potential for gradual rollout. IMO it could also lead to better architected code since feature flags require you to codify the boundaries between the "platform" and the features (and between the features themselves). You also need to limit global state and have good test coverage (which we should do anyway) in order to keep undesired side-effects to a minimum. My previous job also switched to this model and was able to improve their release cadence and sync between features (IIRC, don't have any concrete evidence to back it up unfortunately).
On Tue, Mar 10, 2015 at 11:16 PM, Corey Floyd cfloyd@wikimedia.org wrote:
As luck would have it, a very good tech podcast I subscribe to recently discussed this subject. It is a pretty good listen and discusses trade offs for both feature flags and branching.
http://edgecasesshow.com/123-whats-the-deal-with-nsinteger.html
(Topic is covered in the 2nd half of the show)
On Monday, March 9, 2015, Dan Duvall dduvall@wikimedia.org wrote:
One very simple model that I think works well with distributed software (and should play somewhat nice with Gerrit) is to branch per minor release (assuming somewhat semantic versioning). In this model, master can continue to serve as an integration branch, bug fixes are developed against the release branch and merged following each patch release, and features are developed against master per usual. With Gerrit in the mix, you're essentially left with a two-stage merge for bug fixes which can be a bit of extra work to get them merged into master, but at least the pipeline is greased for getting them released.
I can't say whether this would fit your team's workflow and, as Corey mentioned, there are many different branching models to consider, each with its own focus and drawbacks.[1][2] The one I've outlined above is geared more for stability and maintenance but can probably be tweaked for more frequent releases of features as well. I'm happy to brainstorm further.
[1] http://blog.codinghorror.com/software-branching-and-parallel-universes/ [2] https://msdn.microsoft.com/en-us/library/bb668955.aspx
On Mon, Mar 9, 2015 at 2:30 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Feature flags would help, but they also add an extra development investment to make sure all features are engineered in a way that a flag can shut them off without other bad things happening (not necessarily a bad thing, but require more effort).
Another route to go is to manage this in git using the branches. There are several methodologies for this, and your branching strategy will depend mostly on how the team wants to operate. Pretty much all of them boil down to NOT merging features into master that are not going to be deployed after the current sprint.
On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc tfinc@wikimedia.org wrote:
> Excited to see this. In order for this to be successful we'll want to > developer a health dashboard for the app and set alarming for it > whenever we dip above/below certain thresholds. One of the challenges > that you face when releasing often is that the amount of attention > you > keep during small iterative releases. It's easy to keep very focused > for 2-3 releases but after that attention can drift to just the major > releases. And while it's great to read reviews and find out a > subjective metric of how we're doing we need to get in front of it > with objective metrics. > > Thus having an app health dashboard showcasing: search, readership, > editing, etc can easily show you if you've had any regressions. These > would not only be useful for small bug fix releases but would also > help validate our major product releases. > > I'll leave it with you guys to define what metrics are necessary to > define a healthy app. > > --tomasz > > On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry dgarry@wikimedia.org > wrote: > > Hi everyone, > > > > There is a long time between production release on Android. The > reason for > > this is because we have featured merged into master that are sound > from an > > engineering standpoint, but aren't quite ready for release yet. > These > > features often block production releases. > > > > As product owner, pushing out regular bug fix builds would make me > very > > happy! But there is a requirement that we are able to not push out > features > > that are merged but not ready for production yet. This can > probably me > > managed by feature flags. > > > > Dan Duvall (on cc) from Release Engineering will consult to help > Dmitry and > > Bernd figure out what their process should be for maximum > effectiveness. > > > > Thanks! > > > > Dan > > > > -- > > Dan Garry > > Associate Product Manager, Mobile Apps > > Wikimedia Foundation > > > > _______________________________________________ > > Mobile-l mailing list > > Mobile-l@lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l >
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Duvall Automation Engineer Wikimedia Foundation http://wikimediafoundation.org
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
So it sounds like we have agreement on using feature flags for our features, with the aim of increasing our cadence.
What are our next steps now?
Dan
On 11 March 2015 at 14:01, Dmitry Brant dbrant@wikimedia.org wrote:
Yeah, I'll make sure to listen to the podcast, but generally I'm on the side of feature flags.
On Wed, Mar 11, 2015 at 4:58 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Yeah - after listening to the podcast I was on the fence about it too… I think they bring up good points on the tradeoffs of both (even the hosts were split on their preference at the end).
I was initially attracted towards the git methodology as it gets these business decisions out of the code base - but I also think that having these flags that work at runtime (from a debug menu) is really valuable for testing purposes.
One thing I am not convinced on is that there will be "less work at the end" when integrating features into a release with the feature flag methodology - I feel like that could still be a problem if you are not merging/rebasing. They discussed this in the podcast almost as an aside - but it is entirely possible to develop conflicting features using feature flags if developers do not coordinate (I feel this is really an orthogonal issue to enabling features).
Still - I think the feature flag solution has promise - but I am way more interested in it if we do it as run time option to get the testing/QA benefits.
On Wed, Mar 11, 2015 at 4:08 PM, Bernd Sitzmann bernd@wikimedia.org wrote:
I have not listened to the podcast yet, but my opinion is similar to what Brian described. So, +1 for feature flags.
Bernd
On Wed, Mar 11, 2015 at 1:40 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
Listened to it this morning, gotta say I think I'm on "Team Flag," if it's done well. IMO we should try one (or both) and see how it goes. Here's my take on each approach:
Branches are cheap to implement, but come at the a potentially high cost if you don't continually rebase and try to keep the work scope as small as possible. At a previous job we used git submodules as pseudo feature branches. There were common problems w/ dependencies between branches and between branches and the main repo, which as the hosts mention are often pushed later in the process.
Flags are more expensive to implement up front, but allow for truly continuous integration and delivery—as well as the potential for gradual rollout. IMO it could also lead to better architected code since feature flags require you to codify the boundaries between the "platform" and the features (and between the features themselves). You also need to limit global state and have good test coverage (which we should do anyway) in order to keep undesired side-effects to a minimum. My previous job also switched to this model and was able to improve their release cadence and sync between features (IIRC, don't have any concrete evidence to back it up unfortunately).
On Tue, Mar 10, 2015 at 11:16 PM, Corey Floyd cfloyd@wikimedia.org wrote:
As luck would have it, a very good tech podcast I subscribe to recently discussed this subject. It is a pretty good listen and discusses trade offs for both feature flags and branching.
http://edgecasesshow.com/123-whats-the-deal-with-nsinteger.html
(Topic is covered in the 2nd half of the show)
On Monday, March 9, 2015, Dan Duvall dduvall@wikimedia.org wrote:
One very simple model that I think works well with distributed software (and should play somewhat nice with Gerrit) is to branch per minor release (assuming somewhat semantic versioning). In this model, master can continue to serve as an integration branch, bug fixes are developed against the release branch and merged following each patch release, and features are developed against master per usual. With Gerrit in the mix, you're essentially left with a two-stage merge for bug fixes which can be a bit of extra work to get them merged into master, but at least the pipeline is greased for getting them released.
I can't say whether this would fit your team's workflow and, as Corey mentioned, there are many different branching models to consider, each with its own focus and drawbacks.[1][2] The one I've outlined above is geared more for stability and maintenance but can probably be tweaked for more frequent releases of features as well. I'm happy to brainstorm further.
[1] http://blog.codinghorror.com/software-branching-and-parallel-universes/ [2] https://msdn.microsoft.com/en-us/library/bb668955.aspx
On Mon, Mar 9, 2015 at 2:30 PM, Corey Floyd cfloyd@wikimedia.org wrote:
> Feature flags would help, but they also add an extra development > investment to make sure all features are engineered in a way that a flag > can shut them off without other bad things happening (not necessarily a bad > thing, but require more effort). > > Another route to go is to manage this in git using the branches. > There are several methodologies for this, and your branching strategy will > depend mostly on how the team wants to operate. Pretty much all of them > boil down to NOT merging features into master that are not going to be > deployed after the current sprint. > > > > > On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc tfinc@wikimedia.org > wrote: > >> Excited to see this. In order for this to be successful we'll want >> to >> developer a health dashboard for the app and set alarming for it >> whenever we dip above/below certain thresholds. One of the >> challenges >> that you face when releasing often is that the amount of attention >> you >> keep during small iterative releases. It's easy to keep very focused >> for 2-3 releases but after that attention can drift to just the >> major >> releases. And while it's great to read reviews and find out a >> subjective metric of how we're doing we need to get in front of it >> with objective metrics. >> >> Thus having an app health dashboard showcasing: search, readership, >> editing, etc can easily show you if you've had any regressions. >> These >> would not only be useful for small bug fix releases but would also >> help validate our major product releases. >> >> I'll leave it with you guys to define what metrics are necessary to >> define a healthy app. >> >> --tomasz >> >> On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry dgarry@wikimedia.org >> wrote: >> > Hi everyone, >> > >> > There is a long time between production release on Android. The >> reason for >> > this is because we have featured merged into master that are >> sound from an >> > engineering standpoint, but aren't quite ready for release yet. >> These >> > features often block production releases. >> > >> > As product owner, pushing out regular bug fix builds would make >> me very >> > happy! But there is a requirement that we are able to not push >> out features >> > that are merged but not ready for production yet. This can >> probably me >> > managed by feature flags. >> > >> > Dan Duvall (on cc) from Release Engineering will consult to help >> Dmitry and >> > Bernd figure out what their process should be for maximum >> effectiveness. >> > >> > Thanks! >> > >> > Dan >> > >> > -- >> > Dan Garry >> > Associate Product Manager, Mobile Apps >> > Wikimedia Foundation >> > >> > _______________________________________________ >> > Mobile-l mailing list >> > Mobile-l@lists.wikimedia.org >> > https://lists.wikimedia.org/mailman/listinfo/mobile-l >> > >> >> _______________________________________________ >> Mobile-l mailing list >> Mobile-l@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> > > > > -- > Corey Floyd > Software Engineer > Mobile Apps / iOS > Wikimedia Foundation > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > >
-- Dan Duvall Automation Engineer Wikimedia Foundation http://wikimediafoundation.org
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
My $0.02: planning to determine high-level goals for feature flags, off the top of my head:
- As a developer, I can regularly merge my work into master while developing a feature, *without exposing it to users or causing side effects (*IOW users have to opt-in for experimental features)
- As a product owner, I can regularly ship builds with minor fixes while developing major features in parallel without paying a high synchronization cost between feature & maintenance development
Then, each platform can chart out a road map w/ estimates for how long it might take. There's a fair amount to consider here, at least IMO we should try to knock out a couple of birds w/ this stone (i.e. those "global state" and "modularization" things I mentioned earlier).
On Thu, Mar 12, 2015 at 5:01 PM, Dan Garry dgarry@wikimedia.org wrote:
So it sounds like we have agreement on using feature flags for our features, with the aim of increasing our cadence.
What are our next steps now?
Dan
On 11 March 2015 at 14:01, Dmitry Brant dbrant@wikimedia.org wrote:
Yeah, I'll make sure to listen to the podcast, but generally I'm on the side of feature flags.
On Wed, Mar 11, 2015 at 4:58 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Yeah - after listening to the podcast I was on the fence about it too… I think they bring up good points on the tradeoffs of both (even the hosts were split on their preference at the end).
I was initially attracted towards the git methodology as it gets these business decisions out of the code base - but I also think that having these flags that work at runtime (from a debug menu) is really valuable for testing purposes.
One thing I am not convinced on is that there will be "less work at the end" when integrating features into a release with the feature flag methodology - I feel like that could still be a problem if you are not merging/rebasing. They discussed this in the podcast almost as an aside - but it is entirely possible to develop conflicting features using feature flags if developers do not coordinate (I feel this is really an orthogonal issue to enabling features).
Still - I think the feature flag solution has promise - but I am way more interested in it if we do it as run time option to get the testing/QA benefits.
On Wed, Mar 11, 2015 at 4:08 PM, Bernd Sitzmann bernd@wikimedia.org wrote:
I have not listened to the podcast yet, but my opinion is similar to what Brian described. So, +1 for feature flags.
Bernd
On Wed, Mar 11, 2015 at 1:40 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
Listened to it this morning, gotta say I think I'm on "Team Flag," if it's done well. IMO we should try one (or both) and see how it goes. Here's my take on each approach:
Branches are cheap to implement, but come at the a potentially high cost if you don't continually rebase and try to keep the work scope as small as possible. At a previous job we used git submodules as pseudo feature branches. There were common problems w/ dependencies between branches and between branches and the main repo, which as the hosts mention are often pushed later in the process.
Flags are more expensive to implement up front, but allow for truly continuous integration and delivery—as well as the potential for gradual rollout. IMO it could also lead to better architected code since feature flags require you to codify the boundaries between the "platform" and the features (and between the features themselves). You also need to limit global state and have good test coverage (which we should do anyway) in order to keep undesired side-effects to a minimum. My previous job also switched to this model and was able to improve their release cadence and sync between features (IIRC, don't have any concrete evidence to back it up unfortunately).
On Tue, Mar 10, 2015 at 11:16 PM, Corey Floyd cfloyd@wikimedia.org wrote:
As luck would have it, a very good tech podcast I subscribe to recently discussed this subject. It is a pretty good listen and discusses trade offs for both feature flags and branching.
http://edgecasesshow.com/123-whats-the-deal-with-nsinteger.html
(Topic is covered in the 2nd half of the show)
On Monday, March 9, 2015, Dan Duvall dduvall@wikimedia.org wrote:
> One very simple model that I think works well with distributed > software (and should play somewhat nice with Gerrit) is to branch per minor > release (assuming somewhat semantic versioning). In this model, master can > continue to serve as an integration branch, bug fixes are developed against > the release branch and merged following each patch release, and features > are developed against master per usual. With Gerrit in the mix, you're > essentially left with a two-stage merge for bug fixes which can be a bit of > extra work to get them merged into master, but at least the pipeline is > greased for getting them released. > > I can't say whether this would fit your team's workflow and, as > Corey mentioned, there are many different branching models to consider, > each with its own focus and drawbacks.[1][2] The one I've outlined above is > geared more for stability and maintenance but can probably be tweaked for > more frequent releases of features as well. I'm happy to brainstorm further. > > [1] > http://blog.codinghorror.com/software-branching-and-parallel-universes/ > [2] https://msdn.microsoft.com/en-us/library/bb668955.aspx > > > On Mon, Mar 9, 2015 at 2:30 PM, Corey Floyd cfloyd@wikimedia.org > wrote: > >> Feature flags would help, but they also add an extra development >> investment to make sure all features are engineered in a way that a flag >> can shut them off without other bad things happening (not necessarily a bad >> thing, but require more effort). >> >> Another route to go is to manage this in git using the branches. >> There are several methodologies for this, and your branching strategy will >> depend mostly on how the team wants to operate. Pretty much all of them >> boil down to NOT merging features into master that are not going to be >> deployed after the current sprint. >> >> >> >> >> On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc tfinc@wikimedia.org >> wrote: >> >>> Excited to see this. In order for this to be successful we'll want >>> to >>> developer a health dashboard for the app and set alarming for it >>> whenever we dip above/below certain thresholds. One of the >>> challenges >>> that you face when releasing often is that the amount of attention >>> you >>> keep during small iterative releases. It's easy to keep very >>> focused >>> for 2-3 releases but after that attention can drift to just the >>> major >>> releases. And while it's great to read reviews and find out a >>> subjective metric of how we're doing we need to get in front of it >>> with objective metrics. >>> >>> Thus having an app health dashboard showcasing: search, readership, >>> editing, etc can easily show you if you've had any regressions. >>> These >>> would not only be useful for small bug fix releases but would also >>> help validate our major product releases. >>> >>> I'll leave it with you guys to define what metrics are necessary to >>> define a healthy app. >>> >>> --tomasz >>> >>> On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry dgarry@wikimedia.org >>> wrote: >>> > Hi everyone, >>> > >>> > There is a long time between production release on Android. The >>> reason for >>> > this is because we have featured merged into master that are >>> sound from an >>> > engineering standpoint, but aren't quite ready for release yet. >>> These >>> > features often block production releases. >>> > >>> > As product owner, pushing out regular bug fix builds would make >>> me very >>> > happy! But there is a requirement that we are able to not push >>> out features >>> > that are merged but not ready for production yet. This can >>> probably me >>> > managed by feature flags. >>> > >>> > Dan Duvall (on cc) from Release Engineering will consult to help >>> Dmitry and >>> > Bernd figure out what their process should be for maximum >>> effectiveness. >>> > >>> > Thanks! >>> > >>> > Dan >>> > >>> > -- >>> > Dan Garry >>> > Associate Product Manager, Mobile Apps >>> > Wikimedia Foundation >>> > >>> > _______________________________________________ >>> > Mobile-l mailing list >>> > Mobile-l@lists.wikimedia.org >>> > https://lists.wikimedia.org/mailman/listinfo/mobile-l >>> > >>> >>> _______________________________________________ >>> Mobile-l mailing list >>> Mobile-l@lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >>> >> >> >> >> -- >> Corey Floyd >> Software Engineer >> Mobile Apps / iOS >> Wikimedia Foundation >> >> _______________________________________________ >> Mobile-l mailing list >> Mobile-l@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> >> > > > -- > Dan Duvall > Automation Engineer > Wikimedia Foundation http://wikimediafoundation.org >
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Well, technically all of the new features we're currently developing are properly flagged. So there's nothing blocking a production release (based on the current Beta tag) (Bernd, correct me if I'm wrong). I think the only remaining hesitation at this point is the lack of a formalized QA process. This is what instills a sense of apprehension towards making a production release, and pushes the idea of a release to the back of our minds, until it reaches an arbitrary critical mass.
On Thu, Mar 12, 2015 at 5:01 PM, Dan Garry dgarry@wikimedia.org wrote:
So it sounds like we have agreement on using feature flags for our features, with the aim of increasing our cadence.
What are our next steps now?
Dan
On 11 March 2015 at 14:01, Dmitry Brant dbrant@wikimedia.org wrote:
Yeah, I'll make sure to listen to the podcast, but generally I'm on the side of feature flags.
On Wed, Mar 11, 2015 at 4:58 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Yeah - after listening to the podcast I was on the fence about it too… I think they bring up good points on the tradeoffs of both (even the hosts were split on their preference at the end).
I was initially attracted towards the git methodology as it gets these business decisions out of the code base - but I also think that having these flags that work at runtime (from a debug menu) is really valuable for testing purposes.
One thing I am not convinced on is that there will be "less work at the end" when integrating features into a release with the feature flag methodology - I feel like that could still be a problem if you are not merging/rebasing. They discussed this in the podcast almost as an aside - but it is entirely possible to develop conflicting features using feature flags if developers do not coordinate (I feel this is really an orthogonal issue to enabling features).
Still - I think the feature flag solution has promise - but I am way more interested in it if we do it as run time option to get the testing/QA benefits.
On Wed, Mar 11, 2015 at 4:08 PM, Bernd Sitzmann bernd@wikimedia.org wrote:
I have not listened to the podcast yet, but my opinion is similar to what Brian described. So, +1 for feature flags.
Bernd
On Wed, Mar 11, 2015 at 1:40 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
Listened to it this morning, gotta say I think I'm on "Team Flag," if it's done well. IMO we should try one (or both) and see how it goes. Here's my take on each approach:
Branches are cheap to implement, but come at the a potentially high cost if you don't continually rebase and try to keep the work scope as small as possible. At a previous job we used git submodules as pseudo feature branches. There were common problems w/ dependencies between branches and between branches and the main repo, which as the hosts mention are often pushed later in the process.
Flags are more expensive to implement up front, but allow for truly continuous integration and delivery—as well as the potential for gradual rollout. IMO it could also lead to better architected code since feature flags require you to codify the boundaries between the "platform" and the features (and between the features themselves). You also need to limit global state and have good test coverage (which we should do anyway) in order to keep undesired side-effects to a minimum. My previous job also switched to this model and was able to improve their release cadence and sync between features (IIRC, don't have any concrete evidence to back it up unfortunately).
On Tue, Mar 10, 2015 at 11:16 PM, Corey Floyd cfloyd@wikimedia.org wrote:
As luck would have it, a very good tech podcast I subscribe to recently discussed this subject. It is a pretty good listen and discusses trade offs for both feature flags and branching.
http://edgecasesshow.com/123-whats-the-deal-with-nsinteger.html
(Topic is covered in the 2nd half of the show)
On Monday, March 9, 2015, Dan Duvall dduvall@wikimedia.org wrote:
> One very simple model that I think works well with distributed > software (and should play somewhat nice with Gerrit) is to branch per minor > release (assuming somewhat semantic versioning). In this model, master can > continue to serve as an integration branch, bug fixes are developed against > the release branch and merged following each patch release, and features > are developed against master per usual. With Gerrit in the mix, you're > essentially left with a two-stage merge for bug fixes which can be a bit of > extra work to get them merged into master, but at least the pipeline is > greased for getting them released. > > I can't say whether this would fit your team's workflow and, as > Corey mentioned, there are many different branching models to consider, > each with its own focus and drawbacks.[1][2] The one I've outlined above is > geared more for stability and maintenance but can probably be tweaked for > more frequent releases of features as well. I'm happy to brainstorm further. > > [1] > http://blog.codinghorror.com/software-branching-and-parallel-universes/ > [2] https://msdn.microsoft.com/en-us/library/bb668955.aspx > > > On Mon, Mar 9, 2015 at 2:30 PM, Corey Floyd cfloyd@wikimedia.org > wrote: > >> Feature flags would help, but they also add an extra development >> investment to make sure all features are engineered in a way that a flag >> can shut them off without other bad things happening (not necessarily a bad >> thing, but require more effort). >> >> Another route to go is to manage this in git using the branches. >> There are several methodologies for this, and your branching strategy will >> depend mostly on how the team wants to operate. Pretty much all of them >> boil down to NOT merging features into master that are not going to be >> deployed after the current sprint. >> >> >> >> >> On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc tfinc@wikimedia.org >> wrote: >> >>> Excited to see this. In order for this to be successful we'll want >>> to >>> developer a health dashboard for the app and set alarming for it >>> whenever we dip above/below certain thresholds. One of the >>> challenges >>> that you face when releasing often is that the amount of attention >>> you >>> keep during small iterative releases. It's easy to keep very >>> focused >>> for 2-3 releases but after that attention can drift to just the >>> major >>> releases. And while it's great to read reviews and find out a >>> subjective metric of how we're doing we need to get in front of it >>> with objective metrics. >>> >>> Thus having an app health dashboard showcasing: search, readership, >>> editing, etc can easily show you if you've had any regressions. >>> These >>> would not only be useful for small bug fix releases but would also >>> help validate our major product releases. >>> >>> I'll leave it with you guys to define what metrics are necessary to >>> define a healthy app. >>> >>> --tomasz >>> >>> On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry dgarry@wikimedia.org >>> wrote: >>> > Hi everyone, >>> > >>> > There is a long time between production release on Android. The >>> reason for >>> > this is because we have featured merged into master that are >>> sound from an >>> > engineering standpoint, but aren't quite ready for release yet. >>> These >>> > features often block production releases. >>> > >>> > As product owner, pushing out regular bug fix builds would make >>> me very >>> > happy! But there is a requirement that we are able to not push >>> out features >>> > that are merged but not ready for production yet. This can >>> probably me >>> > managed by feature flags. >>> > >>> > Dan Duvall (on cc) from Release Engineering will consult to help >>> Dmitry and >>> > Bernd figure out what their process should be for maximum >>> effectiveness. >>> > >>> > Thanks! >>> > >>> > Dan >>> > >>> > -- >>> > Dan Garry >>> > Associate Product Manager, Mobile Apps >>> > Wikimedia Foundation >>> > >>> > _______________________________________________ >>> > Mobile-l mailing list >>> > Mobile-l@lists.wikimedia.org >>> > https://lists.wikimedia.org/mailman/listinfo/mobile-l >>> > >>> >>> _______________________________________________ >>> Mobile-l mailing list >>> Mobile-l@lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >>> >> >> >> >> -- >> Corey Floyd >> Software Engineer >> Mobile Apps / iOS >> Wikimedia Foundation >> >> _______________________________________________ >> Mobile-l mailing list >> Mobile-l@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> >> > > > -- > Dan Duvall > Automation Engineer > Wikimedia Foundation http://wikimediafoundation.org >
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Dmitry, that sounds correct.
Dan, In general I think we should strive for a release model similar to what we were aiming for when Yuvi was around [1]. (2 weeks cadence for production, with a 1-2 week lag over beta).
For right now:
I think feature and bug-wise we're pretty much there on Android; and a production release is long overdue from my perspective. We would need another beta release first, though, to make sure the launcher icon change is in there (so we don't change it twice for the main app).
One patch that triggers further need for testing is the move to API level/AppCompat 22. Dmitry has found a few issues already but there might be more. For that I think it would be really valuable to have Elena focus a couple of days on Android.
Bernd
[1] https://www.mediawiki.org/wiki/Wikimedia_Apps/Android_release_policy
On Mar 12, 2015 3:30 PM, "Dmitry Brant" dbrant@wikimedia.org wrote:
Well, technically all of the new features we're currently developing are properly flagged. So there's nothing blocking a production release (based on the current Beta tag) (Bernd, correct me if I'm wrong). I think the only remaining hesitation at this point is the lack of a formalized QA process. This is what instills a sense of apprehension towards making a production release, and pushes the idea of a release to the back of our minds, until it reaches an arbitrary critical mass.
On Thu, Mar 12, 2015 at 5:01 PM, Dan Garry dgarry@wikimedia.org wrote:
So it sounds like we have agreement on using feature flags for our features, with the aim of increasing our cadence.
What are our next steps now?
Dan
On 11 March 2015 at 14:01, Dmitry Brant dbrant@wikimedia.org wrote:
Yeah, I'll make sure to listen to the podcast, but generally I'm on the side of feature flags.
On Wed, Mar 11, 2015 at 4:58 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Yeah - after listening to the podcast I was on the fence about it too… I think they bring up good points on the tradeoffs of both (even the hosts were split on their preference at the end).
I was initially attracted towards the git methodology as it gets these business decisions out of the code base - but I also think that having these flags that work at runtime (from a debug menu) is really valuable for testing purposes.
One thing I am not convinced on is that there will be "less work at the end" when integrating features into a release with the feature flag methodology - I feel like that could still be a problem if you are not merging/rebasing. They discussed this in the podcast almost as an aside - but it is entirely possible to develop conflicting features using feature flags if developers do not coordinate (I feel this is really an orthogonal issue to enabling features).
Still - I think the feature flag solution has promise - but I am way more interested in it if we do it as run time option to get the testing/QA benefits.
On Wed, Mar 11, 2015 at 4:08 PM, Bernd Sitzmann bernd@wikimedia.org wrote:
I have not listened to the podcast yet, but my opinion is similar to what Brian described. So, +1 for feature flags.
Bernd
On Wed, Mar 11, 2015 at 1:40 PM, Brian Gerstle <bgerstle@wikimedia.org
wrote:
Listened to it this morning, gotta say I think I'm on "Team Flag," if it's done well. IMO we should try one (or both) and see how it goes. Here's my take on each approach:
Branches are cheap to implement, but come at the a potentially high cost if you don't continually rebase and try to keep the work scope as small as possible. At a previous job we used git submodules as pseudo feature branches. There were common problems w/ dependencies between branches and between branches and the main repo, which as the hosts mention are often pushed later in the process.
Flags are more expensive to implement up front, but allow for truly continuous integration and delivery—as well as the potential for gradual rollout. IMO it could also lead to better architected code since feature flags require you to codify the boundaries between the "platform" and the features (and between the features themselves). You also need to limit global state and have good test coverage (which we should do anyway) in order to keep undesired side-effects to a minimum. My previous job also switched to this model and was able to improve their release cadence and sync between features (IIRC, don't have any concrete evidence to back it up unfortunately).
On Tue, Mar 10, 2015 at 11:16 PM, Corey Floyd cfloyd@wikimedia.org wrote:
> As luck would have it, a very good tech podcast I subscribe > to recently discussed this subject. It is a pretty good listen and > discusses trade offs for both feature flags and branching. > > > http://edgecasesshow.com/123-whats-the-deal-with-nsinteger.html > > (Topic is covered in the 2nd half of the show) > > > On Monday, March 9, 2015, Dan Duvall dduvall@wikimedia.org wrote: > >> One very simple model that I think works well with distributed >> software (and should play somewhat nice with Gerrit) is to branch per minor >> release (assuming somewhat semantic versioning). In this model, master can >> continue to serve as an integration branch, bug fixes are developed against >> the release branch and merged following each patch release, and features >> are developed against master per usual. With Gerrit in the mix, you're >> essentially left with a two-stage merge for bug fixes which can be a bit of >> extra work to get them merged into master, but at least the pipeline is >> greased for getting them released. >> >> I can't say whether this would fit your team's workflow and, as >> Corey mentioned, there are many different branching models to consider, >> each with its own focus and drawbacks.[1][2] The one I've outlined above is >> geared more for stability and maintenance but can probably be tweaked for >> more frequent releases of features as well. I'm happy to brainstorm further. >> >> [1] >> http://blog.codinghorror.com/software-branching-and-parallel-universes/ >> [2] https://msdn.microsoft.com/en-us/library/bb668955.aspx >> >> >> On Mon, Mar 9, 2015 at 2:30 PM, Corey Floyd cfloyd@wikimedia.org >> wrote: >> >>> Feature flags would help, but they also add an extra development >>> investment to make sure all features are engineered in a way that a flag >>> can shut them off without other bad things happening (not necessarily a bad >>> thing, but require more effort). >>> >>> Another route to go is to manage this in git using the branches. >>> There are several methodologies for this, and your branching strategy will >>> depend mostly on how the team wants to operate. Pretty much all of them >>> boil down to NOT merging features into master that are not going to be >>> deployed after the current sprint. >>> >>> >>> >>> >>> On Mon, Mar 9, 2015 at 5:19 PM, Tomasz Finc tfinc@wikimedia.org >>> wrote: >>> >>>> Excited to see this. In order for this to be successful we'll >>>> want to >>>> developer a health dashboard for the app and set alarming for it >>>> whenever we dip above/below certain thresholds. One of the >>>> challenges >>>> that you face when releasing often is that the amount of >>>> attention you >>>> keep during small iterative releases. It's easy to keep very >>>> focused >>>> for 2-3 releases but after that attention can drift to just the >>>> major >>>> releases. And while it's great to read reviews and find out a >>>> subjective metric of how we're doing we need to get in front of it >>>> with objective metrics. >>>> >>>> Thus having an app health dashboard showcasing: search, >>>> readership, >>>> editing, etc can easily show you if you've had any regressions. >>>> These >>>> would not only be useful for small bug fix releases but would also >>>> help validate our major product releases. >>>> >>>> I'll leave it with you guys to define what metrics are necessary >>>> to >>>> define a healthy app. >>>> >>>> --tomasz >>>> >>>> On Mon, Mar 9, 2015 at 1:31 PM, Dan Garry dgarry@wikimedia.org >>>> wrote: >>>> > Hi everyone, >>>> > >>>> > There is a long time between production release on Android. The >>>> reason for >>>> > this is because we have featured merged into master that are >>>> sound from an >>>> > engineering standpoint, but aren't quite ready for release yet. >>>> These >>>> > features often block production releases. >>>> > >>>> > As product owner, pushing out regular bug fix builds would make >>>> me very >>>> > happy! But there is a requirement that we are able to not push >>>> out features >>>> > that are merged but not ready for production yet. This can >>>> probably me >>>> > managed by feature flags. >>>> > >>>> > Dan Duvall (on cc) from Release Engineering will consult to >>>> help Dmitry and >>>> > Bernd figure out what their process should be for maximum >>>> effectiveness. >>>> > >>>> > Thanks! >>>> > >>>> > Dan >>>> > >>>> > -- >>>> > Dan Garry >>>> > Associate Product Manager, Mobile Apps >>>> > Wikimedia Foundation >>>> > >>>> > _______________________________________________ >>>> > Mobile-l mailing list >>>> > Mobile-l@lists.wikimedia.org >>>> > https://lists.wikimedia.org/mailman/listinfo/mobile-l >>>> > >>>> >>>> _______________________________________________ >>>> Mobile-l mailing list >>>> Mobile-l@lists.wikimedia.org >>>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >>>> >>> >>> >>> >>> -- >>> Corey Floyd >>> Software Engineer >>> Mobile Apps / iOS >>> Wikimedia Foundation >>> >>> _______________________________________________ >>> Mobile-l mailing list >>> Mobile-l@lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >>> >>> >> >> >> -- >> Dan Duvall >> Automation Engineer >> Wikimedia Foundation http://wikimediafoundation.org >> > > > -- > Corey Floyd > Software Engineer > Mobile Apps / iOS > Wikimedia Foundation > > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > >
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation