Hi everyone,
Our current release process does not give anyone outside the team
visibility in to when we are doing releases and what is included in
them. By virtue of our deployments system being apps, we've not got the
same integration as other teams at the WMF. It's difficult for someone like
Erik or Greg to, at a glance, know when things are being deployed in
Appsland. We should strive to integrate our process more so that the senior
management can have visibility in to it. As such, I'm proposing the
following process.
Beta releases [1] are performed as frequently (or infrequently) as required
for testing purposes. The base criteria for a feature going in to beta are
that:
- The feature has some basic functionality that delivers some value to
the user.
- The feature does not crash during basic usage.
- The feature behaves as expected during basic usage.
Production submissions [2] occur on the last Thursday of every month.
Features that are currently in the beta are evaluated to see if they should
be included in the production release. The base criteria for a feature
going in to production are that:
- The feature has functionality which provides value to the user.
- The feature is aimed to advancing us towards meeting our quarterly
goals.
- The feature has been QA tested and found to:
- not crash even under stress.
- behave as expected in typical use cases.
- not introduce regressions.
- The feature has undergone qualitative or quantitative user testing
which validated its objectives.
- The feature has clear criteria for success set which are measurable
(quantitatively *or* qualitatively), such that post-hoc analysis can be
performed about its effectiveness.
In a nutshell, the above is simply a formal documentation of our current
process, except that we've agreed a specific date to do the submissions,
and explicitly included QA (now that we have limited QA resources in The
Specialists Guild, and we are set to get more with Elena and Rummana
joining us).
I plan to officially declare this our release process within a few days,
probably on Friday.
Thoughts?
Dan
[1]: On Android, "beta" refers to the Wikipedia Beta app in Google Play,
whereas on iOS "beta" refers to our latest build in TestFlight.
[2]: As we have a review process on the iOS side due to Apple, which will
stretch releases out an unknown amount, I have specifically denoted this
phase as "submission" and not "release" so that there is congruence
between
the iOS and Android teams.
--
Dan Garry
Associate Product Manager, Mobile Apps
Wikimedia Foundation