Hi everyone,
Our current release process does not give anyone outside the team visibility in to when we are doing releases and what is included in them. By virtue of our deployments system being apps, we've not got the same integration as other teams at the WMF. It's difficult for someone like Erik or Greg to, at a glance, know when things are being deployed in Appsland. We should strive to integrate our process more so that the senior management can have visibility in to it. As such, I'm proposing the following process.
Beta releases [1] are performed as frequently (or infrequently) as required for testing purposes. The base criteria for a feature going in to beta are that:
- The feature has some basic functionality that delivers some value to the user. - The feature does not crash during basic usage. - The feature behaves as expected during basic usage.
Production submissions [2] occur on the last Thursday of every month. Features that are currently in the beta are evaluated to see if they should be included in the production release. The base criteria for a feature going in to production are that:
- The feature has functionality which provides value to the user. - The feature is aimed to advancing us towards meeting our quarterly goals. - The feature has been QA tested and found to: - not crash even under stress. - behave as expected in typical use cases. - not introduce regressions. - The feature has undergone qualitative or quantitative user testing which validated its objectives. - The feature has clear criteria for success set which are measurable (quantitatively *or* qualitatively), such that post-hoc analysis can be performed about its effectiveness.
In a nutshell, the above is simply a formal documentation of our current process, except that we've agreed a specific date to do the submissions, and explicitly included QA (now that we have limited QA resources in The Specialists Guild, and we are set to get more with Elena and Rummana joining us).
I plan to officially declare this our release process within a few days, probably on Friday.
Thoughts?
Dan
[1]: On Android, "beta" refers to the Wikipedia Beta app in Google Play, whereas on iOS "beta" refers to our latest build in TestFlight. [2]: As we have a review process on the iOS side due to Apple, which will stretch releases out an unknown amount, I have specifically denoted this phase as "submission" and not "release" so that there is congruence between the iOS and Android teams.