On Wed, Nov 21, 2012 at 3:45 PM, Quim Gil qgil@wikimedia.org wrote:
Here is a first stab for a draft proposal to organize our volunteer testing activities:
http://www.mediawiki.org/wiki/**Talk:QA/Strategy#Manual_**testing_strategyhttp://www.mediawiki.org/wiki/Talk:QA/Strategy#Manual_testing_strategy
Written after some lousy discussions with Chris and Sumana, and reading a bunch of related wiki pages. Your feedback is welcome.
Ideally this _theory_ will be immediately applicable to some pilots that we can run in the upcoming weeks. The Language and Mobile teams seem to be ready for a try - maybe even before the end of the year. Visual Editor and Editor Engagement teams might come next in January.
This proposal feels detached from reality. Right now features teams mostly do one of the following, in my experience:
1). Product managers and developers do their own manual QA. For PMs this aligns with verifying requirements, for developers it's checking their own work. It can be a pain in the ass but it works for the most part. 2). A lucky few teams have dedicated QA help, like mobile.
In either situation, manual QA tends to be done on a tight deadline, requires an intimiate understanding of the goals and requirements, and within a very specific scope.
I don't have a lot of experience working at a large open source project, as a caveat, so I haven't had the opportunity to see volunteer QA in action. But considering my current working situation, I would rather continue doing my own QA rather than rely on a volunteer who cannot be held to a deadline and is not personally responsible for the quality of our work. The only solutions in my mind are A) much more robust automated testing. B) hiring experienced QA people. Anything else is just going to slow us down.
Steven