This proposal feels detached from reality. Right now features teams mostly
do one of the following, in my experience:
1). Product managers and developers do their own manual QA. For PMs this
aligns with verifying requirements, for developers it's checking their own
work. It can be a pain in the ass but it works for the most part.
2). A lucky few teams have dedicated QA help, like mobile.
I've mentioned this before but I've been pretty quiet about it. QA at WMF
is still a pretty new idea, and we're still getting a lot of bits sorted,
but if your project has a need for software testing/QA, I am always willing
to help, and Zeljko and Michelle are also expert on the subject.
In either situation, manual QA tends to be done on a tight deadline,
requires an intimiate understanding of the goals and
requirements, and
within a very specific scope.
Community QA is less about deadlines and more about organizing around
windows of opportunity. My best example is the testing session that we ran
for AFTv5 just before the first limited release to production. A nice mix
of Wikipedians and outside software testers provided well-considered
testing, and we changed AFTv5 in significant ways before the release as a
result of that feedback.
I don't have a lot of experience working at a
large open source project,
a caveat, so I haven't had the opportunity to see volunteer QA in action.
But considering my current working situation, I would rather continue doing
my own QA rather than rely on a volunteer who cannot be held to a deadline
and is not personally responsible for the quality of our work. The only
solutions in my mind are A) much more robust automated testing. B) hiring
experienced QA people. Anything else is just going to slow us down.
Automated testing is hugely important, and it has been my focus in recent
times, I'll be making some announcements about that very soon. On the
community testing side, it is quite possible to have those who understand
the requirements and desired behavior create guided test "charters" for
those who are not necessarily intimately aware of the project goals.
One of the biggest impediments we have to community testing (or even user
testing by insiders) is the lack of reasonable test environments. The
"test" and "test2" environments are not only misnamed, but are of
marginal
utility. We've been investing in beta labs, and beta is so much better
than it used to be, but we still have a way to go there. As I mentioned on
this list before, the best way to improve beta labs at this point is to use
it.
WMF has a small but dedicated QA staff. My idea of the role of QA/testing
is that QA/testing is a service we may provide to particular projects.
Some projects may not need QA/testing. Some projects may need it from
time to time, but not always. Some projects may need community testing,
and we can support that also.
What I do not want to see is for QA/testing to be some sort of mandatory
gateway/hand-off/quality-police function that everything must pass through.
That way lies madness.
-Chris