On Thu, 2013-01-17 at 10:54 -0700, Chris McMahon wrote:
I'm glad you mentioned this, it's something I'd like to bring up with
Andre and Valerie. Note that much of the backlog for automated tests is the result of fixed BZ tickets http://www.mediawiki.org/wiki/Qa/test_backlog. Fixed bugs are great candidates for regression tests because a) what broke once is more likely to break again and b) an issue fixed may indicate more issues in nearby areas of the feature. Our UploadWizard test is a great example of a single test catching multiple issues in the same area over time.
How do items currently end up on http://www.mediawiki.org/wiki/QA/Browser_testing/Test_backlog#Backlog ? Who thinks "This is a candidate for an automated browser test, I should list it on the wikipage"? QA reading the commit backlogs? Developers who fixed the issue and know about automated tests? Knowing the "target audience" might help finding the best workflow.
So a mechanism by which fixed browser bugs become entered in the automated browser test backlog would be a fine thing.
1) Bugzilla already has a number of keywords such as * need-integration-test (Selenium test should be written for this.) * need-parsertest (bug needs a parsertest written for it.) * need-unittest (bug needs a test written for it.)
We could introduce yet another keyword in Bugzilla to mark reports that could benefit from a regression-test. Then somebody (who?) could set the keyword and QA could query that bug list [1↓]. However, I don't know how actively these three keywords are used. Even more I wonder how many different workflow interpretations exist. "Developer closes bug report as RESOLVED FIXED and adds keyword" followed by "some volunteer finds out how to search for *closed* tickets with these keywords, writes a test, and removes the keyword again"? I'd love to find out to better understand if it's useful.
2) Another option inside of Bugzilla would be creating a dedicated "Automated browser tests to be created" component and filing (or cloning) a ticket under it everytime when a bug got FIXED. Again, who would be expected to create that ticket - QA going through recently fixed reports? Developers who fixed it? Triagers?
I'm not totally convinced by these two yet, as both need buy-in (developers and reporters need to know and remember that QA works on automated browser tests, and should mark issues accordingly), but nothing better came to my mind in the last days. :-/
andre
[1↑] or display the buglist embedded in a wiki page, by using something like http://www.mediawiki.org/wiki/Extension:Bugzilla_Reports