I ran an audit of the existing browser tests for MobileFrontend as part of a mobile team spike [1]. You can see the results here: https://www.mediawiki.org/wiki/Mobile/QA_Test_Audit_March_2014
Our coverage is actually not too bad, but there is definitely room from improvement. That said I have a big concern is around the organisation of the tests and how they are written and what is written - many of the tests could do with being reworded and a lot of them should probably actually be deleted. There is a lot of code duplication.
When auditing I found tests scattered all over the place. This suggests that we could benefit from reorganising the file structure to be more logical, in particular features that relate to special pages should have their own folder (this is particularly useful for clarifying what tests the watch star and what tests the actual watchlist page - Zeljko / Chris is it possible to have subfolders in the features directory that contain features?).
I would also suggest the following actions for improving our test coverage: * Add tests for error handling on login / account creation * Add tests for this page has issues * Improve tests for key editing and upload workflows * Add tests for Language variant support * Add tests for full text search support * Improve the existing watch star tests so they actually check the end result * Add test to check the user can close the left navigation menu * Add tests for logout * Add tests for reference overlay * Add tests for toggling * Add tests for Nearby in skins other than Minerva (mobile skin) * Address hygiene issues on https://www.mediawiki.org/wiki/Mobile/QA_Test_Audit_March_2014
[1] https://wikimedia.mingle.thoughtworks.com/projects/mobile/cards/1687
Head's up that Chris has agreed to carve out some dedicated time to work with us next iteration on resolving some of these. We'll sort out the specifics and get them defined in https://wikimedia.mingle.thoughtworks.com/projects/mobile/cards/1787 as we get closer to kickoff.
On Mon, Mar 3, 2014 at 12:10 PM, Jon Robson jrobson@wikimedia.org wrote:
I ran an audit of the existing browser tests for MobileFrontend as part of a mobile team spike [1]. You can see the results here: https://www.mediawiki.org/wiki/Mobile/QA_Test_Audit_March_2014
Our coverage is actually not too bad, but there is definitely room from improvement. That said I have a big concern is around the organisation of the tests and how they are written and what is written
- many of the tests could do with being reworded and a lot of them
should probably actually be deleted. There is a lot of code duplication.
When auditing I found tests scattered all over the place. This suggests that we could benefit from reorganising the file structure to be more logical, in particular features that relate to special pages should have their own folder (this is particularly useful for clarifying what tests the watch star and what tests the actual watchlist page - Zeljko / Chris is it possible to have subfolders in the features directory that contain features?).
I would also suggest the following actions for improving our test coverage:
- Add tests for error handling on login / account creation
- Add tests for this page has issues
- Improve tests for key editing and upload workflows
- Add tests for Language variant support
- Add tests for full text search support
- Improve the existing watch star tests so they actually check the end
result
- Add test to check the user can close the left navigation menu
- Add tests for logout
- Add tests for reference overlay
- Add tests for toggling
- Add tests for Nearby in skins other than Minerva (mobile skin)
- Address hygiene issues on
https://www.mediawiki.org/wiki/Mobile/QA_Test_Audit_March_2014
[1] https://wikimedia.mingle.thoughtworks.com/projects/mobile/cards/1687
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Excellent news.
On Thu, Mar 20, 2014 at 10:24 AM, Arthur Richards arichards@wikimedia.orgwrote:
Head's up that Chris has agreed to carve out some dedicated time to work with us next iteration on resolving some of these. We'll sort out the specifics and get them defined in https://wikimedia.mingle.thoughtworks.com/projects/mobile/cards/1787 as we get closer to kickoff.
On Mon, Mar 3, 2014 at 12:10 PM, Jon Robson jrobson@wikimedia.org wrote:
I ran an audit of the existing browser tests for MobileFrontend as part of a mobile team spike [1]. You can see the results here: https://www.mediawiki.org/wiki/Mobile/QA_Test_Audit_March_2014
Our coverage is actually not too bad, but there is definitely room from improvement. That said I have a big concern is around the organisation of the tests and how they are written and what is written
- many of the tests could do with being reworded and a lot of them
should probably actually be deleted. There is a lot of code duplication.
When auditing I found tests scattered all over the place. This suggests that we could benefit from reorganising the file structure to be more logical, in particular features that relate to special pages should have their own folder (this is particularly useful for clarifying what tests the watch star and what tests the actual watchlist page - Zeljko / Chris is it possible to have subfolders in the features directory that contain features?).
I would also suggest the following actions for improving our test coverage:
- Add tests for error handling on login / account creation
- Add tests for this page has issues
- Improve tests for key editing and upload workflows
- Add tests for Language variant support
- Add tests for full text search support
- Improve the existing watch star tests so they actually check the end
result
- Add test to check the user can close the left navigation menu
- Add tests for logout
- Add tests for reference overlay
- Add tests for toggling
- Add tests for Nearby in skins other than Minerva (mobile skin)
- Address hygiene issues on
https://www.mediawiki.org/wiki/Mobile/QA_Test_Audit_March_2014
[1] https://wikimedia.mingle.thoughtworks.com/projects/mobile/cards/1687
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
On Mon, Mar 3, 2014 at 8:10 PM, Jon Robson jrobson@wikimedia.org wrote:
When auditing I found tests scattered all over the place. This suggests that we could benefit from reorganising the file structure to be more logical, in particular features that relate to special pages should have their own folder (this is particularly useful for clarifying what tests the watch star and what tests the actual watchlist page - Zeljko / Chris is it possible to have subfolders in the features directory that contain features?).
The short answer is yes[1].
Apologies for the late reply, I have just noticed that I did not reply to this.
Željko -- 1: http://watirmelon.com/2011/07/04/use-cucumber-feature-folders-for-functional...
Update. It's been a bit of a rollercoaster but we are now in a much better state. Also hopefully with the merging of https://gerrit.wikimedia.org/r/125204 and https://gerrit.wikimedia.org/r/125202 all our Jenkins tests builds will pass again.
Going back to my original mail, here's a few updates on each of the problems identified.
Our coverage is actually not too bad, but there is definitely room from improvement. That said I have a big concern is around the organisation of the tests and how they are written and what is written
- many of the tests could do with being reworded and a lot of them
should probably actually be deleted. There is a lot of code duplication.
There is now a lot less code duplication. These browser tests have been massively refactored.
When auditing I found tests scattered all over the place. This suggests that we could benefit from reorganising the file structure to be more logical, in particular features that relate to special pages should have their own folder (this is particularly useful for clarifying what tests the watch star and what tests the actual watchlist page - Zeljko / Chris is it possible to have subfolders in the features directory that contain features?).
This is better but subfolders would still be useful. Željko maybe we could explore this now?
I would also suggest the following actions for improving our test coverage:> * Add tests for this page has issues
Provided in https://gerrit.wikimedia.org/r/125211 - a review needed
- Improve tests for key editing and upload workflows
Our browser tests actually go through the entire workflow now - see Tests merged in https://gerrit.wikimedia.org/r/#/c/118104/ Our lead photo upload workflow could do with a few more tests though, although the fact we test uploads on the uploads page which uses the same code means we are mostly covered.
- Add tests for full text search support
https://gerrit.wikimedia.org/r/124742 provides this - please
- Improve the existing watch star tests so they actually check the end result
Merged in https://gerrit.wikimedia.org/r/#/c/123679/
- Add tests for reference overlay
Merged in Ied8533d42d6b005909347b6c6f7e05bb8cb31364
- Add tests for toggling
Merged in Iab4449e2cffba61a3934b73cb430c0b2c847146a
- Address hygiene issues on
https://www.mediawiki.org/wiki/Mobile/QA_Test_Audit_March_2014
These issues have now been addressed.
This leaves the following features untested - the value of which is debatable:
- Add tests for error handling on login / account creation
- Add tests for Language variant support
- Add tests for logout
- Add test to check the user can close the left navigation menu
(this one should be trivial to cover)
- Add tests for Nearby in skins other than Minerva (mobile skin)
On Thu, Apr 10, 2014 at 6:02 PM, Jon Robson jrobson@wikimedia.org wrote:
This is better but subfolders would still be useful. Željko maybe we could explore this now?
Sorry for the late reply, still catching up after being on parental leave.
You can just move the feature files to subfolders. It should "just work"(tm).
Željko