Update. It's been a bit of a rollercoaster but we are now in a much
better state. Also hopefully with the merging of
https://gerrit.wikimedia.org/r/125204 and
https://gerrit.wikimedia.org/r/125202 all our Jenkins tests builds
will pass again.
Going back to my original mail, here's a few updates on each of the
problems identified.
Our coverage is actually not too bad, but there is
definitely room
from improvement. That said I have a big concern is around the
organisation of the tests and how they are written and what is written
- many of the tests could do with being reworded and a lot of them
should probably actually be deleted. There is a lot of code
duplication.
There is now a lot less code duplication. These browser tests have
been massively refactored.
When auditing I found tests scattered all over the place. This
suggests that we could benefit from reorganising the file structure to
be more logical, in particular features that relate to special pages
should have their own folder (this is particularly useful for
clarifying what tests the watch star and what tests the actual
watchlist page - Zeljko / Chris is it possible to have subfolders in
the features directory that contain features?).
This is better but subfolders would still be useful. Željko maybe we
could explore this now?
I would also suggest the following actions for
improving our test coverage:> * Add tests for this page has issues
Provided in
https://gerrit.wikimedia.org/r/125211 - a review needed
* Improve tests for key editing and upload workflows
Our browser tests actually go through the entire workflow now - see
Tests merged in
https://gerrit.wikimedia.org/r/#/c/118104/
Our lead photo upload workflow could do with a few more tests though,
although the fact we test uploads on the uploads page which uses the
same code means we are mostly covered.
* Add tests for full text search support
https://gerrit.wikimedia.org/r/124742 provides this - please
* Improve the existing watch star tests so they
actually check the end result
Merged in
https://gerrit.wikimedia.org/r/#/c/123679/
* Add tests for reference overlay
Merged in
Ied8533d42d6b005909347b6c6f7e05bb8cb31364
* Add tests for toggling
Merged in
Iab4449e2cffba61a3934b73cb430c0b2c847146a
These issues have
now been addressed.
This leaves the following features untested - the value of which is debatable:
* Add tests for error handling on login / account
creation
* Add tests for Language variant support
* Add tests for logout
* Add test to check the user can close the left navigation menu
(this one should be
trivial to cover)
* Add tests for Nearby in skins other than Minerva
(mobile skin)