[QA] Flow browser tests: issues in improving them

[S Page]

I'd like to meet with the brains in QA about Flow browser tests. Chris
McMahon is going on vacation and Željko might be difficult to schedule. So
here are some topics for a non-meeting.  Thanks for ideas and comments, or
we can meet later.

*Good news:*
    HEADLESS=true MEDIAWIKI_USER=Selenium_user MEDIAWIKI_PASSWORD=XX
MEDIAWIKI_URL=http://localhost/wiki/index.php/ bundle exec cucumber features

all tests pass!

And almost all tests are passing in integration.wikimedia.org!  Thanks Jon
Robson et al.

*Improvements*

*Speed up test setup*

 Background runs once per scenario, when really I only need it once per
feature (from Matt Flaschen)

* *generally useful?* Speed up the "Given I am logged in" by using an API
call to login.

If you look at e.g.
https://saucelabs.com/jobs/465f257ffdeb422e95b029c5f8df3446 , after login
the test does
                        POST elements
                                               using: "xpath"
                                               value: ".//a"
then spends the next 8 seconds getting dozens of attribute hrefs from the
Main_Page. And then it seems to repeat this. What is it doing?

* Reuse existing content instead of always creating new topics and posts.
E.g. for Load More/infinite scroll, instead of creating 11 topics, use
"Pending there are 11 topics"; for sorting, use "Pending there are more
than 2 topics" and add a reply to one of them; etc.

*Catch general failures*

*generally useful?* Always check for ResourceLoader errors

   - We want a way (@RLcheck annotation?) to assert
        And page has no ResourceLoader errors
   on start and end of every single page


*generally useful?* Always check for a pink errorbox on the page
https://bugzilla.wikimedia.org/show_bug.cgi?id=61304

   - Similarly this should be a default assertion.


*Increase Flow browser test reliability*

Problem: Typical test creates a topic or post on Talk:Flow_QA, then checks
elements in the first topic or post on the board. But in Continuous
Integration (and sometimes on shared labs servers), other browsers are
simultaneously hitting the same page, and we get weird failures.

Solutions:

   - Start a new board for some tests. This would mean adding a dedicated
   Flow_QA namespace on test hosts, and adding it to $wgFlowOccupyNamespaces.


   - Change tests to target *the topic or post they just created.* Tests
   know what the text of the topic is, so search for that and look within it
   for elements.


Problem: Flow tests require MEDIAWIKI_USER to have admin and oversight
rights because they assume block user, delete topic/post, and suppress
topic/post are available. Maybe tests should skip if the MEDIAWIKI_USER
doesn't have the right. Or could we grant the Selenium_user these rights
but only in a dedicated Flow_QA namespace?
https://bugzilla.wikimedia.org/show_bug.cgi?id=67158


*New features for new tests*

*generally useful*? To test Thank, mentions appearing in the
thanked/mentioned user's notifications, and future subscription features,
we need a second QA user.


Thanks,
-- 
=S Page  Features engineer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140702/eebb4612/attachment.html>