[QA] Tests timing out when running via Jenkins/SauceLabs

[S Page]

On Sat, Nov 16, 2013 at 3:22 PM, Chris McMahon <cmcmahon at wikimedia.org>wrote:

>
> On Fri, Nov 15, 2013 at 12:28 PM, Jeff Hall <jhall at wikimedia.org> wrote:
>
>>  I spent some time today looking at automated test failures via
>> CloudBees/Jenkins (https://wmf.ci.cloudbees.com), and a pretty common
>> theme of tests that fail inconsistently is "Watir::Wait::TimeoutError"
>> issues.  Here's an example of a recent failure that falls into this
>> category:
>>
>>
>> https://wmf.ci.cloudbees.com/job/browsertests-commons.wikimedia.beta.wmflabs.org-linux-chrome/463/testReport/(root)/UploadWizard/Navigate_to_Describe_page/
>>
>
> Yes, this is a legitimate timeout issue, unlike the UW test failure on Nov
> 14, another of those maddening unexplained js errors from a ResourceLoader
> call:
> https://wmf.ci.cloudbees.com/job/browsertests-commons.wikimedia.beta.wmflabs.org-linux-chrome/461/
>

As some may not know, visiting a URL with ?debug=1 (or setting the cookie)
may report JS errors in the browser console.  See
https://www.mediawiki.org/wiki/ResourceLoader/Developing_with_ResourceLoader#Toggle_debug_mode

How did you determine it's a JS error? Was it by:
Click cloudbees job, navigate to failing test.
Click

Sauce Labs job URL: http://saucelabs.com/jobs/a31c0d53909e436cb6d31fb264933615

link.
In [Commands] tab search for "error", find
click to view 4 console messages ( 1 error )
Expand it and see
 [200 OK] GET
http://bits.beta.wmflabs.org/commons.wikimedia.beta.wmflabs.org/load.php?debug=false&lang=en&modules=jquery.ui.button%2Ccore%2Cdatepicker%2Cdialog%2Cdraggable%2Cmouse%2Cposition%2Cprogressbar%2Cresizable%2Cselectable%2Cwidget&skin=vector&version=20130920T230134Z&*
  Uncaught SyntaxError: Unexpected end of input

There's an explicit test for RL errors in
features/step_definitions/create_account_steps.rb in
qa/browsertests.git has a
  Then(/^page has no ResourceLoader errors$/) do

I think that assertion should be performed when visiting every page.



> I agree that explicit sleep() calls are bad design,
>
Are there any implicit sleep() calls in tests? When I watch WebDriver run
tests, it sometimes seems to wait a long time before moving to the next
step, it's not clear what it's doing. The few sleep calls I've seen in
tests are just 1 or 2 seconds.

(Is there a verbose mode for cucumber?)


> but I like polling for conditions:
>
> when_present() polls until an element can be engaged
> wait_until() polls until a condition returns true
> wait_while() polls until a condition returns false
>

That's gold, Jerry. Gold!  I added it to
https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Running_and_writing_tests#Timeouts


-- 
=S Page  Features engineer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20131118/e31cfa04/attachment-0001.html>