<div dir="ltr">On Sat, Nov 16, 2013 at 3:22 PM, Chris McMahon <span dir="ltr"><<a href="mailto:cmcmahon@wikimedia.org" target="_blank">cmcmahon@wikimedia.org</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra">On Fri, Nov 15, 2013 at 12:28 PM, Jeff Hall <span dir="ltr"><<a href="mailto:jhall@wikimedia.org" target="_blank">jhall@wikimedia.org</a>></span> wrote:<br>
<div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
I spent some time today looking at automated test failures via
CloudBees/Jenkins (<a href="https://wmf.ci.cloudbees.com" target="_blank">https://wmf.ci.cloudbees.com</a>), and a pretty
common theme of tests that fail inconsistently is
"Watir::Wait::TimeoutError" issues. Here's an example of a recent
failure that falls into this category:<br>
<br>
<a href="https://wmf.ci.cloudbees.com/job/browsertests-commons.wikimedia.beta.wmflabs.org-linux-chrome/463/testReport/%28root%29/UploadWizard/Navigate_to_Describe_page/" target="_blank">https://wmf.ci.cloudbees.com/job/browsertests-commons.wikimedia.beta.wmflabs.org-linux-chrome/463/testReport/(root)/UploadWizard/Navigate_to_Describe_page/</a></div>
</blockquote><div><br></div></div><div>Yes, this is a legitimate timeout issue, unlike the UW test failure on Nov 14, another of those maddening unexplained js errors from a ResourceLoader call: <a href="https://wmf.ci.cloudbees.com/job/browsertests-commons.wikimedia.beta.wmflabs.org-linux-chrome/461/" target="_blank">https://wmf.ci.cloudbees.com/job/browsertests-commons.wikimedia.beta.wmflabs.org-linux-chrome/461/</a></div>
</div></div></div></blockquote><div><br></div><div>As some may not know, visiting a URL with ?debug=1 (or setting the cookie) may report JS errors in the browser console. See <a href="https://www.mediawiki.org/wiki/ResourceLoader/Developing_with_ResourceLoader#Toggle_debug_mode">https://www.mediawiki.org/wiki/ResourceLoader/Developing_with_ResourceLoader#Toggle_debug_mode</a><br>
<br></div><div>How did you determine it's a JS error? Was it by:<br></div><div>Click cloudbees job, navigate to failing test.<br>Click <br><pre>Sauce Labs job URL: <a href="http://saucelabs.com/jobs/a31c0d53909e436cb6d31fb264933615">http://saucelabs.com/jobs/a31c0d53909e436cb6d31fb264933615</a></pre>
link.<br>In [Commands] tab search for "error", find<br><a class=""><span style="color:rgb(255,0,0)">click to view 4 console messages
( 1 error )</span><br>Expand it and see</a> <br><div class="">
<div class="">
<span class="">[200 OK]
GET
<a href="http://bits.beta.wmflabs.org/commons.wikimedia.beta.wmflabs.org/load.php?debug=false&lang=en&modules=jquery.ui.button%2Ccore%2Cdatepicker%2Cdialog%2Cdraggable%2Cmouse%2Cposition%2Cprogressbar%2Cresizable%2Cselectable%2Cwidget&skin=vector&version=20130920T230134Z&*">http://bits.beta.wmflabs.org/commons.wikimedia.beta.wmflabs.org/load.php?debug=false&lang=en&modules=jquery.ui.button%2Ccore%2Cdatepicker%2Cdialog%2Cdraggable%2Cmouse%2Cposition%2Cprogressbar%2Cresizable%2Cselectable%2Cwidget&skin=vector&version=20130920T230134Z&*</a></span>
</div>
</div>
<div class="">
<div class="">
<span class="">Uncaught SyntaxError: Unexpected end of input</span>
</div>
</div><br>There's an explicit test for RL errors in features/step_definitions/create_account_steps.rb in <br>qa/browsertests.git has a<br> Then(/^page has no ResourceLoader errors$/) do<br>
<br></div><div>I think that assertion should be performed when visiting every page.<br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div>I agree that explicit sleep() calls are bad design,</div></div></div></blockquote><div>Are there any implicit sleep() calls in tests? When I watch WebDriver run tests, it sometimes seems to wait a long time before moving to the next step, it's not clear what it's doing. The few sleep calls I've seen in tests are just 1 or 2 seconds.<br>
<br></div><div>(Is there a verbose mode for cucumber?)<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra">
<div class="gmail_quote"><div> but I like polling for conditions:</div><div><br></div><div>when_present() polls until an element can be engaged</div>
<div>wait_until() polls until a condition returns true</div><div>wait_while() polls until a condition returns false</div></div></div></div></blockquote><div><br></div><div>That's gold, Jerry. Gold! I added it to <a href="https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Running_and_writing_tests#Timeouts">https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Running_and_writing_tests#Timeouts</a><br>
<br></div></div><br>-- <br><div dir="ltr">=S Page Features engineer<br></div>
</div></div>