[QA] Fixing Jenkins jobs Re: How to contribute?

[Chris McMahon]

On Thu, Jun 6, 2013 at 10:42 AM, Carol Sanders <carol at carolsanders.org>wrote:

> I would like to help fix Jenkins jobs if the task hasn't been taken. I'm
> just starting to implement Jenkins at work so this would help my learning
> curve.
>

Hi Carol,

One of the things that makes browser testing/UI testing unusual is that
unlike unit tests, UI tests can fail for all sorts of reasons.  Analyzing
test failures is interesting and sometimes challenging work.

We recently made some major changes to how we do builds, and we're just now
sorting some of the technical debt from that move, so the timing is good
for this sort of thing.

To get started, take a look at https://wmf.ci.cloudbees.com/

You'll see that our builds are arranged according to target test
environment and browser.  You'll see that most of the Firefox/Chrome builds
are (usually) green, and a number of IE builds are (usually) red.

To diagnose a failed build, do this:

* Find a red build at https://wmf.ci.cloudbees.com/ and click it in Jenkins
* Click "Latest test result" where it shows at least one failure
* Clck whatever "Test Name" interests you under "All failed tests"
* This is a critical page, pay close attention to the "Message" section at
the bottom of the page, this is the test framework saying what went wrong
with the test run.
* With the failure message in mind, click to the Sauce Labs page.
* On the Sauce Labs page you can find a video of the test run and screen
shots taken throughout the course of the test run.  These are often
valuable when figuring out what went wrong in a particular test.

Some common modes of failure:

* Some page element is visible but should not be
* Some page element is not visible but should be
* The test times out waiting for some condition
* Some system error occurs

If the test failure indicates a bug, discussing it here is a great idea.
 Then we can file an issue in Bugzilla using the "browser-test-bug"
keyword, for example like
https://bugzilla.wikimedia.org/show_bug.cgi?id=47837

Keeping the builds green for IE is a challenge because the Selenium
Webdriver "IEDriver" is less robust than the drivers for Firefox, Chrome,
and Opera.  IEDriver gets significantly less attention than the other
browser drivers, and is quite a bit flakier.  In particular, we find that
our UploadWizard application tends to work reasonable well in IE for real
users, but automated tests don't work particularly well, and we suspect
that IEDriver is the culprit.

-Chris





>
> I open to any method of collaboration: IRC/Skype/Google Hangout.
> I would prefer as a first time to 'hangout' with someone more experienced
> in this environment as not to make more work for fellow helpers.
>
> Thanks,
>
> Carol
>
> Carol A. Sanders
> Sr, Network QA Engineer, Mad Scientist, Student
> "Escape the Boundaries of Your Mentality"
>
>
> On Tue, Jun 4, 2013 at 10:46 AM, Željko Filipin <zfilipin at wikimedia.org>wrote:
>
>> Hi everybody,
>>
>> a few people said they would like to contribute. I guess some of you
>> already know how you would like to contribute, like helping organising
>> event(s).
>>
>> I know that at least a few of you need a bit of guidance for the first
>> few contributions, because you have told me so. :) Chris has already sent a
>> few posts saying how you could get involved. I have a few smaller and more
>> focused tasks that could be a good way for somebody to get involved.
>>
>> You can do all tasks on your own time, or we could arrange a pairing
>> session on IRC/Skype/Google Hangout. Tasks are sorted roughly by how hard
>> they are.
>>
>> #1 fix class name
>>
>> visual_editor_page.rb[1] file has wrong class name, UserPage. It should
>> be VisualEditorPage. The task is to rename the class. It should be trivial
>> to do. Depending on your Git and/or Github/Gerrit skills, submitting the
>> code would probably be harder than fixing it.
>>
>> #2 fix failing Jenkins job(s) for desktop browsers
>>
>> Chris and I are doing our best to keep Jenkins jobs running tests on
>> desktop browsers[2] green, but things change a lot and the tests start
>> failing. Additionally, we recently made a few significant changes to
>> Jenkins jobs (targeting more environments) and a lot of tests are failing
>> at the moment. If you would to help us fix a test/job (or six), let me
>> know. Sometimes the fix is as simple as disabling a test that fails because
>> it by mistake tries to test a feature that is not enabled at certain
>> environment. As with task #1 the fix is trivial, and the harder part would
>> probably be submitting code to Github/Gerrit.
>>
>> #3 fix failing Jenkins job(s) for mobile browsers
>>
>> All Jenkins jobs running tests for mobile browsers[3] are disabled
>> because they started failing and nobody had the time to investigate and fix
>> the problems. If you are interested in testing on mobile browsers and/or
>> devices, this is a task for you. I can help you pick and fix at least one
>> test.
>>
>>
>> Regarding tasks #2 and #3: I feels great to fix a broken build and to see
>> a job going from red to green. Trust me.
>>
>> Željko
>> --
>> 1:
>> https://github.com/wikimedia/qa-browsertests/blob/master/features/support/pages/visual_editor_page.rb#L1
>> 2: https://wmf.ci.cloudbees.com/view/r-bt/
>> 3: https://wmf.ci.cloudbees.com/view/r-mf/
>>
>> _______________________________________________
>> QA mailing list
>> QA at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/qa
>>
>>
>
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20130607/02423713/attachment-0001.html>