On 25/03/15 21:37, Greg Grossmeier wrote:

Does the mobile team have the browser test pass/failure announcements sent to it's IRC channel now?

The MobileFrontend jobs send emails to chris and qa-alerts . It is configured JJB integration/config.git /jjb/browsertests.yaml via YAML aliases [1].

The project 'MobileFrontend' has:

recipients: *emails-qa

One would need to define a new alias for Mobile at the top of the file. Maybe the mobile team as an alert

IRC notifications are only send to #wikimedia-releng. It needs a bit more work to get make it customizable.

[1] https://git.wikimedia.org/blob/integration%2Fconfig/0b91d99/jjb%2Fbrowsertes...

On Thu, Mar 26, 2015 at 3:38 PM, Jon Robson jdlrobson@gmail.com wrote:

The lack of replies from any mobile team members seems to suggest I'm the only one watching this. In that case I'd rather we split up the big MobileFrontend job into various jobs based on certain features. How can we get more people caring about browser tests? Naming and shaming?

We have a couple projects in the works that will hopefully help.[1][2] They don't go so far as to name and shame (we'll leave that up to you :), but they should promote more ownership over browser tests, better communication of failure, and analysis of failure over time and by test function (feature vs smoke vs regression).

If many of these browser tests are serving as regression/smoke tests, it might be worthwhile to not only separate them out, but to remove some of the layers of abstraction to make tests more maintainable. I often try to think about tests in terms of a value-to-debt ratio (i.e. "How likely is it that I'll have to fix or refactor this test down the road and is it worth it?") and, while I find quite a bit of value in Cucumber for the sake of acceptance(-esque) testing (it keeps me mindful of the UX),[3] it introduces quite a bit of debt in its layers of abstraction and indirection (it's always a red flag when you have to grep to find your test implementation). Even its creators believe the value of Cucumber lies in its collaboration/planning capacities, not in its function as a test runner.[4]

It's entirely possible, as of the 1.0.0 release, to use MW-Selenium without Cucumber, so perhaps we could experiment with simple RSpec examples for these types of tests.

Tooling aside, there are broader discussions that need to take place among those that are practicing or have practiced TDD/BDD in the organization and how we might (or might not) promote theses practices among others. I'll be trying to form such a group next quarter (will it be a 'guild'?, a 'league'?, no need for tough decisions just yet[5]), so let's maintain a dialogue on that front if you're interested and willing.

[1] https://phabricator.wikimedia.org/T88706 [2] https://phabricator.wikimedia.org/T89375 [3] https://gerrit.wikimedia.org/r/#/c/196492/10/features/role_settings.feature [4] https://cukes.info/blog/2014/03/03/the-worlds-most-misunderstood-collaborati... [5] https://www.youtube.com/watch?v=6KSiyaqnZYs

Practically, the current big job [1] we have has far too many false positives and it is just noise to me. I was paying attention to the smoke tests [2] but I was told that was an upstream bug and haven't been watching it since. Any idea why that has been failing recently? Nothing has broken here...

[1] https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFron... [2] https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFron...

Have you tried to reproduce these failures by executing the tests locally, against either Beta or your local MW? I'll be in the office tomorrow if you want advice on how to best go about it.

Dan