It really saddens me how very few engineers seem to care about browser tests. Our browser tests are failing all over the place. I just saw this bug [1] which has been sitting around for ages and denying us green tests in Echo one of our most important features.
How can we change this anti-pattern?
Dan Duval, would it make sense to do a survey as you did with Vagrant to understand how our developers think of these? Such as who owns them... who is responsible for a test failing... who writes them... who doesn't understand them.. why they don't understand them etc...?
[1] https://phabricator.wikimedia.org/T72298
On Thu, Mar 26, 2015 at 5:54 PM, Dan Duvall dduvall@wikimedia.org wrote:
On Thu, Mar 26, 2015 at 3:38 PM, Jon Robson jdlrobson@gmail.com wrote:
The lack of replies from any mobile team members seems to suggest I'm the only one watching this. In that case I'd rather we split up the big MobileFrontend job into various jobs based on certain features. How can
we
get more people caring about browser tests? Naming and shaming?
We have a couple projects in the works that will hopefully help.[1][2] They don't go so far as to name and shame (we'll leave that up to you :), but they should promote more ownership over browser tests, better communication of failure, and analysis of failure over time and by test function (feature vs smoke vs regression).
If many of these browser tests are serving as regression/smoke tests, it might be worthwhile to not only separate them out, but to remove some of the layers of abstraction to make tests more maintainable. I often try to think about tests in terms of a value-to-debt ratio (i.e. "How likely is it that I'll have to fix or refactor this test down the road and is it worth it?") and, while I find quite a bit of value in Cucumber for the sake of acceptance(-esque) testing (it keeps me mindful of the UX),[3] it introduces quite a bit of debt in its layers of abstraction and indirection (it's always a red flag when you have to grep to find your test implementation). Even its creators believe the value of Cucumber lies in its collaboration/planning capacities, not in its function as a test runner.[4]
It's entirely possible, as of the 1.0.0 release, to use MW-Selenium without Cucumber, so perhaps we could experiment with simple RSpec examples for these types of tests.
Tooling aside, there are broader discussions that need to take place among those that are practicing or have practiced TDD/BDD in the organization and how we might (or might not) promote theses practices among others. I'll be trying to form such a group next quarter (will it be a 'guild'?, a 'league'?, no need for tough decisions just yet[5]), so let's maintain a dialogue on that front if you're interested and willing.
[1] https://phabricator.wikimedia.org/T88706 [2] https://phabricator.wikimedia.org/T89375 [3] https://gerrit.wikimedia.org/r/#/c/196492/10/features/role_settings.feature [4] https://cukes.info/blog/2014/03/03/the-worlds-most-misunderstood-collaborati... [5] https://www.youtube.com/watch?v=6KSiyaqnZYs
Practically, the current big job [1] we have has far too many false positives and it is just noise to me. I was paying attention to the smoke tests [2] but I was told that was an upstream bug and haven't been
watching
it since. Any idea why that has been failing recently? Nothing has broken here...
[1]
https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFron...
[2]
https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFron...
Have you tried to reproduce these failures by executing the tests locally, against either Beta or your local MW? I'll be in the office tomorrow if you want advice on how to best go about it.
Dan
-- Dan Duvall Automation Engineer Wikimedia Foundation
On Mon, Mar 30, 2015 at 10:30 AM, Jon Robson jdlrobson@gmail.com wrote:
It really saddens me how very few engineers seem to care about browser tests. Our browser tests are failing all over the place. I just saw this bug [1] which has been sitting around for ages and denying us green tests in Echo one of our most important features.
How can we change this anti-pattern?
That's exactly what I'd like to explore with you and other like minds.
Dan Duval, would it make sense to do a survey as you did with Vagrant to understand how our developers think of these? Such as who owns them... who is responsible for a test failing... who writes them... who doesn't understand them.. why they don't understand them etc...?
Great idea! I suspect that the number of false positives in a given repo's test suite is inversely related to the number of developers on the team actually writing tests, and the affordance by managers to do so. If you're not regularly writing tests, you're probably not going to feel comfortable troubleshooting and refactoring someone else's. If TDD isn't factored in to your team's velocity, you may feel like the investment in writing tests (or learning to write them) isn't worth it or comes at the risk of missing deadlines.
A survey could definitely help us to verify (or disprove) these relationships.
Some other questions I can think of:
- How valuable are unit tests to the health/quality of a software project? - How valuable are browser tests to the health/quality of a software project? - How much experience do you have with TDD? - Would you like more time to learn or practice TDD? - How often do you write tests when developing a new feature? - What kinds of test? (% of unit test vs. browser test) - How often do you write tests to verify a bugfix? - What kinds of test? (% of unit test vs. browser test) - When would you typically write a unit test? (before implementation, after implementation, when stuff breaks) - When would you typically write a browser test? (during conception, before implementation, after implementation, when stuff breaks) - What are the largest barriers to writing/running unit tests? (test framework, documentation/examples, execution time, CI, structure of my code, structure of code I depend on) - What are the largest barriers to writing/running browser tests? (test framework, documentation/examples, execution time, CI) - What are the largest barriers to debugging test failure? (test framework, confusing errors/stack traces, documentation/examples, debugging tools)
I'll create a Phab task to track it. :)
https://phabricator.wikimedia.org/T94472
On Mon, Mar 30, 2015 at 12:39 PM, Dan Duvall dduvall@wikimedia.org wrote:
On Mon, Mar 30, 2015 at 10:30 AM, Jon Robson jdlrobson@gmail.com wrote:
It really saddens me how very few engineers seem to care about browser tests. Our browser tests are failing all over the place. I just saw this bug [1] which has been sitting around for ages and denying us green tests in Echo one of our most important features.
How can we change this anti-pattern?
That's exactly what I'd like to explore with you and other like minds.
Dan Duval, would it make sense to do a survey as you did with Vagrant to understand how our developers think of these? Such as who owns them... who is responsible for a test failing... who writes them... who doesn't understand them.. why they don't understand them etc...?
Great idea! I suspect that the number of false positives in a given repo's test suite is inversely related to the number of developers on the team actually writing tests, and the affordance by managers to do so. If you're not regularly writing tests, you're probably not going to feel comfortable troubleshooting and refactoring someone else's. If TDD isn't factored in to your team's velocity, you may feel like the investment in writing tests (or learning to write them) isn't worth it or comes at the risk of missing deadlines.
A survey could definitely help us to verify (or disprove) these relationships.
Some other questions I can think of:
- How valuable are unit tests to the health/quality of a software project?
- How valuable are browser tests to the health/quality of a software project?
- How much experience do you have with TDD?
- Would you like more time to learn or practice TDD?
- How often do you write tests when developing a new feature?
- What kinds of test? (% of unit test vs. browser test)
- How often do you write tests to verify a bugfix?
- What kinds of test? (% of unit test vs. browser test)
- When would you typically write a unit test? (before implementation,
after implementation, when stuff breaks)
- When would you typically write a browser test? (during conception,
before implementation, after implementation, when stuff breaks)
- What are the largest barriers to writing/running unit tests? (test
framework, documentation/examples, execution time, CI, structure of my code, structure of code I depend on)
- What are the largest barriers to writing/running browser tests?
(test framework, documentation/examples, execution time, CI)
- What are the largest barriers to debugging test failure? (test
framework, confusing errors/stack traces, documentation/examples, debugging tools)
I'll create a Phab task to track it. :)
-- Dan Duvall Automation Engineer Wikimedia Foundation
Dan, Jon,
I got caught up in meetings yesterday – you'll see this email a lot during Q4 ;) – so I delayed sending this email, so forgive the repetitions of some of Dan's points/questions:
Here are a few ways I can think of:
- include feedback on browser tests – or lack thereof – during code
review
- make browser test failures even more visible than they currently are
– but maybe not the success reports, eh?
- can these reports be made to point at a bunch of candidate changes
that may have broken 'em?
- hold a browser-test-athon with the team and any volunteers at the
{Lyon,Wikimania} hackathon
- make it trivial to run 'em, if it isn't already
From what little experience I have of trying to establish team practices,
I'd say that it's best to advocate for <practice> and demonstrate its value*, rather than criticise. I'd love to see you funnel your passion for browser testing into a talk or series of talks for the mobile team – the org, maybe? – or maybe you've got some recommended reading or talks you'd like to share that'll inspire.
–Sam
* If you'd like to hear my opinions about browser testing, then insert one beer and wind me up a little
On Mon, Mar 30, 2015 at 8:47 PM, Dan Duvall dduvall@wikimedia.org wrote:
https://phabricator.wikimedia.org/T94472
On Mon, Mar 30, 2015 at 12:39 PM, Dan Duvall dduvall@wikimedia.org wrote:
On Mon, Mar 30, 2015 at 10:30 AM, Jon Robson jdlrobson@gmail.com
wrote:
It really saddens me how very few engineers seem to care about browser tests. Our browser tests are failing all over the place. I just saw
this bug
[1] which has been sitting around for ages and denying us green tests in Echo one of our most important features.
How can we change this anti-pattern?
That's exactly what I'd like to explore with you and other like minds.
Dan Duval, would it make sense to do a survey as you did with Vagrant to understand how our developers think of these? Such as who owns them...
who
is responsible for a test failing... who writes them... who doesn't understand them.. why they don't understand them etc...?
Great idea! I suspect that the number of false positives in a given repo's test suite is inversely related to the number of developers on the team actually writing tests, and the affordance by managers to do so. If you're not regularly writing tests, you're probably not going to feel comfortable troubleshooting and refactoring someone else's. If TDD isn't factored in to your team's velocity, you may feel like the investment in writing tests (or learning to write them) isn't worth it or comes at the risk of missing deadlines.
A survey could definitely help us to verify (or disprove) these
relationships.
Some other questions I can think of:
- How valuable are unit tests to the health/quality of a software
project?
- How valuable are browser tests to the health/quality of a software
project?
- How much experience do you have with TDD?
- Would you like more time to learn or practice TDD?
- How often do you write tests when developing a new feature?
- What kinds of test? (% of unit test vs. browser test)
- How often do you write tests to verify a bugfix?
- What kinds of test? (% of unit test vs. browser test)
- When would you typically write a unit test? (before implementation,
after implementation, when stuff breaks)
- When would you typically write a browser test? (during conception,
before implementation, after implementation, when stuff breaks)
- What are the largest barriers to writing/running unit tests? (test
framework, documentation/examples, execution time, CI, structure of my code, structure of code I depend on)
- What are the largest barriers to writing/running browser tests?
(test framework, documentation/examples, execution time, CI)
- What are the largest barriers to debugging test failure? (test
framework, confusing errors/stack traces, documentation/examples, debugging tools)
I'll create a Phab task to track it. :)
-- Dan Duvall Automation Engineer Wikimedia Foundation
-- Dan Duvall Automation Engineer Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Personally, I think one team needs to get this completely right and demonstrate the difference e.g. fewer bugs, iterating fast, quicker code review time etc..
Talks can come off the back of that. The majority of people I speak to seem to advocate a TDD approach but I think we don't make life easy enough for them to do that and we lack the discipline to do that. We need to work on both of those two.
I'm confident if we do a survey we'll identify and prioritise the pain points. For me my top priority would be getting the infrastructure in place on all our existing codebases in a consistent way that make adding tests effortless and prevent code merging when it breaks those tests but there may be more important things we need to sort out first!
On Tue, Mar 31, 2015 at 1:16 AM, Sam Smith samsmith@wikimedia.org wrote:
Dan, Jon,
I got caught up in meetings yesterday – you'll see this email a lot during Q4 ;) – so I delayed sending this email, so forgive the repetitions of some of Dan's points/questions:
Here are a few ways I can think of:
include feedback on browser tests – or lack thereof – during code review
make browser test failures even more visible than they currently are – but maybe not the success reports, eh?
can these reports be made to point at a bunch of candidate changes that may have broken 'em?
hold a browser-test-athon with the team and any volunteers at the {Lyon,Wikimania} hackathon
make it trivial to run 'em, if it isn't already
From what little experience I have of trying to establish team practices, I'd say that it's best to advocate for <practice> and demonstrate its value*, rather than criticise. I'd love to see you funnel your passion for browser testing into a talk or series of talks for the mobile team – the org, maybe? – or maybe you've got some recommended reading or talks you'd like to share that'll inspire.
–Sam
- If you'd like to hear my opinions about browser testing, then insert one
beer and wind me up a little
On Mon, Mar 30, 2015 at 8:47 PM, Dan Duvall dduvall@wikimedia.org wrote:
https://phabricator.wikimedia.org/T94472
On Mon, Mar 30, 2015 at 12:39 PM, Dan Duvall dduvall@wikimedia.org wrote:
On Mon, Mar 30, 2015 at 10:30 AM, Jon Robson jdlrobson@gmail.com wrote:
It really saddens me how very few engineers seem to care about browser tests. Our browser tests are failing all over the place. I just saw this bug [1] which has been sitting around for ages and denying us green tests in Echo one of our most important features.
How can we change this anti-pattern?
That's exactly what I'd like to explore with you and other like minds.
Dan Duval, would it make sense to do a survey as you did with Vagrant to understand how our developers think of these? Such as who owns them... who is responsible for a test failing... who writes them... who doesn't understand them.. why they don't understand them etc...?
Great idea! I suspect that the number of false positives in a given repo's test suite is inversely related to the number of developers on the team actually writing tests, and the affordance by managers to do so. If you're not regularly writing tests, you're probably not going to feel comfortable troubleshooting and refactoring someone else's. If TDD isn't factored in to your team's velocity, you may feel like the investment in writing tests (or learning to write them) isn't worth it or comes at the risk of missing deadlines.
A survey could definitely help us to verify (or disprove) these relationships.
Some other questions I can think of:
- How valuable are unit tests to the health/quality of a software
project?
- How valuable are browser tests to the health/quality of a software
project?
- How much experience do you have with TDD?
- Would you like more time to learn or practice TDD?
- How often do you write tests when developing a new feature?
- What kinds of test? (% of unit test vs. browser test)
- How often do you write tests to verify a bugfix?
- What kinds of test? (% of unit test vs. browser test)
- When would you typically write a unit test? (before implementation,
after implementation, when stuff breaks)
- When would you typically write a browser test? (during conception,
before implementation, after implementation, when stuff breaks)
- What are the largest barriers to writing/running unit tests? (test
framework, documentation/examples, execution time, CI, structure of my code, structure of code I depend on)
- What are the largest barriers to writing/running browser tests?
(test framework, documentation/examples, execution time, CI)
- What are the largest barriers to debugging test failure? (test
framework, confusing errors/stack traces, documentation/examples, debugging tools)
I'll create a Phab task to track it. :)
-- Dan Duvall Automation Engineer Wikimedia Foundation
-- Dan Duvall Automation Engineer Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Personally, I love them, but more often than not when I run them a few things happen that make me sad:
- Tests take too long to run - Even on Gather, which doesn't have too many tests, it takes a really long time, which discourages running all tests every time. - Tests don't block jenkins merges - This would be key for me. Do this. Block jenkins. Then we'll be forced to make it faster, make better browser tests, and everybody will have to care because they will run and block merges.
So there's a few key items that I would love to see:
- Improve performance (needs magnitudes improvements, a lot of work on different fronts) - Fully support headless browsers like phantom (they randomly break with timeout errors and other problems, but are the faster/painless ways of running the tests) - Run the browser tests (or a smoke subset) on jenkins patches as a voting job. Crucial for making everybody care about the tests and to stop regressions.
On Thu, Apr 2, 2015 at 10:18 PM, Jon Robson jdlrobson@gmail.com wrote:
Personally, I think one team needs to get this completely right and demonstrate the difference e.g. fewer bugs, iterating fast, quicker code review time etc..
Talks can come off the back of that. The majority of people I speak to seem to advocate a TDD approach but I think we don't make life easy enough for them to do that and we lack the discipline to do that. We need to work on both of those two.
I'm confident if we do a survey we'll identify and prioritise the pain points. For me my top priority would be getting the infrastructure in place on all our existing codebases in a consistent way that make adding tests effortless and prevent code merging when it breaks those tests but there may be more important things we need to sort out first!
On Tue, Mar 31, 2015 at 1:16 AM, Sam Smith samsmith@wikimedia.org wrote:
Dan, Jon,
I got caught up in meetings yesterday – you'll see this email a lot
during
Q4 ;) – so I delayed sending this email, so forgive the repetitions of
some
of Dan's points/questions:
Here are a few ways I can think of:
include feedback on browser tests – or lack thereof – during code review
make browser test failures even more visible than they currently are –
but
maybe not the success reports, eh?
can these reports be made to point at a bunch of candidate changes that may have broken 'em?
hold a browser-test-athon with the team and any volunteers at the {Lyon,Wikimania} hackathon
make it trivial to run 'em, if it isn't already
From what little experience I have of trying to establish team practices, I'd say that it's best to advocate for <practice> and demonstrate its value*, rather than criticise. I'd love to see you funnel your passion
for
browser testing into a talk or series of talks for the mobile team – the org, maybe? – or maybe you've got some recommended reading or talks you'd like to share that'll inspire.
–Sam
- If you'd like to hear my opinions about browser testing, then insert
one
beer and wind me up a little
On Mon, Mar 30, 2015 at 8:47 PM, Dan Duvall dduvall@wikimedia.org
wrote:
https://phabricator.wikimedia.org/T94472
On Mon, Mar 30, 2015 at 12:39 PM, Dan Duvall dduvall@wikimedia.org wrote:
On Mon, Mar 30, 2015 at 10:30 AM, Jon Robson jdlrobson@gmail.com wrote:
It really saddens me how very few engineers seem to care about
browser
tests. Our browser tests are failing all over the place. I just saw this bug [1] which has been sitting around for ages and denying us green tests in Echo one of our most important features.
How can we change this anti-pattern?
That's exactly what I'd like to explore with you and other like minds.
Dan Duval, would it make sense to do a survey as you did with Vagrant to understand how our developers think of these? Such as who owns
them...
who is responsible for a test failing... who writes them... who doesn't understand them.. why they don't understand them etc...?
Great idea! I suspect that the number of false positives in a given repo's test suite is inversely related to the number of developers on the team actually writing tests, and the affordance by managers to do so. If you're not regularly writing tests, you're probably not going to feel comfortable troubleshooting and refactoring someone else's. If TDD isn't factored in to your team's velocity, you may feel like the investment in writing tests (or learning to write them) isn't worth it or comes at the risk of missing deadlines.
A survey could definitely help us to verify (or disprove) these relationships.
Some other questions I can think of:
- How valuable are unit tests to the health/quality of a software
project?
- How valuable are browser tests to the health/quality of a software
project?
- How much experience do you have with TDD?
- Would you like more time to learn or practice TDD?
- How often do you write tests when developing a new feature?
- What kinds of test? (% of unit test vs. browser test)
- How often do you write tests to verify a bugfix?
- What kinds of test? (% of unit test vs. browser test)
- When would you typically write a unit test? (before implementation,
after implementation, when stuff breaks)
- When would you typically write a browser test? (during conception,
before implementation, after implementation, when stuff breaks)
- What are the largest barriers to writing/running unit tests? (test
framework, documentation/examples, execution time, CI, structure of my code, structure of code I depend on)
- What are the largest barriers to writing/running browser tests?
(test framework, documentation/examples, execution time, CI)
- What are the largest barriers to debugging test failure? (test
framework, confusing errors/stack traces, documentation/examples, debugging tools)
I'll create a Phab task to track it. :)
-- Dan Duvall Automation Engineer Wikimedia Foundation
-- Dan Duvall Automation Engineer Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Jon Robson
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
CC'ing Stephane from the Collaboration team who's keenly interested in this as well.
--tomasz
On Fri, Apr 3, 2015 at 2:30 AM, Joaquin Oltra Hernandez jhernandez@wikimedia.org wrote:
Personally, I love them, but more often than not when I run them a few things happen that make me sad:
Tests take too long to run
Even on Gather, which doesn't have too many tests, it takes a really long time, which discourages running all tests every time.
Tests don't block jenkins merges
This would be key for me. Do this. Block jenkins. Then we'll be forced to make it faster, make better browser tests, and everybody will have to care because they will run and block merges.
So there's a few key items that I would love to see:
Improve performance (needs magnitudes improvements, a lot of work on different fronts) Fully support headless browsers like phantom (they randomly break with timeout errors and other problems, but are the faster/painless ways of running the tests) Run the browser tests (or a smoke subset) on jenkins patches as a voting job. Crucial for making everybody care about the tests and to stop regressions.
On Thu, Apr 2, 2015 at 10:18 PM, Jon Robson jdlrobson@gmail.com wrote:
Personally, I think one team needs to get this completely right and demonstrate the difference e.g. fewer bugs, iterating fast, quicker code review time etc..
Talks can come off the back of that. The majority of people I speak to seem to advocate a TDD approach but I think we don't make life easy enough for them to do that and we lack the discipline to do that. We need to work on both of those two.
I'm confident if we do a survey we'll identify and prioritise the pain points. For me my top priority would be getting the infrastructure in place on all our existing codebases in a consistent way that make adding tests effortless and prevent code merging when it breaks those tests but there may be more important things we need to sort out first!
On Tue, Mar 31, 2015 at 1:16 AM, Sam Smith samsmith@wikimedia.org wrote:
Dan, Jon,
I got caught up in meetings yesterday – you'll see this email a lot during Q4 ;) – so I delayed sending this email, so forgive the repetitions of some of Dan's points/questions:
Here are a few ways I can think of:
include feedback on browser tests – or lack thereof – during code review
make browser test failures even more visible than they currently are – but maybe not the success reports, eh?
can these reports be made to point at a bunch of candidate changes that may have broken 'em?
hold a browser-test-athon with the team and any volunteers at the {Lyon,Wikimania} hackathon
make it trivial to run 'em, if it isn't already
From what little experience I have of trying to establish team practices, I'd say that it's best to advocate for <practice> and demonstrate its value*, rather than criticise. I'd love to see you funnel your passion for browser testing into a talk or series of talks for the mobile team – the org, maybe? – or maybe you've got some recommended reading or talks you'd like to share that'll inspire.
–Sam
- If you'd like to hear my opinions about browser testing, then insert
one beer and wind me up a little
On Mon, Mar 30, 2015 at 8:47 PM, Dan Duvall dduvall@wikimedia.org wrote:
https://phabricator.wikimedia.org/T94472
On Mon, Mar 30, 2015 at 12:39 PM, Dan Duvall dduvall@wikimedia.org wrote:
On Mon, Mar 30, 2015 at 10:30 AM, Jon Robson jdlrobson@gmail.com wrote:
It really saddens me how very few engineers seem to care about browser tests. Our browser tests are failing all over the place. I just saw this bug [1] which has been sitting around for ages and denying us green tests in Echo one of our most important features.
How can we change this anti-pattern?
That's exactly what I'd like to explore with you and other like minds.
Dan Duval, would it make sense to do a survey as you did with Vagrant to understand how our developers think of these? Such as who owns them... who is responsible for a test failing... who writes them... who doesn't understand them.. why they don't understand them etc...?
Great idea! I suspect that the number of false positives in a given repo's test suite is inversely related to the number of developers on the team actually writing tests, and the affordance by managers to do so. If you're not regularly writing tests, you're probably not going to feel comfortable troubleshooting and refactoring someone else's. If TDD isn't factored in to your team's velocity, you may feel like the investment in writing tests (or learning to write them) isn't worth it or comes at the risk of missing deadlines.
A survey could definitely help us to verify (or disprove) these relationships.
Some other questions I can think of:
- How valuable are unit tests to the health/quality of a software
project?
- How valuable are browser tests to the health/quality of a software
project?
- How much experience do you have with TDD?
- Would you like more time to learn or practice TDD?
- How often do you write tests when developing a new feature?
- What kinds of test? (% of unit test vs. browser test)
- How often do you write tests to verify a bugfix?
- What kinds of test? (% of unit test vs. browser test)
- When would you typically write a unit test? (before
implementation, after implementation, when stuff breaks)
- When would you typically write a browser test? (during conception,
before implementation, after implementation, when stuff breaks)
- What are the largest barriers to writing/running unit tests? (test
framework, documentation/examples, execution time, CI, structure of my code, structure of code I depend on)
- What are the largest barriers to writing/running browser tests?
(test framework, documentation/examples, execution time, CI)
- What are the largest barriers to debugging test failure? (test
framework, confusing errors/stack traces, documentation/examples, debugging tools)
I'll create a Phab task to track it. :)
-- Dan Duvall Automation Engineer Wikimedia Foundation
-- Dan Duvall Automation Engineer Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Jon Robson
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Apologies for spam if you are already aware of this, but I have proposed two workshops for this years' hackathons, write the first browsertests/Selenium test[0] and Fix broken browsertests/Selenium Jenkins jobs[1].
Željko -- 0: https://phabricator.wikimedia.org/T94024 1: https://phabricator.wikimedia.org/T94299
On Tue, Apr 14, 2015 at 8:55 PM, Tomasz Finc tfinc@wikimedia.org wrote:
CC'ing Stephane from the Collaboration team who's keenly interested in this as well.
--tomasz
On Fri, Apr 3, 2015 at 2:30 AM, Joaquin Oltra Hernandez jhernandez@wikimedia.org wrote:
Personally, I love them, but more often than not when I run them a few things happen that make me sad:
Tests take too long to run
Even on Gather, which doesn't have too many tests, it takes a really long time, which discourages running all tests every time.
Tests don't block jenkins merges
This would be key for me. Do this. Block jenkins. Then we'll be forced to make it faster, make better browser tests, and everybody will have to
care
because they will run and block merges.
So there's a few key items that I would love to see:
Improve performance (needs magnitudes improvements, a lot of work on different fronts) Fully support headless browsers like phantom (they randomly break with timeout errors and other problems, but are the faster/painless ways of running the tests) Run the browser tests (or a smoke subset) on jenkins patches as a voting job. Crucial for making everybody care about the tests and to stop regressions.
On Thu, Apr 2, 2015 at 10:18 PM, Jon Robson jdlrobson@gmail.com wrote:
Personally, I think one team needs to get this completely right and demonstrate the difference e.g. fewer bugs, iterating fast, quicker code review time etc..
Talks can come off the back of that. The majority of people I speak to seem to advocate a TDD approach but I think we don't make life easy enough for them to do that and we lack the discipline to do that. We need to work on both of those two.
I'm confident if we do a survey we'll identify and prioritise the pain points. For me my top priority would be getting the infrastructure in place on all our existing codebases in a consistent way that make adding tests effortless and prevent code merging when it breaks those tests but there may be more important things we need to sort out first!
On Tue, Mar 31, 2015 at 1:16 AM, Sam Smith samsmith@wikimedia.org
wrote:
Dan, Jon,
I got caught up in meetings yesterday – you'll see this email a lot during Q4 ;) – so I delayed sending this email, so forgive the repetitions of some of Dan's points/questions:
Here are a few ways I can think of:
include feedback on browser tests – or lack thereof – during code review
make browser test failures even more visible than they currently are
–
but maybe not the success reports, eh?
can these reports be made to point at a bunch of candidate changes
that
may have broken 'em?
hold a browser-test-athon with the team and any volunteers at the {Lyon,Wikimania} hackathon
make it trivial to run 'em, if it isn't already
From what little experience I have of trying to establish team practices, I'd say that it's best to advocate for <practice> and demonstrate its value*, rather than criticise. I'd love to see you funnel your passion for browser testing into a talk or series of talks for the mobile team –
the
org, maybe? – or maybe you've got some recommended reading or talks you'd like to share that'll inspire.
–Sam
- If you'd like to hear my opinions about browser testing, then insert
one beer and wind me up a little
On Mon, Mar 30, 2015 at 8:47 PM, Dan Duvall dduvall@wikimedia.org wrote:
https://phabricator.wikimedia.org/T94472
On Mon, Mar 30, 2015 at 12:39 PM, Dan Duvall dduvall@wikimedia.org wrote:
On Mon, Mar 30, 2015 at 10:30 AM, Jon Robson jdlrobson@gmail.com wrote: > It really saddens me how very few engineers seem to care about > browser > tests. Our browser tests are failing all over the place. I just
saw
> this bug > [1] which has been sitting around for ages and denying us green > tests > in > Echo one of our most important features. > > How can we change this anti-pattern?
That's exactly what I'd like to explore with you and other like minds.
> Dan Duval, would it make sense to do a survey as you did with > Vagrant > to > understand how our developers think of these? Such as who owns > them... > who > is responsible for a test failing... who writes them... who
doesn't
> understand them.. why they don't understand them etc...?
Great idea! I suspect that the number of false positives in a given repo's test suite is inversely related to the number of developers
on
the team actually writing tests, and the affordance by managers to
do
so. If you're not regularly writing tests, you're probably not
going
to feel comfortable troubleshooting and refactoring someone else's. If TDD isn't factored in to your team's velocity, you may feel like
the
investment in writing tests (or learning to write them) isn't worth it or comes at the risk of missing deadlines.
A survey could definitely help us to verify (or disprove) these relationships.
Some other questions I can think of:
- How valuable are unit tests to the health/quality of a software
project?
- How valuable are browser tests to the health/quality of a
software
project?
- How much experience do you have with TDD?
- Would you like more time to learn or practice TDD?
- How often do you write tests when developing a new feature?
- What kinds of test? (% of unit test vs. browser test)
- How often do you write tests to verify a bugfix?
- What kinds of test? (% of unit test vs. browser test)
- When would you typically write a unit test? (before
implementation, after implementation, when stuff breaks)
- When would you typically write a browser test? (during
conception,
before implementation, after implementation, when stuff breaks)
- What are the largest barriers to writing/running unit tests?
(test
framework, documentation/examples, execution time, CI, structure of my code, structure of code I depend on)
- What are the largest barriers to writing/running browser tests?
(test framework, documentation/examples, execution time, CI)
- What are the largest barriers to debugging test failure? (test
framework, confusing errors/stack traces, documentation/examples, debugging tools)
I'll create a Phab task to track it. :)
-- Dan Duvall Automation Engineer Wikimedia Foundation
-- Dan Duvall Automation Engineer Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Jon Robson
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
QA mailing list QA@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/qa