I conducted a little bit of research on visual regression testing, mainly for the mobile team for now, but it could be easily reused for other teams. I had a look at three existing solutions that seem somewhat popular and are actively developed:
* Wraith (https://github.com/BBC-News/wraith) * PhantomCSS (https://github.com/Huddle/PhantomCSS) * Huxley (https://github.com/facebook/huxley)
They all have their own pros and cons, but in my opinion they share one important disadvantage: they can't be easily integrated with our current browser testing setup. For all the aforementioned tools we would have to create from scratch a completely separate set of tests just for visual regression testing instead of extending our existing browser tests.
I spent a few hours in my spare time tinkering with an alternative idea which would enable us to add visual regression testing to our Cucumber/Watir tests. The result is a small prototype available at https://github.com/jgonera/photographer. There is no docs or anything yet, but I prepared a simple demo patch for MobileFrontend: https://gerrit.wikimedia.org/r/#/c/126878/.
The idea is to add a new method for Cucumber steps (snap) that takes a screenshot of the current browser state and compares it with a screenshot taken in one of the previous test runs. To update screenshots that are used as a reference you run tests with env var PHOTOGRAPHER=update. If newly taken screenshot differs by too many pixels from an old one, the test will fail.
It's still only an early prototype, but I'd appreciate any comments about this idea.
Great. Eager to see us test with this.
Do we have any of its early screenshot comparisons up?
--tomasz
On Fri, Apr 18, 2014 at 4:33 PM, Juliusz Gonera jgonera@wikimedia.org wrote:
I conducted a little bit of research on visual regression testing, mainly for the mobile team for now, but it could be easily reused for other teams. I had a look at three existing solutions that seem somewhat popular and are actively developed:
- Wraith (https://github.com/BBC-News/wraith)
- PhantomCSS (https://github.com/Huddle/PhantomCSS)
- Huxley (https://github.com/facebook/huxley)
They all have their own pros and cons, but in my opinion they share one important disadvantage: they can't be easily integrated with our current browser testing setup. For all the aforementioned tools we would have to create from scratch a completely separate set of tests just for visual regression testing instead of extending our existing browser tests.
I spent a few hours in my spare time tinkering with an alternative idea which would enable us to add visual regression testing to our Cucumber/Watir tests. The result is a small prototype available at https://github.com/jgonera/photographer. There is no docs or anything yet, but I prepared a simple demo patch for MobileFrontend: https://gerrit.wikimedia.org/r/#/c/126878/.
The idea is to add a new method for Cucumber steps (snap) that takes a screenshot of the current browser state and compares it with a screenshot taken in one of the previous test runs. To update screenshots that are used as a reference you run tests with env var PHOTOGRAPHER=update. If newly taken screenshot differs by too many pixels from an old one, the test will fail.
It's still only an early prototype, but I'd appreciate any comments about this idea.
-- Juliusz
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Nope, it's an early prototype. It basically only makes an updated Cucumber test fail if new screenshot differs from reference screenshot by more than X%. Adding visual comparison of two screenshots would be the next step and would not be too hard. I might work on it one evening this week.
On 04/21/2014 11:42 AM, Tomasz Finc wrote:
Great. Eager to see us test with this.
Do we have any of its early screenshot comparisons up?
--tomasz
On Fri, Apr 18, 2014 at 4:33 PM, Juliusz Gonera jgonera@wikimedia.org wrote:
I conducted a little bit of research on visual regression testing, mainly for the mobile team for now, but it could be easily reused for other teams. I had a look at three existing solutions that seem somewhat popular and are actively developed:
- Wraith (https://github.com/BBC-News/wraith)
- PhantomCSS (https://github.com/Huddle/PhantomCSS)
- Huxley (https://github.com/facebook/huxley)
They all have their own pros and cons, but in my opinion they share one important disadvantage: they can't be easily integrated with our current browser testing setup. For all the aforementioned tools we would have to create from scratch a completely separate set of tests just for visual regression testing instead of extending our existing browser tests.
I spent a few hours in my spare time tinkering with an alternative idea which would enable us to add visual regression testing to our Cucumber/Watir tests. The result is a small prototype available at https://github.com/jgonera/photographer. There is no docs or anything yet, but I prepared a simple demo patch for MobileFrontend: https://gerrit.wikimedia.org/r/#/c/126878/.
The idea is to add a new method for Cucumber steps (snap) that takes a screenshot of the current browser state and compares it with a screenshot taken in one of the previous test runs. To update screenshots that are used as a reference you run tests with env var PHOTOGRAPHER=update. If newly taken screenshot differs by too many pixels from an old one, the test will fail.
It's still only an early prototype, but I'd appreciate any comments about this idea.
-- Juliusz
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l