This looks like a good time to fork this conversation as this is a good problem to fix. How can we notify developers when they break other things in the stack and how? On 7 Mar 2014 05:36, "Bartosz Dziewoński" matma.rex@gmail.com wrote:

(continued, about the browser testing)

(tl;dr where are the tests and how do I know they fail?)

So, we have some slick browser tests. Awesome! But what's not good is that the tests run off-site, the results are not reported back to gerrit nor Bugzilla (unless someone manually files a bug, usually Chris) not IRC nor anywhere else, and are generally non-discoverable until someone shouts at you for breaking them. (As Tim guessed, I did not know about any failures until Jon told me.)

In fact, I still have no idea what exactly the tests encompass (I've heard about some browser tests for VE because I lurk a lot, never heard of any for core) or where to find them or how to run them. Either I'm slow or we have a serious documentation failure here.

Can something be done about it? Can we have the results reported somewhere visible – preferably to gerrit, as jenkins already reports some post-merge checks there? Or maybe we can have automatically filed bug reports if the build breaks? A bot reporting test status on #wikimedia-dev? Anything?

(I understand that the tests take too long to run them in the pre-merge checks.)

(Jon proposed reverting problematic changes outright, but to me that seems like a bit of an overreaction – bugs in tests and false positives happen, let's not make a huge fuss out of that.)

(to be continued: about the deployment)

---

Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l