Thanks to Željko, the next emails we receive for build status should be significantly more useful.
Also, we had an instance of a Jenkins build marked red where no tests failed. We have an open ticket with Cloudbees for this and we will be working with them to find out why. https://wmf.ci.cloudbees.com/job/MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox/275/