Le 10/03/2014 08:51, Martijn Hoekstra a écrit :
If the test infrastructure can't handle running every test on every commit (why can't they by the way, and could that be remedied?) Would it be possible and desireable to make sure a more elaborate test suite with all available tests in run as a gate before cutting the production branch?
Which is more or less what I would like us to achieve. Ie on each change made to core: run the browser tests of the most used extensions we have, making sure they do not break. That is heavy, but we have a scalable infrastructure to do it.
The priority for now is the staging beta cluster on which we run browser tests. It already saved our a**** a bunch of time and are faster to run than having to run all tests on any changes.
I still have to digest this huge thread, might end up writing a summary of what is needed on the continuous integration front.
For the curious, we are using a gate developed by OpenStack which basically plug Gerrit and Jenkins together and support running tests across multiple repositories. Some high level doc is at their upstream site: http://ci.openstack.org/zuul/gating.html