Hello,
We often have the case of a change to an extension depending on a pending patch to MediaWiki core. I have upgraded our CI scheduler - Zuul - a couple weeks ago and it now supports marking dependencies even in different repositories.
Why does it matter? To make sure the dependency is fulfilled one usually either:
* CR-2 the patch until dependent change is merged * write a test that exercise the required patch in MediaWiki.
With the first solution (lack of test), once both are merged, nothing prevent one from cherry picking a patch without its dependent patch. For example for MediaWiki minor releases or Wikimedia deployment branches.
When a test covers the dependency, it will fail until the dependent one is merged which is rather annoying.
Zuul now recognizes the header 'Depends-On' in git messages, similar to 'Change-Id' and 'Bug'. 'Depends-On' takes as parameter a change-id and multiple ones can be added.
When a patch is proposed in Gerrit, Zuul looks for Gerrit changes matching the 'Depends-On' and verify whether any are still open. In such a case, it will craft references for the open patches so all the dependencies can be tested as if they got merged.
Real world example ------------------
The ContentTranslation extension is tested with the Wikidata one and was not passing the test. Wikidata created a patch and we did not want to merge it until we confirm the ContentTranslation one is passing properly.
The Wikidata patch is https://gerrit.wikimedia.org/r/#/c/252227/ Change-Id: I0312c23628d706deb507b5534b868480945b6163
On ContentTranslation we indicated the dependency: https://gerrit.wikimedia.org/r/#/c/252172/1..2//COMMIT_MSG + Depends-On: I0312c23628d706deb507b5534b868480945b6163
Which is the Wikidata patch.
Zuul: * received the patch for ContentTranslation * looked up the change-id and found the Wikidata * created git references in both repo to point to the proper patches
Jenkins: * zuul-cloner cloned both repos and fetched the references created by the Zuul service * run tests * SUCCESS
That confirmed us the Wikidata patch was actually fixing the issue for ContentTranslation. Hence we CR+2 both and all merged fine.
Please take a moment to read upstream documentation:
http://docs.openstack.org/infra/zuul/gating.html#cross-repository-dependenci...
Wikidata/ContentTranslation task: https://phabricator.wikimedia.org/T118263
That sounds really great and very useful! Thanks for this info :)
Best, Florian
-----Original-Nachricht----- Betreff: [Wikitech-l] CI and cross repository dependencies Datum: 2015-11-19T14:19:27+0100 Von: "Antoine Musso" <hashar+wmf@free.fr> An: "wikitech-l@lists.wikimedia.org" <wikitech-l@lists.wikimedia.org>
Hello,
We often have the case of a change to an extension depending on a pending patch to MediaWiki core. I have upgraded our CI scheduler - Zuul - a couple weeks ago and it now supports marking dependencies even in different repositories.
Why does it matter? To make sure the dependency is fulfilled one usually either:
* CR-2 the patch until dependent change is merged * write a test that exercise the required patch in MediaWiki.
With the first solution (lack of test), once both are merged, nothing prevent one from cherry picking a patch without its dependent patch. For example for MediaWiki minor releases or Wikimedia deployment branches.
When a test covers the dependency, it will fail until the dependent one is merged which is rather annoying.
Zuul now recognizes the header 'Depends-On' in git messages, similar to 'Change-Id' and 'Bug'. 'Depends-On' takes as parameter a change-id and multiple ones can be added.
When a patch is proposed in Gerrit, Zuul looks for Gerrit changes matching the 'Depends-On' and verify whether any are still open. In such a case, it will craft references for the open patches so all the dependencies can be tested as if they got merged.
Real world example ------------------
The ContentTranslation extension is tested with the Wikidata one and was not passing the test. Wikidata created a patch and we did not want to merge it until we confirm the ContentTranslation one is passing properly.
The Wikidata patch is https://gerrit.wikimedia.org/r/#/c/252227/ Change-Id: I0312c23628d706deb507b5534b868480945b6163
On ContentTranslation we indicated the dependency: https://gerrit.wikimedia.org/r/#/c/252172/1..2//COMMIT_MSG + Depends-On: I0312c23628d706deb507b5534b868480945b6163
Which is the Wikidata patch.
Zuul: * received the patch for ContentTranslation * looked up the change-id and found the Wikidata * created git references in both repo to point to the proper patches
Jenkins: * zuul-cloner cloned both repos and fetched the references created by the Zuul service * run tests * SUCCESS
That confirmed us the Wikidata patch was actually fixing the issue for ContentTranslation. Hence we CR+2 both and all merged fine.
Please take a moment to read upstream documentation:
http://docs.openstack.org/infra/zuul/gating.html#cross-repository-dependenci...
Wikidata/ContentTranslation task: https://phabricator.wikimedia.org/T118263
Wonderful, thank you, this should be very useful.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2015-11-19 15:19 GMT+02:00 Antoine Musso hashar+wmf@free.fr:
Hello,
We often have the case of a change to an extension depending on a pending patch to MediaWiki core. I have upgraded our CI scheduler - Zuul - a couple weeks ago and it now supports marking dependencies even in different repositories.
Why does it matter? To make sure the dependency is fulfilled one usually either:
- CR-2 the patch until dependent change is merged
- write a test that exercise the required patch in MediaWiki.
With the first solution (lack of test), once both are merged, nothing prevent one from cherry picking a patch without its dependent patch. For example for MediaWiki minor releases or Wikimedia deployment branches.
When a test covers the dependency, it will fail until the dependent one is merged which is rather annoying.
Zuul now recognizes the header 'Depends-On' in git messages, similar to 'Change-Id' and 'Bug'. 'Depends-On' takes as parameter a change-id and multiple ones can be added.
When a patch is proposed in Gerrit, Zuul looks for Gerrit changes matching the 'Depends-On' and verify whether any are still open. In such a case, it will craft references for the open patches so all the dependencies can be tested as if they got merged.
Real world example
The ContentTranslation extension is tested with the Wikidata one and was not passing the test. Wikidata created a patch and we did not want to merge it until we confirm the ContentTranslation one is passing properly.
The Wikidata patch is https://gerrit.wikimedia.org/r/#/c/252227/ Change-Id: I0312c23628d706deb507b5534b868480945b6163
On ContentTranslation we indicated the dependency: https://gerrit.wikimedia.org/r/#/c/252172/1..2//COMMIT_MSG
- Depends-On: I0312c23628d706deb507b5534b868480945b6163
Which is the Wikidata patch.
Zuul:
- received the patch for ContentTranslation
- looked up the change-id and found the Wikidata
- created git references in both repo to point to the proper patches
Jenkins:
- zuul-cloner cloned both repos and fetched the references created by
the Zuul service
- run tests
- SUCCESS
That confirmed us the Wikidata patch was actually fixing the issue for ContentTranslation. Hence we CR+2 both and all merged fine.
Please take a moment to read upstream documentation:
http://docs.openstack.org/infra/zuul/gating.html#cross-repository-dependenci...
Wikidata/ContentTranslation task: https://phabricator.wikimedia.org/T118263
-- Antoine "hashar" Musso
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
This is great! A huge thanks to everyone involved.
-Sam
On Thu, Nov 19, 2015 at 5:26 PM, Amir E. Aharoni < amir.aharoni@mail.huji.ac.il> wrote:
Wonderful, thank you, this should be very useful.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2015-11-19 15:19 GMT+02:00 Antoine Musso hashar+wmf@free.fr:
Hello,
We often have the case of a change to an extension depending on a pending patch to MediaWiki core. I have upgraded our CI scheduler - Zuul - a couple weeks ago and it now supports marking dependencies even in different repositories.
Why does it matter? To make sure the dependency is fulfilled one usually either:
- CR-2 the patch until dependent change is merged
- write a test that exercise the required patch in MediaWiki.
With the first solution (lack of test), once both are merged, nothing prevent one from cherry picking a patch without its dependent patch. For example for MediaWiki minor releases or Wikimedia deployment
branches.
When a test covers the dependency, it will fail until the dependent one is merged which is rather annoying.
Zuul now recognizes the header 'Depends-On' in git messages, similar to 'Change-Id' and 'Bug'. 'Depends-On' takes as parameter a change-id and multiple ones can be added.
When a patch is proposed in Gerrit, Zuul looks for Gerrit changes matching the 'Depends-On' and verify whether any are still open. In such a case, it will craft references for the open patches so all the dependencies can be tested as if they got merged.
Real world example
The ContentTranslation extension is tested with the Wikidata one and was not passing the test. Wikidata created a patch and we did not want to merge it until we confirm the ContentTranslation one is passing properly.
The Wikidata patch is https://gerrit.wikimedia.org/r/#/c/252227/ Change-Id: I0312c23628d706deb507b5534b868480945b6163
On ContentTranslation we indicated the dependency: https://gerrit.wikimedia.org/r/#/c/252172/1..2//COMMIT_MSG
- Depends-On: I0312c23628d706deb507b5534b868480945b6163
Which is the Wikidata patch.
Zuul:
- received the patch for ContentTranslation
- looked up the change-id and found the Wikidata
- created git references in both repo to point to the proper patches
Jenkins:
- zuul-cloner cloned both repos and fetched the references created by
the Zuul service
- run tests
- SUCCESS
That confirmed us the Wikidata patch was actually fixing the issue for ContentTranslation. Hence we CR+2 both and all merged fine.
Please take a moment to read upstream documentation:
http://docs.openstack.org/infra/zuul/gating.html#cross-repository-dependenci...
Wikidata/ContentTranslation task: https://phabricator.wikimedia.org/T118263
-- Antoine "hashar" Musso
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Nice work! Is this at all related to upstream/downstream Jenkins jobs?
On Thu, Nov 19, 2015 at 12:38 PM, Sam Smith samsmith@wikimedia.org wrote:
This is great! A huge thanks to everyone involved.
-Sam
On Thu, Nov 19, 2015 at 5:26 PM, Amir E. Aharoni < amir.aharoni@mail.huji.ac.il> wrote:
Wonderful, thank you, this should be very useful.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2015-11-19 15:19 GMT+02:00 Antoine Musso hashar+wmf@free.fr:
Hello,
We often have the case of a change to an extension depending on a pending patch to MediaWiki core. I have upgraded our CI scheduler - Zuul - a couple weeks ago and it now supports marking dependencies even in different repositories.
Why does it matter? To make sure the dependency is fulfilled one usually either:
- CR-2 the patch until dependent change is merged
- write a test that exercise the required patch in MediaWiki.
With the first solution (lack of test), once both are merged, nothing prevent one from cherry picking a patch without its dependent patch. For example for MediaWiki minor releases or Wikimedia deployment
branches.
When a test covers the dependency, it will fail until the dependent one is merged which is rather annoying.
Zuul now recognizes the header 'Depends-On' in git messages, similar to 'Change-Id' and 'Bug'. 'Depends-On' takes as parameter a change-id and multiple ones can be added.
When a patch is proposed in Gerrit, Zuul looks for Gerrit changes matching the 'Depends-On' and verify whether any are still open. In
such
a case, it will craft references for the open patches so all the dependencies can be tested as if they got merged.
Real world example
The ContentTranslation extension is tested with the Wikidata one and
was
not passing the test. Wikidata created a patch and we did not want to merge it until we confirm the ContentTranslation one is passing
properly.
The Wikidata patch is https://gerrit.wikimedia.org/r/#/c/252227/ Change-Id: I0312c23628d706deb507b5534b868480945b6163
On ContentTranslation we indicated the dependency: https://gerrit.wikimedia.org/r/#/c/252172/1..2//COMMIT_MSG
- Depends-On: I0312c23628d706deb507b5534b868480945b6163
Which is the Wikidata patch.
Zuul:
- received the patch for ContentTranslation
- looked up the change-id and found the Wikidata
- created git references in both repo to point to the proper patches
Jenkins:
- zuul-cloner cloned both repos and fetched the references created by
the Zuul service
- run tests
- SUCCESS
That confirmed us the Wikidata patch was actually fixing the issue for ContentTranslation. Hence we CR+2 both and all merged fine.
Please take a moment to read upstream documentation:
http://docs.openstack.org/infra/zuul/gating.html#cross-repository-dependenci...
Wikidata/ContentTranslation task: https://phabricator.wikimedia.org/T118263
-- Antoine "hashar" Musso
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Le 19/11/2015 18:51, Brian Gerstle a écrit :
Nice work!
Is this at all related to upstream/downstream Jenkins jobs?
The Zuul system does not rely at all on Jenkins upstream/downstream. One can think of Jenkins as a huge collections of independent shell scripts that are waiting to be executed with appropriate parameters.
OpenStack has a spec to get rid of Jenkins entirely and instead have Zuul create an Ansible play book to run on a machine. But really that is another topic.
To elaborate a bit more:
Gerrit does support dependencies between changes, but only in the same repository and branch. You can see that in the Gerrit web interface, and Gerrit will refuse to merge a change for which the parent is not merged yet.
Zuul does the same but independently from Gerrit. It is merely filling the gap of Gerrit lacks of cross repositories dependencies.
When a change is voted +2, it is enqueued in 'gate-and-submit'. Zuul immediately verify whether the dependencies are either merged or ahead in the queue, else it will reject the change and report back in Gerrit.
So if you have change A and change B depending on A. You +2 A then B and the queue is:
A <-- B (depend on A)
A is processed (no dependency) For B, Zuul find the dependency A ahead and thus it is processed.
If A fails the tests, B tests are automatically cancelled and the change dequeued. Zuul knows B depends on A.
Assuming all changes are merged by Zuul (via CR+2), Zuul dependency comes on top of Gerrit and nicely enforce dependencies.
Jenkins as a huge collections of independent shell scripts that are waiting to be executed with appropriate parameters.
Right that made me wonder... but then you said:
OpenStack has a spec to get rid of Jenkins entirely and instead have
Zuul create an Ansible play book to run on a machine. But really that is another topic.
Makes sense that we should drop Jenkins if we're not leveraging it's features.
Congrats again on contributing back to the OSS community!
On Thu, Nov 19, 2015 at 2:27 PM, Antoine Musso hashar+wmf@free.fr wrote:
Le 19/11/2015 18:51, Brian Gerstle a écrit :
Nice work!
Is this at all related to upstream/downstream Jenkins jobs?
The Zuul system does not rely at all on Jenkins upstream/downstream. One can think of Jenkins as a huge collections of independent shell scripts that are waiting to be executed with appropriate parameters.
OpenStack has a spec to get rid of Jenkins entirely and instead have Zuul create an Ansible play book to run on a machine. But really that is another topic.
To elaborate a bit more:
Gerrit does support dependencies between changes, but only in the same repository and branch. You can see that in the Gerrit web interface, and Gerrit will refuse to merge a change for which the parent is not merged yet.
Zuul does the same but independently from Gerrit. It is merely filling the gap of Gerrit lacks of cross repositories dependencies.
When a change is voted +2, it is enqueued in 'gate-and-submit'. Zuul immediately verify whether the dependencies are either merged or ahead in the queue, else it will reject the change and report back in Gerrit.
So if you have change A and change B depending on A. You +2 A then B and the queue is:
A <-- B (depend on A)
A is processed (no dependency) For B, Zuul find the dependency A ahead and thus it is processed.
If A fails the tests, B tests are automatically cancelled and the change dequeued. Zuul knows B depends on A.
Assuming all changes are merged by Zuul (via CR+2), Zuul dependency comes on top of Gerrit and nicely enforce dependencies.
-- Antoine "hashar" Musso
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
This is great. Just tried it out on https://gerrit.wikimedia.org/r/#/c/238992/ and hopefully it will soon be passing. This should hopefully help bubble up important dependent patches that might be overlooked if developers are not watching certain repositories and simply ignoring patches to Jenkins -2ing which can be for a variety of reasons :)
Now I just need to get out of the habit of writing "Dependency:" in favour of "Depends-On:" :)
On Thu, Nov 19, 2015 at 2:51 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
Jenkins as a huge collections of independent shell scripts that are waiting to be executed with appropriate parameters.
Right that made me wonder... but then you said:
OpenStack has a spec to get rid of Jenkins entirely and instead have
Zuul create an Ansible play book to run on a machine. But really that is another topic.
Makes sense that we should drop Jenkins if we're not leveraging it's features.
Congrats again on contributing back to the OSS community!
On Thu, Nov 19, 2015 at 2:27 PM, Antoine Musso hashar+wmf@free.fr wrote:
Le 19/11/2015 18:51, Brian Gerstle a écrit :
Nice work!
Is this at all related to upstream/downstream Jenkins jobs?
The Zuul system does not rely at all on Jenkins upstream/downstream. One can think of Jenkins as a huge collections of independent shell scripts that are waiting to be executed with appropriate parameters.
OpenStack has a spec to get rid of Jenkins entirely and instead have Zuul create an Ansible play book to run on a machine. But really that is another topic.
To elaborate a bit more:
Gerrit does support dependencies between changes, but only in the same repository and branch. You can see that in the Gerrit web interface, and Gerrit will refuse to merge a change for which the parent is not merged yet.
Zuul does the same but independently from Gerrit. It is merely filling the gap of Gerrit lacks of cross repositories dependencies.
When a change is voted +2, it is enqueued in 'gate-and-submit'. Zuul immediately verify whether the dependencies are either merged or ahead in the queue, else it will reject the change and report back in Gerrit.
So if you have change A and change B depending on A. You +2 A then B and the queue is:
A <-- B (depend on A)
A is processed (no dependency) For B, Zuul find the dependency A ahead and thus it is processed.
If A fails the tests, B tests are automatically cancelled and the change dequeued. Zuul knows B depends on A.
Assuming all changes are merged by Zuul (via CR+2), Zuul dependency comes on top of Gerrit and nicely enforce dependencies.
-- Antoine "hashar" Musso
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Thanks a ton, Antoine, for this! It has just made our lives sooo much easier for properly testing https://gerrit.wikimedia.org/r/#/c/254086/ :)
On 2 December 2015 at 18:44, Jon Robson jdlrobson@gmail.com wrote:
This is great. Just tried it out on https://gerrit.wikimedia.org/r/#/c/238992/ and hopefully it will soon be passing. This should hopefully help bubble up important dependent patches that might be overlooked if developers are not watching certain repositories and simply ignoring patches to Jenkins -2ing which can be for a variety of reasons :)
Now I just need to get out of the habit of writing "Dependency:" in favour of "Depends-On:" :)
True. My favourite line was:
Note: depends on <changeset-id>
In my case, I just need to drop "note:" and change the casing :P
Cheers, Marko
On Thu, Nov 19, 2015 at 2:51 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
Jenkins as a huge collections of independent shell scripts that are waiting to be executed with appropriate parameters.
Right that made me wonder... but then you said:
OpenStack has a spec to get rid of Jenkins entirely and instead have
Zuul create an Ansible play book to run on a machine. But really that is another topic.
Makes sense that we should drop Jenkins if we're not leveraging it's features.
Congrats again on contributing back to the OSS community!
On Thu, Nov 19, 2015 at 2:27 PM, Antoine Musso hashar+wmf@free.fr
wrote:
Le 19/11/2015 18:51, Brian Gerstle a écrit :
Nice work!
Is this at all related to upstream/downstream Jenkins jobs?
The Zuul system does not rely at all on Jenkins upstream/downstream. One can think of Jenkins as a huge collections of independent shell scripts that are waiting to be executed with appropriate parameters.
OpenStack has a spec to get rid of Jenkins entirely and instead have Zuul create an Ansible play book to run on a machine. But really that is another topic.
To elaborate a bit more:
Gerrit does support dependencies between changes, but only in the same repository and branch. You can see that in the Gerrit web interface, and Gerrit will refuse to merge a change for which the parent is not merged yet.
Zuul does the same but independently from Gerrit. It is merely filling the gap of Gerrit lacks of cross repositories dependencies.
When a change is voted +2, it is enqueued in 'gate-and-submit'. Zuul immediately verify whether the dependencies are either merged or ahead in the queue, else it will reject the change and report back in Gerrit.
So if you have change A and change B depending on A. You +2 A then B and the queue is:
A <-- B (depend on A)
A is processed (no dependency) For B, Zuul find the dependency A ahead and thus it is processed.
If A fails the tests, B tests are automatically cancelled and the change dequeued. Zuul knows B depends on A.
Assuming all changes are merged by Zuul (via CR+2), Zuul dependency comes on top of Gerrit and nicely enforce dependencies.
-- Antoine "hashar" Musso
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- Jon Robson
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
This doesn't seem to work for https://gerrit.wikimedia.org/r/#/c/238992/ Any ideas why? Is it because the Gather patch is on a different branch or because I'm not using the commit message right?
On Wed, Dec 2, 2015 at 2:36 PM, Marko Obrovac mobrovac@wikimedia.org wrote:
Thanks a ton, Antoine, for this! It has just made our lives sooo much easier for properly testing https://gerrit.wikimedia.org/r/#/c/254086/ :)
On 2 December 2015 at 18:44, Jon Robson jdlrobson@gmail.com wrote:
This is great. Just tried it out on https://gerrit.wikimedia.org/r/#/c/238992/ and hopefully it will soon be passing. This should hopefully help bubble up important dependent patches that might be overlooked if developers are not watching certain repositories and simply ignoring patches to Jenkins -2ing which can be for a variety of reasons :)
Now I just need to get out of the habit of writing "Dependency:" in favour of "Depends-On:" :)
True. My favourite line was:
Note: depends on <changeset-id>
In my case, I just need to drop "note:" and change the casing :P
Cheers, Marko
On Thu, Nov 19, 2015 at 2:51 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
Jenkins as a huge collections of independent shell scripts that are waiting to be executed with appropriate parameters.
Right that made me wonder... but then you said:
OpenStack has a spec to get rid of Jenkins entirely and instead have
Zuul create an Ansible play book to run on a machine. But really that is another topic.
Makes sense that we should drop Jenkins if we're not leveraging it's features.
Congrats again on contributing back to the OSS community!
On Thu, Nov 19, 2015 at 2:27 PM, Antoine Musso hashar+wmf@free.fr
wrote:
Le 19/11/2015 18:51, Brian Gerstle a écrit :
Nice work!
Is this at all related to upstream/downstream Jenkins jobs?
The Zuul system does not rely at all on Jenkins upstream/downstream. One can think of Jenkins as a huge collections of independent shell scripts that are waiting to be executed with appropriate parameters.
OpenStack has a spec to get rid of Jenkins entirely and instead have Zuul create an Ansible play book to run on a machine. But really that is another topic.
To elaborate a bit more:
Gerrit does support dependencies between changes, but only in the same repository and branch. You can see that in the Gerrit web interface, and Gerrit will refuse to merge a change for which the parent is not merged yet.
Zuul does the same but independently from Gerrit. It is merely filling the gap of Gerrit lacks of cross repositories dependencies.
When a change is voted +2, it is enqueued in 'gate-and-submit'. Zuul immediately verify whether the dependencies are either merged or ahead in the queue, else it will reject the change and report back in Gerrit.
So if you have change A and change B depending on A. You +2 A then B and the queue is:
A <-- B (depend on A)
A is processed (no dependency) For B, Zuul find the dependency A ahead and thus it is processed.
If A fails the tests, B tests are automatically cancelled and the change dequeued. Zuul knows B depends on A.
Assuming all changes are merged by Zuul (via CR+2), Zuul dependency comes on top of Gerrit and nicely enforce dependencies.
-- Antoine "hashar" Musso
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- Jon Robson
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- Marko Obrovac, PhD Senior Services Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Le 03/12/2015 02:17, Jon Robson a écrit :
This doesn't seem to work for https://gerrit.wikimedia.org/r/#/c/238992/ Any ideas why? Is it because the Gather patch is on a different branch or because I'm not using the commit message right?
Hello,
Indeed the patch from Gather dev branch is not found, because the job consider the branch to be 'master'.
Long read, one feel free to copy to the wiki!
context =======
https://gerrit.wikimedia.org/r/#/c/238992/ MobileFrontend patch against master
It relies on some Javascripts introduced in Gather dev branch by: https://gerrit.wikimedia.org/r/#/c/250826/
The job mediawiki-extensions-qunit on MobileFrontend fail. That job clones a subset of extensions deployed on Wikimedia cluster and run the whole QUnit test suite.
What Zuul does? ===============
Prepare patches ---------------
When the patch for MobileFrontend is received, Zuul detects it depends on the Gather change and thus build the dependency chain. It then ask for zuul-merger to craft merge commits for each patches against the tip of their branches and tag them. The output from the merge system is:
CreateZuulRef dev/Zfb21011d8e154089a0657b3a419d739e on Gather CreateZuulRef master/Zfb21011d8e154089a0657b3a419d739e on MobileFrontend
The refs are prefixed with the branch the patch has been merged against. Note the ideas are the same, that is the unique identifier for this specific build.
The references are exposed in git under refs/zuul/*
Retrieval of refs -----------------
Zuul triggers the builds with the parameters:
ZUUL_REF=refs/zuul/master/Zfb21011d8e154089a0657b3a419d739e ZUUL_BRANCH=master
'master' because that is the tip of MobileFrontend.
zuul-cloner takes care of cloning the repositories and checking out the patch / branch. The order of lookups are:
1) Zuul reference for the indicated branch 2) Zuul reference for the master branch 3) The tip of the indicated branch 4) The tip of the master branch
Ref: http://docs.openstack.org/infra/zuul/cloner.html#clone-order
Where "indicated branch" is ZUUL_BRANCH or 'master'.
When looking at the zuul-cloner output:
Creating repo mediawiki/extensions/Gather upstream repo has branch master does not have ref refs/zuul/master/Zfb21011d8e154089a0657b3a419d739e Falling back to branch master Checking out remotes/origin/master
The master/Z... ref is not found for Gather cause the patch was againt dev branch and the actual ref is dev/Z... So it fallback to master and your patch is not included.
Creating repo mediawiki/extensions/MobileFrontend from upstream upstream repo has branch master Fetched ref refs/zuul/master/Zfb21011d8e154089a0657b3a419d739e
What?? ------
Since MobileFrontend@master is tested with Gather@master, one would need the patch to dev to be cherry picked to master and that would work.
The same issue will occur for Gather@dev, other repositories will use the tip of master since dev branches do not exists.
You can override the branch either for all projects (--branch) or on a per project basis (--project-branch). So in theory, we could have a second job that properly fetch from dev, but the other job will still fail.
What do I do now? -----------------
Once your Gather patch in 'dev' is reviewed and merged, you can cherry pick against master (the Change-Id will be the same). Then 'recheck' the MobileFrontend patch which should be happy. You can +2 it, it is not going to merge till the change to Gather is merged in master.
Then, we can cut the wmf branches, deploy on the cluster with high confident and process with the next feature :-}
cheers,
Le 19/11/2015 18:38, Sam Smith a écrit :
This is great! A huge thanks to everyone involved.
-Sam
To be fair, I have not done much. The feature has been written by James E. Blair for the OpenStack Fundation.
On my side, I merely carefully reviewed six months worth of code and made sure the feature wasn't going to over load our Gerrit :-}
wikitech-l@lists.wikimedia.org