I'm trying to recreate a CI build to debug things locally. I've followed the instructions under https://doc.wikimedia.org/quibble/#reproducing-a-ci-build, but the command for listing Docker images doesn't work. There are no images with "quibble" in the name. Where can I find the available Quibble images?
*Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE)
E-post/*E-Mail*: sebastian.berlin@wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84
Hi, indeed the API request mentioned in the docs doesn't show up the quibble images.
The docker registry mentioned in the docs is this one here [1] so you can search in this page for the available versions of the quibble images. For example this one [2] is built using debian buster and php 8.1.
[1] https://docker-registry.wikimedia.org/ [2] https://docker-registry.wikimedia.org/releng/quibble-buster-php81/tags/
On Fri, Apr 19, 2024 at 10:43 AM Sebastian Berlin < sebastian.berlin@wikimedia.se> wrote:
I'm trying to recreate a CI build to debug things locally. I've followed the instructions under https://doc.wikimedia.org/quibble/#reproducing-a-ci-build, but the command for listing Docker images doesn't work. There are no images with "quibble" in the name. Where can I find the available Quibble images?
*Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE)
E-post/*E-Mail*: sebastian.berlin@wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84 _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Hi Sebastian,
I've also been playing around with debugging CI images on my local machine recently. I've pushed an update to the Quibble docs on Mediawiki [1]. Hopefully that's helpful - feedback and changes welcome!
Thanks,
Arthur
[1] https://www.mediawiki.org/wiki/Continuous_integration/Quibble
Am Fr., 19. Apr. 2024 um 10:56 Uhr schrieb Yiannis Giannelos < jgiannelos@wikimedia.org>:
Hi, indeed the API request mentioned in the docs doesn't show up the quibble images.
The docker registry mentioned in the docs is this one here [1] so you can search in this page for the available versions of the quibble images. For example this one [2] is built using debian buster and php 8.1.
[1] https://docker-registry.wikimedia.org/ [2] https://docker-registry.wikimedia.org/releng/quibble-buster-php81/tags/
On Fri, Apr 19, 2024 at 10:43 AM Sebastian Berlin < sebastian.berlin@wikimedia.se> wrote:
I'm trying to recreate a CI build to debug things locally. I've followed the instructions under https://doc.wikimedia.org/quibble/#reproducing-a-ci-build, but the command for listing Docker images doesn't work. There are no images with "quibble" in the name. Where can I find the available Quibble images?
*Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE)
E-post/*E-Mail*: sebastian.berlin@wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84 _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Thank you both 🙂
I'll have a look at the updated documentation.
*Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE)
E-post/*E-Mail*: sebastian.berlin@wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84
On Fri, 19 Apr 2024 at 11:03, Arthur Taylor arthur.taylor@wikimedia.de wrote:
Hi Sebastian,
I've also been playing around with debugging CI images on my local machine recently. I've pushed an update to the Quibble docs on Mediawiki [1]. Hopefully that's helpful - feedback and changes welcome!
Thanks,
Arthur
[1] https://www.mediawiki.org/wiki/Continuous_integration/Quibble
Am Fr., 19. Apr. 2024 um 10:56 Uhr schrieb Yiannis Giannelos < jgiannelos@wikimedia.org>:
Hi, indeed the API request mentioned in the docs doesn't show up the quibble images.
The docker registry mentioned in the docs is this one here [1] so you can search in this page for the available versions of the quibble images. For example this one [2] is built using debian buster and php 8.1.
[1] https://docker-registry.wikimedia.org/ [2] https://docker-registry.wikimedia.org/releng/quibble-buster-php81/tags/
On Fri, Apr 19, 2024 at 10:43 AM Sebastian Berlin < sebastian.berlin@wikimedia.se> wrote:
I'm trying to recreate a CI build to debug things locally. I've followed the instructions under https://doc.wikimedia.org/quibble/#reproducing-a-ci-build, but the command for listing Docker images doesn't work. There are no images with "quibble" in the name. Where can I find the available Quibble images?
*Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE)
E-post/*E-Mail*: sebastian.berlin@wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84 _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
-- Arthur Taylor Senior Software Engineer
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30-577 11 62-0 https://wikimedia.de
Keep up to date! Current news and exciting stories about Wikimedia, Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now https://www.wikimedia.de/newsletter/.
Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de
Wikimedia Deutschland — Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Charlottenburg, VR 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. Geschäftsführende Vorstände: Franziska Heine, Dr. Christian Humborg _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Le 19/04/2024 à 11:03, Arthur Taylor a écrit :
Hi Sebastian,
I've also been playing around with debugging CI images on my local machine recently. I've pushed an update to the Quibble docs on Mediawiki [1]. Hopefully that's helpful - feedback and changes welcome!
Thanks,
Arthur
[1] https://www.mediawiki.org/wiki/Continuous_integration/Quibble
Hello,
That documentation was merely an helper for people maintaining the CI stack linking to the releasing doc which is in the repository, how to rebuild the CI and then switch the jobs.
The Quibble manual / user documentation is inside the source repository under the doc/source directory https://gerrit.wikimedia.org/g/integration/quibble/+/refs/heads/master/doc/s... . It ends up being published at https://doc.wikimedia.org/quibble/ which is the canonical place.
The wiki edit at https://www.mediawiki.org/w/index.php?title=Continuous_integration/Quibble&a... https://www.mediawiki.org/w/index.php?title=Continuous_integration/Quibble&diff=prev&oldid=6476384 is amazing, and it would be really great to have it moved to the source repository. I like the Quabble.py script to pause before running tests, though one can also ask Quibble to run bash as a test command: `quibble -c bash`.
Antoine "hashar" Musso
Hi Antoine,
Thanks for the positive feedback! I'll make a PR to move my documentation edits over to the Quibble repo - thanks for the pointer there.
Happy you like the Quabble script - it's definitely not intended as a long-term solution for anything - I just wrote it as a quick way to get my local setup to do what I wanted. I would be happy to take a look at contributing to Quibble itself and making it a bit more friendly for debugging purposes.
I tried `quibble -c bash` this morning, or more completely I tried:
``` docker run --entrypoint=quibble-with-supervisord --tmpfs /workspace/db:size=320M --volume "$(pwd)"/src:/workspace/src --volume "$(pwd)"/cache:/cache --volume "$(pwd)"/log:/workspace/log --volume "$(pwd)"/ref:/srv/git:ro --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --env-file=env-Wikibase-Master-20240410 --init --rm docker-registry.wikimedia.org/releng/quibble-buster-php74:1.7.0-s1 --packages-source composer --db mysql --db-dir /workspace/db --git-parallel=8 -c bash ```
and didn't get a shell. The command just exited immediately:
``` ... INFO:quibble.commands:>>> Start: User commands: bash INFO:quibble.commands:User commands, working directory: /workspace/src INFO:quibble.commands:bash INFO:quibble.commands:<<< Finish: User commands: bash, in 0.006 s INFO:quibble.commands:Shutting down backends: <ExternalWebserver http://127.0.0.1:9413 /workspace/src> <Xvfb :94> <ChromeWebDriver :94> ... ```
so I'm not sure that addresses my wish to have an interactive debugging environment with the backends running. But maybe I'm doing something wrong there. Pointers welcome.
Will try and get to the documentation this week,
Many thanks,
Arthur
Am Fr., 19. Apr. 2024 um 14:50 Uhr schrieb Antoine Musso hashar@free.fr:
Le 19/04/2024 à 11:03, Arthur Taylor a écrit :
Hi Sebastian,
I've also been playing around with debugging CI images on my local machine recently. I've pushed an update to the Quibble docs on Mediawiki [1]. Hopefully that's helpful - feedback and changes welcome!
Thanks,
Arthur
[1] https://www.mediawiki.org/wiki/Continuous_integration/Quibble
Hello,
That documentation was merely an helper for people maintaining the CI stack linking to the releasing doc which is in the repository, how to rebuild the CI and then switch the jobs.
The Quibble manual / user documentation is inside the source repository under the doc/source directory
https://gerrit.wikimedia.org/g/integration/quibble/+/refs/heads/master/doc/s... . It ends up being published at https://doc.wikimedia.org/quibble/ which is the canonical place.
The wiki edit at
https://www.mediawiki.org/w/index.php?title=Continuous_integration/Quibble&a... < https://www.mediawiki.org/w/index.php?title=Continuous_integration/Quibble&a...
is amazing, and it would be really great to have it moved to the source repository. I like the Quabble.py script to pause before running tests, though one can also ask Quibble to run bash as a test command: `quibble -c bash`.
Antoine "hashar" Musso
Hi again,
My bad - `quibble -c bash` does launch a shell with the services running. I just needed to add `-it` to the docker invocation. That makes things much simpler - thank you!
Arthur
Am Mo., 22. Apr. 2024 um 09:03 Uhr schrieb Arthur Taylor < arthur.taylor@wikimedia.de>:
Hi Antoine,
Thanks for the positive feedback! I'll make a PR to move my documentation edits over to the Quibble repo - thanks for the pointer there.
Happy you like the Quabble script - it's definitely not intended as a long-term solution for anything - I just wrote it as a quick way to get my local setup to do what I wanted. I would be happy to take a look at contributing to Quibble itself and making it a bit more friendly for debugging purposes.
I tried `quibble -c bash` this morning, or more completely I tried:
docker run --entrypoint=quibble-with-supervisord --tmpfs /workspace/db:size=320M --volume "$(pwd)"/src:/workspace/src --volume "$(pwd)"/cache:/cache --volume "$(pwd)"/log:/workspace/log --volume "$(pwd)"/ref:/srv/git:ro --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --env-file=env-Wikibase-Master-20240410 --init --rm docker-registry.wikimedia.org/releng/quibble-buster-php74:1.7.0-s1 --packages-source composer --db mysql --db-dir /workspace/db --git-parallel=8 -c bash
and didn't get a shell. The command just exited immediately:
... INFO:quibble.commands:>>> Start: User commands: bash INFO:quibble.commands:User commands, working directory: /workspace/src INFO:quibble.commands:bash INFO:quibble.commands:<<< Finish: User commands: bash, in 0.006 s INFO:quibble.commands:Shutting down backends: <ExternalWebserver http://127.0.0.1:9413 /workspace/src> <Xvfb :94> <ChromeWebDriver :94> ...
so I'm not sure that addresses my wish to have an interactive debugging environment with the backends running. But maybe I'm doing something wrong there. Pointers welcome.
Will try and get to the documentation this week,
Many thanks,
Arthur
Am Fr., 19. Apr. 2024 um 14:50 Uhr schrieb Antoine Musso hashar@free.fr:
Le 19/04/2024 à 11:03, Arthur Taylor a écrit :
Hi Sebastian,
I've also been playing around with debugging CI images on my local machine recently. I've pushed an update to the Quibble docs on Mediawiki [1]. Hopefully that's helpful - feedback and changes welcome!
Thanks,
Arthur
[1] https://www.mediawiki.org/wiki/Continuous_integration/Quibble
Hello,
That documentation was merely an helper for people maintaining the CI stack linking to the releasing doc which is in the repository, how to rebuild the CI and then switch the jobs.
The Quibble manual / user documentation is inside the source repository under the doc/source directory
https://gerrit.wikimedia.org/g/integration/quibble/+/refs/heads/master/doc/s... . It ends up being published at https://doc.wikimedia.org/quibble/ which is the canonical place.
The wiki edit at
https://www.mediawiki.org/w/index.php?title=Continuous_integration/Quibble&a... < https://www.mediawiki.org/w/index.php?title=Continuous_integration/Quibble&a...
is amazing, and it would be really great to have it moved to the source repository. I like the Quabble.py script to pause before running tests, though one can also ask Quibble to run bash as a test command: `quibble -c bash`.
Antoine "hashar" Musso
-- Arthur Taylor Senior Software Engineer
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30-577 11 62-0 https://wikimedia.de
Keep up to date! Current news and exciting stories about Wikimedia, Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now https://www.wikimedia.de/newsletter/.
Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de
Wikimedia Deutschland — Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Charlottenburg, VR 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. Geschäftsführende Vorstände: Franziska Heine, Dr. Christian Humborg
Le 19/04/2024 à 10:42, Sebastian Berlin a écrit :
I'm trying to recreate a CI build to debug things locally. I've followed the instructions under https://doc.wikimedia.org/quibble/#reproducing-a-ci-build, but the command for listing Docker images doesn't work. There are no images with "quibble" in the name. Where can I find the available Quibble images?
Hello,
Quibble is not very user friendly. I started it during the 2017 Vienna hackathon as a quick hack to unify the complicated stack of repos, shell, python, Jenkins job into a single place. That has served to reduce the CI complexity and made the jobs quite simpler.
Overall I can't say it is very pleasant to use as a human, there are a few oddities such as:
- cleaning the dependencies and reinstalling everytime (use --skip-deps) - misnamed options such as --skip-zuul to prevent refetching and doing a git-clean of everything - one need to set ZUUL_PROJECT to trigger the logic that runs extension/skin tests And more :)
I have sent a few patches recently to slightly improve it such as:
- automatically deleting the autogenerated LocalSettings.php instead of having the MediaWiki installer to bail out after some minutes.
- Adding --change <gerrit change number> which retrieve from Gerrit the metadata of the given change and initialize the ZUUL_* environment variables (though I think one still has to set ZUUL_PROJECT to trigger the proper logic ...)
Back on topic, the documentation about reproducing a CI build deserves to be refreshed:
- the /v2/_catalog registry endpoint has evolved and is paginated nowadays, see https://distribution.github.io/distribution/spec/api/#listing-repositories or look at which image is currently defined in CI ( https://gerrit.wikimedia.org/g/integration/config/+/refs/heads/master/jjb/me... ).
Note that CI does a merge commit of the propose patch or patches against the tip of the target branch. That is done by processes named zuul-merger which exposes the resulting merges using git-daemon. Those repositories are not publicly available, though they should be reachable from a WMCS instance.
I am more than happy to assist and chat about Quibble. I will very warmly welcome any help on that front :)
Antoine "hashar" Musso
wikitech-l@lists.wikimedia.org