Hello all,
Turning my frustration into something more constructive has resulted in a script to download & setup a local Jenkins install that mirrors the Wikimedia Jenkins install. This means it's now actually easy (or well, easier) to debug Jenkins issues & to develop new jobs without access to WMF servers.
mkjenkins [1] downloads Jenkins, the WMF Jenkins configuration (integration/jenkins.git, integration/jenkins-job-builder, integration/jenkins-job-builder-config) and installs the Jenkins job builder jobs.
In addition, it patches around two configuration issues: - jobs assume Jenkins is in /var/lib/jenkins. This is symlinked to $HOME/.jenkins - the git config assumes there are Zuul repositories in /var/lib/zuul/git. These are re-written to the on-line gerrit repositories.
Antoine suggested it might be interesting to make this into a Vagrant script, which would have the added advantage that it should be relatively easy to also include Zuul. However, my Vagrant-fu is weak and my hard drive full, so I'm not sure if I will have time to work on this anytime soon :-)
Best. Merlijn