Hey all,
I recently had a new repository created; and I wanted to create some jobs for it.
I dutifully created and had merged: https://gerrit.wikimedia.org/r/#/c/115968/ https://gerrit.wikimedia.org/r/#/c/115967/
Hashar told me I then needed to follow the instructions on [1] to push the jobs to jenkins. Running the script myself was only pain; it kept erroring out while trying to create the job. Marktraceur managed to create the jobs after much "kicking down the door" aka running the script multiple times.
It appears that the problem is that https://integration.mediawiki.org/ci/createItem?name=mwext-FundraisingChart-... to https://integration.mediawiki.org/?...
So that's a problem? We're still not sure why Mark was able to create the jobs with perseverance though.
Another problem that I'm seeing is in responsibilities -- supposedly only jenkins admins (wmf developers) can submit jobs (and then only when it works). And then, only people with root on galium can apply the Zuul configs. To me this is clearly not something the average developer is supposed to be doing.
Would it make sense to have QChris / ^demon create the standard jobs when they create the repository?
[1] https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Adding_a_Med...
~Matt Walker Wikimedia Foundation Fundraising Technology Team
On Thu, Feb 27, 2014 at 04:28:52PM -0800, Matthew Walker wrote:
Hashar told me I then needed to follow the instructions on [1] to push the jobs to jenkins. Running the script myself was only pain; it kept erroring out while trying to create the job. Marktraceur managed to create the jobs after much "kicking down the door" aka running the script multiple times.
Hilariously, the jobs still don't run.
I don't see the code getting checked out on Gallium, and the jobs are all marked "LOST" with no logs. I'm hopeful that this is an issue related to the repository still being empty, but this may be too-wishful thinking.
https://gerrit.wikimedia.org/r/116008
On Thu, Feb 27, 2014 at 4:36 PM, Mark Holmquist mtraceur@member.fsf.orgwrote:
On Thu, Feb 27, 2014 at 04:28:52PM -0800, Matthew Walker wrote:
Hashar told me I then needed to follow the instructions on [1] to push
the
jobs to jenkins. Running the script myself was only pain; it kept
erroring
out while trying to create the job. Marktraceur managed to create the
jobs
after much "kicking down the door" aka running the script multiple times.
Hilariously, the jobs still don't run.
I don't see the code getting checked out on Gallium, and the jobs are all marked "LOST" with no logs. I'm hopeful that this is an issue related to the repository still being empty, but this may be too-wishful thinking.
No such luck; we pushed the initial rough code into the repo; and I rebased the above on top of it -- the jobs still fail.
Le 28/02/2014 01:36, Mark Holmquist a écrit :
I don't see the code getting checked out on Gallium, and the jobs are all marked "LOST" with no logs. I'm hopeful that this is an issue related to the repository still being empty, but this may be too-wishful thinking.
A job reported as LOST is always because Zuul could not find the build result in Jenkins. This can be caused by various situations:
- Jenkins died and thus not reporting anything back
- The job is not registered in the Gearman bus (the server is integrated in Zuul and Jenkins is a client of it, Jenkins is supposed to register jobs to Zuul Gearman server)
In this case, the created jobs have not been properly registered. Which is because the job creation via Jenkins Job Builder did not work as expected. Although the job did get created, the Jenkins hook to register the job in Gearman did not trigger :-(
Le 28/02/2014 01:28, Matthew Walker a écrit :
Hey all,
I recently had a new repository created; and I wanted to create some jobs for it.
I dutifully created and had merged: https://gerrit.wikimedia.org/r/#/c/115968/ https://gerrit.wikimedia.org/r/#/c/115967/
Hashar told me I then needed to follow the instructions on [1] to push the jobs to jenkins. Running the script myself was only pain; it kept erroring out while trying to create the job. Marktraceur managed to create the jobs after much "kicking down the door" aka running the script multiple times.
It appears that the problem is that https://integration.mediawiki.org/ci/createItem?name=mwext-FundraisingChart-... to https://integration.mediawiki.org/?...
So that's a problem? We're still not sure why Mark was able to create the jobs with perseverance though.
<snip>
The proper URL is https://integration.wikimedia.org/ci/ , the integration.mediawiki.org redirects to the / (though it does not discard the query string which is a bug).
I have updated the wiki page, the jenkins_jobs.ini file should have:
[jenkins] url=https://integration.wikimedia.org/ci/ user=... password=... # actually an user API token
While deploying some job to day, I have been hit by the issue of the jobs being created but not registered in Gearman. When posting to the Jenkins API, it issue a redirect to a status page which is cached by the misc Varnish. So we need to send headers to prevent page caching :/
On Fri, Feb 28, 2014 at 04:58:51PM +0100, Antoine Musso wrote:
The proper URL is https://integration.wikimedia.org/ci/ , the integration.mediawiki.org redirects to the / (though it does not discard the query string which is a bug).
I have updated the wiki page, the jenkins_jobs.ini file should have:
[jenkins] url=https://integration.wikimedia.org/ci/ user=... password=... # actually an user API token
I have had this the entire time we were trying to create the jobs - it did not help, I still saw the issue.
Le 28/02/2014 18:39, Mark Holmquist a écrit :
I have had this the entire time we were trying to create the jobs - it did not help, I still saw the issue.
Got any trace to share? On job creation, a POST is sent which is then redirected to a GET which has been cached by varnish previously and says the job does not work. That cause Jenkins Job Builder to choke with an error saying the created job does not exist :-(
Workaround: disable Varnish caching entirely..
On Fri, Feb 28, 2014 at 06:43:04PM +0100, Antoine Musso wrote:
Got any trace to share?
marktraceur@midvalley-the-hornfreak:~/projects/wikimedia/integration/jenkins-job-builder$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ 'mwext-MultimediaViewer-do-something' INFO:root:Updating jobs in config/ (['mwext-MultimediaViewer-do-something']) INFO:jenkins_jobs.builder:Creating jenkins job mwext-MultimediaViewer-do-something https://integration.wikimedia.org/ci/createItem?name=mwext-MultimediaViewer-... Traceback (most recent call last): File "/usr/local/bin/jenkins-jobs", line 9, in <module> load_entry_point('jenkins-job-builder==0.0.584.07fa712', 'console_scripts', 'jenkins-jobs')() File "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/cmd.py", line 127, in main jobs = builder.update_job(options.path, options.names) File "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/builder.py", line 581, in update_job self.jenkins.update_job(job.name, job.output()) File "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/builder.py", line 476, in update_job self.jenkins.create_job(job_name, xml) File "/usr/local/lib/python2.7/dist-packages/python_jenkins-0.2.1-py2.7.egg/jenkins/__init__.py", line 400, in create_job raise JenkinsException('create[%s] failed' % (name)) jenkins.JenkinsException: create[mwext-MultimediaViewer-do-something] failed marktraceur@midvalley-the-hornfreak:~/projects/wikimedia/integration/jenkins-job-builder$
I just made a dummy job - commit here: https://gerrit.wikimedia.org/r/116123
Obviously nothing special, but the issue is in the HTTP request code anyway.
Cheers,
Yep; that's what I was saying above; the REST call to create the job 301 redirects back to integration.wikimedia.org/ when it should redirect to integration.wikimedia.org/ci.
~Matt Walker Wikimedia Foundation Fundraising Technology Team
On Fri, Feb 28, 2014 at 9:55 AM, Mark Holmquist mtraceur@member.fsf.orgwrote:
On Fri, Feb 28, 2014 at 06:43:04PM +0100, Antoine Musso wrote:
Got any trace to share?
marktraceur@midvalley-the-hornfreak:~/projects/wikimedia/integration/jenkins-job-builder$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ 'mwext-MultimediaViewer-do-something' INFO:root:Updating jobs in config/ (['mwext-MultimediaViewer-do-something']) INFO:jenkins_jobs.builder:Creating jenkins job mwext-MultimediaViewer-do-something
https://integration.wikimedia.org/ci/createItem?name=mwext-MultimediaViewer-... Traceback (most recent call last): File "/usr/local/bin/jenkins-jobs", line 9, in <module> load_entry_point('jenkins-job-builder==0.0.584.07fa712', 'console_scripts', 'jenkins-jobs')() File "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/cmd.py", line 127, in main jobs = builder.update_job(options.path, options.names) File "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/builder.py", line 581, in update_job self.jenkins.update_job(job.name, job.output()) File "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/builder.py", line 476, in update_job self.jenkins.create_job(job_name, xml) File "/usr/local/lib/python2.7/dist-packages/python_jenkins-0.2.1-py2.7.egg/jenkins/__init__.py", line 400, in create_job raise JenkinsException('create[%s] failed' % (name)) jenkins.JenkinsException: create[mwext-MultimediaViewer-do-something] failed marktraceur@midvalley-the-hornfreak :~/projects/wikimedia/integration/jenkins-job-builder$
I just made a dummy job - commit here: https://gerrit.wikimedia.org/r/116123
Obviously nothing special, but the issue is in the HTTP request code anyway.
Cheers,
-- Mark Holmquist Software Engineer, Multimedia Wikimedia Foundation mtraceur@member.fsf.org https://wikimediafoundation.org/wiki/User:MHolmquist
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux)
iQIbBAEBAgAGBQJTEM2sAAoJEEPl+wghkjzxctAP+MR4obFT1V2rhj0cCr2fs2sI /HqB4/3jT2berFZSrQROmS8BKKneNZa/15tvSE3CBOxna5dBAu5LPaR8oLSiD6G1 Ncd3zC5gt8jxsf7CGDi4Op9hUvZ4jFpq7YqunFO84wns9kEO6BTwGZrpR8pLDJ87 I6QyFPQmPE/oS9gib1q8c1VMSBgQOyonlWjk91uI1UkRpDO/v2QE3zteWmhTgfWj wp0XziQ4prbiwTHP2lXe+4GcOF0XFbTalqUbBrkbjPhqBeEEbBY4ER5lYsn+swS5 sd6yGKM4WlCgYIjIBZDa4amoAUoj9hu5DvS3LtYdcWptS5NmlWMYkp1utlnNjdpa dzekPueAsDr7DcBvQ26O1XnwJ80zC6czZGvNIVAhC92i1rIv2ufAWZeqPYsU29Xb VvRtplq5DC3iiblWqyTuinHOUPcU/tXNsH093NkSfwYilSyF8+AbOm4C8ncejOO/ FT0RbVjosHsBrxzVEMi7FBTBVqssj2dNuiloC//qu8GSaRD/V+B/Ma+SWOgfSjC5 vacRZvcU8m/lZ13cdTndEKSD6q2a9C2Ld2+iUd3GNayXEnughhi7VIbMFF3ogFR5 VQ1hKOpvAoGMEykm9tflVaDVsK+O4s/cj1zaDcxnUwaLOvYevojSNLgjZTNHPSZH LI11xv7omFaNz6caYvk= =z2n8 -----END PGP SIGNATURE-----
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Le 28/02/2014 18:55, Mark Holmquist a écrit :
marktraceur@midvalley-the-hornfreak:~/projects/wikimedia/integration/jenkins-job-builder$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ 'mwext-MultimediaViewer-do-something' INFO:root:Updating jobs in config/ (['mwext-MultimediaViewer-do-something']) INFO:jenkins_jobs.builder:Creating jenkins job mwext-MultimediaViewer-do-something https://integration.wikimedia.org/ci/createItem?name=mwext-MultimediaViewer-...
<snip stack trace>
jenkins.JenkinsException: create[mwext-MultimediaViewer-do-something] failed marktraceur@midvalley-the-hornfreak:~/projects/wikimedia/integration/jenkins-job-builder$
I just made a dummy job - commit here: https://gerrit.wikimedia.org/r/116123
Obviously nothing special, but the issue is in the HTTP request code anyway.
Hi,
Jenkins jobs builder use the python-jenkins module check for the existence of jobs using a simple GET request:
GET /ci/job/mwext-MultimediaViewer-do-something/api/json?tree=name
That throws a 404 and JJB then create the job:
POST /ci/createItem?name=mwext-MultimediaViewer-do-something
python-jenkins then verify the job got created using the GET request above. I found out tonight that our misc Varnish caches the 404 error for up to a minute, and hence the second GET is being served the cached 404 by Varnish. End result JJB consider the job hasn't been created and bails out with the above stack trace.
The way to fix it: stop caching 404 on the misc varnish, at least when using the gallium backend. There is a cache4xx parameters to the varnish::instance puppet class. Will have to checkout out with some Varnish guru how to best fix it.
Meanwhile I am entering sleep() mode..
Le 28/02/2014 23:10, Antoine Musso a écrit :
python-jenkins then verify the job got created using the GET request above. I found out tonight that our misc Varnish caches the 404 error for up to a minute, and hence the second GET is being served the cached 404 by Varnish. End result JJB consider the job hasn't been created and bails out with the above stack trace.
Hello,
This is now fixed. I made the Apache proxy in front of Jenkins to send Cache-Control: no-cache,must-revalidate headers following a suggestion by Mark Bergsma.
Related change is https://gerrit.wikimedia.org/r/116748
I confirmed jobs can now be created properly by JJB.
Thanks Matthew to have raised the issue!
Le 28/02/2014 01:28, Matthew Walker wrote:
Would it make sense to have QChris / ^demon create the standard jobs when they create the repository?
Hello,
That is a good idea. Moreover we could ensure Bugzilla has a component. We might want to automatize a lot of the workflow as well.
Two things that could help a bit is to run actions after a merge to the Zuul and Jenkins Jobs configuration repositories. Ie reload Zuul automatically and generate jobs on post merge.
cheers,
Hi,
On Thu, Feb 27, 2014 at 04:28:52PM -0800, Matthew Walker wrote:
[ git repository creation and jenkins ]
Would it make sense to have QChris / ^demon create the standard jobs when they create the repository?
Yes, have ^demon do it :-)
I know nothing about our job creation, but sounds like I should start toying around with it. I like the idea of automatically setting up CI for new extensions.
Have fun, Christian
wikitech-l@lists.wikimedia.org