Adventures in creating new repos / jenkins jobs

List overview All Threads
Download

newer

older

Re: [Wikitech-l] Fw: Captcha Idea...

python bot/script with javascript...

Matthew Walker

27 Feb 2014 27 Feb '14

7:28 p.m.

Hey all, I recently had a new repository created; and I wanted to create some jobs for it. I dutifully created and had merged: https://gerrit.wikimedia.org/r/#/c/115968/ https://gerrit.wikimedia.org/r/#/c/115967/ Hashar told me I then needed to follow the instructions on [1] to push the jobs to jenkins. Running the script myself was only pain; it kept erroring out while trying to create the job. Marktraceur managed to create the jobs after much "kicking down the door" aka running the script multiple times. It appears that the problem is that https://integration.mediawiki.org/ci/createItem?name=mwext-FundraisingChart… to https://integration.mediawiki.org/?... So that's a problem? We're still not sure why Mark was able to create the jobs with perseverance though. Another problem that I'm seeing is in responsibilities -- supposedly only jenkins admins (wmf developers) can submit jobs (and then only when it works). And then, only people with root on galium can apply the Zuul configs. To me this is clearly not something the average developer is supposed to be doing. Would it make sense to have QChris / ^demon create the standard jobs when they create the repository? [1] https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Adding_a_Me… ~Matt Walker Wikimedia Foundation Fundraising Technology Team

Show replies by thread

Mark Holmquist

27 Feb 27 Feb

7:36 p.m.

On Thu, Feb 27, 2014 at 04:28:52PM -0800, Matthew Walker wrote:

...

Hashar told me I then needed to follow the instructions on [1] to push the jobs to jenkins. Running the script myself was only pain; it kept erroring out while trying to create the job. Marktraceur managed to create the jobs after much "kicking down the door" aka running the script multiple times.

Hilariously, the jobs still don't run. I don't see the code getting checked out on Gallium, and the jobs are all marked "LOST" with no logs. I'm hopeful that this is an issue related to the repository still being empty, but this may be too-wishful thinking. https://gerrit.wikimedia.org/r/116008 -- Mark Holmquist Software Engineer, Multimedia Wikimedia Foundation mtraceur(a)member.fsf.org https://wikimediafoundation.org/wiki/User:MHolmquist

Matthew Walker

8:56 p.m.

On Thu, Feb 27, 2014 at 4:36 PM, Mark Holmquist <mtraceur(a)member.fsf.org>wrote;wrote:

...

On Thu, Feb 27, 2014 at 04:28:52PM -0800, Matthew Walker wrote:

Hashar told me I then needed to follow the instructions on [1] to push

the

jobs to jenkins. Running the script myself was only pain; it kept

erroring

out while trying to create the job. Marktraceur managed to create the

jobs

after much "kicking down the door" aka running the script multiple times.

No such luck; we pushed the initial rough code into the repo; and I rebased the above on top of it -- the jobs still fail.

Antoine Musso

28 Feb 28 Feb

4:26 a.m.

Le 28/02/2014 01:36, Mark Holmquist a écrit :

...

I don't see the code getting checked out on Gallium, and the jobs are all marked "LOST" with no logs. I'm hopeful that this is an issue related to the repository still being empty, but this may be too-wishful thinking. https://gerrit.wikimedia.org/r/116008

A job reported as LOST is always because Zuul could not find the build result in Jenkins. This can be caused by various situations: - Jenkins died and thus not reporting anything back - The job is not registered in the Gearman bus (the server is integrated in Zuul and Jenkins is a client of it, Jenkins is supposed to register jobs to Zuul Gearman server) In this case, the created jobs have not been properly registered. Which is because the job creation via Jenkins Job Builder did not work as expected. Although the job did get created, the Jenkins hook to register the job in Gearman did not trigger :-( -- Antoine "hashar" Musso

Antoine Musso

10:58 a.m.

Le 28/02/2014 01:28, Matthew Walker a écrit :

...

<snip> The proper URL is https://integration.wikimedia.org/ci/ , the integration.mediawiki.org redirects to the / (though it does not discard the query string which is a bug). I have updated the wiki page, the jenkins_jobs.ini file should have: [jenkins] url=https://integration.wikimedia.org/ci/ user=... password=... # actually an user API token While deploying some job to day, I have been hit by the issue of the jobs being created but not registered in Gearman. When posting to the Jenkins API, it issue a redirect to a status page which is cached by the misc Varnish. So we need to send headers to prevent page caching :/ -- Antoine "hashar" Musso

Antoine Musso

11:02 a.m.

Le 28/02/2014 01:28, Matthew Walker wrote:

...

Would it make sense to have QChris / ^demon create the standard jobs when they create the repository?

Hello, That is a good idea. Moreover we could ensure Bugzilla has a component. We might want to automatize a lot of the workflow as well. Two things that could help a bit is to run actions after a merge to the Zuul and Jenkins Jobs configuration repositories. Ie reload Zuul automatically and generate jobs on post merge. cheers, -- Antoine "hashar" Musso

Mark Holmquist

12:39 p.m.

On Fri, Feb 28, 2014 at 04:58:51PM +0100, Antoine Musso wrote:

...

The proper URL is https://integration.wikimedia.org/ci/ , the integration.mediawiki.org redirects to the / (though it does not discard the query string which is a bug). I have updated the wiki page, the jenkins_jobs.ini file should have: [jenkins] url=https://integration.wikimedia.org/ci/ user=... password=... # actually an user API token

I have had this the entire time we were trying to create the jobs - it did not help, I still saw the issue. -- Mark Holmquist Software Engineer, Multimedia Wikimedia Foundation mtraceur(a)member.fsf.org https://wikimediafoundation.org/wiki/User:MHolmquist

Antoine Musso

12:43 p.m.

Le 28/02/2014 18:39, Mark Holmquist a écrit :

...

I have had this the entire time we were trying to create the jobs - it did not help, I still saw the issue.

Got any trace to share? On job creation, a POST is sent which is then redirected to a GET which has been cached by varnish previously and says the job does not work. That cause Jenkins Job Builder to choke with an error saying the created job does not exist :-( Workaround: disable Varnish caching entirely.. -- Antoine "hashar" Musso

Mark Holmquist

12:55 p.m.

On Fri, Feb 28, 2014 at 06:43:04PM +0100, Antoine Musso wrote:

...

Got any trace to share?

marktraceur@midvalley-the-hornfreak:~/projects/wikimedia/integration/jenkins-job-builder$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ 'mwext-MultimediaViewer-do-something' INFO:root:Updating jobs in config/ (['mwext-MultimediaViewer-do-something']) INFO:jenkins_jobs.builder:Creating jenkins job mwext-MultimediaViewer-do-something https://integration.wikimedia.org/ci/createItem?name=mwext-MultimediaViewer… Traceback (most recent call last): File "/usr/local/bin/jenkins-jobs", line 9, in <module> load_entry_point('jenkins-job-builder==0.0.584.07fa712', 'console_scripts', 'jenkins-jobs')() File "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/cmd.py", line 127, in main jobs = builder.update_job(options.path, options.names) File "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/builder.py", line 581, in update_job self.jenkins.update_job(job.name, job.output()) File "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/builder.py", line 476, in update_job self.jenkins.create_job(job_name, xml) File "/usr/local/lib/python2.7/dist-packages/python_jenkins-0.2.1-py2.7.egg/jenkins/__init__.py", line 400, in create_job raise JenkinsException('create[%s] failed' % (name)) jenkins.JenkinsException: create[mwext-MultimediaViewer-do-something] failed marktraceur@midvalley-the-hornfreak:~/projects/wikimedia/integration/jenkins-job-builder$ I just made a dummy job - commit here: https://gerrit.wikimedia.org/r/116123 Obviously nothing special, but the issue is in the HTTP request code anyway. Cheers, -- Mark Holmquist Software Engineer, Multimedia Wikimedia Foundation mtraceur(a)member.fsf.org https://wikimediafoundation.org/wiki/User:MHolmquist

Matthew Walker

2:07 p.m.

Yep; that's what I was saying above; the REST call to create the job 301 redirects back to integration.wikimedia.org/ when it should redirect to integration.wikimedia.org/ci. ~Matt Walker Wikimedia Foundation Fundraising Technology Team On Fri, Feb 28, 2014 at 9:55 AM, Mark Holmquist <mtraceur(a)member.fsf.org>wrote;wrote:

...

On Fri, Feb 28, 2014 at 06:43:04PM +0100, Antoine Musso wrote:

Got any trace to share?

marktraceur@midvalley-the-hornfreak:~/projects/wikimedia/integration/jenkins-job-builder$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ 'mwext-MultimediaViewer-do-something' INFO:root:Updating jobs in config/ (['mwext-MultimediaViewer-do-something']) INFO:jenkins_jobs.builder:Creating jenkins job mwext-MultimediaViewer-do-something https://integration.wikimedia.org/ci/createItem?name=mwext-MultimediaViewer… Traceback (most recent call last): File "/usr/local/bin/jenkins-jobs", line 9, in <module> load_entry_point('jenkins-job-builder==0.0.584.07fa712', 'console_scripts', 'jenkins-jobs')() File "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/cmd.py", line 127, in main jobs = builder.update_job(options.path, options.names) File "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/builder.py", line 581, in update_job self.jenkins.update_job(job.name, job.output()) File "/home/marktraceur/projects/wikimedia/integration/jenkins-job-builder/jenkins_jobs/builder.py", line 476, in update_job self.jenkins.create_job(job_name, xml) File "/usr/local/lib/python2.7/dist-packages/python_jenkins-0.2.1-py2.7.egg/jenkins/__init__.py", line 400, in create_job raise JenkinsException('create[%s] failed' % (name)) jenkins.JenkinsException: create[mwext-MultimediaViewer-do-something] failed marktraceur@midvalley-the-hornfreak :~/projects/wikimedia/integration/jenkins-job-builder$ I just made a dummy job - commit here: https://gerrit.wikimedia.org/r/116123 Obviously nothing special, but the issue is in the HTTP request code anyway. Cheers, -- Mark Holmquist Software Engineer, Multimedia Wikimedia Foundation mtraceur(a)member.fsf.org https://wikimediafoundation.org/wiki/User:MHolmquist

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iQIbBAEBAgAGBQJTEM2sAAoJEEPl+wghkjzxctAP+MR4obFT1V2rhj0cCr2fs2sI /HqB4/3jT2berFZSrQROmS8BKKneNZa/15tvSE3CBOxna5dBAu5LPaR8oLSiD6G1 Ncd3zC5gt8jxsf7CGDi4Op9hUvZ4jFpq7YqunFO84wns9kEO6BTwGZrpR8pLDJ87 I6QyFPQmPE/oS9gib1q8c1VMSBgQOyonlWjk91uI1UkRpDO/v2QE3zteWmhTgfWj wp0XziQ4prbiwTHP2lXe+4GcOF0XFbTalqUbBrkbjPhqBeEEbBY4ER5lYsn+swS5 sd6yGKM4WlCgYIjIBZDa4amoAUoj9hu5DvS3LtYdcWptS5NmlWMYkp1utlnNjdpa dzekPueAsDr7DcBvQ26O1XnwJ80zC6czZGvNIVAhC92i1rIv2ufAWZeqPYsU29Xb VvRtplq5DC3iiblWqyTuinHOUPcU/tXNsH093NkSfwYilSyF8+AbOm4C8ncejOO/ FT0RbVjosHsBrxzVEMi7FBTBVqssj2dNuiloC//qu8GSaRD/V+B/Ma+SWOgfSjC5 vacRZvcU8m/lZ13cdTndEKSD6q2a9C2Ld2+iUd3GNayXEnughhi7VIbMFF3ogFR5 VQ1hKOpvAoGMEykm9tflVaDVsK+O4s/cj1zaDcxnUwaLOvYevojSNLgjZTNHPSZH LI11xv7omFaNz6caYvk= =z2n8 -----END PGP SIGNATURE-----

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Antoine Musso

5:10 p.m.

Le 28/02/2014 18:55, Mark Holmquist a écrit :

...

...

jenkins.JenkinsException: create[mwext-MultimediaViewer-do-something] failed marktraceur@midvalley-the-hornfreak:~/projects/wikimedia/integration/jenkins-job-builder$ I just made a dummy job - commit here: https://gerrit.wikimedia.org/r/116123 Obviously nothing special, but the issue is in the HTTP request code anyway.

Hi, Jenkins jobs builder use the python-jenkins module check for the existence of jobs using a simple GET request: GET /ci/job/mwext-MultimediaViewer-do-something/api/json?tree=name That throws a 404 and JJB then create the job: POST /ci/createItem?name=mwext-MultimediaViewer-do-something python-jenkins then verify the job got created using the GET request above. I found out tonight that our misc Varnish caches the 404 error for up to a minute, and hence the second GET is being served the cached 404 by Varnish. End result JJB consider the job hasn't been created and bails out with the above stack trace. The way to fix it: stop caching 404 on the misc varnish, at least when using the gallium backend. There is a cache4xx parameters to the varnish::instance puppet class. Will have to checkout out with some Varnish guru how to best fix it. Meanwhile I am entering sleep() mode.. -- Antoine "hashar" Musso

Christian Aistleitner

4 Mar 4 Mar

4:49 a.m.

Hi, On Thu, Feb 27, 2014 at 04:28:52PM -0800, Matthew Walker wrote:

...

[ git repository creation and jenkins ] Would it make sense to have QChris / ^demon create the standard jobs when they create the repository?

Yes, have ^demon do it :-) I know nothing about our job creation, but sounds like I should start toying around with it. I like the idea of automatically setting up CI for new extensions. Have fun, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian(a)quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

Antoine Musso

5 Mar 5 Mar

4:53 a.m.

Le 28/02/2014 23:10, Antoine Musso a écrit :

...

python-jenkins then verify the job got created using the GET request above. I found out tonight that our misc Varnish caches the 404 error for up to a minute, and hence the second GET is being served the cached 404 by Varnish. End result JJB consider the job hasn't been created and bails out with the above stack trace.

Hello, This is now fixed. I made the Apache proxy in front of Jenkins to send Cache-Control: no-cache,must-revalidate headers following a suggestion by Mark Bergsma. Related change is https://gerrit.wikimedia.org/r/116748 I confirmed jobs can now be created properly by JJB. Thanks Matthew to have raised the issue! -- Antoine "hashar" Musso

3698

days inactive

3703

days old

wikitech-l@lists.wikimedia.org

Manage subscription

12 comments

4 participants

tags (0)

participants (4)

Antoine Musso
Christian Aistleitner
Mark Holmquist
Matthew Walker