[QA] Zuul / jenkins is broken

[Antoine Musso]

Le 06/01/2015 02:55, Chris McMahon a écrit :
> 
> In Jenkins I clicked "prepare for shutdown", then cancelled the
> operation, hoping to unstick jenkins. I saw the beta-scap-eqiad job run
> after that. 
> 
> Then I disabled/enabled gearman
> 
> I made a trivial update to a gerrit patch but did not see zuul pick up
> the change. 
> 
> I seem to recall that at one point I had access to the "restart Jenkins"
> control in the Jenkins UI, but that no longer seems to be the case. 

Hello,

That was the proper course of action, unfortunately it would not solve
this issue.  The root cause is Zuul ended up being stalled completely
while reporting a change back to Gerrit. That is a blocking operation
with no timeout.

The next step in this case would have been to restart Zuul entirely.

I have documented the incident on wikitech:

https://wikitech.wikimedia.org/wiki/Incident_documentation/20150106-Zuul



-- 
Antoine "hashar" Musso