Le 10/05/13 00:00, Antoine Musso a écrit :

Hello,

Jenkins crashed again today. The first time at 6am UTC, I got it fixed. And again between 9pm and 10pm UTC.

This has been a recurring event since we have upgraded our installation and the bug is: https://bugzilla.wikimedia.org/show_bug.cgi?id=48025

Tonight I got Jenkins access log enabled and made Zuul query jenkins directly instead of passing via SSL + an Apache frontend proxy. That will help a little bit.

The root cause is some weird issue in Jenkins where one of its thread will use 100% CPU. I have yet to determine what that thread is doing though nor what trigger the exact issue. Whenever I get some useful informations I will fill a bug upstream and make sure it get attention.

So I went to bed, and in the morning Jenkins was unsurprisingly stuck again. Enjoying coffee and croissant, my morning newspapers have been replaced by obscure web browsers windows titled: "how to read a java heap dump" "help reading a 2GB head dump" (trivia: you need a ton of memory) "java stack trace" "google: enable java debugging symbols" "Garbage Collection in the Java HotSpot Virtual Machine" [1]

All of that while breaking the #1 WMF rule: "do not work in pyjama".

I found out the Java Heap memory was full.

Also took time to look at a Jenkins notice that is warning about some mysterious old data format. After some reading, they are XML elements from the history build files which points to non existent entry points in Jenkins. That can happens when a plugin is removed.

When Jenkins parse the build history, it will record an in memory entry for each occurrences, with the thousands of builds we keep, that turns in a memory killer.

Jenkins offer the possibility to clean the, now invalid, elements for us but it is eventually terribly slow. I thus resurrected my sed skills and altered the XML file. That ran from 12:25am UTC till 17:19am UTC.

The invalid data gone, I hope Jenkins is not going to fill its memory again :-] I will monitor that tonight and on Monday then probably call it done.

I am really sorry for the multiple inconveniences since the upgrade on May 2nd and for the long time it took me to figure out the issue :(

Thanks Chad for the helpful tips regarding Java Heap memory size and thank you Timo for the Java Melody monitoring system.

The bug report: https://bugzilla.wikimedia.org/show_bug.cgi?id=48025#c19

[1] http://www.devx.com/Java/Article/21977 recommended reading