[Labs-admin] tools: main page (and possibly other tools) just died, then (after about half an hour) restarted itself (?)

[Chase Pettet]

I thought both admin and xtools ran on k8s?  Maybe some issue with new
webservice package?

Thanks Alex

On Nov 20, 2016 1:32 AM, "Alex Monk" <amonk at wikimedia.org> wrote:

> [06:33:00] <icinga-wm> PROBLEM - tools homepage -admin tool- on
> tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Not
> Available - 531 bytes in 0.021 second response time
> [06:34:03] <shinken-wm> PROBLEM - ToolLabs Home Page on toollabs is
> CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Not Available - string
> 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 531 bytes in 0.031
> second response time
>
> I started looking into this
> * Checked a couple of tools, other things e.g. GUC appear up (so didn't
> SMS any ops as I'm not sure the main page is that important)
> * Found it runs on the grid and tried `qmod -rj lighttpd-admin`
> * It appears up after this, but only briefly, then it's gone again
> * I try to figure out how to start it
> * Attempted 'webservice start', which looked OK, but 'webservice status'
> would always say 'Your webservice is not running'
> * ~07:13:24ish - it mysteriously appears online again
> * 07:16:52 - Matthew Bowker informs me that xTools was down too (no
> monitoring from shinken or icinga alerted IRC of this, but possibly
> connected) - he says the error from 'webservice restart' was
> https://www.irccloud.com/pastebin/w6AfLja7/
>
> I was looking at /data/project/.system/gridengine/spool/qmaster/messages
> while this was happening, I see quite a few 'host
> "tools-cron-01.tools.eqiad.wmflabs" is no admin host' errors in there
> though I have no reason to believe that's connected.
>
> _______________________________________________
> Labs-admin mailing list
> Labs-admin at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-admin
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-admin/attachments/20161120/56a59298/attachment.html>