DeltaQuad wrote: It's about 9 p.m. on Monday evening right now for me. https://toolserver.org/~mzmcbride/watcher/ and other similar URLs were 404ing for me yesterday (Sunday) evening. And then they suddenly started working again without explanation. It seems to be an intermittent issue. Maybe it's related to the start of a new UTC day and load? Or maybe it's just an intermittent issue. Probably needs to be investigated if it continues to happen, though. MZMcBride

On Tue, 26 Feb 2013, Johannes Kroll wrote: At which time did you try about? Did you try the whole night? Or which time? And which databases seemed to answer slower? The problem is that the head nodes are doing SQL forwarding too. So if the active one is fishy you might not even have SQL connections. But the phenomenon should have occured between about 0:30 and 1:30 am UTC (1:30 and 2:30 CET). If you tried outside of this timeframe it would be good to know if you had any other errors and what they looked like. Cheers nosy

On Tue, 26 Feb 2013 05:19:32 -0600 legoktm <legoktm.wikipedia(a)gmail.com> wrote: Curious: now 'ps aux' on nightshade hangs after displaying some 30 or so processes. Some lines from strace when it was hanging: connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.24.1.18")}, 16) = 0 poll([{fd=6, events=POLLOUT}], 1, 0) = 1 ([{fd=6, revents=POLLOUT}]) sendto(6, "\3353\1\0\0\1\0\0\0\0\0\0\4ldap\3esi\ntoolserver"..., 41, MSG_NOSIGNAL, NULL, 0) = 41 poll([{fd=6, events=POLLIN|POLLOUT}], 1, 5000) = 1 ([{fd=6, revents=POLLOUT}]) sendto(6, "\23I\1\0\0\1\0\0\0\0\0\0\4ldap\3esi\ntoolserver"..., 41, MSG_NOSIGNAL, NULL, 0) = 41 poll([{fd=6, events=POLLIN}], 1, 4999) = 0 (Timeout) Port 53 is DNS? So it looks like some DNS query timed out? Now it seems to be working again. I didn't log the whole strace run, but I saved the lines that I still had in the terminal buffer... I can send it if anybody needs it.