[Labs-l] Catscan2 out of action?

[Jeremy Baron]

(moved from commons-l)

On Mon, Sep 29, 2014 at 7:59 AM, Fæ <faewik at gmail.com> wrote:
> What's the story on Catscan2? It seems to have been out of action for
> a week, I'm starting to doubt it will come back anytime soon and I
> suspect that many of us have a routines reliant on it.
>
> http://tools.wmflabs.org/catscan2/quick_intersection.php

https://tools.wmflabs.org/?status told me it's running on
tools-webgrid-03. Which I noticed had a recent spike in load avg:

tools-webgrid-03:~$ uptime
 08:16:11 up 90 days, 16:22,  1 user,  load average: 41.74, 14.04, 7.33

but that seems to be a red herring. monitored CPU for that user's
processes as well as overall host CPU, load, vmstat during a page
load. doesn't seem to be either IO bound nor CPU bound. uses some CPU
then none then uses some again. In betweeen I guess must be waiting on
network?

not much else one can do without either root on the tools project or
access to that tool.

btw, graphs:

https://graphite.wmflabs.org/render?width=586&from=-2hours&until=now&height=310&target=tools.tools-webgrid-03.loadavg.01.value&target=tools.tools-webgrid-03.loadavg.05.value&target=tools.tools-webgrid-03.loadavg.15.value&target=tools.tools-webgrid-03.loadavg.processes_running.value

https://graphite.wmflabs.org/render/?width=586&height=310&target=tools.tools-webgrid-03.cpu.total.user.value&target=tools.tools-webgrid-03.cpu.total.system.value&target=tools.tools-webgrid-03.cpu.total.steal.value&target=tools.tools-webgrid-03.cpu.total.nice.value&target=tools.tools-webgrid-03.cpu.total.iowait.value

-Jeremy

P.S. what's up with all these gam_server instances?

oh, and now I see some iowait...

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0 3108660  87616 10156948    0    0     0     0 1174 2286
0  0 100  0
 0  0      0 3108660  87616 10156948    0    0     0    44 1179 2270
0  0 100  0
 0  0      0 3108660  87624 10156940    0    0     0    32 1241 2397  0  0 99  0
 8  0      0 3088800  87624 10165096    0    0     0     0 4307 4606 15  6 74  3
 8  0      0 3036196  87624 10170008    0    0     0     0 4564 5211 58  8 29  1
 3  0      0 3041648  87624 10170012    0    0     0     0 4867 6671 57  7 32  0
 0  1      0 3119492  87624 10170012    0    0     0     0 2245 3436 14  2 77  6
 0  1      0 3119244  87624 10170012    0    0     0    12 1355 2438  0  1 86 13
 0  1      0 3119244  87624 10170012    0    0     0     0 1256 2334  0  1 87 13
 0  1      0 3119244  87624 10170012    0    0     0     0 1220 2319  0  1 86 13
 0  1      0 3119244  87624 10170012    0    0     0     0 1229 2314  0  0 87 13
 0  1      0 3119244  87624 10170012    0    0     0     0 1231 2259  0  1 87 12
 0  1      0 3119368  87624 10170012    0    0     0     0 1223 2276  0  1 87 13
 0  1      0 3119368  87632 10170012    0    0     0    56 1263 2294  1  1 86 13
 0  1      0 3119368  87632 10170012    0    0     0     0 1248 2306  0  1 86 13
^C

earlier:
 2  1      0 3190744  87560 10146460    0    0     0     0 4026 6356 14  5 80  1
 2  1      0 3189904  87560 10146460    0    0     0     0 5241 6523 23  5 60 12
 3  0      0 3149648  87564 10146504    0    0     0   104 4581 5060 30  3 65  1
 1  0      0 3079196  87564 10144588    0    0     0     0 5485 6641 26  7 67  0
 2  0      0 3075548  87564 10144968    0    0     0     0 3774 6088  8  5 86  0