That is a sge scheduler problem.
I could not commend your sge ticket because jira does not accept my jira
token. The load limit is set ok because we use np_load_* values which is
the load divided by the number of cores on this host. So e.g. sge stop
scheduling jobs on nightshade if host load is more than 20. So i think
increasing this value does not make sense.
You output below contains load adjustments:
queue instance "longrun-lx(a)yarrow.toolserver.org" dropped because it
is overloaded: np_load_short=3.215000 (= 0.015000 + 0.8 * 16.000000 with
nproc=4) >= 3.1
means that there is a normalized host load of 0.015000 on yarrow and 16
jobs are started within the last 4,5 minutes (=load_adjustment_time).
sge temporary (for the first 4,5 minutes of a job lifetime) adds some
expected load for new jobs to be not overloaded in future. Most new jobs
normally needs some starting until they really use all need resources.
This prevents scheduling to much jobs at once to one execd client.
But as you can also see in real there are no new jobs. This is problem
the response from master:
$qping -info damiana 536 qmaster 1
05/17/2013 07:03:14:
SIRM version: 0.1
SIRM message id: 1
start time: 05/15/2013 23:47:49 (1368661669)
run time [s]: 112525
messages in read buffer: 0
messages in write buffer: 0
nr. of connected clients: 8
status: 1
info: MAIN: E (112524.48) | signaler000: E
(112523.98) | event_master000: E (0.27) | timer000: E (4.27) |
worker000: E (7.05) | worker001: E (8.93) | listener000: E (1.03) |
scheduler000: E (8.93) | listener001: E (5.03) | WARNING
All theads are in error state including the scheduler thread. So the
schedular does not accept status updates send by all execd and so it
does not know about finished jobs and load updates. Thats why you see on
qstat output an (not existing) overload problem and no running jobs
(although some old long running jobs are still running).
I think this could be solved by restarting the master scheduler process.
That is why i (as sge operator) send a kill command to the scheduler on
damiana and hoped that the ha_cluster automatically restarts this
process/service. But this is sadly not the case. So we have to wait
until a ts admin can restart this service manually.
In between submitting new jobs will return an error, sorry for that.
All running or queued jobs are not affected and will keep running or queued.
Merlissimo
Am 17.05.2013 03:41, schrieb Tim Landscheidt:
Hi,
a "qstat -j" of a simple job yields inter alia:
| scheduling info: queue instance
"longrun-sol(a)willow.toolserver.org" dropped because it is temporarily not
available
| queue instance "short-sol(a)willow.toolserver.org"
dropped because it is temporarily not available
| queue instance
"medium-lx(a)mayapple.toolserver.org" dropped because it is temporarily not
available
| queue instance
"longrun3-sol(a)willow.toolserver.org" dropped because it is temporarily not
available
| queue instance
"longrun2-sol(a)clematis.toolserver.org" dropped because it is disabled
| queue instance
"longrun2-sol(a)hawthorn.toolserver.org" dropped because it is disabled
| queue instance
"medium-sol(a)ortelius.toolserver.org" dropped because it is overloaded:
np_load_short=0.791601 (= 0.391601 + 0.8 * 2.000000 with nproc=4) >= 0.75
| queue instance "medium-lx(a)yarrow.toolserver.org"
dropped because it is overloaded: np_load_short=1.215000 (= 0.015000 + 0.8 * 6.000000 with
nproc=4) >= 1.2
| queue instance
"medium-lx(a)nightshade.toolserver.org" dropped because it is overloaded:
np_load_short=1.227500 (= 0.127500 + 0.8 * 11.000000 with nproc=8) >= 1.2
| queue instance
"medium-sol(a)wolfsbane.toolserver.org" dropped because it is overloaded:
np_load_short=0.778613 (= 0.078613 + 0.8 * 7.000000 with nproc=8) >= 0.75
| queue instance
"short-sol(a)wolfsbane.toolserver.org" dropped because it is overloaded:
np_load_short=1.278613 (= 0.078613 + 0.8 * 12.000000 with nproc=8) >= 1.2
| queue instance
"short-sol(a)ortelius.toolserver.org" dropped because it is overloaded:
np_load_short=1.391601 (= 0.391601 + 0.8 * 5.000000 with nproc=4) >= 1.2
| queue instance "longrun-lx(a)yarrow.toolserver.org"
dropped because it is overloaded: np_load_short=3.215000 (= 0.015000 + 0.8 * 16.000000
with nproc=4) >= 3.1
| queue instance
"longrun-lx(a)nightshade.toolserver.org" dropped because it is overloaded:
mem_free=-420765696.524288 (= 14098.726562M - 500M * 29.000000) <= 500
At the moment, we have /no/ jobs scheduled by SGE running.
Meanwhile, the hosts are idling:
| queuename qtype resv/used/tot. load_avg arch states
| ---------------------------------------------------------------------------------
| short-sol(a)ortelius.toolserver. B 0/0/8 1.52 sol-amd64
| ---------------------------------------------------------------------------------
| short-sol(a)willow.toolserver.or B 0/0/8 -NA- sol-amd64 au
| ---------------------------------------------------------------------------------
| short-sol(a)wolfsbane.toolserver B 0/0/12 0.64 sol-amd64
| ---------------------------------------------------------------------------------
| medium-lx(a)mayapple.toolserver. B 0/0/32 -NA- linux-x64 adu
| ---------------------------------------------------------------------------------
| medium-lx(a)nightshade.toolserve B 0/0/8 1.05 linux-x64
| ---------------------------------------------------------------------------------
| medium-lx(a)yarrow.toolserver.or B 0/0/8 0.02 linux-x64
| ---------------------------------------------------------------------------------
| longrun-lx(a)nightshade.toolserv BI 0/0/64 1.05 linux-x64
| ---------------------------------------------------------------------------------
| longrun-lx(a)yarrow.toolserver.o BI 0/0/64 0.02 linux-x64
| ---------------------------------------------------------------------------------
| longrun-sol(a)willow.toolserver. BI 0/0/64 -NA- sol-amd64 au
| ---------------------------------------------------------------------------------
| medium-sol(a)ortelius.toolserver B 0/0/4 1.52 sol-amd64
| ---------------------------------------------------------------------------------
| medium-sol(a)wolfsbane.toolserve B 0/0/4 0.64 sol-amd64
| ---------------------------------------------------------------------------------
| longrun2-sol(a)clematis.toolserv B 0/0/8 0.03 sol-amd64 d
| ---------------------------------------------------------------------------------
| longrun2-sol(a)hawthorn.toolserv B 0/0/8 0.23 sol-amd64 d
| ---------------------------------------------------------------------------------
| longrun3-sol(a)willow.toolserver B 0/0/4 -NA- sol-amd64 aduE
I filed
https://jira.toolserver.org/browse/TS-1650 on Monday
to no avail so far.
Tim
_______________________________________________
Toolserver-l mailing list (Toolserver-l(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette