Hi,
Some of the jobs I submit to the grid take a long time (say, 30-60 minutes) and I would like to check on their status without having to log back into the labs.
I was hoping I could run shell_exec('qstat') in PHP and display its output on a web page. While shell_exec() works with other commands—e.g. echo shell_exec('whoami') correctly displays my tool's username on the webpage—for reasons I cannot explain, the output of shell_exec('qstat') is always blank. When I run qstat on the CLI at that same time, it does show me the familiar table output of ongoing jobs.
Any idea why that is the case? Has anyone already created a solution for fetching a list of active jobs from the grid and displaying it on a web-based status page for your tool?
Thanks, Huji
I’d suggest using the "-xml” argument, since the output will be easier to parse for web use, I think. That would also never have empty output so you could see exactly what you are getting back.
“Empty” output with the -xml argument looks like: <?xml version='1.0'?> <job_info xmlns:xsd="http://arc.liv.ac.uk/repos/darcs/sge/source/dist/util/resources/schemas/qsta... <queue_info> </queue_info> <job_info> </job_info> </job_info>
So maybe that will help.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org
On May 11, 2021, at 1:15 PM, Huji Lee huji.huji@gmail.com wrote:
Hi,
Some of the jobs I submit to the grid take a long time (say, 30-60 minutes) and I would like to check on their status without having to log back into the labs.
I was hoping I could run shell_exec('qstat') in PHP and display its output on a web page. While shell_exec() works with other commands—e.g. echo shell_exec('whoami') correctly displays my tool's username on the webpage—for reasons I cannot explain, the output of shell_exec('qstat') is always blank. When I run qstat on the CLI at that same time, it does show me the familiar table output of ongoing jobs.
Any idea why that is the case? Has anyone already created a solution for fetching a list of active jobs from the grid and displaying it on a web-based status page for your tool?
Thanks, Huji _______________________________________________ Wikimedia Cloud Services mailing list To unsubscribe send an email to cloud-leave@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
On Tue, May 11, 2021 at 2:16 PM Huji Lee huji.huji@gmail.com wrote:
Hi,
Some of the jobs I submit to the grid take a long time (say, 30-60 minutes) and I would like to check on their status without having to log back into the labs.
I was hoping I could run shell_exec('qstat') in PHP and display its output on a web page. While shell_exec() works with other commands—e.g. echo shell_exec('whoami') correctly displays my tool's username on the webpage—for reasons I cannot explain, the output of shell_exec('qstat') is always blank. When I run qstat on the CLI at that same time, it does show me the familiar table output of ongoing jobs.
Any idea why that is the case? Has anyone already created a solution for fetching a list of active jobs from the grid and displaying it on a web-based status page for your tool?
I've not done it for a single tool, but I did it for all tools: * https://sge-status.toolforge.org/ * https://sge-jobs.toolforge.org/
The source for both are linked from their page footers.
You can fetch the current grid job status as json too via https://sge-status.toolforge.org/api/v1/
Bryan
@Bryan: this is great! In fact, https://sge-jobs.toolforge.org/tool/huji completely serves my current needs. I added the links to the Grid page on the Wikitech wiki https://wikitech.wikimedia.org/w/index.php?title=Help%3AToolforge%2FGrid&type=revision&diff=1911745&oldid=1907595 .
@Brooke: thanks for the -xml idea. As it turns out, that is what Bryan's code uses too. Nevertheless, I cannot get it to work. The PHP code shown below returns a blank page at https://huji.toolforge.org/status.php
<?php $out = shell_exec('qstat -xml'); print_r($out);
In fact, I even went so far as using the exact command I found in Bryan's code which is /usr/bin/qstat -xml -j '*' | /bin/sed -e 's/JATASK:[^>]*/jatask/g' but even that results in a blank output on the web. On the command line, both of those return an XML as expected. Any ideas as to how that could be the case?
On Tue, May 11, 2021 at 6:18 PM Bryan Davis bd808@wikimedia.org wrote:
On Tue, May 11, 2021 at 2:16 PM Huji Lee huji.huji@gmail.com wrote:
Hi,
Some of the jobs I submit to the grid take a long time (say, 30-60
minutes) and I would like to check on their status without having to log back into the labs.
I was hoping I could run shell_exec('qstat') in PHP and display its
output on a web page. While shell_exec() works with other commands—e.g. echo shell_exec('whoami') correctly displays my tool's username on the webpage—for reasons I cannot explain, the output of shell_exec('qstat') is always blank. When I run qstat on the CLI at that same time, it does show me the familiar table output of ongoing jobs.
Any idea why that is the case? Has anyone already created a solution for
fetching a list of active jobs from the grid and displaying it on a web-based status page for your tool?
I've not done it for a single tool, but I did it for all tools:
The source for both are linked from their page footers.
You can fetch the current grid job status as json too via https://sge-status.toolforge.org/api/v1/
Bryan
Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808 _______________________________________________ Wikimedia Cloud Services mailing list To unsubscribe send an email to cloud-leave@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
On Tue, May 11, 2021 at 8:45 PM Huji Lee huji.huji@gmail.com wrote:
@Bryan: this is great! In fact, https://sge-jobs.toolforge.org/tool/huji completely serves my current needs. I added the links to the Grid page on the Wikitech wiki.
@Brooke: thanks for the -xml idea. As it turns out, that is what Bryan's code uses too. Nevertheless, I cannot get it to work. The PHP code shown below returns a blank page at https://huji.toolforge.org/status.php
<?php $out = shell_exec('qstat -xml'); print_r($out);
In fact, I even went so far as using the exact command I found in Bryan's code which is /usr/bin/qstat -xml -j '*' | /bin/sed -e 's/JATASK:[^>]*/jatask/g' but even that results in a blank output on the web. On the command line, both of those return an XML as expected. Any ideas as to how that could be the case?
There's no qstat command on k8s which your webservice is running from:
02:25:25 1 ✗ zhuyifei1999@tools-sgebastion-08: ~$ cat ~tools.huji/service.manifest # This file is used by Toolforge infrastructure. # Please do not edit manually at this time. # Mon Oct 12 19:49:38 2020 backend: kubernetes distribution: debian version: 4 web: php7.2 02:25:29 0 ✓ zhuyifei1999@tools-sgebastion-08: ~$ sudo -- sudo -Hu tools.huji kubectl get po NAME READY STATUS RESTARTS AGE huji-[...] 1/1 Running 0 74d 02:25:36 0 ✓ zhuyifei1999@tools-sgebastion-08: ~$ sudo -- sudo -Hu tools.huji kubectl exec -it huji-[..] /bin/bash tools.huji@huji-[...]:~$ qstat bash: qstat: command not found
You must run on the grid to talk to the grid.
YiFei Zhu
That explains it! Thanks!
On Tue, May 11, 2021 at 10:29 PM YiFei Zhu zhuyifei1999@gmail.com wrote:
On Tue, May 11, 2021 at 8:45 PM Huji Lee huji.huji@gmail.com wrote:
@Bryan: this is great! In fact, https://sge-jobs.toolforge.org/tool/huji
completely serves my current needs. I added the links to the Grid page on the Wikitech wiki.
@Brooke: thanks for the -xml idea. As it turns out, that is what Bryan's
code uses too. Nevertheless, I cannot get it to work. The PHP code shown below returns a blank page at https://huji.toolforge.org/status.php
<?php $out = shell_exec('qstat -xml'); print_r($out);
In fact, I even went so far as using the exact command I found in
Bryan's code which is /usr/bin/qstat -xml -j '*' | /bin/sed -e 's/JATASK:[^>]*/jatask/g' but even that results in a blank output on the web. On the command line, both of those return an XML as expected. Any ideas as to how that could be the case?
There's no qstat command on k8s which your webservice is running from:
02:25:25 1 ✗ zhuyifei1999@tools-sgebastion-08: ~$ cat ~tools.huji/service.manifest # This file is used by Toolforge infrastructure. # Please do not edit manually at this time. # Mon Oct 12 19:49:38 2020 backend: kubernetes distribution: debian version: 4 web: php7.2 02:25:29 0 ✓ zhuyifei1999@tools-sgebastion-08: ~$ sudo -- sudo -Hu tools.huji kubectl get po NAME READY STATUS RESTARTS AGE huji-[...] 1/1 Running 0 74d 02:25:36 0 ✓ zhuyifei1999@tools-sgebastion-08: ~$ sudo -- sudo -Hu tools.huji kubectl exec -it huji-[..] /bin/bash tools.huji@huji-[...]:~$ qstat bash: qstat: command not found
You must run on the grid to talk to the grid.
YiFei Zhu _______________________________________________ Wikimedia Cloud Services mailing list To unsubscribe send an email to cloud-leave@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s