On Tue, Nov 23, 2021 at 12:06 PM YiFei Zhu <zhuyifei1999@gmail.com> wrote:

On Wed, Nov 17, 2021 at 1:04 AM YiFei Zhu <zhuyifei1999@gmail.com> wrote:
> On Tue, Nov 16, 2021 at 6:38 PM Huji Lee <huji.huji@gmail.com> wrote:
> >
> > I went back and reactivated the line in .bash_profile which enabled zsh ("exec zsh" as the last line of .bash_profile)
> >
> > Then I submitted the job to the grid, using a command like this:
> >
> > jsub -N "n" -once -o ~/err/nightly.out -e ~/err/nightly.err ~/grid/jobs/nightly.sh
> >
> > I did it three ways. First, I used the nightly.sh file as is (see source). Second, I replaced "source" with "." and third I replaced "source" with "bash". In all three cases, it failed, without even producing an output or error. The nightly.out and nightly.err files were created of course, but were empty.
> >
> > Next, I added a "#!/bin/bash" shabang and ran it again all three ways. Result was the same.
> >
> > Running qstat many times shows that the job gets into a queued state ("qw") and after a few seconds, it goes into the run state ("r") and immediately stops.
> >
> > Removing the "exec zsh" command from .bash_profile will make things work again.
> >
> > Finally, I decided maybe the problem is that zsh is available for me, but not on the grid. So I change the .bash_profile ending from a single "exec zsh" command to this:
> >
> > if [ -f /usr/bin/zsh ]; then
> > zsh
> > fi
> >
> > Under this config, jobs on the grid worked, and when I used "become" to login as my tool, I ended with zsh. Obviously, I am happy with this workaround. But I am still curious as to the root cause.
> >
> > Is it really that zsh is not available on the grid, and the grid tries to replicate my environment first and reaches the "exec zsh" command and falls apart somehow?
> >
>
> This is consistent with what I described earlier:
>
> > Since you have "exec zsh" in your
> > .bash_profile, bash will run it as startup as a login shell, which in
> > theory would immediately replace itself with zsh with no arguments.
> > zsh will then see it has no arguments, attempts to read script from
> > stdin and get nothing, and immediately exit, stopping the job in grid.
>
> However, now that you have "zsh" instead of "exec zsh", the "replace"
> is not done. bash as the login shell executes zsh as a subshell, and
> zsh, having no inputs, immediately exits. The execution continues as
> if nothing had ever happened.
>
> I just tested the behavior of a how bash invokes .bash_profile by
> adding a sleep 60 to .bash_profile, and have my test.sh have a
> shebang, a a job is submitted for both with explicit 'bash' and
> without, and it looks like .bash_profile is executed in bath cases:
>
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> sgeadmin 762 0.4 0.1 111020 16056 ? Sl Mar25 1383:08
> /usr/lib/gridengine/sge_execd
> [...]
> sgeadmin 20388 0.0 0.1 51468 8540 ? S 07:57 0:00 \_
> /usr/lib/gridengine/sge_shepherd -bg
> tools.z+ 20390 0.0 0.0 23580 3196 ? Ss 07:57 0:00
> \_ -bash -c /data/project/zhuyifei1999-test/test.sh
> tools.z+ 20393 0.0 0.0 5796 672 ? S 07:57 0:00
> \_ sleep 60
>
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> sgeadmin 752 0.3 0.1 115112 16100 ? Sl Mar25 1313:16
> /usr/lib/gridengine/sge_execd
> [...]
> sgeadmin 8715 0.0 0.1 51468 8688 ? S 07:57 0:00 \_
> /usr/lib/gridengine/sge_shepherd -bg
> tools.z+ 8717 0.0 0.0 23580 3324 ? Ss 07:57 0:00
> \_ -bash -c /bin/bash /data/project/zhuyifei1999-test/test.sh
> tools.z+ 8720 0.0 0.0 5796 656 ? S 07:57 0:00
> \_ sleep 60
>
> It did take me by surprise that it's still bash that invokes the given
> command, because bash was not in the process tree for a usual "jsub
> [...] python script.sh". For example, a non-continuous job typically
> looks like this:
>
> sgeadmin 28386 0.0 0.1 51468 8588 ? S Nov15 0:00 \_
> /usr/lib/gridengine/sge_shepherd -bg
> tools.f+ 28388 7.2 3.5 427144 293024 ? Ss Nov15 210:55 |
> \_ /usr/bin/python pycore/pwb.py pycore/fawikibot/rade.py -newcat:10
>
> And a continuous one:
>
> sgeadmin 3699 0.0 0.0 51464 4540 ? S Apr19 0:00 \_
> /usr/lib/gridengine/sge_shepherd -bg
> tools.b+ 3701 0.0 0.0 4280 68 ? SNs Apr19 0:00 |
> \_ /bin/sh /var/spool/gridengine/execd/tools-sgeexec-0942/job_scripts/1302451
> tools.b+ 3702 0.2 2.8 505104 231092 ? SNl Apr19 674:45 |
> \_ /usr/bin/python bot2.py
>
> There is no `-bash -c "python script.sh"`
>
> However, if you trace what's going on, for a non-interactive bash that
> only receives a single command, it will directly execve that command:
>
> $ strace -e clone,execve bash -c '/bin/true'
> execve("/bin/bash", ["bash", "-c", "/bin/true"], [/* 26 vars */]) = 0
> execve("/bin/true", ["/bin/true"], [/* 25 vars */]) = 0
> +++ exited with 0 +++
>
> It does not involve child processes from the fork-exec model you'd
> expect. Therefore, we can say that no matter what you do with the job
> submission, a bash non-interactive login shell will be executed to run
> the command you specified to jsub. And the mess of "bash replace
> itself with zsh which immediately exits because stdin is empty" will
> apply.
>
> I think it is important to clarify that a shell like bash has 4 modes
> of execution, defined by whether it is an interactive shell, and
> whether it is a login shell. The details for the modes in the case of
> bash you can find in its man page [1]. But tl;dr:
>
> Login shells:
> - Upon startup, sources /etc/profile, then the first one among
> ~/.bash_profile, ~/.bash_login, and ~/.profile, that exists.
> - `bash -l` and `-bash` (note the dash sign at the front) makes bash a
> login shell
>
> Non-login shells:
> - If also interactive, upon startup, sources ~/.bashrc
>
> Interactive shells:
> - DIsplays a prompt for each command
>
> Non-interactive shells:
> - Upon startup, sources $BASH_ENV if it exists
> - As we saw above, if the command is given in the command string in -c
> and there is only one command, bash does not fork-exec the command but
> execs the command directly.
>
> So you might wonder why the separation of login shells (profile) vs
> non-login shells (rc). The reason is some environments are inherited
> by subshells while others are not. Environment variables are
> inherited:
>
> $ export FOO=bar
> $ echo $FOO
> bar
> $ bash
> $ echo $FOO
> bar
>
> While things like aliases are not:
>
> $ alias foo='echo bar'
> $ foo
> bar
> $ bash
> $ foo
> bash: foo: command not found
>
> There are environment setups that get inherited but you do not want it
> to be executed over and over by subshells. For example, appending to
> $PATH (`export PATH="$PATH:/path/to/bin"`). If it is in rc instead of
> profile, every time you run an interactive bash subshell PATH gets
> longer and more redundant; hence $PATH setups normally go to profile
> instead of rc. Non-inheritable setups like aliases go to rc. And the
> separation between .bash_profile and .profile is just so that you can
> have a .bash_profile that uses bash-specific syntax. I never needed
> any so I always use .profile.
>
> And to have bash login shells also get the initialization from rc,
> .profile usually has a header like this:
>
> # if running bash
> if [ -n "$BASH_VERSION" ]; then
> # include .bashrc if it exists
> if [ -f "$HOME/.bashrc" ]; then
> . "$HOME/.bashrc"
> fi
> fi
>
> And .bashrc:
>
> # Test for an interactive shell
> if [[ $- != *i* ]] ; then
> # Shell is non-interactive. Be done now!
> return
> fi
>
> I hope this makes sense. Let me know if not.
>
> Back to your question, let's see in what scenarios you would want to invoke zsh:
> - Non-interactive shells: No, you don't want `bash command.sh` randomly exec zsh
> - Interactive non-login shells: No, if you explicitly run `bash`, you
> want bash not zsh.
> - Interactive login shells. Yes, this is what `become tool` runs
> initially and you want bash here.
>
> Hence, to run in a login shell environment you'd want the .profile or
> .bash_profile. And interactive guard is simply [[ $- = *i* ]] in bash
> syntax, so what you want, expressed in code, is in .bash_profile:
>
> if [[ $- = *i* ]]; then
> exec zsh
> fi
>
> As a side note, yes zsh exists on the grid hosts:
>
> zhuyifei1999@tools-sgeexec-0901: ~$ ls -l {/usr,}/bin/zsh
> -rwxr-xr-x 1 root root 819744 Dec 1 2020 /bin/zsh
> lrwxrwxrwx 1 root root 8 Nov 22 2018 /usr/bin/zsh -> /bin/zsh
>
> [1] https://man7.org/linux/man-pages/man1/bash.1.html#INVOCATION
>
> YiFei Zhu

Have you had a chance to take a look at it yet?

YiFei Zhu
_______________________________________________
Cloud mailing list -- cloud@lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/