[Cloud] Re: Source command not working with zsh

On Thu, Dec 2, 2021 at 7:14 PM Huji Lee <huji.huji(a)gmail.com> wrote: Yep, things like terminals and shells are more complicated than they look ;) Sounds good to me YiFei Zhu > On Tue, Nov 23, 2021 at 12:06 PM YiFei Zhu <zhuyifei1999(a)gmail.com> wrote: >> >> On Wed, Nov 17, 2021 at 1:04 AM YiFei Zhu <zhuyifei1999(a)gmail.com> wrote: >> > On Tue, Nov 16, 2021 at 6:38 PM Huji Lee <huji.huji(a)gmail.com> wrote: >> > > >> > > I went back and reactivated the line in .bash_profile which enabled zsh ("exec zsh" as the last line of .bash_profile) >> > > >> > > Then I submitted the job to the grid, using a command like this: >> > > >> > > jsub -N "n" -once -o ~/err/nightly.out -e ~/err/nightly.err ~/grid/jobs/nightly.sh >> > > >> > > I did it three ways. First, I used the nightly.sh file as is (see source). Second, I replaced "source" with "." and third I replaced "source" with "bash". In all three cases, it failed, without even producing an output or error. The nightly.out and nightly.err files were created of course, but were empty. >> > > >> > > Next, I added a "#!/bin/bash" shabang and ran it again all three ways. Result was the same. >> > > >> > > Running qstat many times shows that the job gets into a queued state ("qw") and after a few seconds, it goes into the run state ("r") and immediately stops. >> > > >> > > Removing the "exec zsh" command from .bash_profile will make things work again. >> > > >> > > Finally, I decided maybe the problem is that zsh is available for me, but not on the grid. So I change the .bash_profile ending from a single "exec zsh" command to this: >> > > >> > > if [ -f /usr/bin/zsh ]; then >> > > zsh >> > > fi >> > > >> > > Under this config, jobs on the grid worked, and when I used "become" to login as my tool, I ended with zsh. Obviously, I am happy with this workaround. But I am still curious as to the root cause. >> > > >> > > Is it really that zsh is not available on the grid, and the grid tries to replicate my environment first and reaches the "exec zsh" command and falls apart somehow? >> > > >> > >> > This is consistent with what I described earlier: >> > >> > > Since you have "exec zsh" in your >> > > .bash_profile, bash will run it as startup as a login shell, which in >> > > theory would immediately replace itself with zsh with no arguments. >> > > zsh will then see it has no arguments, attempts to read script from >> > > stdin and get nothing, and immediately exit, stopping the job in grid. >> > >> > However, now that you have "zsh" instead of "exec zsh", the "replace" >> > is not done. bash as the login shell executes zsh as a subshell, and >> > zsh, having no inputs, immediately exits. The execution continues as >> > if nothing had ever happened. >> > >> > I just tested the behavior of a how bash invokes .bash_profile by >> > adding a sleep 60 to .bash_profile, and have my test.sh have a >> > shebang, a a job is submitted for both with explicit 'bash' and >> > without, and it looks like .bash_profile is executed in bath cases: >> > >> > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND >> > sgeadmin 762 0.4 0.1 111020 16056 ? Sl Mar25 1383:08 >> > /usr/lib/gridengine/sge_execd >> > [...] >> > sgeadmin 20388 0.0 0.1 51468 8540 ? S 07:57 0:00 \_ >> > /usr/lib/gridengine/sge_shepherd -bg >> > tools.z+ 20390 0.0 0.0 23580 3196 ? Ss 07:57 0:00 >> > \_ -bash -c /data/project/zhuyifei1999-test/test.sh >> > tools.z+ 20393 0.0 0.0 5796 672 ? S 07:57 0:00 >> > \_ sleep 60 >> > >> > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND >> > sgeadmin 752 0.3 0.1 115112 16100 ? Sl Mar25 1313:16 >> > /usr/lib/gridengine/sge_execd >> > [...] >> > sgeadmin 8715 0.0 0.1 51468 8688 ? S 07:57 0:00 \_ >> > /usr/lib/gridengine/sge_shepherd -bg >> > tools.z+ 8717 0.0 0.0 23580 3324 ? Ss 07:57 0:00 >> > \_ -bash -c /bin/bash /data/project/zhuyifei1999-test/test.sh >> > tools.z+ 8720 0.0 0.0 5796 656 ? S 07:57 0:00 >> > \_ sleep 60 >> > >> > It did take me by surprise that it's still bash that invokes the given >> > command, because bash was not in the process tree for a usual "jsub >> > [...] python script.sh". For example, a non-continuous job typically >> > looks like this: >> > >> > sgeadmin 28386 0.0 0.1 51468 8588 ? S Nov15 0:00 \_ >> > /usr/lib/gridengine/sge_shepherd -bg >> > tools.f+ 28388 7.2 3.5 427144 293024 ? Ss Nov15 210:55 | >> > \_ /usr/bin/python pycore/pwb.py pycore/fawikibot/rade.py -newcat:10 >> > >> > And a continuous one: >> > >> > sgeadmin 3699 0.0 0.0 51464 4540 ? S Apr19 0:00 \_ >> > /usr/lib/gridengine/sge_shepherd -bg >> > tools.b+ 3701 0.0 0.0 4280 68 ? SNs Apr19 0:00 | >> > \_ /bin/sh /var/spool/gridengine/execd/tools-sgeexec-0942/job_scripts/1302451 >> > tools.b+ 3702 0.2 2.8 505104 231092 ? SNl Apr19 674:45 | >> > \_ /usr/bin/python bot2.py >> > >> > There is no `-bash -c "python script.sh"` >> > >> > However, if you trace what's going on, for a non-interactive bash that >> > only receives a single command, it will directly execve that command: >> > >> > $ strace -e clone,execve bash -c '/bin/true' >> > execve("/bin/bash", ["bash", "-c", "/bin/true"], [/* 26 vars */]) = 0 >> > execve("/bin/true", ["/bin/true"], [/* 25 vars */]) = 0 >> > +++ exited with 0 +++ >> > >> > It does not involve child processes from the fork-exec model you'd >> > expect. Therefore, we can say that no matter what you do with the job >> > submission, a bash non-interactive login shell will be executed to run >> > the command you specified to jsub. And the mess of "bash replace >> > itself with zsh which immediately exits because stdin is empty" will >> > apply. >> > >> > I think it is important to clarify that a shell like bash has 4 modes >> > of execution, defined by whether it is an interactive shell, and >> > whether it is a login shell. The details for the modes in the case of >> > bash you can find in its man page [1]. But tl;dr: >> > >> > Login shells: >> > - Upon startup, sources /etc/profile, then the first one among >> > ~/.bash_profile, ~/.bash_login, and ~/.profile, that exists. >> > - `bash -l` and `-bash` (note the dash sign at the front) makes bash a >> > login shell >> > >> > Non-login shells: >> > - If also interactive, upon startup, sources ~/.bashrc >> > >> > Interactive shells: >> > - DIsplays a prompt for each command >> > >> > Non-interactive shells: >> > - Upon startup, sources $BASH_ENV if it exists >> > - As we saw above, if the command is given in the command string in -c >> > and there is only one command, bash does not fork-exec the command but >> > execs the command directly. >> > >> > So you might wonder why the separation of login shells (profile) vs >> > non-login shells (rc). The reason is some environments are inherited >> > by subshells while others are not. Environment variables are >> > inherited: >> > >> > $ export FOO=bar >> > $ echo $FOO >> > bar >> > $ bash >> > $ echo $FOO >> > bar >> > >> > While things like aliases are not: >> > >> > $ alias foo='echo bar' >> > $ foo >> > bar >> > $ bash >> > $ foo >> > bash: foo: command not found >> > >> > There are environment setups that get inherited but you do not want it >> > to be executed over and over by subshells. For example, appending to >> > $PATH (`export PATH="$PATH:/path/to/bin"`). If it is in rc instead of >> > profile, every time you run an interactive bash subshell PATH gets >> > longer and more redundant; hence $PATH setups normally go to profile >> > instead of rc. Non-inheritable setups like aliases go to rc. And the >> > separation between .bash_profile and .profile is just so that you can >> > have a .bash_profile that uses bash-specific syntax. I never needed >> > any so I always use .profile. >> > >> > And to have bash login shells also get the initialization from rc, >> > .profile usually has a header like this: >> > >> > # if running bash >> > if [ -n "$BASH_VERSION" ]; then >> > # include .bashrc if it exists >> > if [ -f "$HOME/.bashrc" ]; then >> > . "$HOME/.bashrc" >> > fi >> > fi >> > >> > And .bashrc: >> > >> > # Test for an interactive shell >> > if [[ $- != *i* ]] ; then >> > # Shell is non-interactive. Be done now! >> > return >> > fi >> > >> > I hope this makes sense. Let me know if not. >> > >> > Back to your question, let's see in what scenarios you would want to invoke zsh: >> > - Non-interactive shells: No, you don't want `bash command.sh` randomly exec zsh >> > - Interactive non-login shells: No, if you explicitly run `bash`, you >> > want bash not zsh. >> > - Interactive login shells. Yes, this is what `become tool` runs >> > initially and you want bash here. >> > >> > Hence, to run in a login shell environment you'd want the .profile or >> > .bash_profile. And interactive guard is simply [[ $- = *i* ]] in bash >> > syntax, so what you want, expressed in code, is in .bash_profile: >> > >> > if [[ $- = *i* ]]; then >> > exec zsh >> > fi >> > >> > As a side note, yes zsh exists on the grid hosts: >> > >> > zhuyifei1999@tools-sgeexec-0901: ~$ ls -l {/usr,}/bin/zsh >> > -rwxr-xr-x 1 root root 819744 Dec 1 2020 /bin/zsh >> > lrwxrwxrwx 1 root root 8 Nov 22 2018 /usr/bin/zsh -> /bin/zsh >> > >> > [1] https://man7.org/linux/man-pages/man1/bash.1.html#INVOCATION >> > >> > YiFei Zhu >> >> Have you had a chance to take a look at it yet? >> >> YiFei Zhu >> _______________________________________________ >> Cloud mailing list -- cloud(a)lists.wikimedia.org >> List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ > > _______________________________________________ > Cloud mailing list -- cloud(a)lists.wikimedia.org > List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/