[Labs-admin] ** PROBLEM alert - tools-exec-1433/Puppet run is CRITICAL **

[Bryan Davis]

On Wed, Apr 19, 2017 at 5:59 PM, Chase Pettet <cpettet at wikimedia.org> wrote:
> Well I found this on tools-exec-1433 to coincide with the Puppet run alert:
>
> Notice:
> /Stage[main]/Role::Labs::Nfsclient/Labstore::Nfs_mount[tools-project-on-labstore-secondary]/Exec[ensure-nfs-tools-project-on-labstore-secondary]/returns:
> mount.nfs: mount(2): Device or resource busy
> Notice:
> /Stage[main]/Role::Labs::Nfsclient/Labstore::Nfs_mount[tools-project-on-labstore-secondary]/Exec[ensure-nfs-tools-project-on-labstore-secondary]/returns:
> mount.nfs: /mnt/nfs/labstore-secondary-tools-project is busy or already
> mounted
> Notice:
> /Stage[main]/Role::Labs::Nfsclient/Labstore::Nfs_mount[tools-project-on-labstore-secondary]/Exec[ensure-nfs-tools-project-on-labstore-secondary]/returns:
> mount.nfs: trying text-based options
> 'vers=4,bg,intr,sec=sys,proto=tcp,port=0,lookupcache=all,nofsc,hard,addr=10.64.37.18,clientaddr=10.68.21.151'
> Error: /usr/local/sbin/nfs-mount-manager mount
> /mnt/nfs/labstore-secondary-tools-project returned 32 instead of one of [0]
> Error:
> /Stage[main]/Role::Labs::Nfsclient/Labstore::Nfs_mount[tools-project-on-labstore-secondary]/Exec[ensure-nfs-tools-project-on-labstore-secondary]/returns:
> change from notrun to 0 failed: /usr/local/sbin/nfs-mount-manager mount
> /mnt/nfs/labstore-secondary-tools-project returned 32 instead of one of [0]
> Notice: /Stage[main]/Role::Labs::Nfsclient/File[/data/project]: Dependency
> Exec[ensure-nfs-tools-project-on-labstore-secondary] has failures: true
> Warning: /Stage[main]/Role::Labs::Nfsclient/File[/data/project]: Skipping
> because of failed dependencies
>
>
> So I guess we can stop laying the blame on wsexport (or all of it anyway).
> I don't get this transient failure but something is up here still it seems.

So if I'm reading nfs-mount-manager correctly, it sounds like
`/usr/bin/timeout -k 10s 20s ls $2` is failing in the
`nfs-mount-manager check` for the Exec resource and then the Exec
tries to mount the export again? Would that mount command ever
actually work on an active host?

Bryan
-- 
Bryan Davis              Wikimedia Foundation    <bd808 at wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Cloud Services          Boise, ID USA
irc: bd808                                        v:415.839.6885 x6855