<div dir="ltr">Well I found this on tools-exec-1433 to coincide with the Puppet run alert:<div><br></div><div><div>Notice: /Stage[main]/Role::Labs::Nfsclient/Labstore::Nfs_mount[tools-project-on-labstore-secondary]/Exec[ensure-nfs-tools-project-on-labstore-secondary]/returns: mount.nfs: mount(2): Device or resource busy</div><div>Notice: /Stage[main]/Role::Labs::Nfsclient/Labstore::Nfs_mount[tools-project-on-labstore-secondary]/Exec[ensure-nfs-tools-project-on-labstore-secondary]/returns: mount.nfs: /mnt/nfs/labstore-secondary-tools-project is busy or already mounted</div><div>Notice: /Stage[main]/Role::Labs::Nfsclient/Labstore::Nfs_mount[tools-project-on-labstore-secondary]/Exec[ensure-nfs-tools-project-on-labstore-secondary]/returns: mount.nfs: trying text-based options 'vers=4,bg,intr,sec=sys,proto=tcp,port=0,lookupcache=all,nofsc,hard,addr=10.64.37.18,clientaddr=10.68.21.151'</div><div>Error: /usr/local/sbin/nfs-mount-manager mount /mnt/nfs/labstore-secondary-tools-project returned 32 instead of one of [0]</div><div>Error: /Stage[main]/Role::Labs::Nfsclient/Labstore::Nfs_mount[tools-project-on-labstore-secondary]/Exec[ensure-nfs-tools-project-on-labstore-secondary]/returns: change from notrun to 0 failed: /usr/local/sbin/nfs-mount-manager mount /mnt/nfs/labstore-secondary-tools-project returned 32 instead of one of [0]</div><div>Notice: /Stage[main]/Role::Labs::Nfsclient/File[/data/project]: Dependency Exec[ensure-nfs-tools-project-on-labstore-secondary] has failures: true</div><div>Warning: /Stage[main]/Role::Labs::Nfsclient/File[/data/project]: Skipping because of failed dependencies</div></div><div><br></div><div><br></div><div>So I guess we can stop laying the blame on wsexport (or all of it anyway).  I don't get this transient failure but something is up here still it seems.</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 19, 2017 at 6:54 PM, shinken <span dir="ltr"><<a href="mailto:shinken@shinken-01.shinken.eqiad.wmflabs" target="_blank">shinken@shinken-01.shinken.eqiad.wmflabs</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Notification Type: PROBLEM<br>
<br>
Service: Puppet run<br>
Host: tools-exec-1433<br>
Address: 10.68.21.151<br>
State: CRITICAL<br>
<br>
Date/Time: Wed 19 Apr 23:54:16 UTC 2017<br>
<br>
Additional Info:<br>
<br>
CRITICAL: 60.00% of data above the critical threshold [0.0]</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Chase Pettet<br></div>Engineering Manager -- Labs<br></div><div>chasemp on <a href="https://phabricator.wikimedia.org/p/chasemp/" target="_blank">phabricator</a> and IRC<br></div></div></div></div></div></div>
</div></div>