[Labs-l] Recovering from role::labs::vagrant data loss caused by puppet today

[Bryan Davis]

I made a patch to migrate the MediaWiki-Vagrant clone used by
role::labs::vagrant from /mnt/vagrant to /srv/vagrant [0] that was
merged into operations/puppet.git today. Unfortunately, this patch has
a bug that leads to non-deterministic behavior. The outcome in the
crap-bryan-why-did-you-break-that case is that the contents of the old
/mnt/vagrant directory were copied to /srv/vagrant *before* the new
disk partition was mounted. This happened to at least 2 labs-vagrant
hosts that I know of. When it does happen, all is not lost!

== Recovering old /mnt/vagrant contents ==
# Becoming root just makes typing the rest of this easier
$ su -i

# Make a temporary mount point
$ mkdir /mnt/bd808-is-a-dork

# Re-mount the root disk at the temporary mount point
$ mount --bind / /mnt/bd808-is-a-dork

# Chdir to the shadowed /srv directory
$ cd /mnt/bd808-is-a-dork/srv

# Move the "new" copy of MediaWiki-Vagrant out of the way
$ mv /srv/vagrant /srv/vagrant-that-bd808-messed-up

# Copy the prior vagrant directory to /srv/vagrant
$ tar cf - vagrant | (cd /srv; tar xvf -)

# Re-provision via labs-vagrant
$ cd /vagrant
$ labs-vagrant provision

# Check to make sure things are ok again.
# Your instance may have had other content in /srv that you wish to restore.

# Clean up the mess.
$ rm -rf /mnt/bd808-is-a-dork/srv/vagrant
$ umount /mnt/bd808-is-a-dork
$ rm -rf /srv/vagrant-that-bd808-messed-up


[0]: https://gerrit.wikimedia.org/r/#/c/145974

Bryan
-- 
Bryan Davis              Wikimedia Foundation    <bd808 at wikimedia.org>
[[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
irc: bd808                                        v:415.839.6885 x6855