[Labs-l] Recovering from role::labs::vagrant data loss caused by puppet today

[Brion Vibber]

(actually I had to copy /srv/images too, since I had uploads on my
instance. now it's fixed :D)

-- brion


On Tue, Jul 15, 2014 at 2:43 PM, Brion Vibber <bvibber at wikimedia.org> wrote:

> Thanks, those directions fixed my instance!
>
> -- brion
>
>
> On Tue, Jul 15, 2014 at 2:21 PM, Bryan Davis <bd808 at wikimedia.org> wrote:
>
>> I made a patch to migrate the MediaWiki-Vagrant clone used by
>> role::labs::vagrant from /mnt/vagrant to /srv/vagrant [0] that was
>> merged into operations/puppet.git today. Unfortunately, this patch has
>> a bug that leads to non-deterministic behavior. The outcome in the
>> crap-bryan-why-did-you-break-that case is that the contents of the old
>> /mnt/vagrant directory were copied to /srv/vagrant *before* the new
>> disk partition was mounted. This happened to at least 2 labs-vagrant
>> hosts that I know of. When it does happen, all is not lost!
>>
>> == Recovering old /mnt/vagrant contents ==
>> # Becoming root just makes typing the rest of this easier
>> $ su -i
>>
>> # Make a temporary mount point
>> $ mkdir /mnt/bd808-is-a-dork
>>
>> # Re-mount the root disk at the temporary mount point
>> $ mount --bind / /mnt/bd808-is-a-dork
>>
>> # Chdir to the shadowed /srv directory
>> $ cd /mnt/bd808-is-a-dork/srv
>>
>> # Move the "new" copy of MediaWiki-Vagrant out of the way
>> $ mv /srv/vagrant /srv/vagrant-that-bd808-messed-up
>>
>> # Copy the prior vagrant directory to /srv/vagrant
>> $ tar cf - vagrant | (cd /srv; tar xvf -)
>>
>> # Re-provision via labs-vagrant
>> $ cd /vagrant
>> $ labs-vagrant provision
>>
>> # Check to make sure things are ok again.
>> # Your instance may have had other content in /srv that you wish to
>> restore.
>>
>> # Clean up the mess.
>> $ rm -rf /mnt/bd808-is-a-dork/srv/vagrant
>> $ umount /mnt/bd808-is-a-dork
>> $ rm -rf /srv/vagrant-that-bd808-messed-up
>>
>>
>> [0]: https://gerrit.wikimedia.org/r/#/c/145974
>>
>> Bryan
>> --
>> Bryan Davis              Wikimedia Foundation    <bd808 at wikimedia.org>
>> [[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
>> irc: bd808                                        v:415.839.6885 x6855
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/labs-l/attachments/20140715/602e8bad/attachment-0001.html>