[Labs-l] Servers using role::labs::mediawiki_vagrant need manual intervention

[Andrew Bogott]

On 2/18/16 10:56 AM, Bryan Davis wrote:
> On Thu, Jan 21, 2016 at 3:48 PM, Bryan Davis <bd808 at wikimedia.org> wrote:
>> The Vagrant managed LXC container setup by
>> role::labs::mediawiki_vagrant [0] does not automatically restart when
>> the instance is rebooted. Since everything in Labs was rebooted today
>> to apply an urgent kernel patch [1] anyone who hasn't yet done so will
>> need to manually restart their Vagrant containers:
>>
>>    $ ssh my-instance.eqiad.wmflabs
>>    $ cd /srv/mediawiki-vagrant
>>    $ vagrant up
>>
>> Sorry for the inconvenience. If you have troubles with an instance
>> that was working before the reboot (or even one that wasn't), come to
>> #wikimedia-labs for help. Feel free to ping me there (bd808) if you
>> can't find others available to help. I may not get back to you
>> immediately but my bouncer will help me find your questions later if
>> I'm offline for lame things like food and sleep.
>>
>> [0]: https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Labs
>> [1]: https://lists.wikimedia.org/pipermail/labs-l/2016-January/004247.html
> /me performs thread necromancy
>
> Yuvi reminded me of this issue earlier this week. This morning Andrew
> merged a change [2] for me that adds an Upstart script that will check
> for an existing but stopped MediaWiki-Vagrant LXC container when the
> Labs instance boots. If one is found then it will attempt to boot it
> using the normal `vagrant up` command. This change should propagate to
> all hosts using role::labs::mediawiki_vagrant over the next 20 minutes
> or so. If you'd like to make sure that your instances have it, ssh in
> and run `sudo puppet agent --test --verbose` to force a Puppet run and
> look for the file /etc/init/mediawiki-vagrant.conf afterwards.
Thanks for writing this, Bryan.  At the rate that security exploits are 
appearing these days, bolstering instances against arbitrary reboots is 
something we should all be thinking about.

-A


>
> [2]: https://gerrit.wikimedia.org/r/#/c/271171/
>
> Bryan