[Labs-announce] rolling outages and reboots on Wednesday, including tools-bastion-02

[Andrew Bogott]

This is now done.  Please let me know if you detect any lingering 
ill-effects.

-Andrew


On 3/22/17 8:29 AM, Andrew Bogott wrote:
> Reminder:  All of this will start happening in about 30 minutes.
>
> On 3/20/17 6:19 PM, Andrew Bogott wrote:
>> One of our virtualization hosts (labvirt1001) is exhibiting troubling 
>> behavior.  Nothing hosted on that box is currently affected, but to 
>> be on the safe side we're going to evacuate its VMs and run it 
>> through some tests.
>>
>> The evacuation process is a bit slow and haphazard -- while being 
>> transferred, any one VM may experience 30-60 minutes of downtime, and 
>> will be fully rebooted at the end.  The VMs that will be affected by 
>> this process are:
>>
>> tools-bastion-02 | tools
>> citoidtest       | services
>> clm-web-01       | community-labs-monitoring
>> db               | cognate
>> elastic1         | analytics
>> etherpad01       | etherpad
>> maps-tiles2      | maps
>> osmit-tre        | osmit
>> perf-testing     | mobile
>> puppet-mailman   | puppet
>> test-spm-1       | project-proxy
>> wikidata-mobile  | wikidata-dev
>> wmt-exec         | wmt
>>
>>
>> Note that tools-bastion-02 is on that list!  I will be moving that 
>> system first, at 14:00 UTC on Wednesday the 22nd (that's 7AM San 
>> Francisco time.)
>>
>> The other instances will be evacuated over the course of that day, in 
>> no particular order and at no particular time.  If you require a more 
>> specific migration window, please respond and let me know. 
>> Alternatively, if you totally don't care about downtime and are happy 
>> to have your instance migrated tomorrow instead of Wednesday, that 
>> would be good to know as well.
>>
>> Thank you, and sorry for the inconvenience.
>>
>> -Andrew + the Labs Team
>>
>