[Labs-l] Reminder: Labs Homedir Calamity coming Monday (was Re: Moving away from NFS homedir instance on 12/17)

[Andrew Bogott]

Last warning:  This switchover will start in about an hour.

-Andrew


On 12/14/12 2:52 PM, Andrew Bogott wrote:
> Just a reminder that all labs homedirs will become read-only at some 
> point on Monday.  When everything is done I'll send an 'all clear' 
> message, at which point instances will need a reboot in order to 
> restore writes.
>
> -Andrew
>
>
>
> On 12/11/12 4:26 PM, Andrew Bogott wrote:
>> This change is back on the calendar! I'll be migrating homedirs from 
>> NFS to gluster on Monday the 17th.
>>
>> The good:  After this, home directories won't fill up so often.
>>
>> The bad:  Starting Monday (around 18:00UTC) homedirs will be 
>> read-only, and will remain so until an instance reboot.
>>
>> Read below for details.
>>
>> -Andrew
>>
>>
>> On 10/25/12 5:25 PM, Ryan Lane wrote:
>>> A few weeks ago I changed the /home mount to be a direct, rather than
>>> an indirect, automount, which means that all of /home is mounted
>>> rather than directories for each user. I did this in anticipation of
>>> switching to glustefs for homedirs, but specifically I did it in
>>> anticipation of enabling pam_mkhomedir. Today I enabled pam_mkhomedir.
>>> So, if you log into an instance and your home directory didn't already
>>> exist in the project, the instance will create it automatically.
>>>
>>> This is one step towards getting rid of the centralized NFS instance,
>>> which is really small, causes us performance issues, and often fills
>>> up, causing lots of annoying issues.
>>>
>>> Tuesday night (Oct 30) I'll switch the home directories over to
>>> glusterfs storage, rather than the centralized home directories. This
>>> isn't an easy switch. I'm going to take the following steps:
>>>
>>> 1. Change the NFS home directories to be read-only
>>> 2. Copy each project's data to its private gluster share
>>> 3. Change the autofs information in LDAP to point the /home mount to
>>> the private gluster share
>>>
>>> After doing so, instances will need to be rebooted to use the new
>>> mount. Until your instance is rebooted your home directories will be
>>> read-only. After all instances are switched off of NFS, I'm going to
>>> delete the NFS instance.
>>>
>>> - Ryan
>>>
>>> _______________________________________________
>>> Labs-l mailing list
>>> Labs-l at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>