[Labs-l] Reminder: Labs Homedir Calamity coming Monday (was Re: Moving away from NFS homedir instance on 12/17)

[Andrew Bogott]

Just a reminder that all labs homedirs will become read-only at some 
point on Monday.  When everything is done I'll send an 'all clear' 
message, at which point instances will need a reboot in order to restore 
writes.

-Andrew



On 12/11/12 4:26 PM, Andrew Bogott wrote:
> This change is back on the calendar! I'll be migrating homedirs from 
> NFS to gluster on Monday the 17th.
>
> The good:  After this, home directories won't fill up so often.
>
> The bad:  Starting Monday (around 18:00UTC) homedirs will be 
> read-only, and will remain so until an instance reboot.
>
> Read below for details.
>
> -Andrew
>
>
> On 10/25/12 5:25 PM, Ryan Lane wrote:
>> A few weeks ago I changed the /home mount to be a direct, rather than
>> an indirect, automount, which means that all of /home is mounted
>> rather than directories for each user. I did this in anticipation of
>> switching to glustefs for homedirs, but specifically I did it in
>> anticipation of enabling pam_mkhomedir. Today I enabled pam_mkhomedir.
>> So, if you log into an instance and your home directory didn't already
>> exist in the project, the instance will create it automatically.
>>
>> This is one step towards getting rid of the centralized NFS instance,
>> which is really small, causes us performance issues, and often fills
>> up, causing lots of annoying issues.
>>
>> Tuesday night (Oct 30) I'll switch the home directories over to
>> glusterfs storage, rather than the centralized home directories. This
>> isn't an easy switch. I'm going to take the following steps:
>>
>> 1. Change the NFS home directories to be read-only
>> 2. Copy each project's data to its private gluster share
>> 3. Change the autofs information in LDAP to point the /home mount to
>> the private gluster share
>>
>> After doing so, instances will need to be rebooted to use the new
>> mount. Until your instance is rebooted your home directories will be
>> read-only. After all instances are switched off of NFS, I'm going to
>> delete the NFS instance.
>>
>> - Ryan
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>