[Labs-l] Finished: Labs Homedir Calamity

[Andrew Bogott]

I have now copied all labs homedirs into shiny, futuristic gluster volumes.

In order to start using the new /home setup, you will need to reboot 
your instances.  In the meantime, your home directory is read-only. 
That's going to cause you real trouble if you try to add new people to 
your project, so you'll want to arrange a reboot as soon as possible.

Project sysadmins can reboot from the commandline or using the 'reboot' 
link here: https://labsconsole.wikimedia.org/wiki/Special:NovaInstance 
.  I recommend you check in with your fellow project members before 
yanking out the rug.

If anyone notices irregularities after reboot (for example if you have 
no $HOME or your $HOME is suddenly empty where it was previously full of 
files) please notify me or Ryan immediately!

Thanks for your patience during this downtime.

-Anrew



On 12/14/12 2:52 PM, Andrew Bogott wrote:
>> Just a reminder that all labs homedirs will become read-only at some 
>> point on Monday.  When everything is done I'll send an 'all clear' 
>> message, at which point instances will need a reboot in order to 
>> restore writes.
>>
>> -Andrew
>>
>>
>>
>> On 12/11/12 4:26 PM, Andrew Bogott wrote:
>>> This change is back on the calendar! I'll be migrating homedirs from 
>>> NFS to gluster on Monday the 17th.
>>>
>>> The good:  After this, home directories won't fill up so often.
>>>
>>> The bad:  Starting Monday (around 18:00UTC) homedirs will be 
>>> read-only, and will remain so until an instance reboot.
>>>
>>> Read below for details.
>>>
>>> -Andrew
>>>
>>>
>>> On 10/25/12 5:25 PM, Ryan Lane wrote:
>>>> A few weeks ago I changed the /home mount to be a direct, rather than
>>>> an indirect, automount, which means that all of /home is mounted
>>>> rather than directories for each user. I did this in anticipation of
>>>> switching to glustefs for homedirs, but specifically I did it in
>>>> anticipation of enabling pam_mkhomedir. Today I enabled pam_mkhomedir.
>>>> So, if you log into an instance and your home directory didn't already
>>>> exist in the project, the instance will create it automatically.
>>>>
>>>> This is one step towards getting rid of the centralized NFS instance,
>>>> which is really small, causes us performance issues, and often fills
>>>> up, causing lots of annoying issues.
>>>>
>>>> Tuesday night (Oct 30) I'll switch the home directories over to
>>>> glusterfs storage, rather than the centralized home directories. This
>>>> isn't an easy switch. I'm going to take the following steps:
>>>>
>>>> 1. Change the NFS home directories to be read-only
>>>> 2. Copy each project's data to its private gluster share
>>>> 3. Change the autofs information in LDAP to point the /home mount to
>>>> the private gluster share
>>>>
>>>> After doing so, instances will need to be rebooted to use the new
>>>> mount. Until your instance is rebooted your home directories will be
>>>> read-only. After all instances are switched off of NFS, I'm going to
>>>> delete the NFS instance.
>>>>
>>>> - Ryan
>>>>
>>>> _______________________________________________
>>>> Labs-l mailing list
>>>> Labs-l at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>
>>
>