[Labs-l] [Wikimedia Labs][Announce] NFS (only labs projects) maintenance on 2017-01-18

[Madhumitha Viswanathan]

Update: NFS for labs projects is back to read write now.

There was an incident in the process of the migration that caused /home in
a significant number of labs instances (where /home was not on NFS) to be
replaced by an erroneous symlink. We are currently working on restoring all
the home directories - it looks like it may take at-least a couple hours.
I'll keep the list updated on progress.


On Wed, Jan 18, 2017 at 11:34 AM, Madhumitha Viswanathan <
mviswanathan at wikimedia.org> wrote:

> Update: this is still in progress. There's some ongoing issues with home
> directories on non-nfs /home across Labs (not tools or maps) - we are
> working on it, and will update soon. Do feel free to reach out
> #wikimedia-labs if you have any questions or concerns.
>
> On Wed, Jan 18, 2017 at 8:38 AM, Madhumitha Viswanathan <
> mviswanathan at wikimedia.org> wrote:
>
>> Reminder: This is starting soon, in ~22 minutes.
>>
>> On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan <
>> mviswanathan at wikimedia.org> wrote:
>>
>>> Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC).
>>>
>>> On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan <
>>> mviswanathan at wikimedia.org> wrote:
>>>
>>>> Hello,
>>>>
>>>> Continuing the storage redundancy and reliability efforts for Labs (
>>>> https://phabricator.wikimedia.org/T126083), the final migration to the
>>>> new NFS storage cluster for Labs projects with NFS enabled is upcoming. The
>>>> migration is planned to happen 2017-01-18 starting 09:00 PST(16:00
>>>> UTC). This *does not* affect tools, maps or any other projects that don't
>>>> have /home or /data/project mounted. The migration window is expected to be
>>>> fairly short (<3 hours) - but could last up to 6 hours.
>>>>
>>>> During the migration, no new data will be written to NFS (/home or
>>>> /data/project), but existing data will be accessible in Read-only mode for
>>>> the most part. Post migration, any services or jobs that were running on
>>>> top of NFS (/home or /data/project) will require manual restarts. Jobs
>>>> running on top of /scratch or /public/dumps will be unaffected. I will keep
>>>> the lists and #wikimedia-labs updated on progress during and after the
>>>> migration.
>>>>
>>>> The list of labs projects that will be affected in this migration are:
>>>>
>>>>
>>>>    - catgraph
>>>>
>>>>
>>>>    - account-creation-assistance
>>>>
>>>>
>>>>    - contributors
>>>>
>>>>
>>>>    - wikidata-topicmaps
>>>>
>>>>
>>>>    - sugarcrm
>>>>
>>>>
>>>>    - wikidumpparse
>>>>
>>>>
>>>>    - video
>>>>
>>>>
>>>>    - openstack
>>>>
>>>>
>>>>    - testlabs
>>>>
>>>>
>>>>    - wikidata-dev
>>>>
>>>>
>>>>    - quarry
>>>>
>>>>
>>>>    - huggle
>>>>
>>>>
>>>>    - editor-engagement
>>>>
>>>>
>>>>    - utrs
>>>>
>>>>
>>>>    - wmt
>>>>
>>>>
>>>>    - cvn
>>>>
>>>>
>>>>    - fastcci
>>>>
>>>>
>>>>    - toolsbeta
>>>>
>>>>
>>>>    - project-proxy
>>>>
>>>>
>>>>    - dumps
>>>>
>>>>
>>>>    - bots
>>>>
>>>>
>>>>    - snuggle
>>>>
>>>>
>>>>    - math
>>>>
>>>>
>>>>    - wikisource-tools
>>>>
>>>>
>>>> The tracking task on phabricator is here - https://phabricator.wikimedi
>>>> a.org/T154336. Let us know if you have any questions or concerns on
>>>> the list or on #wikimedia-labs.
>>>>
>>>> --
>>>> Madhu Viswanathan
>>>> Operations Engineer, Wikimedia Labs
>>>>
>>>
>>>
>>>
>>> --
>>> --Madhu :)
>>>
>>
>>
>>
>> --
>> --Madhu :)
>>
>
>
>
> --
> --Madhu :)
>



-- 
--Madhu :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20170118/03dcdada/attachment-0001.html>