[Labs-l] [Wikimedia Labs][Announce] NFS (only labs projects) maintenance on 2017-01-18

[Madhumitha Viswanathan]

Hi all,

Update - most of the home directories have been restored at this point -
we're down to the last few, that seem to have had a lot of data hence
taking quite some time.

These are the pending instances(currently being restored):

data.eqiad.wmflabs
ldfclient.eqiad.wmflabs
mathosphere.eqiad.wmflabs
mwoffliner4.eqiad.wmflabs
nomad.eqiad.wmflabs
reading.eqiad.wmflabs
recommendations.eqiad.wmflabs
captcha-apiproxy-03.eqiad.wmflabs
dashboardchat.eqiad.wmflabs
maps-cruncher.eqiad.wmflabs
pdf.eqiad.wmflabs
mwoffliner[2-3].eqiad.wmflabs
phlogiston-2.eqiad.wmflabs
pole.eqiad.wmflabs

Other than this, it should all look good. Do let us know if something seems
amiss here or on #wikimedia-labs. Thanks for being patient!


On Wed, Jan 18, 2017 at 3:45 PM, Madhumitha Viswanathan <
mviswanathan at wikimedia.org> wrote:

> Update: NFS for labs projects is back to read write now.
>
> There was an incident in the process of the migration that caused /home in
> a significant number of labs instances (where /home was not on NFS) to be
> replaced by an erroneous symlink. We are currently working on restoring all
> the home directories - it looks like it may take at-least a couple hours.
> I'll keep the list updated on progress.
>
>
> On Wed, Jan 18, 2017 at 11:34 AM, Madhumitha Viswanathan <
> mviswanathan at wikimedia.org> wrote:
>
>> Update: this is still in progress. There's some ongoing issues with home
>> directories on non-nfs /home across Labs (not tools or maps) - we are
>> working on it, and will update soon. Do feel free to reach out
>> #wikimedia-labs if you have any questions or concerns.
>>
>> On Wed, Jan 18, 2017 at 8:38 AM, Madhumitha Viswanathan <
>> mviswanathan at wikimedia.org> wrote:
>>
>>> Reminder: This is starting soon, in ~22 minutes.
>>>
>>> On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan <
>>> mviswanathan at wikimedia.org> wrote:
>>>
>>>> Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC).
>>>>
>>>> On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan <
>>>> mviswanathan at wikimedia.org> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Continuing the storage redundancy and reliability efforts for Labs (
>>>>> https://phabricator.wikimedia.org/T126083), the final migration to
>>>>> the new NFS storage cluster for Labs projects with NFS enabled is upcoming.
>>>>> The migration is planned to happen 2017-01-18 starting 09:00
>>>>> PST(16:00 UTC). This *does not* affect tools, maps or any other projects
>>>>> that don't have /home or /data/project mounted. The migration window is
>>>>> expected to be fairly short (<3 hours) - but could last up to 6 hours.
>>>>>
>>>>> During the migration, no new data will be written to NFS (/home or
>>>>> /data/project), but existing data will be accessible in Read-only mode for
>>>>> the most part. Post migration, any services or jobs that were running on
>>>>> top of NFS (/home or /data/project) will require manual restarts. Jobs
>>>>> running on top of /scratch or /public/dumps will be unaffected. I will keep
>>>>> the lists and #wikimedia-labs updated on progress during and after the
>>>>> migration.
>>>>>
>>>>> The list of labs projects that will be affected in this migration are:
>>>>>
>>>>>
>>>>>    - catgraph
>>>>>
>>>>>
>>>>>    - account-creation-assistance
>>>>>
>>>>>
>>>>>    - contributors
>>>>>
>>>>>
>>>>>    - wikidata-topicmaps
>>>>>
>>>>>
>>>>>    - sugarcrm
>>>>>
>>>>>
>>>>>    - wikidumpparse
>>>>>
>>>>>
>>>>>    - video
>>>>>
>>>>>
>>>>>    - openstack
>>>>>
>>>>>
>>>>>    - testlabs
>>>>>
>>>>>
>>>>>    - wikidata-dev
>>>>>
>>>>>
>>>>>    - quarry
>>>>>
>>>>>
>>>>>    - huggle
>>>>>
>>>>>
>>>>>    - editor-engagement
>>>>>
>>>>>
>>>>>    - utrs
>>>>>
>>>>>
>>>>>    - wmt
>>>>>
>>>>>
>>>>>    - cvn
>>>>>
>>>>>
>>>>>    - fastcci
>>>>>
>>>>>
>>>>>    - toolsbeta
>>>>>
>>>>>
>>>>>    - project-proxy
>>>>>
>>>>>
>>>>>    - dumps
>>>>>
>>>>>
>>>>>    - bots
>>>>>
>>>>>
>>>>>    - snuggle
>>>>>
>>>>>
>>>>>    - math
>>>>>
>>>>>
>>>>>    - wikisource-tools
>>>>>
>>>>>
>>>>> The tracking task on phabricator is here -
>>>>> https://phabricator.wikimedia.org/T154336. Let us know if you have
>>>>> any questions or concerns on the list or on #wikimedia-labs.
>>>>>
>>>>> --
>>>>> Madhu Viswanathan
>>>>> Operations Engineer, Wikimedia Labs
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --Madhu :)
>>>>
>>>
>>>
>>>
>>> --
>>> --Madhu :)
>>>
>>
>>
>>
>> --
>> --Madhu :)
>>
>
>
>
> --
> --Madhu :)
>



-- 
--Madhu :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20170118/de5a25b0/attachment-0001.html>