[Labs-l] [Wikimedia Labs][Announce] NFS (only labs projects) maintenance on 2017-01-18

[Madhumitha Viswanathan]

Update: 2 instances are still being restored - taxonbot.dwl and
pole.wikidata-query, the rest on the previous list are done.

On Wed, Jan 18, 2017 at 11:20 PM, Madhumitha Viswanathan <
mviswanathan at wikimedia.org> wrote:

> Hi all,
>
> Update - most of the home directories have been restored at this point -
> we're down to the last few, that seem to have had a lot of data hence
> taking quite some time.
>
> These are the pending instances(currently being restored):
>
> data.eqiad.wmflabs
> ldfclient.eqiad.wmflabs
> mathosphere.eqiad.wmflabs
> mwoffliner4.eqiad.wmflabs
> nomad.eqiad.wmflabs
> reading.eqiad.wmflabs
> recommendations.eqiad.wmflabs
> captcha-apiproxy-03.eqiad.wmflabs
> dashboardchat.eqiad.wmflabs
> maps-cruncher.eqiad.wmflabs
> pdf.eqiad.wmflabs
> mwoffliner[2-3].eqiad.wmflabs
> phlogiston-2.eqiad.wmflabs
> pole.eqiad.wmflabs
>
> Other than this, it should all look good. Do let us know if something
> seems amiss here or on #wikimedia-labs. Thanks for being patient!
>
>
> On Wed, Jan 18, 2017 at 3:45 PM, Madhumitha Viswanathan <
> mviswanathan at wikimedia.org> wrote:
>
>> Update: NFS for labs projects is back to read write now.
>>
>> There was an incident in the process of the migration that caused /home
>> in a significant number of labs instances (where /home was not on NFS) to
>> be replaced by an erroneous symlink. We are currently working on restoring
>> all the home directories - it looks like it may take at-least a couple
>> hours. I'll keep the list updated on progress.
>>
>>
>> On Wed, Jan 18, 2017 at 11:34 AM, Madhumitha Viswanathan <
>> mviswanathan at wikimedia.org> wrote:
>>
>>> Update: this is still in progress. There's some ongoing issues with home
>>> directories on non-nfs /home across Labs (not tools or maps) - we are
>>> working on it, and will update soon. Do feel free to reach out
>>> #wikimedia-labs if you have any questions or concerns.
>>>
>>> On Wed, Jan 18, 2017 at 8:38 AM, Madhumitha Viswanathan <
>>> mviswanathan at wikimedia.org> wrote:
>>>
>>>> Reminder: This is starting soon, in ~22 minutes.
>>>>
>>>> On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan <
>>>> mviswanathan at wikimedia.org> wrote:
>>>>
>>>>> Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC).
>>>>>
>>>>> On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan <
>>>>> mviswanathan at wikimedia.org> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Continuing the storage redundancy and reliability efforts for Labs (
>>>>>> https://phabricator.wikimedia.org/T126083), the final migration to
>>>>>> the new NFS storage cluster for Labs projects with NFS enabled is upcoming.
>>>>>> The migration is planned to happen 2017-01-18 starting 09:00
>>>>>> PST(16:00 UTC). This *does not* affect tools, maps or any other projects
>>>>>> that don't have /home or /data/project mounted. The migration window is
>>>>>> expected to be fairly short (<3 hours) - but could last up to 6 hours.
>>>>>>
>>>>>> During the migration, no new data will be written to NFS (/home or
>>>>>> /data/project), but existing data will be accessible in Read-only mode for
>>>>>> the most part. Post migration, any services or jobs that were running on
>>>>>> top of NFS (/home or /data/project) will require manual restarts. Jobs
>>>>>> running on top of /scratch or /public/dumps will be unaffected. I will keep
>>>>>> the lists and #wikimedia-labs updated on progress during and after the
>>>>>> migration.
>>>>>>
>>>>>> The list of labs projects that will be affected in this migration are:
>>>>>>
>>>>>>
>>>>>>    - catgraph
>>>>>>
>>>>>>
>>>>>>    - account-creation-assistance
>>>>>>
>>>>>>
>>>>>>    - contributors
>>>>>>
>>>>>>
>>>>>>    - wikidata-topicmaps
>>>>>>
>>>>>>
>>>>>>    - sugarcrm
>>>>>>
>>>>>>
>>>>>>    - wikidumpparse
>>>>>>
>>>>>>
>>>>>>    - video
>>>>>>
>>>>>>
>>>>>>    - openstack
>>>>>>
>>>>>>
>>>>>>    - testlabs
>>>>>>
>>>>>>
>>>>>>    - wikidata-dev
>>>>>>
>>>>>>
>>>>>>    - quarry
>>>>>>
>>>>>>
>>>>>>    - huggle
>>>>>>
>>>>>>
>>>>>>    - editor-engagement
>>>>>>
>>>>>>
>>>>>>    - utrs
>>>>>>
>>>>>>
>>>>>>    - wmt
>>>>>>
>>>>>>
>>>>>>    - cvn
>>>>>>
>>>>>>
>>>>>>    - fastcci
>>>>>>
>>>>>>
>>>>>>    - toolsbeta
>>>>>>
>>>>>>
>>>>>>    - project-proxy
>>>>>>
>>>>>>
>>>>>>    - dumps
>>>>>>
>>>>>>
>>>>>>    - bots
>>>>>>
>>>>>>
>>>>>>    - snuggle
>>>>>>
>>>>>>
>>>>>>    - math
>>>>>>
>>>>>>
>>>>>>    - wikisource-tools
>>>>>>
>>>>>>
>>>>>> The tracking task on phabricator is here -
>>>>>> https://phabricator.wikimedia.org/T154336. Let us know if you have
>>>>>> any questions or concerns on the list or on #wikimedia-labs.
>>>>>>
>>>>>> --
>>>>>> Madhu Viswanathan
>>>>>> Operations Engineer, Wikimedia Labs
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --Madhu :)
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --Madhu :)
>>>>
>>>
>>>
>>>
>>> --
>>> --Madhu :)
>>>
>>
>>
>>
>> --
>> --Madhu :)
>>
>
>
>
> --
> --Madhu :)
>



-- 
--Madhu :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20170119/8b4b0478/attachment-0001.html>