[Labs-l] [Wikimedia Labs][Announce] NFS (only labs projects) maintenance on 2017-01-18

[Chase Pettet]

For onlookers, we caught up on IRC and things are good now.

Thank you for the honest feedback Maximillian.  This was an unfortunate and
unplanned outcome that we struggled to recover from.  A handful of users we
were not able to restore /home for completely and it's been an uphill
battle.

On Mon, Jan 23, 2017 at 10:42 AM, Maximilian Doerr <
maximilian.doerr at gmail.com> wrote:

> The migration went horribly wrong on my project, which deleted roughly 60%
> of the project, including the .my.cnf file.  All the permissions and file
> ownerships were screwed up on the remaining files, and completely broke the
> bot tasks running on it.  I have restored the permissions, ownership and
> files, but I’m still missing my db cnf file to access the DB with.  It was
> not fun having to fix this.
>
> Cyberpower678
> English Wikipedia Account Creation Team
> English Wikipedia Administrator
> Global User Renamer
>
> On Jan 19, 2017, at 13:03, Madhumitha Viswanathan <
> mviswanathan at wikimedia.org> wrote:
>
> Update: 2 instances are still being restored - taxonbot.dwl and
> pole.wikidata-query, the rest on the previous list are done.
>
> On Wed, Jan 18, 2017 at 11:20 PM, Madhumitha Viswanathan <mviswanathan@
> wikimedia.org> wrote:
>
>> Hi all,
>>
>> Update - most of the home directories have been restored at this point -
>> we're down to the last few, that seem to have had a lot of data hence
>> taking quite some time.
>>
>> These are the pending instances(currently being restored):
>>
>> data.eqiad.wmflabs
>> ldfclient.eqiad.wmflabs
>> mathosphere.eqiad.wmflabs
>> mwoffliner4.eqiad.wmflabs
>> nomad.eqiad.wmflabs
>> reading.eqiad.wmflabs
>> recommendations.eqiad.wmflabs
>> captcha-apiproxy-03.eqiad.wmflabs
>> dashboardchat.eqiad.wmflabs
>> maps-cruncher.eqiad.wmflabs
>> pdf.eqiad.wmflabs
>> mwoffliner[2-3].eqiad.wmflabs
>> phlogiston-2.eqiad.wmflabs
>> pole.eqiad.wmflabs
>>
>> Other than this, it should all look good. Do let us know if something
>> seems amiss here or on #wikimedia-labs. Thanks for being patient!
>>
>>
>> On Wed, Jan 18, 2017 at 3:45 PM, Madhumitha Viswanathan <mviswanathan@
>> wikimedia.org> wrote:
>>
>>> Update: NFS for labs projects is back to read write now.
>>>
>>> There was an incident in the process of the migration that caused /home
>>> in a significant number of labs instances (where /home was not on NFS) to
>>> be replaced by an erroneous symlink. We are currently working on restoring
>>> all the home directories - it looks like it may take at-least a couple
>>> hours. I'll keep the list updated on progress.
>>>
>>>
>>> On Wed, Jan 18, 2017 at 11:34 AM, Madhumitha Viswanathan <mviswanathan@
>>> wikimedia.org> wrote:
>>>
>>>> Update: this is still in progress. There's some ongoing issues with
>>>> home directories on non-nfs /home across Labs (not tools or maps) - we are
>>>> working on it, and will update soon. Do feel free to reach out
>>>> #wikimedia-labs if you have any questions or concerns.
>>>>
>>>> On Wed, Jan 18, 2017 at 8:38 AM, Madhumitha Viswanathan <mviswanathan@
>>>> wikimedia.org> wrote:
>>>>
>>>>> Reminder: This is starting soon, in ~22 minutes.
>>>>>
>>>>> On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan <
>>>>> mviswanathan at wikimedia.org> wrote:
>>>>>
>>>>>> Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC).
>>>>>>
>>>>>> On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan <mviswanathan@
>>>>>> wikimedia.org> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Continuing the storage redundancy and reliability efforts for Labs (
>>>>>>> https://phabricator.wikimedia.org/T126083), the final migration to
>>>>>>> the new NFS storage cluster for Labs projects with NFS enabled is upcoming.
>>>>>>> The migration is planned to happen 2017-01-18 starting 09:00
>>>>>>> PST(16:00 UTC). This *does not* affect tools, maps or any other projects
>>>>>>> that don't have /home or /data/project mounted. The migration window is
>>>>>>> expected to be fairly short (<3 hours) - but could last up to 6 hours.
>>>>>>>
>>>>>>> During the migration, no new data will be written to NFS (/home or
>>>>>>> /data/project), but existing data will be accessible in Read-only mode for
>>>>>>> the most part. Post migration, any services or jobs that were running on
>>>>>>> top of NFS (/home or /data/project) will require manual restarts. Jobs
>>>>>>> running on top of /scratch or /public/dumps will be unaffected. I will keep
>>>>>>> the lists and #wikimedia-labs updated on progress during and after the
>>>>>>> migration.
>>>>>>>
>>>>>>> The list of labs projects that will be affected in this migration
>>>>>>> are:
>>>>>>>
>>>>>>>
>>>>>>>    - catgraph
>>>>>>>
>>>>>>>
>>>>>>>    - account-creation-assistance
>>>>>>>
>>>>>>>
>>>>>>>    - contributors
>>>>>>>
>>>>>>>
>>>>>>>    - wikidata-topicmaps
>>>>>>>
>>>>>>>
>>>>>>>    - sugarcrm
>>>>>>>
>>>>>>>
>>>>>>>    - wikidumpparse
>>>>>>>
>>>>>>>
>>>>>>>    - video
>>>>>>>
>>>>>>>
>>>>>>>    - openstack
>>>>>>>
>>>>>>>
>>>>>>>    - testlabs
>>>>>>>
>>>>>>>
>>>>>>>    - wikidata-dev
>>>>>>>
>>>>>>>
>>>>>>>    - quarry
>>>>>>>
>>>>>>>
>>>>>>>    - huggle
>>>>>>>
>>>>>>>
>>>>>>>    - editor-engagement
>>>>>>>
>>>>>>>
>>>>>>>    - utrs
>>>>>>>
>>>>>>>
>>>>>>>    - wmt
>>>>>>>
>>>>>>>
>>>>>>>    - cvn
>>>>>>>
>>>>>>>
>>>>>>>    - fastcci
>>>>>>>
>>>>>>>
>>>>>>>    - toolsbeta
>>>>>>>
>>>>>>>
>>>>>>>    - project-proxy
>>>>>>>
>>>>>>>
>>>>>>>    - dumps
>>>>>>>
>>>>>>>
>>>>>>>    - bots
>>>>>>>
>>>>>>>
>>>>>>>    - snuggle
>>>>>>>
>>>>>>>
>>>>>>>    - math
>>>>>>>
>>>>>>>
>>>>>>>    - wikisource-tools
>>>>>>>
>>>>>>>
>>>>>>> The tracking task on phabricator is here -
>>>>>>> https://phabricator.wikimedia.org/T154336. Let us know if you have
>>>>>>> any questions or concerns on the list or on #wikimedia-labs.
>>>>>>>
>>>>>>> --
>>>>>>> Madhu Viswanathan
>>>>>>> Operations Engineer, Wikimedia Labs
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --Madhu :)
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --Madhu :)
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --Madhu :)
>>>>
>>>
>>>
>>>
>>> --
>>> --Madhu :)
>>>
>>
>>
>>
>> --
>> --Madhu :)
>>
>
>
>
> --
> --Madhu :)
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
>


-- 
Chase Pettet
Engineering Manager -- Labs
chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and
IRC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20170123/d91f2d0c/attachment-0001.html>