[Labs-l] [Wikimedia Labs][Announce] NFS (only labs projects) maintenance on 2017-01-18

Mon Jan 23 19:05:22 UTC 2017

Thank you Chase.  :-)  Everything on my instance is now restored and working, thank you local backups.

Cyberpower678
English Wikipedia Account Creation Team
English Wikipedia Administrator
Global User Renamer

> On Jan 23, 2017, at 12:45, Chase Pettet <cpettet at wikimedia.org> wrote:
> 
> For onlookers, we caught up on IRC and things are good now.
> 
> Thank you for the honest feedback Maximillian.  This was an unfortunate and unplanned outcome that we struggled to recover from.  A handful of users we were not able to restore /home for completely and it's been an uphill battle. 
> 
>> On Mon, Jan 23, 2017 at 10:42 AM, Maximilian Doerr <maximilian.doerr at gmail.com> wrote:
>> The migration went horribly wrong on my project, which deleted roughly 60% of the project, including the .my.cnf file.  All the permissions and file ownerships were screwed up on the remaining files, and completely broke the bot tasks running on it.  I have restored the permissions, ownership and files, but I’m still missing my db cnf file to access the DB with.  It was not fun having to fix this.
>> 
>> Cyberpower678
>> English Wikipedia Account Creation Team
>> English Wikipedia Administrator
>> Global User Renamer
>> 
>>> On Jan 19, 2017, at 13:03, Madhumitha Viswanathan <mviswanathan at wikimedia.org> wrote:
>>> 
>>> Update: 2 instances are still being restored - taxonbot.dwl and pole.wikidata-query, the rest on the previous list are done.
>>> 
>>> On Wed, Jan 18, 2017 at 11:20 PM, Madhumitha Viswanathan <mviswanathan at wikimedia.org> wrote:
>>>> Hi all,
>>>> 
>>>> Update - most of the home directories have been restored at this point - we're down to the last few, that seem to have had a lot of data hence taking quite some time.
>>>> 
>>>> These are the pending instances(currently being restored):
>>>> 
>>>> data.eqiad.wmflabs
>>>> ldfclient.eqiad.wmflabs
>>>> mathosphere.eqiad.wmflabs
>>>> mwoffliner4.eqiad.wmflabs
>>>> nomad.eqiad.wmflabs
>>>> reading.eqiad.wmflabs
>>>> recommendations.eqiad.wmflabs
>>>> captcha-apiproxy-03.eqiad.wmflabs
>>>> dashboardchat.eqiad.wmflabs
>>>> maps-cruncher.eqiad.wmflabs
>>>> pdf.eqiad.wmflabs
>>>> mwoffliner[2-3].eqiad.wmflabs
>>>> phlogiston-2.eqiad.wmflabs
>>>> pole.eqiad.wmflabs
>>>> 
>>>> Other than this, it should all look good. Do let us know if something seems amiss here or on #wikimedia-labs. Thanks for being patient!
>>>> 
>>>> 
>>>>> On Wed, Jan 18, 2017 at 3:45 PM, Madhumitha Viswanathan <mviswanathan at wikimedia.org> wrote:
>>>>> Update: NFS for labs projects is back to read write now. 
>>>>> 
>>>>> There was an incident in the process of the migration that caused /home in a significant number of labs instances (where /home was not on NFS) to be replaced by an erroneous symlink. We are currently working on restoring all the home directories - it looks like it may take at-least a couple hours. I'll keep the list updated on progress.
>>>>> 
>>>>> 
>>>>>> On Wed, Jan 18, 2017 at 11:34 AM, Madhumitha Viswanathan <mviswanathan at wikimedia.org> wrote:
>>>>>> Update: this is still in progress. There's some ongoing issues with home directories on non-nfs /home across Labs (not tools or maps) - we are working on it, and will update soon. Do feel free to reach out #wikimedia-labs if you have any questions or concerns.
>>>>>> 
>>>>>>> On Wed, Jan 18, 2017 at 8:38 AM, Madhumitha Viswanathan <mviswanathan at wikimedia.org> wrote:
>>>>>>> Reminder: This is starting soon, in ~22 minutes.
>>>>>>> 
>>>>>>>> On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan <mviswanathan at wikimedia.org> wrote:
>>>>>>>> Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC).
>>>>>>>> 
>>>>>>>>> On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan <mviswanathan at wikimedia.org> wrote:
>>>>>>>>> Hello,
>>>>>>>>> 
>>>>>>>>> Continuing the storage redundancy and reliability efforts for Labs (https://phabricator.wikimedia.org/T126083), the final migration to the new NFS storage cluster for Labs projects with NFS enabled is upcoming. The migration is planned to happen 2017-01-18 starting 09:00 PST(16:00 UTC). This *does not* affect tools, maps or any other projects that don't have /home or /data/project mounted. The migration window is expected to be fairly short (<3 hours) - but could last up to 6 hours.
>>>>>>>>> 
>>>>>>>>> During the migration, no new data will be written to NFS (/home or /data/project), but existing data will be accessible in Read-only mode for the most part. Post migration, any services or jobs that were running on top of NFS (/home or /data/project) will require manual restarts. Jobs running on top of /scratch or /public/dumps will be unaffected. I will keep the lists and #wikimedia-labs updated on progress during and after the migration.
>>>>>>>>> 
>>>>>>>>> The list of labs projects that will be affected in this migration are:
>>>>>>>>>     
>>>>>>>>> catgraph
>>>>>>>>> account-creation-assistance
>>>>>>>>> contributors
>>>>>>>>> wikidata-topicmaps
>>>>>>>>> sugarcrm
>>>>>>>>> wikidumpparse
>>>>>>>>> video
>>>>>>>>> openstack
>>>>>>>>> testlabs
>>>>>>>>> wikidata-dev
>>>>>>>>> quarry
>>>>>>>>> huggle
>>>>>>>>> editor-engagement
>>>>>>>>> utrs
>>>>>>>>> wmt
>>>>>>>>> cvn
>>>>>>>>> fastcci
>>>>>>>>> toolsbeta
>>>>>>>>> project-proxy
>>>>>>>>> dumps
>>>>>>>>> bots
>>>>>>>>> snuggle
>>>>>>>>> math
>>>>>>>>> wikisource-tools
>>>>>>>>> 
>>>>>>>>> The tracking task on phabricator is here - https://phabricator.wikimedia.org/T154336. Let us know if you have any questions or concerns on the list or on #wikimedia-labs.
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Madhu Viswanathan
>>>>>>>>> Operations Engineer, Wikimedia Labs
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -- 
>>>>>>>> --Madhu :)
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -- 
>>>>>>> --Madhu :)
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> --Madhu :)
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> --Madhu :)
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> --Madhu :)
>>> 
>>> 
>>> 
>>> -- 
>>> --Madhu :)
>>> _______________________________________________
>>> Labs-l mailing list
>>> Labs-l at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>> 
>> 
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>> 
> 
> 
> 
> -- 
> Chase Pettet
> Engineering Manager -- Labs
> chasemp on phabricator and IRC
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20170123/b0feca90/attachment-0001.html>