[Labs-l] [Wikimedia Labs][Announce] NFS (only labs projects) maintenance on 2017-01-18

[Maximilian Doerr]

The migration went horribly wrong on my project, which deleted roughly 60% of the project, including the .my.cnf file.  All the permissions and file ownerships were screwed up on the remaining files, and completely broke the bot tasks running on it.  I have restored the permissions, ownership and files, but I’m still missing my db cnf file to access the DB with.  It was not fun having to fix this.

Cyberpower678
English Wikipedia Account Creation Team
English Wikipedia Administrator
Global User Renamer

> On Jan 19, 2017, at 13:03, Madhumitha Viswanathan <mviswanathan at wikimedia.org> wrote:
> 
> Update: 2 instances are still being restored - taxonbot.dwl and pole.wikidata-query, the rest on the previous list are done.
> 
> On Wed, Jan 18, 2017 at 11:20 PM, Madhumitha Viswanathan <mviswanathan at wikimedia.org <mailto:mviswanathan at wikimedia.org>> wrote:
> Hi all,
> 
> Update - most of the home directories have been restored at this point - we're down to the last few, that seem to have had a lot of data hence taking quite some time.
> 
> These are the pending instances(currently being restored):
> 
> data.eqiad.wmflabs
> ldfclient.eqiad.wmflabs
> mathosphere.eqiad.wmflabs
> mwoffliner4.eqiad.wmflabs
> nomad.eqiad.wmflabs
> reading.eqiad.wmflabs
> recommendations.eqiad.wmflabs
> captcha-apiproxy-03.eqiad.wmflabs
> dashboardchat.eqiad.wmflabs
> maps-cruncher.eqiad.wmflabs
> pdf.eqiad.wmflabs
> mwoffliner[2-3].eqiad.wmflabs
> phlogiston-2.eqiad.wmflabs
> pole.eqiad.wmflabs
> 
> Other than this, it should all look good. Do let us know if something seems amiss here or on #wikimedia-labs. Thanks for being patient!
> 
> 
> On Wed, Jan 18, 2017 at 3:45 PM, Madhumitha Viswanathan <mviswanathan at wikimedia.org <mailto:mviswanathan at wikimedia.org>> wrote:
> Update: NFS for labs projects is back to read write now. 
> 
> There was an incident in the process of the migration that caused /home in a significant number of labs instances (where /home was not on NFS) to be replaced by an erroneous symlink. We are currently working on restoring all the home directories - it looks like it may take at-least a couple hours. I'll keep the list updated on progress.
> 
> 
> On Wed, Jan 18, 2017 at 11:34 AM, Madhumitha Viswanathan <mviswanathan at wikimedia.org <mailto:mviswanathan at wikimedia.org>> wrote:
> Update: this is still in progress. There's some ongoing issues with home directories on non-nfs /home across Labs (not tools or maps) - we are working on it, and will update soon. Do feel free to reach out #wikimedia-labs if you have any questions or concerns.
> 
> On Wed, Jan 18, 2017 at 8:38 AM, Madhumitha Viswanathan <mviswanathan at wikimedia.org <mailto:mviswanathan at wikimedia.org>> wrote:
> Reminder: This is starting soon, in ~22 minutes.
> 
> On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan <mviswanathan at wikimedia.org <mailto:mviswanathan at wikimedia.org>> wrote:
> Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC).
> 
> On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan <mviswanathan at wikimedia.org <mailto:mviswanathan at wikimedia.org>> wrote:
> Hello,
> 
> Continuing the storage redundancy and reliability efforts for Labs (https://phabricator.wikimedia.org/T126083 <https://phabricator.wikimedia.org/T126083>), the final migration to the new NFS storage cluster for Labs projects with NFS enabled is upcoming. The migration is planned to happen 2017-01-18 starting 09:00 PST(16:00 UTC). This *does not* affect tools, maps or any other projects that don't have /home or /data/project mounted. The migration window is expected to be fairly short (<3 hours) - but could last up to 6 hours.
> 
> During the migration, no new data will be written to NFS (/home or /data/project), but existing data will be accessible in Read-only mode for the most part. Post migration, any services or jobs that were running on top of NFS (/home or /data/project) will require manual restarts. Jobs running on top of /scratch or /public/dumps will be unaffected. I will keep the lists and #wikimedia-labs updated on progress during and after the migration.
> 
> The list of labs projects that will be affected in this migration are:
>     
> catgraph
> account-creation-assistance
> contributors
> wikidata-topicmaps
> sugarcrm
> wikidumpparse
> video
> openstack
> testlabs
> wikidata-dev
> quarry
> huggle
> editor-engagement
> utrs
> wmt
> cvn
> fastcci
> toolsbeta
> project-proxy
> dumps
> bots
> snuggle
> math
> wikisource-tools
> 
> The tracking task on phabricator is here - https://phabricator.wikimedia.org/T154336 <https://phabricator.wikimedia.org/T154336>. Let us know if you have any questions or concerns on the list or on #wikimedia-labs.
> 
> --
> Madhu Viswanathan
> Operations Engineer, Wikimedia Labs
> 
> 
> 
> -- 
> --Madhu :)
> 
> 
> 
> -- 
> --Madhu :)
> 
> 
> 
> -- 
> --Madhu :)
> 
> 
> 
> -- 
> --Madhu :)
> 
> 
> 
> -- 
> --Madhu :)
> 
> 
> 
> -- 
> --Madhu :)
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org <mailto:Labs-l at lists.wikimedia.org>
> https://lists.wikimedia.org/mailman/listinfo/labs-l <https://lists.wikimedia.org/mailman/listinfo/labs-l>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20170123/6088dbc1/attachment-0001.html>