Hello,
Continuing the storage redundancy and reliability efforts for Labs ( https://phabricator.wikimedia.org/T126083), the final migration to the new NFS storage cluster for Labs projects with NFS enabled is upcoming. The migration is planned to happen 2017-01-18 starting 09:00 PST(16:00 UTC). This *does not* affect tools, maps or any other projects that don't have /home or /data/project mounted. The migration window is expected to be fairly short (<3 hours) - but could last up to 6 hours.
During the migration, no new data will be written to NFS (/home or /data/project), but existing data will be accessible in Read-only mode for the most part. Post migration, any services or jobs that were running on top of NFS (/home or /data/project) will require manual restarts. Jobs running on top of /scratch or /public/dumps will be unaffected. I will keep the lists and #wikimedia-labs updated on progress during and after the migration.
The list of labs projects that will be affected in this migration are:
- catgraph
- account-creation-assistance
- contributors
- wikidata-topicmaps
- sugarcrm
- wikidumpparse
- video
- openstack
- testlabs
- wikidata-dev
- quarry
- huggle
- editor-engagement
- utrs
- wmt
- cvn
- fastcci
- toolsbeta
- project-proxy
- dumps
- bots
- snuggle
- math
- wikisource-tools
The tracking task on phabricator is here - https://phabricator.wikimedia.org/T154336. Let us know if you have any questions or concerns on the list or on #wikimedia-labs.
-- Madhu Viswanathan Operations Engineer, Wikimedia Labs
Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC).
On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Hello,
Continuing the storage redundancy and reliability efforts for Labs ( https://phabricator.wikimedia.org/T126083), the final migration to the new NFS storage cluster for Labs projects with NFS enabled is upcoming. The migration is planned to happen 2017-01-18 starting 09:00 PST(16:00 UTC). This *does not* affect tools, maps or any other projects that don't have /home or /data/project mounted. The migration window is expected to be fairly short (<3 hours) - but could last up to 6 hours.
During the migration, no new data will be written to NFS (/home or /data/project), but existing data will be accessible in Read-only mode for the most part. Post migration, any services or jobs that were running on top of NFS (/home or /data/project) will require manual restarts. Jobs running on top of /scratch or /public/dumps will be unaffected. I will keep the lists and #wikimedia-labs updated on progress during and after the migration.
The list of labs projects that will be affected in this migration are:
catgraph
account-creation-assistance
contributors
wikidata-topicmaps
sugarcrm
wikidumpparse
video
openstack
testlabs
wikidata-dev
quarry
huggle
editor-engagement
utrs
wmt
cvn
fastcci
toolsbeta
project-proxy
dumps
bots
snuggle
math
wikisource-tools
The tracking task on phabricator is here - https://phabricator. wikimedia.org/T154336. Let us know if you have any questions or concerns on the list or on #wikimedia-labs.
-- Madhu Viswanathan Operations Engineer, Wikimedia Labs
Reminder: This is starting soon, in ~22 minutes.
On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC).
On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Hello,
Continuing the storage redundancy and reliability efforts for Labs ( https://phabricator.wikimedia.org/T126083), the final migration to the new NFS storage cluster for Labs projects with NFS enabled is upcoming. The migration is planned to happen 2017-01-18 starting 09:00 PST(16:00 UTC). This *does not* affect tools, maps or any other projects that don't have /home or /data/project mounted. The migration window is expected to be fairly short (<3 hours) - but could last up to 6 hours.
During the migration, no new data will be written to NFS (/home or /data/project), but existing data will be accessible in Read-only mode for the most part. Post migration, any services or jobs that were running on top of NFS (/home or /data/project) will require manual restarts. Jobs running on top of /scratch or /public/dumps will be unaffected. I will keep the lists and #wikimedia-labs updated on progress during and after the migration.
The list of labs projects that will be affected in this migration are:
catgraph
account-creation-assistance
contributors
wikidata-topicmaps
sugarcrm
wikidumpparse
video
openstack
testlabs
wikidata-dev
quarry
huggle
editor-engagement
utrs
wmt
cvn
fastcci
toolsbeta
project-proxy
dumps
bots
snuggle
math
wikisource-tools
The tracking task on phabricator is here - https://phabricator.wikimedi a.org/T154336. Let us know if you have any questions or concerns on the list or on #wikimedia-labs.
-- Madhu Viswanathan Operations Engineer, Wikimedia Labs
-- --Madhu :)
Update: this is still in progress. There's some ongoing issues with home directories on non-nfs /home across Labs (not tools or maps) - we are working on it, and will update soon. Do feel free to reach out #wikimedia-labs if you have any questions or concerns.
On Wed, Jan 18, 2017 at 8:38 AM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Reminder: This is starting soon, in ~22 minutes.
On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC).
On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Hello,
Continuing the storage redundancy and reliability efforts for Labs ( https://phabricator.wikimedia.org/T126083), the final migration to the new NFS storage cluster for Labs projects with NFS enabled is upcoming. The migration is planned to happen 2017-01-18 starting 09:00 PST(16:00 UTC). This *does not* affect tools, maps or any other projects that don't have /home or /data/project mounted. The migration window is expected to be fairly short (<3 hours) - but could last up to 6 hours.
During the migration, no new data will be written to NFS (/home or /data/project), but existing data will be accessible in Read-only mode for the most part. Post migration, any services or jobs that were running on top of NFS (/home or /data/project) will require manual restarts. Jobs running on top of /scratch or /public/dumps will be unaffected. I will keep the lists and #wikimedia-labs updated on progress during and after the migration.
The list of labs projects that will be affected in this migration are:
catgraph
account-creation-assistance
contributors
wikidata-topicmaps
sugarcrm
wikidumpparse
video
openstack
testlabs
wikidata-dev
quarry
huggle
editor-engagement
utrs
wmt
cvn
fastcci
toolsbeta
project-proxy
dumps
bots
snuggle
math
wikisource-tools
The tracking task on phabricator is here - https://phabricator.wikimedi a.org/T154336. Let us know if you have any questions or concerns on the list or on #wikimedia-labs.
-- Madhu Viswanathan Operations Engineer, Wikimedia Labs
-- --Madhu :)
-- --Madhu :)
Update: NFS for labs projects is back to read write now.
There was an incident in the process of the migration that caused /home in a significant number of labs instances (where /home was not on NFS) to be replaced by an erroneous symlink. We are currently working on restoring all the home directories - it looks like it may take at-least a couple hours. I'll keep the list updated on progress.
On Wed, Jan 18, 2017 at 11:34 AM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Update: this is still in progress. There's some ongoing issues with home directories on non-nfs /home across Labs (not tools or maps) - we are working on it, and will update soon. Do feel free to reach out #wikimedia-labs if you have any questions or concerns.
On Wed, Jan 18, 2017 at 8:38 AM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Reminder: This is starting soon, in ~22 minutes.
On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC).
On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Hello,
Continuing the storage redundancy and reliability efforts for Labs ( https://phabricator.wikimedia.org/T126083), the final migration to the new NFS storage cluster for Labs projects with NFS enabled is upcoming. The migration is planned to happen 2017-01-18 starting 09:00 PST(16:00 UTC). This *does not* affect tools, maps or any other projects that don't have /home or /data/project mounted. The migration window is expected to be fairly short (<3 hours) - but could last up to 6 hours.
During the migration, no new data will be written to NFS (/home or /data/project), but existing data will be accessible in Read-only mode for the most part. Post migration, any services or jobs that were running on top of NFS (/home or /data/project) will require manual restarts. Jobs running on top of /scratch or /public/dumps will be unaffected. I will keep the lists and #wikimedia-labs updated on progress during and after the migration.
The list of labs projects that will be affected in this migration are:
catgraph
account-creation-assistance
contributors
wikidata-topicmaps
sugarcrm
wikidumpparse
video
openstack
testlabs
wikidata-dev
quarry
huggle
editor-engagement
utrs
wmt
cvn
fastcci
toolsbeta
project-proxy
dumps
bots
snuggle
math
wikisource-tools
The tracking task on phabricator is here - https://phabricator.wikimedi a.org/T154336. Let us know if you have any questions or concerns on the list or on #wikimedia-labs.
-- Madhu Viswanathan Operations Engineer, Wikimedia Labs
-- --Madhu :)
-- --Madhu :)
-- --Madhu :)
Hi all,
Update - most of the home directories have been restored at this point - we're down to the last few, that seem to have had a lot of data hence taking quite some time.
These are the pending instances(currently being restored):
data.eqiad.wmflabs ldfclient.eqiad.wmflabs mathosphere.eqiad.wmflabs mwoffliner4.eqiad.wmflabs nomad.eqiad.wmflabs reading.eqiad.wmflabs recommendations.eqiad.wmflabs captcha-apiproxy-03.eqiad.wmflabs dashboardchat.eqiad.wmflabs maps-cruncher.eqiad.wmflabs pdf.eqiad.wmflabs mwoffliner[2-3].eqiad.wmflabs phlogiston-2.eqiad.wmflabs pole.eqiad.wmflabs
Other than this, it should all look good. Do let us know if something seems amiss here or on #wikimedia-labs. Thanks for being patient!
On Wed, Jan 18, 2017 at 3:45 PM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Update: NFS for labs projects is back to read write now.
There was an incident in the process of the migration that caused /home in a significant number of labs instances (where /home was not on NFS) to be replaced by an erroneous symlink. We are currently working on restoring all the home directories - it looks like it may take at-least a couple hours. I'll keep the list updated on progress.
On Wed, Jan 18, 2017 at 11:34 AM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Update: this is still in progress. There's some ongoing issues with home directories on non-nfs /home across Labs (not tools or maps) - we are working on it, and will update soon. Do feel free to reach out #wikimedia-labs if you have any questions or concerns.
On Wed, Jan 18, 2017 at 8:38 AM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Reminder: This is starting soon, in ~22 minutes.
On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC).
On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Hello,
Continuing the storage redundancy and reliability efforts for Labs ( https://phabricator.wikimedia.org/T126083), the final migration to the new NFS storage cluster for Labs projects with NFS enabled is upcoming. The migration is planned to happen 2017-01-18 starting 09:00 PST(16:00 UTC). This *does not* affect tools, maps or any other projects that don't have /home or /data/project mounted. The migration window is expected to be fairly short (<3 hours) - but could last up to 6 hours.
During the migration, no new data will be written to NFS (/home or /data/project), but existing data will be accessible in Read-only mode for the most part. Post migration, any services or jobs that were running on top of NFS (/home or /data/project) will require manual restarts. Jobs running on top of /scratch or /public/dumps will be unaffected. I will keep the lists and #wikimedia-labs updated on progress during and after the migration.
The list of labs projects that will be affected in this migration are:
catgraph
account-creation-assistance
contributors
wikidata-topicmaps
sugarcrm
wikidumpparse
video
openstack
testlabs
wikidata-dev
quarry
huggle
editor-engagement
utrs
wmt
cvn
fastcci
toolsbeta
project-proxy
dumps
bots
snuggle
math
wikisource-tools
The tracking task on phabricator is here - https://phabricator.wikimedia.org/T154336. Let us know if you have any questions or concerns on the list or on #wikimedia-labs.
-- Madhu Viswanathan Operations Engineer, Wikimedia Labs
-- --Madhu :)
-- --Madhu :)
-- --Madhu :)
-- --Madhu :)
Update: 2 instances are still being restored - taxonbot.dwl and pole.wikidata-query, the rest on the previous list are done.
On Wed, Jan 18, 2017 at 11:20 PM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Hi all,
Update - most of the home directories have been restored at this point - we're down to the last few, that seem to have had a lot of data hence taking quite some time.
These are the pending instances(currently being restored):
data.eqiad.wmflabs ldfclient.eqiad.wmflabs mathosphere.eqiad.wmflabs mwoffliner4.eqiad.wmflabs nomad.eqiad.wmflabs reading.eqiad.wmflabs recommendations.eqiad.wmflabs captcha-apiproxy-03.eqiad.wmflabs dashboardchat.eqiad.wmflabs maps-cruncher.eqiad.wmflabs pdf.eqiad.wmflabs mwoffliner[2-3].eqiad.wmflabs phlogiston-2.eqiad.wmflabs pole.eqiad.wmflabs
Other than this, it should all look good. Do let us know if something seems amiss here or on #wikimedia-labs. Thanks for being patient!
On Wed, Jan 18, 2017 at 3:45 PM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Update: NFS for labs projects is back to read write now.
There was an incident in the process of the migration that caused /home in a significant number of labs instances (where /home was not on NFS) to be replaced by an erroneous symlink. We are currently working on restoring all the home directories - it looks like it may take at-least a couple hours. I'll keep the list updated on progress.
On Wed, Jan 18, 2017 at 11:34 AM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Update: this is still in progress. There's some ongoing issues with home directories on non-nfs /home across Labs (not tools or maps) - we are working on it, and will update soon. Do feel free to reach out #wikimedia-labs if you have any questions or concerns.
On Wed, Jan 18, 2017 at 8:38 AM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Reminder: This is starting soon, in ~22 minutes.
On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Reminder: This is happening tomorrow, starting 09:00 PST(16:00 UTC).
On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan < mviswanathan@wikimedia.org> wrote:
Hello,
Continuing the storage redundancy and reliability efforts for Labs ( https://phabricator.wikimedia.org/T126083), the final migration to the new NFS storage cluster for Labs projects with NFS enabled is upcoming. The migration is planned to happen 2017-01-18 starting 09:00 PST(16:00 UTC). This *does not* affect tools, maps or any other projects that don't have /home or /data/project mounted. The migration window is expected to be fairly short (<3 hours) - but could last up to 6 hours.
During the migration, no new data will be written to NFS (/home or /data/project), but existing data will be accessible in Read-only mode for the most part. Post migration, any services or jobs that were running on top of NFS (/home or /data/project) will require manual restarts. Jobs running on top of /scratch or /public/dumps will be unaffected. I will keep the lists and #wikimedia-labs updated on progress during and after the migration.
The list of labs projects that will be affected in this migration are:
catgraph
account-creation-assistance
contributors
wikidata-topicmaps
sugarcrm
wikidumpparse
video
openstack
testlabs
wikidata-dev
quarry
huggle
editor-engagement
utrs
wmt
cvn
fastcci
toolsbeta
project-proxy
dumps
bots
snuggle
math
wikisource-tools
The tracking task on phabricator is here - https://phabricator.wikimedia.org/T154336. Let us know if you have any questions or concerns on the list or on #wikimedia-labs.
-- Madhu Viswanathan Operations Engineer, Wikimedia Labs
-- --Madhu :)
-- --Madhu :)
-- --Madhu :)
-- --Madhu :)
-- --Madhu :)
wikitech-l@lists.wikimedia.org