[Labs-l] labmon1001 downtime scheduled for 2016-06-23 from 15:00 GMT to 23:00 GMT

[Maximilian Doerr]

Thank you for the update.  It's great to see labs moving to SSDs in some areas.

Cyberpower678
English Wikipedia Account Creation Team
ACC Mailing List Moderator
Global User Renamer

> On Jun 24, 2016, at 00:00, Rob Halsell <rhalsell at wikimedia.org> wrote:
> 
> Please note that the expended downtime for labmon1001 has been extended to 2016-06-24 @ 17:00 GMT.  Details are noted on https://phabricator.wikimedia.org/T137924.
> 
> Basically I'm planning to watch it migrate data the rest of this evening (its been at it all day.)  If it completes before I go to bed, I'll bring its services back online so we collect more data, and it becomes available.  
> 
> The labmon1001 system has been reimaged, and the 893GB of data is being restored to the new SSDs on the system.  During the restoration process, puppet and all services are halted on labmon1001, so it won't start attempting to write new data during the restoration.
> 
> Apologies for the extended downtime!
> 
> 
> 
>> On Wed, Jun 22, 2016 at 8:26 AM, Rob Halsell <rhalsell at wikimedia.org> wrote:
>> Labs users,
>> 
>> As many of you may recall, (mostly) Yuvi and (slightly) myself worked on labmon1001 a couple of weeks back.  Unfortunately, the work performed wasn't enough (as it still left the host with spinning disks) and the graphite service hammering them bogs down the server.  As a solution, we will be reinstalling the system on 2016-06-23 from 15:00 GMT to 23:00 GMT.  I don't expect to use the majority of this window, since much of the data was migrated backup previously.  However, it is a possibility if we have to re-sync all the data (without differential).
>> 
>> Details of this work can be viewed on https://phabricator.wikimedia.org/T137924.
>> 
>> Once the SSDs are installed, their ability to handle the iops generated from graphite is expected to bring load levels on labmon1001 down to sane levels.
>> 
>> Please let me know if there are any questions or concerns.  They can be raised via this email thread, or by comment on the linked phabricator task.
>> 
>> Thanks,
>> 
>> -- 
>> Rob Halsell
>> Operations Engineer
>> Wikimedia Foundation, Inc.
>> E-Mail: rhalsell at wikimedia.org
>> Key fingerprint = CB1F C7E7 0FF8 5DB2 6820  9C7E 75ED 14C7 0245 D22A
>> Office: 415.839.6885 x6620
>> Fax: 415.882.0495
> 
> 
> 
> -- 
> Rob Halsell
> Operations Engineer
> Wikimedia Foundation, Inc.
> E-Mail: rhalsell at wikimedia.org
> Key fingerprint = CB1F C7E7 0FF8 5DB2 6820  9C7E 75ED 14C7 0245 D22A
> Office: 415.839.6885 x6620
> Fax: 415.882.0495
> 
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20160624/ae76c993/attachment.html>