<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Thank you for the update. It's great to see labs moving to SSDs in some areas.<br><br>Cyberpower678<div>English Wikipedia Account Creation Team</div><div>ACC Mailing List Moderator</div><div>Global User Renamer</div></div><div><br>On Jun 24, 2016, at 00:00, Rob Halsell <<a href="mailto:rhalsell@wikimedia.org">rhalsell@wikimedia.org</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr"><div><div>Please note that the expended downtime for labmon1001 has been extended to 2016-06-24 @ 17:00 GMT. Details are noted on <a href="https://phabricator.wikimedia.org/T137924">https://phabricator.wikimedia.org/T137924</a>.<br><br></div><div>Basically I'm planning to watch it migrate data the rest of this evening (its been at it all day.) If it completes before I go to bed, I'll bring its services back online so we collect more data, and it becomes available. <br></div><div><br></div>The labmon1001 system has been reimaged, and the 893GB of data is being restored to the new SSDs on the system. During the restoration process, puppet and all services are halted on labmon1001, so it won't start attempting to write new data during the restoration.<br><br></div><div>Apologies for the extended downtime!<br></div><div><br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 22, 2016 at 8:26 AM, Rob Halsell <span dir="ltr"><<a href="mailto:rhalsell@wikimedia.org" target="_blank">rhalsell@wikimedia.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Labs users,<br><br>As many of you may recall, (mostly) Yuvi and (slightly) myself worked on labmon1001 a couple of weeks back. Unfortunately, the work performed wasn't enough (as it still left the host with spinning disks) and the graphite service hammering them bogs down the server. As a solution, we will be reinstalling the system on 2016-06-23 from 15:00 GMT to 23:00 GMT. I don't expect to use the majority of this window, since much of the data was migrated backup previously. However, it is a possibility if we have to re-sync all the data (without differential).<br><br>Details of this work can be viewed on <a href="https://phabricator.wikimedia.org/T137924" target="_blank">https://phabricator.wikimedia.org/T137924</a>.<br><br>Once the SSDs are installed, their ability to handle the iops generated from graphite is expected to bring load levels on labmon1001 down to sane levels.<br><br>Please let me know if there are any questions or concerns. They can be raised via this email thread, or by comment on the linked phabricator task.<br><br></div>Thanks,<span class="HOEnZb"><font color="#888888"><br clear="all"><div><br>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div>Rob Halsell<br>Operations Engineer<br>Wikimedia Foundation, Inc.<br>E-Mail: <a href="mailto:rhalsell@wikimedia.org" target="_blank">rhalsell@wikimedia.org</a><br>Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7 <i>0245 D22A<br></i>Office: 415.839.6885 x6620<br>Fax: 415.882.0495<br><br></div></div></div>
</div></font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Rob Halsell<br>Operations Engineer<br>Wikimedia Foundation, Inc.<br>E-Mail: <a href="mailto:rhalsell@wikimedia.org" target="_blank">rhalsell@wikimedia.org</a><br>Key fingerprint = CB1F C7E7 0FF8 5DB2 6820 9C7E 75ED 14C7 <i>0245 D22A<br></i>Office: 415.839.6885 x6620<br>Fax: 415.882.0495<br><br></div></div></div>
</div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Labs-l mailing list</span><br><span><a href="mailto:Labs-l@lists.wikimedia.org">Labs-l@lists.wikimedia.org</a></span><br><span><a href="https://lists.wikimedia.org/mailman/listinfo/labs-l">https://lists.wikimedia.org/mailman/listinfo/labs-l</a></span><br></div></blockquote></body></html>