<html><head/><body><html><head>
<meta name="viewport" content="width=device-width" /><meta http-equiv="Content-Type" content="text/vnd.ui.secure+html;charset=utf-8" /></head><body><div class="mail_android_message" style="line-height: 1; padding: 0.5em">Hi!<br>
<br>
The window was not so long as planned before. <br>
<br>
Thanks a lot for the fast and good job.<br>
<br>
Kind regards,<br>
Martin<br>
<br>
-- <br>
Diese Nachricht wurde von meinem Android Mobiltelefon mit GMX Mail gesendet.</div><div class="mail_android_quote" style="line-height: 1; padding: 0.3em">Am 15.11.2016, 06:39, Madhumitha Viswanathan <mviswanathan@wikimedia.org> schrieb:<blockquote class="gmail_quote" style="margin: 0.8ex 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div>
Hi all,
<div>
<br />
</div>
<div>
Tools is now back to read write, and all of tools should work as expected. Webservice logs aren't being written to disk currently, and will not be written for ~12 more hours, while we make sure everything else is working alright.
</div>
<div>
<br />
</div>
<div>
The storage infrastructure is brand new, so we recommend being a bit gentle for the next few days, as we monitor performance and make sure it's all okay. Do let us know if things aren't working like you'd expect, on #wikimedia-labs/on list or phabricator.
</div>
<div>
<br />
</div>
<div>
Thanks for being patient through this long maintenance window!
</div>
<div>
<br />
</div>
</div>
<div class="gmail_extra">
<br />
<div class="gmail_quote">
On Mon, Nov 14, 2016 at 10:15 AM, Madhumitha Viswanathan<span> <<a href="mailto:mviswanathan@wikimedia.org" target="_blank">mviswanathan@wikimedia.org</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex;border-left: 1.0px rgb(204,204,204) solid;padding-left: 1.0ex;">
<div>
Most of tools should be back up now. Expected behavior for the next many hours (Expected ~12-14 hours based on last sync of tools data, but might be much longer)
<div>
<br />
</div>
<div>
Quoting from previous announcement [1]:
</div>
<div>
<br />
</div>
<div>
<blockquote class="gmail_quote" style="margin: 0.0px 0.0px 0.0px 0.8ex;border-left: 1.0px solid rgb(204,204,204);padding-left: 1.0ex;">
<font face="arial, helvetica, sans-serif">* The tools NFS share (/data/project and /home) will be read-only for the duration of the maintenance, so no new data or logs will get written to it.<br />
</font><font face="arial, helvetica, sans-serif">* New jobs cannot be submitted for the whole maintenance window - this means submitting jobs through cron or tools-mail will not function, although tools-mail can continue to send emails.<br />
</font><font face="arial, helvetica, sans-serif">* Current jobs might keep running, but won't get rescheduled if they die. If they do not die and aren't writing to NFS they should be fine.</font>
</blockquote>
<div>
</div>
</div>
<div>
[1]:<a href="https://lists.wikimedia.org/pipermail/labs-l/2016-October/004698.html" target="_blank"> https://lists.wikimedia.org/pipermail/labs-l/2016-October/004698.html</a><br />
</div>
</div>
<div class="gmail_extra">
<div>
<div class="h5">
<br />
<div class="gmail_quote">
On Mon, Nov 14, 2016 at 9:31 AM, Madhumitha Viswanathan<span> <<a href="mailto:mviswanathan@wikimedia.org" target="_blank">mviswanathan@wikimedia.org</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex;border-left: 1.0px rgb(204,204,204) solid;padding-left: 1.0ex;">
<div>
All of tools is down currently - unexpected side effect of the ongoing maintenance - we are working on it, I will update once it's back up. Apologies for the inconvenience caused.
</div>
<div class="gmail_extra">
<span><br />
<div class="gmail_quote">
On Mon, Nov 14, 2016 at 7:59 AM, Madhumitha Viswanathan<span> <<a href="mailto:mviswanathan@wikimedia.org" target="_blank">mviswanathan@wikimedia.org</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex;border-left: 1.0px rgb(204,204,204) solid;padding-left: 1.0ex;">
<div>
Hi all,
<div>
<br />
</div>
<div>
This is starting in a few minutes. You can follow along progress on #wikimedia-labs or at <a href="https://phabricator.wikimedia.org/T146154" style="font-size: 12.8px;" target="_blank">T146154</a>.
</div>
<div>
<br />
</div>
</div>
<div class="gmail_extra">
<span><br />
<div class="gmail_quote">
On Fri, Nov 11, 2016 at 10:16 AM, Madhumitha Viswanathan<span> <<a href="mailto:mviswanathan@wikimedia.org" target="_blank">mviswanathan@wikimedia.org</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex;border-left: 1.0px rgb(204,204,204) solid;padding-left: 1.0ex;">
<div>
Hi all,
<div>
<br />
</div>
<div>
This migration is scheduled to happened on Monday 11/14. We will start at 08:00 PST/16:00 UTC. More details here - <a href="https://phabricator.wikimedia.org/T146154" target="_blank">https://phabricator.wikimedia.org/T146154</a>.
</div>
</div>
<div class="gmail_extra">
<div>
<div class="m_8043260073835487680m_-7593300809719582187m_8861644900626944798h5">
<br />
<div class="gmail_quote">
On Tue, Nov 1, 2016 at 11:47 AM, Madhumitha Viswanathan<span> <<a href="mailto:mviswanathan@wikimedia.org" target="_blank">mviswanathan@wikimedia.org</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex;border-left: 1.0px rgb(204,204,204) solid;padding-left: 1.0ex;">
<div>
Hi all,
<div>
<br />
</div>
<div>
We had previously announced a disruptive maintenance for tools[1] on 11/2(tomorrow), but we are going to have to postpone it tentatively to 11/14/2016.
</div>
<div>
<br />
</div>
<div>
Apologies for the late notice, but the server in our secondary datacenter in Dallas that hosts data backups is having issues since yesterday[2], and we are currently still debugging and fixing the problems there. We've decided not to proceed with this migration until our backups there are online.
</div>
<div>
<br />
</div>
<div>
As always, do let us know if there are any questions/concerns here or on #wikimedia-labs.
</div>
<div>
<br />
</div>
<div>
[1]: <a href="https://lists.wikimedia.org/pipermail/labs-l/2016-October/004698.html" target="_blank">https://lists.wikimedia.org/pipermail/labs-l/2016-October/004698.html</a>
</div>
<div>
[2]: <a href="https://phabricator.wikimedia.org/T149567" target="_blank">https://phabricator.wikimedia.org/T149567</a>
</div>
</div>
</blockquote>
</div>
<br />
<br clear="all" />
<div>
<br />
</div>
</div>
</div>
<span class="m_8043260073835487680m_-7593300809719582187m_8861644900626944798HOEnZb"><font color="#888888">--<br />
<div class="m_8043260073835487680m_-7593300809719582187m_8861644900626944798m_-5170187048180779485gmail_signature">
<div>
--Madhu :)
</div>
</div>
</font></span>
</div>
</blockquote>
</div>
<br />
<br clear="all" />
<div>
<br />
</div>
</span><span class="m_8043260073835487680m_-7593300809719582187HOEnZb"><font color="#888888">--<br />
<div class="m_8043260073835487680m_-7593300809719582187m_8861644900626944798gmail_signature">
<div>
--Madhu :)
</div>
</div>
</font></span>
</div>
</blockquote>
</div>
<br />
<br clear="all" />
<div>
<br />
</div>
</span><span class="m_8043260073835487680HOEnZb"><font color="#888888">--<br />
<div class="m_8043260073835487680m_-7593300809719582187gmail_signature">
<div>
--Madhu :)
</div>
</div>
</font></span>
</div>
</blockquote>
</div>
<br />
<br clear="all" />
<div>
<br />
</div>
</div>
</div>
<span class="HOEnZb"><font color="#888888">--<br />
<div class="m_8043260073835487680gmail_signature">
<div>
--Madhu :)
</div>
</div>
</font></span>
</div>
</blockquote>
</div>
<br />
<br clear="all" />
<div>
<br />
</div>
--<br />
<div class="gmail_signature">
<div>
--Madhu :)
</div>
</div>
</div>
_______________________________________________ Labs-l mailing list Labs-l@lists.wikimedia.org<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank"> https://lists.wikimedia.org/mailman/listinfo/labs-l</a>
</blockquote></div></body></html></body></html>