[Labs-l] Tools NFS disruptive maintenance scheduled on 11/2 is being postponed to 11/14

[Madhumitha Viswanathan]

Hi all,

Tools is now back to read write, and all of tools should work as expected.
Webservice logs aren't being written to disk currently, and will not be
written for ~12 more hours, while we make sure everything else is working
alright.

The storage infrastructure is brand new, so we recommend being a bit gentle
for the next few days, as we monitor performance and make sure it's all
okay. Do let us know if things aren't working like you'd expect, on
#wikimedia-labs/on list or phabricator.

Thanks for being patient through this long maintenance window!


On Mon, Nov 14, 2016 at 10:15 AM, Madhumitha Viswanathan <
mviswanathan at wikimedia.org> wrote:

> Most of tools should be back up now. Expected behavior for the next many
> hours (Expected ~12-14 hours based on last sync of tools data, but might be
> much longer)
>
> Quoting from previous announcement [1]:
>
> * The tools NFS share (/data/project and /home) will be read-only for the
>> duration of the maintenance, so no new data or logs will get written to it.
>> * New jobs cannot be submitted for the whole maintenance window - this
>> means submitting jobs through cron or tools-mail will not function,
>> although tools-mail can continue to send emails.
>> * Current jobs might keep running, but won't get rescheduled if they die.
>> If they do not die and aren't writing to NFS they should be fine.
>
>
> [1]: https://lists.wikimedia.org/pipermail/labs-l/2016-October/004698.html
>
> On Mon, Nov 14, 2016 at 9:31 AM, Madhumitha Viswanathan <
> mviswanathan at wikimedia.org> wrote:
>
>> All of tools is down currently - unexpected side effect of the ongoing
>> maintenance - we are working on it, I will update once it's back up.
>> Apologies for the inconvenience caused.
>>
>> On Mon, Nov 14, 2016 at 7:59 AM, Madhumitha Viswanathan <
>> mviswanathan at wikimedia.org> wrote:
>>
>>> Hi all,
>>>
>>> This is starting in a few minutes. You can follow along progress on
>>> #wikimedia-labs or at T146154
>>> <https://phabricator.wikimedia.org/T146154>.
>>>
>>>
>>> On Fri, Nov 11, 2016 at 10:16 AM, Madhumitha Viswanathan <
>>> mviswanathan at wikimedia.org> wrote:
>>>
>>>> Hi all,
>>>>
>>>> This migration is scheduled to happened on Monday 11/14. We will start
>>>> at 08:00 PST/16:00 UTC. More details here -
>>>> https://phabricator.wikimedia.org/T146154.
>>>>
>>>> On Tue, Nov 1, 2016 at 11:47 AM, Madhumitha Viswanathan <
>>>> mviswanathan at wikimedia.org> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> We had previously announced a disruptive maintenance for tools[1] on
>>>>> 11/2(tomorrow), but we are going to have to postpone it tentatively to
>>>>> 11/14/2016.
>>>>>
>>>>> Apologies for the late notice, but the server in our secondary
>>>>> datacenter in Dallas that hosts data backups is having issues since
>>>>> yesterday[2], and we are currently still debugging and fixing the problems
>>>>> there. We've decided not to proceed with this migration until our backups
>>>>> there are online.
>>>>>
>>>>> As always, do let us know if there are any questions/concerns here or
>>>>> on #wikimedia-labs.
>>>>>
>>>>> [1]: https://lists.wikimedia.org/pipermail/labs-l/2016-Octob
>>>>> er/004698.html
>>>>> [2]: https://phabricator.wikimedia.org/T149567
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --Madhu :)
>>>>
>>>
>>>
>>>
>>> --
>>> --Madhu :)
>>>
>>
>>
>>
>> --
>> --Madhu :)
>>
>
>
>
> --
> --Madhu :)
>



-- 
--Madhu :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20161114/8586cf28/attachment-0001.html>