[Labs-l] labsdb1002 scheduled downtime maintenance (was: labsdb1002 possible issues)

[Jaime Crespo]

Labs users,

We have completed the intended maintenance on labsdb1002, and *.labsdb
should point to the original place now or in a short period of time. You
can use that instance in the same way you use the others.

We will continue working on improving the service, but it should not create
more downtime in the foreseeable future.

Regards,

On Thu, Jun 11, 2015 at 8:27 AM, Jaime Crespo <jcrespo at wikimedia.org> wrote:

> Maintenance update:
>
> We have performed the desired maintenance, including an upgrade, reboot
> and filesystem check, and MariaDB is available again for connection. We
> believe to have fixed the origin of the problems.
>
> However, as the issues had caused some database integrity problems, we are
> refreshing and checking the production wikis data. Until further notice,
> later this week, I would recommend you to continue using labsdb1001 and
> labsdb1003 or you will run the risk of working with stale data.
>
> The user's own databases are available and are not affected by this.
>
> Thank you for your understanding,
>
> On Mon, Jun 8, 2015 at 4:46 PM, Yuvi Panda <yuvipanda at gmail.com> wrote:
>
>> We will redirect *.labsdb to point to another database before the
>> beginning of the window. However this might still affect you if you
>> have a user database on labsdb1002.
>>
>> On Mon, Jun 8, 2015 at 2:38 PM, Jaime Crespo <jcrespo at wikimedia.org>
>> wrote:
>> > As a follow up, labsdb1002 has been scheduled for downtime maintenance,
>> > starting on 2015-06-10 07:00 UTC and will be unavailable until further
>> > notice.
>> >
>> > While we expect that the disruption will take only 1 day or less, we are
>> > unsure if we will found more complicated issues during the outage that
>> would
>> > force us to prolong the maintenance.
>> >
>> > I would recommend to switch your queries before that date temporarily to
>> > lasbdb1001 or labsdb1003, which will not be affected by the
>> maintenance. I
>> > will send here again what is the status when the works are finished.
>> >
>> > Regards,
>> >
>> > On Sun, Jun 7, 2015 at 9:49 AM, Sean Pringle <springle at wikimedia.org>
>> wrote:
>> >>
>> >> https://phabricator.wikimedia.org/T101567
>> >>
>> >> labsdb1002 had a conniption, possibly related to XFS, and causing some
>> >> unsettling DB replication glitches. Full investigation and (probable)
>> >> resync yet to be done.
>> >>
>> >> So, heads up. If you spot anything odd, best to switch to one of the
>> >> other replicas: lasbdb1001 or labsdb1003
>> >>
>> >> -s
>> >>
>> >> _______________________________________________
>> >> Labs-l mailing list
>> >> Labs-l at lists.wikimedia.org
>> >> https://lists.wikimedia.org/mailman/listinfo/labs-l
>> >
>> >
>> >
>> >
>> > --
>> > Jaime Crespo
>> > <http://wikimedia.org>
>> >
>> > _______________________________________________
>> > Labs-l mailing list
>> > Labs-l at lists.wikimedia.org
>> > https://lists.wikimedia.org/mailman/listinfo/labs-l
>> >
>>
>>
>>
>> --
>> Yuvi Panda T
>> http://yuvi.in/blog
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>
>
>
> --
> Jaime Crespo
> <http://wikimedia.org>
>



-- 
Jaime Crespo
<http://wikimedia.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20150612/1f6ad0c6/attachment.html>