[Labs-admin] Labsdb migration (was: Report from ops meeting)

[Yuvi Panda]

Anyone object?

On Tue, Jan 31, 2017 at 12:58 PM, Jaime Crespo <jcrespo at wikimedia.org> wrote:
> Ok, too.
>
> On Tue, Jan 31, 2017 at 9:52 PM, Yuvi Panda <yuvipanda at gmail.com> wrote:
>>
>> Actually I've just been told that Feb 14 is valentine's day and I
>> might be tasked with other duties on that day. Sorry! Feb 15?
>>
>> On Mon, Jan 30, 2017 at 10:57 AM, Jaime Crespo <jcrespo at wikimedia.org>
>> wrote:
>> > Ok to me.
>> >
>> > On Mon, Jan 30, 2017 at 7:54 PM, Yuvi Panda <yuvipanda at gmail.com> wrote:
>> >>
>> >> How about Feb 14? That gives us two weeks.
>> >>
>> >> On Mon, Jan 30, 2017 at 10:33 AM, Jaime Crespo <jcrespo at wikimedia.org>
>> >> wrote:
>> >> > As an admin, everything you should know about the upcoming labsdb1005
>> >> > reimage:
>> >> >
>> >> > * We are ready (DBAs) to do this at any time, we just need to tell
>> >> > users
>> >> > in
>> >> > advance of potential outages/degradations of service
>> >> > * For 99% of the users, we will just switchover them transparently to
>> >> > the
>> >> > slave (should not cause issues). As usual, if their application does
>> >> > not
>> >> > retry to reconnect, there will be problems.
>> >> > * For 3 users (databases), there will be full outage because they
>> >> > have
>> >> > such
>> >> > a heavy usage that we cannot replicate them in real time. They were
>> >> > made
>> >> > aware of this limitation months ago, so it should not come as a
>> >> > surprise:
>> >> > https://phabricator.wikimedia.org/T127164 The users's databases are
>> >> > documented at:
>> >> >
>> >> >
>> >> > https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/templates/mariadb/tools.my.cnf.erb;f21ce599fe626e7c96010a5d0335370ebe510ca7$65
>> >> > * Data will be copied away, server will be reimaged, then data will
>> >> > be
>> >> > copied back That normally takes 3 hours, but things could go wrong...
>> >> > * People could complain for a 10.0 upgrade (?). But some people
>> >> > actually
>> >> > complained already for the lack of 5.5 -> 10 upgrade.
>> >> > https://phabricator.wikimedia.org/T138517#2796682
>> >> > * On switch-back, again bad-programmed application may temporarily
>> >> > fail,
>> >> > but
>> >> > good ones should just switch transparently; unavailable dbs should be
>> >> > available again
>> >> >
>> >> > That should be enough background to schedule and send an email to
>> >> > users
>> >> > :-)
>> >> >
>> >> > ---------- Forwarded message ----------
>> >> > From: Yuvi Panda <yuvipanda at gmail.com>
>> >> > Date: Mon, Jan 30, 2017 at 7:14 PM
>> >> > Subject: [Labs-admin] Report from ops meeting
>> >> > To: Labs admin list for infrastructure and discussion
>> >> > <labs-admin at lists.wikimedia.org>
>> >> >
>> >> >
>> >> > 1. Faidon talking about ip space discussions wrt asia dc discussion,
>> >> > and mentioned we might / should renumber labs IP space. Not sure
>> >> > about
>> >> > more details.
>> >> > 2. Ping on labsdb migration to Jessie
>> >> > 3. Mid-year review of annual goals coming up, need status about OGE
>> >> > migration
>> >> >
>> >> > That's it.
>> >> >
>> >> > --
>> >> > Yuvi Panda T
>> >> > http://yuvi.in/blog
>> >> >
>> >> > _______________________________________________
>> >> > Labs-admin mailing list
>> >> > Labs-admin at lists.wikimedia.org
>> >> > https://lists.wikimedia.org/mailman/listinfo/labs-admin
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Jaime Crespo
>> >> > <http://wikimedia.org>
>> >> >
>> >> > _______________________________________________
>> >> > Labs-admin mailing list
>> >> > Labs-admin at lists.wikimedia.org
>> >> > https://lists.wikimedia.org/mailman/listinfo/labs-admin
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Yuvi Panda T
>> >> http://yuvi.in/blog
>> >>
>> >> _______________________________________________
>> >> Labs-admin mailing list
>> >> Labs-admin at lists.wikimedia.org
>> >> https://lists.wikimedia.org/mailman/listinfo/labs-admin
>> >
>> >
>> >
>> >
>> > --
>> > Jaime Crespo
>> > <http://wikimedia.org>
>>
>>
>>
>> --
>> Yuvi Panda T
>> http://yuvi.in/blog
>
>
>
>
> --
> Jaime Crespo
> <http://wikimedia.org>



-- 
Yuvi Panda T
http://yuvi.in/blog