[Labs-admin] Labsdb migration (was: Report from ops meeting)

Mon Feb 6 05:57:48 UTC 2017

I emailed the announcement!

On Tue, Jan 31, 2017 at 11:27 PM, Manuel Arostegui
<marostegui at wikimedia.org> wrote:
>
>
> On Tue, Jan 31, 2017 at 11:07 PM, Yuvi Panda <yuvipanda at gmail.com> wrote:
>>
>> I've drafted https://etherpad.wikimedia.org/p/toolsdb-upgrade. Things
>> to get from DBA:
>>
>> 1. Time - does 5PM UTC (which will be 9AM PST) ok, or do you want it
>> to be earlier? We can probably make it earlier if chase or andrew or
>> bd808 (who are on an earlier TZ) can be around, or if the DBAs are ok
>> with doing this without us being around. Worst case I can wake up
>> really early :)
>
>
> 5PM UTC works for me!
>
>>
>> 2. Total duration - I've said '6h' as a conservative estimate. Too
>> much / too little?
>>
>
>
> We'd need to:
> - copy the data somewhere (it is 1.6T, so I guess it will take around
> 1.40-1.45h, let's make it 2h)
> - reimage (let's say 15-20 minutes?)
> - copy the data back (again, I guess we can make it 2h to be on the safe
> side)
>
> I would say 6h is enough, but I have no idea how many unexpected things we
> are afraid to find here. As per my chats with Yuvi, many :-)
> So 6h sounds reasonable to me!
>
> Just my opinion!
> Thanks!!
>
> Manuel.
>
>
>>
>> Thanks!
>>
>>
>>
>> On Tue, Jan 31, 2017 at 12:59 PM, Yuvi Panda <yuvipanda at gmail.com> wrote:
>> > Anyone object?
>> >
>> > On Tue, Jan 31, 2017 at 12:58 PM, Jaime Crespo <jcrespo at wikimedia.org>
>> > wrote:
>> >> Ok, too.
>> >>
>> >> On Tue, Jan 31, 2017 at 9:52 PM, Yuvi Panda <yuvipanda at gmail.com>
>> >> wrote:
>> >>>
>> >>> Actually I've just been told that Feb 14 is valentine's day and I
>> >>> might be tasked with other duties on that day. Sorry! Feb 15?
>> >>>
>> >>> On Mon, Jan 30, 2017 at 10:57 AM, Jaime Crespo <jcrespo at wikimedia.org>
>> >>> wrote:
>> >>> > Ok to me.
>> >>> >
>> >>> > On Mon, Jan 30, 2017 at 7:54 PM, Yuvi Panda <yuvipanda at gmail.com>
>> >>> > wrote:
>> >>> >>
>> >>> >> How about Feb 14? That gives us two weeks.
>> >>> >>
>> >>> >> On Mon, Jan 30, 2017 at 10:33 AM, Jaime Crespo
>> >>> >> <jcrespo at wikimedia.org>
>> >>> >> wrote:
>> >>> >> > As an admin, everything you should know about the upcoming
>> >>> >> > labsdb1005
>> >>> >> > reimage:
>> >>> >> >
>> >>> >> > * We are ready (DBAs) to do this at any time, we just need to
>> >>> >> > tell
>> >>> >> > users
>> >>> >> > in
>> >>> >> > advance of potential outages/degradations of service
>> >>> >> > * For 99% of the users, we will just switchover them
>> >>> >> > transparently to
>> >>> >> > the
>> >>> >> > slave (should not cause issues). As usual, if their application
>> >>> >> > does
>> >>> >> > not
>> >>> >> > retry to reconnect, there will be problems.
>> >>> >> > * For 3 users (databases), there will be full outage because they
>> >>> >> > have
>> >>> >> > such
>> >>> >> > a heavy usage that we cannot replicate them in real time. They
>> >>> >> > were
>> >>> >> > made
>> >>> >> > aware of this limitation months ago, so it should not come as a
>> >>> >> > surprise:
>> >>> >> > https://phabricator.wikimedia.org/T127164 The users's databases
>> >>> >> > are
>> >>> >> > documented at:
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> > https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/templates/mariadb/tools.my.cnf.erb;f21ce599fe626e7c96010a5d0335370ebe510ca7$65
>> >>> >> > * Data will be copied away, server will be reimaged, then data
>> >>> >> > will
>> >>> >> > be
>> >>> >> > copied back That normally takes 3 hours, but things could go
>> >>> >> > wrong...
>> >>> >> > * People could complain for a 10.0 upgrade (?). But some people
>> >>> >> > actually
>> >>> >> > complained already for the lack of 5.5 -> 10 upgrade.
>> >>> >> > https://phabricator.wikimedia.org/T138517#2796682
>> >>> >> > * On switch-back, again bad-programmed application may
>> >>> >> > temporarily
>> >>> >> > fail,
>> >>> >> > but
>> >>> >> > good ones should just switch transparently; unavailable dbs
>> >>> >> > should be
>> >>> >> > available again
>> >>> >> >
>> >>> >> > That should be enough background to schedule and send an email to
>> >>> >> > users
>> >>> >> > :-)
>> >>> >> >
>> >>> >> > ---------- Forwarded message ----------
>> >>> >> > From: Yuvi Panda <yuvipanda at gmail.com>
>> >>> >> > Date: Mon, Jan 30, 2017 at 7:14 PM
>> >>> >> > Subject: [Labs-admin] Report from ops meeting
>> >>> >> > To: Labs admin list for infrastructure and discussion
>> >>> >> > <labs-admin at lists.wikimedia.org>
>> >>> >> >
>> >>> >> >
>> >>> >> > 1. Faidon talking about ip space discussions wrt asia dc
>> >>> >> > discussion,
>> >>> >> > and mentioned we might / should renumber labs IP space. Not sure
>> >>> >> > about
>> >>> >> > more details.
>> >>> >> > 2. Ping on labsdb migration to Jessie
>> >>> >> > 3. Mid-year review of annual goals coming up, need status about
>> >>> >> > OGE
>> >>> >> > migration
>> >>> >> >
>> >>> >> > That's it.
>> >>> >> >
>> >>> >> > --
>> >>> >> > Yuvi Panda T
>> >>> >> > http://yuvi.in/blog
>> >>> >> >
>> >>> >> > _______________________________________________
>> >>> >> > Labs-admin mailing list
>> >>> >> > Labs-admin at lists.wikimedia.org
>> >>> >> > https://lists.wikimedia.org/mailman/listinfo/labs-admin
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> > --
>> >>> >> > Jaime Crespo
>> >>> >> > <http://wikimedia.org>
>> >>> >> >
>> >>> >> > _______________________________________________
>> >>> >> > Labs-admin mailing list
>> >>> >> > Labs-admin at lists.wikimedia.org
>> >>> >> > https://lists.wikimedia.org/mailman/listinfo/labs-admin
>> >>> >> >
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Yuvi Panda T
>> >>> >> http://yuvi.in/blog
>> >>> >>
>> >>> >> _______________________________________________
>> >>> >> Labs-admin mailing list
>> >>> >> Labs-admin at lists.wikimedia.org
>> >>> >> https://lists.wikimedia.org/mailman/listinfo/labs-admin
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Jaime Crespo
>> >>> > <http://wikimedia.org>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Yuvi Panda T
>> >>> http://yuvi.in/blog
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Jaime Crespo
>> >> <http://wikimedia.org>
>> >
>> >
>> >
>> > --
>> > Yuvi Panda T
>> > http://yuvi.in/blog
>>
>>
>>
>> --
>> Yuvi Panda T
>> http://yuvi.in/blog
>
>

-- 
Yuvi Panda T
http://yuvi.in/blog