[Engineering] [Analytics] Eventlogging Mysql consumers downtime

[Andrew Otto]

Backfilling is done on the master, but I’m not sure about slave resyncing.
I think it is mostly done, except for large tables.  Tracking this here:
https://phabricator.wikimedia.org/T120187

On Thu, Jan 28, 2016 at 7:34 AM, Marcel Ruiz Forns <mforns at wikimedia.org>
wrote:

> Yes, it has been completely backfilled.
>
> Cheers!
>
> On Thu, Jan 28, 2016 at 8:31 AM, Amir Aharoni <aaharoni at wikimedia.org>
> wrote:
>
>> Is the backfilling done? I see some data, but I can't be sure whether
>> it's complete.
>>
>> 2016-01-25 20:34 GMT+02:00 Nuria Ruiz <nuria at wikimedia.org>:
>>
>>> We are working at this time on backfilling data for the scheduled
>>> maintenance, we shall report when we think backlogging has finished.
>>>
>>> Thanks,
>>>
>>> Nuria
>>>
>>> On Mon, Jan 25, 2016 at 5:34 AM, Oliver Keyes <okeyes at wikimedia.org>
>>> wrote:
>>>
>>>> Awesome; thanks :). Anything on the switchon time? (I appreciate it's
>>>> only barely Monday ;p)
>>>>
>>>> On 21 January 2016 at 11:36, Andrew Otto <otto at wikimedia.org> wrote:
>>>> > FYI, This has started.  I have turned off the MySQL processes.  No
>>>> new data
>>>> > will be inserted into MySQL until we turn them back on.
>>>> >
>>>> > On Wed, Jan 20, 2016 at 4:43 PM, Andrew Otto <otto at wikimedia.org>
>>>> wrote:
>>>> >>
>>>> >> (+Jaime)
>>>> >>
>>>> >> It’s hard to tell how long it will take, but I imagine for quite a
>>>> while.
>>>> >> I guess a day max, but probably much less.  If Jaime let’s me know
>>>> that he’s
>>>> >> finished early, and I am close enough to a computer, I’ll turn the
>>>> >> EventLogging MySQL processes back on asap.
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, Jan 20, 2016 at 4:12 PM, Oliver Keyes <okeyes at wikimedia.org>
>>>> >> wrote:
>>>> >>>
>>>> >>> Thanks Madhu! Awesome to see the transparency and announces around
>>>> this
>>>> >>> :D.
>>>> >>>
>>>> >>> Do you know how long the reimport should take on Monday? I want to
>>>> >>> make sure I know when to expect we can safely launch all our
>>>> scripts.
>>>> >>>
>>>> >>> On 20 January 2016 at 13:08, Madhumitha Viswanathan
>>>> >>> <mviswanathan at wikimedia.org> wrote:
>>>> >>> > Hi all,
>>>> >>> >
>>>> >>> > In order to convert tables on db1046 to the TokuDB engine - we
>>>> have to
>>>> >>> > schedule some downtime on the Eventlogging databases between
>>>> tomorrow,
>>>> >>> > Thursday Jan 21, 2016 at 16:00 UTC to Monday Jan 25, 2016 16:00
>>>> UTC.
>>>> >>> >
>>>> >>> > What this means for EL users:
>>>> >>> >
>>>> >>> > 1. Eventlogging will still receive data and it will be available
>>>> in
>>>> >>> > Kafka.
>>>> >>> > The data will continue to be imported into Hadoop and files.
>>>> >>> > 2. The Mysql consumers of Eventlogging will be stopped, so no
>>>> data will
>>>> >>> > get
>>>> >>> > imported into the master(db1046 or m4-master) and by extension to
>>>> the
>>>> >>> > analytics-store.
>>>> >>> > 3. Querying existing data from analytics-store will still be
>>>> available,
>>>> >>> > but
>>>> >>> > data for the next 4 days won't be available.
>>>> >>> > 4. On Monday, after the maintenance window we'll restart the Mysql
>>>> >>> > consumers, and all the data should get reimported from Kafka.
>>>> >>> >
>>>> >>> > Analytics and Ops(DBA) will work on this together.
>>>> >>> >
>>>> >>> > Feel free to reach out to us here or on #wikimedia-analytics if
>>>> you
>>>> >>> > have any
>>>> >>> > concerns/questions.
>>>> >>> >
>>>> >>> >
>>>> >>> > -- Madhu Viswanathan
>>>> >>> > Software Engineer, Analytics
>>>> >>> >
>>>> >>> > _______________________________________________
>>>> >>> > Engineering mailing list
>>>> >>> > Engineering at lists.wikimedia.org
>>>> >>> > https://lists.wikimedia.org/mailman/listinfo/engineering
>>>> >>> >
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Oliver Keyes
>>>> >>> Count Logula
>>>> >>> Wikimedia Foundation
>>>> >>>
>>>> >>> _______________________________________________
>>>> >>> Engineering mailing list
>>>> >>> Engineering at lists.wikimedia.org
>>>> >>> https://lists.wikimedia.org/mailman/listinfo/engineering
>>>> >>
>>>> >>
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Oliver Keyes
>>>> Count Logula
>>>> Wikimedia Foundation
>>>>
>>>> _______________________________________________
>>>> Analytics mailing list
>>>> Analytics at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/analytics
>>>>
>>>
>>>
>>> _______________________________________________
>>> Engineering mailing list
>>> Engineering at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/engineering
>>>
>>>
>>
>>
>> --
>> Amir Elisha Aharoni‏ ። אָמִיר אֱלִישָׁע אַהֲרוֹנִי
>> Language Engineering‏ ። הַנְדָּסָה לְשׁוֹנִית
>> Wikimedia Foundation‏ ። קֶרֶן וִיקִימֶדְיָה
>>
>> _______________________________________________
>> Analytics mailing list
>> Analytics at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/analytics
>>
>>
>
>
> --
> *Marcel Ruiz Forns*
> Analytics Developer
> Wikimedia Foundation
>
> _______________________________________________
> Analytics mailing list
> Analytics at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/analytics
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/engineering/attachments/20160128/0a1228ce/attachment-0001.html>