Hi all,
In order to convert tables on db1046 to the TokuDB engine - we have to schedule some downtime on the Eventlogging databases between tomorrow, Thursday Jan 21, 2016 at 16:00 UTC to Monday Jan 25, 2016 16:00 UTC.
What this means for EL users:
1. Eventlogging will still receive data and it will be available in Kafka. The data will continue to be imported into Hadoop and files. 2. The Mysql consumers of Eventlogging will be stopped, so no data will get imported into the master(db1046 or m4-master) and by extension to the analytics-store. 3. Querying existing data from analytics-store will still be available, but data for the next 4 days won't be available. 4. On Monday, after the maintenance window we'll restart the Mysql consumers, and all the data should get reimported from Kafka.
Analytics and Ops(DBA) will work on this together.
Feel free to reach out to us here or on #wikimedia-analytics if you have any concerns/questions.
-- Madhu Viswanathan Software Engineer, Analytics
Thanks Madhu! Awesome to see the transparency and announces around this :D.
Do you know how long the reimport should take on Monday? I want to make sure I know when to expect we can safely launch all our scripts.
On 20 January 2016 at 13:08, Madhumitha Viswanathan mviswanathan@wikimedia.org wrote:
Hi all,
In order to convert tables on db1046 to the TokuDB engine - we have to schedule some downtime on the Eventlogging databases between tomorrow, Thursday Jan 21, 2016 at 16:00 UTC to Monday Jan 25, 2016 16:00 UTC.
What this means for EL users:
- Eventlogging will still receive data and it will be available in Kafka.
The data will continue to be imported into Hadoop and files. 2. The Mysql consumers of Eventlogging will be stopped, so no data will get imported into the master(db1046 or m4-master) and by extension to the analytics-store. 3. Querying existing data from analytics-store will still be available, but data for the next 4 days won't be available. 4. On Monday, after the maintenance window we'll restart the Mysql consumers, and all the data should get reimported from Kafka.
Analytics and Ops(DBA) will work on this together.
Feel free to reach out to us here or on #wikimedia-analytics if you have any concerns/questions.
-- Madhu Viswanathan Software Engineer, Analytics
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
(+Jaime)
It’s hard to tell how long it will take, but I imagine for quite a while. I guess a day max, but probably much less. If Jaime let’s me know that he’s finished early, and I am close enough to a computer, I’ll turn the EventLogging MySQL processes back on asap.
On Wed, Jan 20, 2016 at 4:12 PM, Oliver Keyes okeyes@wikimedia.org wrote:
Thanks Madhu! Awesome to see the transparency and announces around this :D.
Do you know how long the reimport should take on Monday? I want to make sure I know when to expect we can safely launch all our scripts.
On 20 January 2016 at 13:08, Madhumitha Viswanathan mviswanathan@wikimedia.org wrote:
Hi all,
In order to convert tables on db1046 to the TokuDB engine - we have to schedule some downtime on the Eventlogging databases between tomorrow, Thursday Jan 21, 2016 at 16:00 UTC to Monday Jan 25, 2016 16:00 UTC.
What this means for EL users:
- Eventlogging will still receive data and it will be available in
Kafka.
The data will continue to be imported into Hadoop and files. 2. The Mysql consumers of Eventlogging will be stopped, so no data will
get
imported into the master(db1046 or m4-master) and by extension to the analytics-store. 3. Querying existing data from analytics-store will still be available,
but
data for the next 4 days won't be available. 4. On Monday, after the maintenance window we'll restart the Mysql consumers, and all the data should get reimported from Kafka.
Analytics and Ops(DBA) will work on this together.
Feel free to reach out to us here or on #wikimedia-analytics if you have
any
concerns/questions.
-- Madhu Viswanathan Software Engineer, Analytics
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Oliver Keyes Count Logula Wikimedia Foundation
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
FYI, This has started. I have turned off the MySQL processes. No new data will be inserted into MySQL until we turn them back on.
On Wed, Jan 20, 2016 at 4:43 PM, Andrew Otto otto@wikimedia.org wrote:
(+Jaime)
It’s hard to tell how long it will take, but I imagine for quite a while. I guess a day max, but probably much less. If Jaime let’s me know that he’s finished early, and I am close enough to a computer, I’ll turn the EventLogging MySQL processes back on asap.
On Wed, Jan 20, 2016 at 4:12 PM, Oliver Keyes okeyes@wikimedia.org wrote:
Thanks Madhu! Awesome to see the transparency and announces around this :D.
Do you know how long the reimport should take on Monday? I want to make sure I know when to expect we can safely launch all our scripts.
On 20 January 2016 at 13:08, Madhumitha Viswanathan mviswanathan@wikimedia.org wrote:
Hi all,
In order to convert tables on db1046 to the TokuDB engine - we have to schedule some downtime on the Eventlogging databases between tomorrow, Thursday Jan 21, 2016 at 16:00 UTC to Monday Jan 25, 2016 16:00 UTC.
What this means for EL users:
- Eventlogging will still receive data and it will be available in
Kafka.
The data will continue to be imported into Hadoop and files. 2. The Mysql consumers of Eventlogging will be stopped, so no data will
get
imported into the master(db1046 or m4-master) and by extension to the analytics-store. 3. Querying existing data from analytics-store will still be available,
but
data for the next 4 days won't be available. 4. On Monday, after the maintenance window we'll restart the Mysql consumers, and all the data should get reimported from Kafka.
Analytics and Ops(DBA) will work on this together.
Feel free to reach out to us here or on #wikimedia-analytics if you
have any
concerns/questions.
-- Madhu Viswanathan Software Engineer, Analytics
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Oliver Keyes Count Logula Wikimedia Foundation
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Awesome; thanks :). Anything on the switchon time? (I appreciate it's only barely Monday ;p)
On 21 January 2016 at 11:36, Andrew Otto otto@wikimedia.org wrote:
FYI, This has started. I have turned off the MySQL processes. No new data will be inserted into MySQL until we turn them back on.
On Wed, Jan 20, 2016 at 4:43 PM, Andrew Otto otto@wikimedia.org wrote:
(+Jaime)
It’s hard to tell how long it will take, but I imagine for quite a while. I guess a day max, but probably much less. If Jaime let’s me know that he’s finished early, and I am close enough to a computer, I’ll turn the EventLogging MySQL processes back on asap.
On Wed, Jan 20, 2016 at 4:12 PM, Oliver Keyes okeyes@wikimedia.org wrote:
Thanks Madhu! Awesome to see the transparency and announces around this :D.
Do you know how long the reimport should take on Monday? I want to make sure I know when to expect we can safely launch all our scripts.
On 20 January 2016 at 13:08, Madhumitha Viswanathan mviswanathan@wikimedia.org wrote:
Hi all,
In order to convert tables on db1046 to the TokuDB engine - we have to schedule some downtime on the Eventlogging databases between tomorrow, Thursday Jan 21, 2016 at 16:00 UTC to Monday Jan 25, 2016 16:00 UTC.
What this means for EL users:
- Eventlogging will still receive data and it will be available in
Kafka. The data will continue to be imported into Hadoop and files. 2. The Mysql consumers of Eventlogging will be stopped, so no data will get imported into the master(db1046 or m4-master) and by extension to the analytics-store. 3. Querying existing data from analytics-store will still be available, but data for the next 4 days won't be available. 4. On Monday, after the maintenance window we'll restart the Mysql consumers, and all the data should get reimported from Kafka.
Analytics and Ops(DBA) will work on this together.
Feel free to reach out to us here or on #wikimedia-analytics if you have any concerns/questions.
-- Madhu Viswanathan Software Engineer, Analytics
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Oliver Keyes Count Logula Wikimedia Foundation
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
We are working at this time on backfilling data for the scheduled maintenance, we shall report when we think backlogging has finished.
Thanks,
Nuria
On Mon, Jan 25, 2016 at 5:34 AM, Oliver Keyes okeyes@wikimedia.org wrote:
Awesome; thanks :). Anything on the switchon time? (I appreciate it's only barely Monday ;p)
On 21 January 2016 at 11:36, Andrew Otto otto@wikimedia.org wrote:
FYI, This has started. I have turned off the MySQL processes. No new
data
will be inserted into MySQL until we turn them back on.
On Wed, Jan 20, 2016 at 4:43 PM, Andrew Otto otto@wikimedia.org wrote:
(+Jaime)
It’s hard to tell how long it will take, but I imagine for quite a
while.
I guess a day max, but probably much less. If Jaime let’s me know that
he’s
finished early, and I am close enough to a computer, I’ll turn the EventLogging MySQL processes back on asap.
On Wed, Jan 20, 2016 at 4:12 PM, Oliver Keyes okeyes@wikimedia.org wrote:
Thanks Madhu! Awesome to see the transparency and announces around this :D.
Do you know how long the reimport should take on Monday? I want to make sure I know when to expect we can safely launch all our scripts.
On 20 January 2016 at 13:08, Madhumitha Viswanathan mviswanathan@wikimedia.org wrote:
Hi all,
In order to convert tables on db1046 to the TokuDB engine - we have
to
schedule some downtime on the Eventlogging databases between
tomorrow,
Thursday Jan 21, 2016 at 16:00 UTC to Monday Jan 25, 2016 16:00 UTC.
What this means for EL users:
- Eventlogging will still receive data and it will be available in
Kafka. The data will continue to be imported into Hadoop and files. 2. The Mysql consumers of Eventlogging will be stopped, so no data
will
get imported into the master(db1046 or m4-master) and by extension to the analytics-store. 3. Querying existing data from analytics-store will still be
available,
but data for the next 4 days won't be available. 4. On Monday, after the maintenance window we'll restart the Mysql consumers, and all the data should get reimported from Kafka.
Analytics and Ops(DBA) will work on this together.
Feel free to reach out to us here or on #wikimedia-analytics if you have any concerns/questions.
-- Madhu Viswanathan Software Engineer, Analytics
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Oliver Keyes Count Logula Wikimedia Foundation
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Oliver Keyes Count Logula Wikimedia Foundation
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Is the backfilling done? I see some data, but I can't be sure whether it's complete.
2016-01-25 20:34 GMT+02:00 Nuria Ruiz nuria@wikimedia.org:
We are working at this time on backfilling data for the scheduled maintenance, we shall report when we think backlogging has finished.
Thanks,
Nuria
On Mon, Jan 25, 2016 at 5:34 AM, Oliver Keyes okeyes@wikimedia.org wrote:
Awesome; thanks :). Anything on the switchon time? (I appreciate it's only barely Monday ;p)
On 21 January 2016 at 11:36, Andrew Otto otto@wikimedia.org wrote:
FYI, This has started. I have turned off the MySQL processes. No new
data
will be inserted into MySQL until we turn them back on.
On Wed, Jan 20, 2016 at 4:43 PM, Andrew Otto otto@wikimedia.org
wrote:
(+Jaime)
It’s hard to tell how long it will take, but I imagine for quite a
while.
I guess a day max, but probably much less. If Jaime let’s me know
that he’s
finished early, and I am close enough to a computer, I’ll turn the EventLogging MySQL processes back on asap.
On Wed, Jan 20, 2016 at 4:12 PM, Oliver Keyes okeyes@wikimedia.org wrote:
Thanks Madhu! Awesome to see the transparency and announces around
this
:D.
Do you know how long the reimport should take on Monday? I want to make sure I know when to expect we can safely launch all our scripts.
On 20 January 2016 at 13:08, Madhumitha Viswanathan mviswanathan@wikimedia.org wrote:
Hi all,
In order to convert tables on db1046 to the TokuDB engine - we have
to
schedule some downtime on the Eventlogging databases between
tomorrow,
Thursday Jan 21, 2016 at 16:00 UTC to Monday Jan 25, 2016 16:00 UTC.
What this means for EL users:
- Eventlogging will still receive data and it will be available in
Kafka. The data will continue to be imported into Hadoop and files. 2. The Mysql consumers of Eventlogging will be stopped, so no data
will
get imported into the master(db1046 or m4-master) and by extension to
the
analytics-store. 3. Querying existing data from analytics-store will still be
available,
but data for the next 4 days won't be available. 4. On Monday, after the maintenance window we'll restart the Mysql consumers, and all the data should get reimported from Kafka.
Analytics and Ops(DBA) will work on this together.
Feel free to reach out to us here or on #wikimedia-analytics if you have any concerns/questions.
-- Madhu Viswanathan Software Engineer, Analytics
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Oliver Keyes Count Logula Wikimedia Foundation
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Oliver Keyes Count Logula Wikimedia Foundation
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Yes, it has been completely backfilled.
Cheers!
On Thu, Jan 28, 2016 at 8:31 AM, Amir Aharoni aaharoni@wikimedia.org wrote:
Is the backfilling done? I see some data, but I can't be sure whether it's complete.
2016-01-25 20:34 GMT+02:00 Nuria Ruiz nuria@wikimedia.org:
We are working at this time on backfilling data for the scheduled maintenance, we shall report when we think backlogging has finished.
Thanks,
Nuria
On Mon, Jan 25, 2016 at 5:34 AM, Oliver Keyes okeyes@wikimedia.org wrote:
Awesome; thanks :). Anything on the switchon time? (I appreciate it's only barely Monday ;p)
On 21 January 2016 at 11:36, Andrew Otto otto@wikimedia.org wrote:
FYI, This has started. I have turned off the MySQL processes. No new
data
will be inserted into MySQL until we turn them back on.
On Wed, Jan 20, 2016 at 4:43 PM, Andrew Otto otto@wikimedia.org
wrote:
(+Jaime)
It’s hard to tell how long it will take, but I imagine for quite a
while.
I guess a day max, but probably much less. If Jaime let’s me know
that he’s
finished early, and I am close enough to a computer, I’ll turn the EventLogging MySQL processes back on asap.
On Wed, Jan 20, 2016 at 4:12 PM, Oliver Keyes okeyes@wikimedia.org wrote:
Thanks Madhu! Awesome to see the transparency and announces around
this
:D.
Do you know how long the reimport should take on Monday? I want to make sure I know when to expect we can safely launch all our scripts.
On 20 January 2016 at 13:08, Madhumitha Viswanathan mviswanathan@wikimedia.org wrote: > Hi all, > > In order to convert tables on db1046 to the TokuDB engine - we
have to
> schedule some downtime on the Eventlogging databases between
tomorrow,
> Thursday Jan 21, 2016 at 16:00 UTC to Monday Jan 25, 2016 16:00
UTC.
> > What this means for EL users: > > 1. Eventlogging will still receive data and it will be available in > Kafka. > The data will continue to be imported into Hadoop and files. > 2. The Mysql consumers of Eventlogging will be stopped, so no data
will
> get > imported into the master(db1046 or m4-master) and by extension to
the
> analytics-store. > 3. Querying existing data from analytics-store will still be
available,
> but > data for the next 4 days won't be available. > 4. On Monday, after the maintenance window we'll restart the Mysql > consumers, and all the data should get reimported from Kafka. > > Analytics and Ops(DBA) will work on this together. > > Feel free to reach out to us here or on #wikimedia-analytics if you > have any > concerns/questions. > > > -- Madhu Viswanathan > Software Engineer, Analytics > > _______________________________________________ > Engineering mailing list > Engineering@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/engineering >
-- Oliver Keyes Count Logula Wikimedia Foundation
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Oliver Keyes Count Logula Wikimedia Foundation
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Amir Elisha Aharoni ። אָמִיר אֱלִישָׁע אַהֲרוֹנִי Language Engineering ። הַנְדָּסָה לְשׁוֹנִית Wikimedia Foundation ። קֶרֶן וִיקִימֶדְיָה
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Backfilling is done on the master, but I’m not sure about slave resyncing. I think it is mostly done, except for large tables. Tracking this here: https://phabricator.wikimedia.org/T120187
On Thu, Jan 28, 2016 at 7:34 AM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
Yes, it has been completely backfilled.
Cheers!
On Thu, Jan 28, 2016 at 8:31 AM, Amir Aharoni aaharoni@wikimedia.org wrote:
Is the backfilling done? I see some data, but I can't be sure whether it's complete.
2016-01-25 20:34 GMT+02:00 Nuria Ruiz nuria@wikimedia.org:
We are working at this time on backfilling data for the scheduled maintenance, we shall report when we think backlogging has finished.
Thanks,
Nuria
On Mon, Jan 25, 2016 at 5:34 AM, Oliver Keyes okeyes@wikimedia.org wrote:
Awesome; thanks :). Anything on the switchon time? (I appreciate it's only barely Monday ;p)
On 21 January 2016 at 11:36, Andrew Otto otto@wikimedia.org wrote:
FYI, This has started. I have turned off the MySQL processes. No
new data
will be inserted into MySQL until we turn them back on.
On Wed, Jan 20, 2016 at 4:43 PM, Andrew Otto otto@wikimedia.org
wrote:
(+Jaime)
It’s hard to tell how long it will take, but I imagine for quite a
while.
I guess a day max, but probably much less. If Jaime let’s me know
that he’s
finished early, and I am close enough to a computer, I’ll turn the EventLogging MySQL processes back on asap.
On Wed, Jan 20, 2016 at 4:12 PM, Oliver Keyes okeyes@wikimedia.org wrote: > > Thanks Madhu! Awesome to see the transparency and announces around
this
> :D. > > Do you know how long the reimport should take on Monday? I want to > make sure I know when to expect we can safely launch all our
scripts.
> > On 20 January 2016 at 13:08, Madhumitha Viswanathan > mviswanathan@wikimedia.org wrote: > > Hi all, > > > > In order to convert tables on db1046 to the TokuDB engine - we
have to
> > schedule some downtime on the Eventlogging databases between
tomorrow,
> > Thursday Jan 21, 2016 at 16:00 UTC to Monday Jan 25, 2016 16:00
UTC.
> > > > What this means for EL users: > > > > 1. Eventlogging will still receive data and it will be available
in
> > Kafka. > > The data will continue to be imported into Hadoop and files. > > 2. The Mysql consumers of Eventlogging will be stopped, so no
data will
> > get > > imported into the master(db1046 or m4-master) and by extension to
the
> > analytics-store. > > 3. Querying existing data from analytics-store will still be
available,
> > but > > data for the next 4 days won't be available. > > 4. On Monday, after the maintenance window we'll restart the Mysql > > consumers, and all the data should get reimported from Kafka. > > > > Analytics and Ops(DBA) will work on this together. > > > > Feel free to reach out to us here or on #wikimedia-analytics if
you
> > have any > > concerns/questions. > > > > > > -- Madhu Viswanathan > > Software Engineer, Analytics > > > > _______________________________________________ > > Engineering mailing list > > Engineering@lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/engineering > > > > > > -- > Oliver Keyes > Count Logula > Wikimedia Foundation > > _______________________________________________ > Engineering mailing list > Engineering@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/engineering
-- Oliver Keyes Count Logula Wikimedia Foundation
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Amir Elisha Aharoni ። אָמִיר אֱלִישָׁע אַהֲרוֹנִי Language Engineering ። הַנְדָּסָה לְשׁוֹנִית Wikimedia Foundation ። קֶרֶן וִיקִימֶדְיָה
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- *Marcel Ruiz Forns* Analytics Developer Wikimedia Foundation
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics