There seems to be issues with the Edit_13457736 table on the db log, on the db1047 host.
db1047 may be known better by some of you as you as "s1-analytics-slave" or "analytics-slave". You can use dbstore1002 for the time being ("analytics-store" or "sX-analytics-slave", where X is 2-7) to read from that table. Other "schemas" are not affected.
I am fixing those right now, but it will take some time, as I may have to handle (again) the recently purged rows on that table. Worst case scenario, I may have to do a quick reboot of that machine.
I will update the status soon to communicate next steps.
Thanks for the heads up.
On Tue, Mar 29, 2016 at 3:44 AM, Jaime Crespo jcrespo@wikimedia.org wrote:
There seems to be issues with the Edit_13457736 table on the db log, on the db1047 host.
db1047 may be known better by some of you as you as "s1-analytics-slave" or "analytics-slave". You can use dbstore1002 for the time being ("analytics-store" or "sX-analytics-slave", where X is 2-7) to read from that table. Other "schemas" are not affected.
I am fixing those right now, but it will take some time, as I may have to handle (again) the recently purged rows on that table. Worst case scenario, I may have to do a quick reboot of that machine.
I will update the status soon to communicate next steps.
-- Jaime Crespo http://wikimedia.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On Tue, Mar 29, 2016 at 3:44 AM, Jaime Crespo jcrespo@wikimedia.org wrote:
There seems to be issues with the Edit_13457736 table on the db log, on the db1047 host.
db1047 may be known better by some of you as you as "s1-analytics-slave" or "analytics-slave". You can use dbstore1002 for the time being ("analytics-store" or "sX-analytics-slave", where X is 2-7) to read from that table. Other "schemas" are not affected.
I am fixing those right now, but it will take some time, as I may have to handle (again) the recently purged rows on that table. Worst case scenario, I may have to do a quick reboot of that machine.
Did that happen? I just had a query on that server abort with "ERROR 1053 (08S01) at line 5: Server shutdown in progress". It had been running since Sunday :(
I will update the status soon to communicate next steps.
-- Jaime Crespo http://wikimedia.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
You can call it restart, or you can call it (un)controlled crash. The data issues where affecting the whole server, and I had to drop the previously mentioned table (which mysql refused to do while running).
Things seem back to normal now, but of course tables will take some time to get refilled, specially Edit_13457736.
On Wed, Mar 30, 2016 at 10:27 AM, Tilman Bayer tbayer@wikimedia.org wrote:
On Tue, Mar 29, 2016 at 3:44 AM, Jaime Crespo jcrespo@wikimedia.org wrote:
There seems to be issues with the Edit_13457736 table on the db log, on the db1047 host.
db1047 may be known better by some of you as you as "s1-analytics-slave" or "analytics-slave". You can use dbstore1002 for the time being ("analytics-store" or "sX-analytics-slave", where X is 2-7) to read from that table. Other "schemas" are not affected.
I am fixing those right now, but it will take some time, as I may have to handle (again) the recently purged rows on that table. Worst case scenario, I may have to do a quick reboot of that machine.
Did that happen? I just had a query on that server abort with "ERROR 1053 (08S01) at line 5: Server shutdown in progress". It had been running since Sunday :(
I will update the status soon to communicate next steps.
-- Jaime Crespo http://wikimedia.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Jaime:
Seems that there are still problems. Please see: https://phabricator.wikimedia.org/T131236
Thanks,
Nuria
On Wed, Mar 30, 2016 at 1:54 AM, Jaime Crespo jcrespo@wikimedia.org wrote:
You can call it restart, or you can call it (un)controlled crash. The data issues where affecting the whole server, and I had to drop the previously mentioned table (which mysql refused to do while running).
Things seem back to normal now, but of course tables will take some time to get refilled, specially Edit_13457736.
On Wed, Mar 30, 2016 at 10:27 AM, Tilman Bayer tbayer@wikimedia.org wrote:
On Tue, Mar 29, 2016 at 3:44 AM, Jaime Crespo jcrespo@wikimedia.org
wrote:
There seems to be issues with the Edit_13457736 table on the db log, on the db1047 host.
db1047 may be known better by some of you as you as "s1-analytics-slave" or "analytics-slave". You can use dbstore1002 for the time being ("analytics-store" or "sX-analytics-slave", where X is 2-7) to read from that table. Other "schemas" are not affected.
I am fixing those right now, but it will take some time, as I may have to handle (again) the recently purged rows on that table. Worst case scenario, I may have to do a quick reboot of that machine.
Did that happen? I just had a query on that server abort with "ERROR 1053 (08S01) at line 5: Server shutdown in progress". It had been running since Sunday :(
I will update the status soon to communicate next steps.
-- Jaime Crespo http://wikimedia.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Jaime Crespo http://wikimedia.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On Wed, Mar 30, 2016 at 10:24 AM, Nuria Ruiz nuria@wikimedia.org wrote:
Jaime:
Seems that there are still problems. Please see: https://phabricator.wikimedia.org/T131236
Thanks Nuria - actually I filed that bug on Jaime's request (i.e. he already knows about it), and the data discrepancies described there concern other tables and have mostly been observed two weeks ago already.
Thanks,
Nuria
On Wed, Mar 30, 2016 at 1:54 AM, Jaime Crespo jcrespo@wikimedia.org wrote:
You can call it restart, or you can call it (un)controlled crash. The data issues where affecting the whole server, and I had to drop the previously mentioned table (which mysql refused to do while running).
Things seem back to normal now, but of course tables will take some time to get refilled, specially Edit_13457736.
On Wed, Mar 30, 2016 at 10:27 AM, Tilman Bayer tbayer@wikimedia.org wrote:
On Tue, Mar 29, 2016 at 3:44 AM, Jaime Crespo jcrespo@wikimedia.org wrote:
There seems to be issues with the Edit_13457736 table on the db log, on the db1047 host.
db1047 may be known better by some of you as you as "s1-analytics-slave" or "analytics-slave". You can use dbstore1002 for the time being ("analytics-store" or "sX-analytics-slave", where X is 2-7) to read from that table. Other "schemas" are not affected.
I am fixing those right now, but it will take some time, as I may have to handle (again) the recently purged rows on that table. Worst case scenario, I may have to do a quick reboot of that machine.
Did that happen? I just had a query on that server abort with "ERROR 1053 (08S01) at line 5: Server shutdown in progress". It had been running since Sunday :(
I will update the status soon to communicate next steps.
-- Jaime Crespo http://wikimedia.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Jaime Crespo http://wikimedia.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
For anyone who has reports affected by this (a LOT of people), please let us know if you'd like them to be re-run once the table stabilizes.
On Wed, Mar 30, 2016 at 1:30 PM, Tilman Bayer tbayer@wikimedia.org wrote:
On Wed, Mar 30, 2016 at 10:24 AM, Nuria Ruiz nuria@wikimedia.org wrote:
Jaime:
Seems that there are still problems. Please see: https://phabricator.wikimedia.org/T131236
Thanks Nuria - actually I filed that bug on Jaime's request (i.e. he already knows about it), and the data discrepancies described there concern other tables and have mostly been observed two weeks ago already.
Thanks,
Nuria
On Wed, Mar 30, 2016 at 1:54 AM, Jaime Crespo jcrespo@wikimedia.org
wrote:
You can call it restart, or you can call it (un)controlled crash. The data issues where affecting the whole server, and I had to drop the previously mentioned table (which mysql refused to do while running).
Things seem back to normal now, but of course tables will take some time to get refilled, specially Edit_13457736.
On Wed, Mar 30, 2016 at 10:27 AM, Tilman Bayer tbayer@wikimedia.org wrote:
On Tue, Mar 29, 2016 at 3:44 AM, Jaime Crespo jcrespo@wikimedia.org wrote:
There seems to be issues with the Edit_13457736 table on the db log, on the db1047 host.
db1047 may be known better by some of you as you as "s1-analytics-slave" or "analytics-slave". You can use dbstore1002
for
the time being ("analytics-store" or "sX-analytics-slave", where X is 2-7) to read from that table. Other "schemas" are not affected.
I am fixing those right now, but it will take some time, as I may
have
to handle (again) the recently purged rows on that table. Worst case scenario, I may have to do a quick reboot of that machine.
Did that happen? I just had a query on that server abort with "ERROR 1053 (08S01) at line 5: Server shutdown in progress". It had been running since Sunday :(
I will update the status soon to communicate next steps.
-- Jaime Crespo http://wikimedia.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Jaime Crespo http://wikimedia.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics