Hello all,
I will switch sql-s1-rr from thyme to rosemary and back after
19:00 UTC this evening.
You should notice no difference, but everything at sql-s1 will be a little bit slower while sql-s1-rr is on rosemary (and all queries running on thyme will be killed). The background of the switches is that I finally extracted the mysql-files from the wmf-binary dump and so I can re-setup thyme now with an unbroken version of s1. If everything works as planed I will handle rosemary during the next week too, so the hole s1-cluster will be unbroken again.
Just to let you know.
Sincerely, DaB.
Hello all,
I just started the thyme with a fresh dump from the WMF. The import took longer than thought because some files had to decompressed first. I let it catch-up in replication for the next hours. The next steps are the following: I will switch sql-s1-user to read-only at
tomorrow, Monday after 20:05 UTC
and than dump the user-databases (and import them on thyme). I do not know how long this will take, but I guess the hole European night. When the import is done somewhen on Tuesday I will switch sql-s1-user to thyme and switch it to read/write. Later (or in parallel to the user-database-dumping, not sure yet) Nosy or I will dump commons from rosemary too for a later re-import. When everything is dumped and imported I will re-setup rosemary with the fresh dump too.
Sincerely, DaB.
On 11/11/12 17:37, DaB. wrote:
I will switch sql-s1-user to read-only at
tomorrow, Monday after 20:05 UTC
and than dump the user-databases (and import them on thyme). I do not know how long this will take, but I guess the hole European night. When the import is done somewhen on Tuesday I will switch sql-s1-user to thyme and switch it to read/write. Later (or in parallel to the user-database-dumping, not sure yet) Nosy or I will dump commons from rosemary too for a later re-import. When everything is dumped and imported I will re-setup rosemary with the fresh dump too.
Sincerely, DaB.
Remember that there is a trick where you dump the user-databases with the binlog enabled (and noting its position), leaving the database writable. You then transfer and import the dump. Finally you make it readonly, copy the changes from the binlog (which will be much smaller and thus results in a shorter read-only interval) and point the dns to the other server.
Regards
Hello, At Sunday 11 November 2012 23:54:39 DaB. wrote:
Remember that there is a trick where you dump the user-databases with the binlog enabled (and noting its position), leaving the database writable. You then transfer and import the dump. Finally you make it readonly, copy the changes from the binlog (which will be much smaller and thus results in a shorter read-only interval) and point the dns to the other server.
AFAIK this would need a complete LOCK of all tables during the dumping (to get a consistent state) which would be the same as switching the server read-only. Also somebody would have to write a program that read the bin-log and filter out all non-user-database-entries (enwp, commons and heartbeat in this case). The 3rd disadvantage would be that you would have to wait for the dump to complete before you can start the import, The only advantage I see it that sql-s1-user would be writable between the end of the dump and the time I awake in the morning.
Sincerely, DaB.
On 12/11/12 00:04, DaB. wrote:
Hello, At Sunday 11 November 2012 23:54:39 DaB. wrote:
Remember that there is a trick where you dump the user-databases with the binlog enabled (and noting its position), leaving the database writable. You then transfer and import the dump. Finally you make it readonly, copy the changes from the binlog (which will be much smaller and thus results in a shorter read-only interval) and point the dns to the other server.
AFAIK this would need a complete LOCK of all tables during the dumping (to get a consistent state) which would be the same as switching the server read-only. Also somebody would have to write a program that read the bin-log and filter out all non-user-database-entries (enwp, commons and heartbeat in this case). The 3rd disadvantage would be that you would have to wait for the dump to complete before you can start the import, The only advantage I see it that sql-s1-user would be writable between the end of the dump and the time I awake in the morning.
Sincerely, DaB.
You are completely right. Still, I suspect it *is* possible to do so efficiently. Starting a global read transaction and then resuming writes (I am assuming that all will be InnoDB, which is very likely). Maybe it needs a special tool.
I am copying Asher in case he can enlighten us.
There are tools for the binlog filtering, but it is not a problem in this case, since we could just stop replication before the export and let the lag rise during the backup. In the general case we would reenable them and let the server pick up after it finishes, but in this case we are just going to overwrite it :)
How much data is in user tables and how long does it take to import the (uncompressed, ready) wiki tablespaces? It might turn out to be faster to switch the tablespaces than moving the dbs around.
Hello all, At Monday 12 November 2012 22:31:44 DaB. wrote:
I will switch sql-s1-user to read-only at
tomorrow, Monday after 20:05 UTC
this is postponed to unknown timestamp. We have problems to reach thyme since 14 o'clock. Looks like we will need a manually reboot there.
Sincerely, DaB.
toolserver-l@lists.wikimedia.org