Hi.
There's an outstanding ticket (https://jira.toolserver.org/browse/TS-1414) about thyme's copy of enwiki_p being corrupt. Now I'm noticing apparent corruption with rosemary's copy of enwiki_p. Example below:
--- mzmcbride@willow:~$ mysql -hrosemary enwiki_p; mysql> select * from user where user_name = 'GoodPotato'\G Empty set (0.01 sec) ---
An empty set is the wrong result. Compare:
--- mzmcbride@willow:~$ mysql -hthyme enwiki_p; mysql> select * from user where user_name = 'GoodPotato'\G *************************** 1. row *************************** user_id: 17766406 user_name: GoodPotato user_registration: 20121026010740 user_editcount: 2 user_email_authenticated: 0 1 row in set (0.00 sec) ---
This indicates that in addition to thyme's copy of enwiki_p being corrupt, rosemary's copy of enwiki_p is now also corrupt, leaving us with no complete copy of enwiki.
Can someone please let me know what needs to happen in order for us to get a working copy of enwiki back in production? If there's a hold-up somewhere (particularly with the Wikimedia Foundation), please let me know what that hold-up is and I'll do what I can to move things forward.
MZMcBride
Hello, At Sunday 04 November 2012 00:29:50 DaB. wrote:
Can someone please let me know what needs to happen in order for us to get a working copy of enwiki back in production? If there's a hold-up somewhere (particularly with the Wikimedia Foundation), please let me know what that hold-up is and I'll do what I can to move things forward.
the situation is like the following at the moment: WMF and WMDE finally agreed on the conditions under which the toolserver can get new dumps ~2 weeks ago (in most manners that's how some private fields in the toolserver database are hidden). Shortly after Asher Feldman provided a dump in the new xtrabackup-binary- format the wmf uses for dumps. It took ~2 days to transmit this dump to the toolserver. It is the first time the toolserver has to handle this new format and it took some days to find the needed tools (the wmf uses Ubuntu for their database- servers while we uses Solaris). Asher told me that the dump is compressed in some way, but I could not decompress it. While I tried to figure out how to decompress the file, I accidentally overwritten the file at Thursday. I immediately mailed Asher that he should not delete the file on his side and restarted the transfer. Asher mailed back little time later and told me that the dump would be quite old already and asked if I would like to get a fresh one. I accepted of corse and stopped the transfer. Few hours later I got an eMail by Erik Möller asking the toolserver-roots to provide (again) how we hid/delete the private fields in our database; I answered him and Nosy did too with more details. Until now there is no new dump to transmit.
Sincerely, DaB.
On Nov 3, 2012 5:06 PM, "DaB." WP@daniel.baur4.info wrote:
While I tried to figure out how to decompress the file, I accidentally overwritten the file at Thursday.
Sorry to hear this. Shit happens - we'll help when we can. My most recent inquiry to you and Nosy was unrelated (I asked for the configs so we can compare them as we begin setting up replication in Labs.)
Erik
On 04/11/12 02:29, Erik Moeller wrote:
My most recent inquiry to you and Nosy was unrelated (I asked for the configs so we can compare them as we begin setting up replication in Labs.)
Erik
It's the first notice I get that labs replication is being set up. Who is handling this? Asher? I would expect Ryan to be involved, but he is currently on vacation...
On Sun, Nov 4, 2012 at 6:42 AM, Platonides platonides@gmail.com wrote:
It's the first notice I get that labs replication is being set up. Who is handling this? Asher?
Yes, Asher is working on it this month. We've got most of the hardware provisioned. Ryan will work on user grants when he's back. No promises yet that we'll exceed our original deadline (which is launch of this functionality in the first quarter of 2013) but we'll do what we can to move it forward.
Erik
Yes, Asher is working on it this month. We've got most of the hardware provisioned. Ryan will work on user grants when he's back. No promises yet that we'll exceed our original deadline (which is launch of this functionality in the first quarter of 2013) but we'll do what we can to move it forward.
Any updates regarding the possibility to perform joins between wiki DBs and user DBs? Or wikipedia DBs and the Commons database? As it surely has been mentioned, this is a vital feature of the toolserver DB replication setup and would be much appreciated (i.e. desperately needed) on labs. Daniel
Hello, At Sunday 04 November 2012 17:58:45 DaB. wrote:
Sorry to hear this. Shit happens - we'll help when we can.
ok, good to hear.
My most recent inquiry to you and Nosy was unrelated (I asked for the configs so we can compare them as we begin setting up replication in Labs.)
ok, than sorry for the bad faith.
Sincerely, DaB.
On 04/11/12 00:56, DaB. wrote:
Shortly after Asher Feldman provided a dump in the new xtrabackup-binary- format the wmf uses for dumps. It took ~2 days to transmit this dump to the toolserver.
Was the rate-limiting issue fixed?
It is the first time the toolserver has to handle this new format and it took some days to find the needed tools (the wmf uses Ubuntu for their database- servers while we uses Solaris).
This may be a good time to migrate one of rosemary/thyme to Debian.
Asher told me that the dump is compressed in some way, but I could not decompress it. While I tried to figure out how to decompress the file, I accidentally overwritten the file at Thursday.
Failures happen. Looking aroound, it isn't obvious the way they get compressed. Given you could not decompress it initially, it probably used xbstream (the other common compression format with xtrabackup seems to be using a gzipped tar). I would have expected Asher to provide you a full page documenting how to restore the backups, though. "How to restore a backup" is a critical piece which can't be left undocumented (or untested).
Thanks for the update, DaB.
Hello, At Sunday 04 November 2012 17:50:37 DaB. wrote:
On 04/11/12 00:56, DaB. wrote:
Shortly after Asher Feldman provided a dump in the new xtrabackup-binary- format the wmf uses for dumps. It took ~2 days to transmit this dump to the toolserver.
Was the rate-limiting issue fixed?
We worked-around that using another target-host.
It is the first time the toolserver has to handle this new format and it took some days to find the needed tools (the wmf uses Ubuntu for their database- servers while we uses Solaris).
This may be a good time to migrate one of rosemary/thyme to Debian.
That would be possible, yes.
Asher told me that the dump is compressed in some way, but I could not decompress it. While I tried to figure out how to decompress the file, I accidentally overwritten the file at Thursday.
Failures happen. Looking aroound, it isn't obvious the way they get compressed. Given you could not decompress it initially, it probably used xbstream (the other common compression format with xtrabackup seems to be using a gzipped tar). I would have expected Asher to provide you a full page documenting how to restore the backups, though. "How to restore a backup" is a critical piece which can't be left undocumented (or untested).
Asher told be that I have to use xbstream. The problem is that I can't find a binary for Solaris (neither in the solaris-package from percona nor in the web) and the source-package is not compiling (a known problem with the linker). As a work-around I plan to use a Debian-host now (at least for the decompressing), but I have to get a dump first :-).
Thanks for the update, DaB.
No problem.
Sincerely, DaB.
toolserver-l@lists.wikimedia.org