Hello everyone,
in the past weeks, I put together "xmldumps-test"---a test suite for Ariel's xmldumps-backup software. xmldumps-test tries to assure that the MySQL database, MediaWiki, and xmldumps-backup play nicely together.
xmldumps-test injects data into the database (so do not use it on a live database), starts xmldumps-backup, and compares the generated XML dumps against pre-verified data. Using xmldumps-test I hope to catch problems caused by modifications to MediaWiki or xmldumps-backup /before/ they hit Wikimedia's production servers dumping enwiki, ...
The code is up for review at https://gerrit.wikimedia.org/r/p/operations/dumps/test.git . README serves as general point of entry to the documentation. README.installation shows you how to set up xmldumps-test. After setup is completed, ./run_tests.sh runs all available tests.
xmldumps-test comes with tests for REL1_1{7,8,9} and trunk.
I'd love to get some feedback, or comments on the scripts.
Best regards, Christian
Awesome! I have been looking forward to this QA tool for a long time. Presumably this will also shorten dump job suspension after new software rollouts?!.
Erik Zachte
-----Original Message----- From: wikitech-l-bounces@lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of Christian Aistleitner Sent: Friday, March 16, 2012 8:02 PM To: wikitech-l@lists.wikimedia.org Subject: [Wikitech-l] Test suite for dumping MediaWikis using xmldumps-backup
Hello everyone,
in the past weeks, I put together "xmldumps-test"---a test suite for Ariel's xmldumps-backup software. xmldumps-test tries to assure that the MySQL database, MediaWiki, and xmldumps-backup play nicely together.
xmldumps-test injects data into the database (so do not use it on a live database), starts xmldumps-backup, and compares the generated XML dumps against pre-verified data. Using xmldumps-test I hope to catch problems caused by modifications to MediaWiki or xmldumps-backup /before/ they hit Wikimedia's production servers dumping enwiki, ...
The code is up for review at https://gerrit.wikimedia.org/r/p/operations/dumps/test.git . README serves as general point of entry to the documentation. README.installation shows you how to set up xmldumps-test. After setup is completed, ./run_tests.sh runs all available tests.
xmldumps-test comes with tests for REL1_1{7,8,9} and trunk.
I'd love to get some feedback, or comments on the scripts.
Best regards, Christian
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian@quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------
Christian Aistleitner christian@quelltextlich.at wrote:
--===============2205038051751942713== Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Q68bSM7Ycu6FN28Q" Content-Disposition: inline
--Q68bSM7Ycu6FN28Q Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable
Hello everyone,
in the past weeks, I put together "xmldumps-test"---a test suite for Ariel's xmldumps-backup software. xmldumps-test tries to assure that the MySQL database, MediaWiki, and xmldumps-backup play nicely together.
xmldumps-test injects data into the database (so do not use it on a live database), starts xmldumps-backup, and compares the generated XML dumps against pre-verified data. Using xmldumps-test I hope to catch problems caused by modifications to MediaWiki or xmldumps-backup /before/ they hit Wikimedia's production servers dumping enwiki, ...
The code is up for review at https://gerrit.wikimedia.org/r/p/operations/dumps/test.git =2E README serves as general point of entry to the documentation. README.installation shows you how to set up xmldumps-test.
One question:
in the https://gerrit.wikimedia.org/r/p/operations/dumps/test.git repository there are two branches, "master" and "ariel" and the README me says we should use "ariel".
"master" however see to be also attached to a gerrit project
I was able to check it out using
ssh://saper@gerrit.wikimedia.pl/operations/dumps/test.git
port 29418
Which shall we use? It seems that I can propose patches using gerrit only to "master" while "ariel" seems to be a bit more active.
Second thing - I was fixing recently few nuts and bolts for seamless PostgreSQL support, so I'd love to have that for PostgreSQL too. Once I sort out outstanding installer/updater issues I am willing to help, of course.
We already ran a PostgreSQL testsuite on jenkins and I think we should check dumps too.
//Saper
Hi Saper,
On Sat, Mar 17, 2012 at 01:59:33PM +0000, Marcin Cieslak wrote:
[ Announcing xmldumps-test ]
The code is up for review at https://gerrit.wikimedia.org/r/p/operations/dumps/test.git
[ Confusion of URLs and branches ]
yes, the current situation with the "ariel" branch is suboptimal.
Which shall we use?
The current xmldumps-test is stored in the "master" branch of https://gerrit.wikimedia.org/r/p/operations/dumps/test.git
The current xmldumps-backup is stored in the "ariel" branch of https://gerrit.wikimedia.org/r/p/operations/dumps.git
(Note the different URLs)
If you find documentation referring to other branches, or other branches being used anywhere, please let me know.
[ adding PostgreSQL support ]
As far as I know, xmldumps-backup (the software that xmldumps-test tests) can only deal with MySQL databases.
However, if you manage to add PostgreSQL support to xmldumps-backup, adapting xmldumps-test to use PostgreSQL should be doable.
Kind regards, Christian
Στις 17-03-2012, ημέρα Σαβ, και ώρα 16:45 +0100, ο/η Christian Aistleitner έγραψε:
Hi Saper,
On Sat, Mar 17, 2012 at 01:59:33PM +0000, Marcin Cieslak wrote:
[ Announcing xmldumps-test ]
The code is up for review at https://gerrit.wikimedia.org/r/p/operations/dumps/test.git
[ Confusion of URLs and branches ]
yes, the current situation with the "ariel" branch is suboptimal.
Which shall we use?
The current xmldumps-test is stored in the "master" branch of https://gerrit.wikimedia.org/r/p/operations/dumps/test.git
The current xmldumps-backup is stored in the "ariel" branch of https://gerrit.wikimedia.org/r/p/operations/dumps.git
(Note the different URLs)
I should really move everything over to master soon and use my branch just for development. Maybe I'll do that before the end of the month.
Ariel
Christian Aistleitner christian@quelltextlich.at wrote:
--===============8638225475146652871== Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="sm4nu43k4a2Rpi4c" Content-Disposition: inline
--sm4nu43k4a2Rpi4c Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable
Hi Saper,
On Sat, Mar 17, 2012 at 01:59:33PM +0000, Marcin Cieslak wrote:
[ Announcing xmldumps-test ]
The code is up for review at https://gerrit.wikimedia.org/r/p/operations/dumps/test.git
[ Confusion of URLs and branches ]
yes, the current situation with the "ariel" branch is suboptimal.
I don't mind that, actually. I have confused names and repositories.
Which shall we use?
The current xmldumps-test is stored in the "master" branch of https://gerrit.wikimedia.org/r/p/operations/dumps/test.git
The current xmldumps-backup is stored in the "ariel" branch of https://gerrit.wikimedia.org/r/p/operations/dumps.git
Ok so those are two separate pieces of software, indeed when checking out "ariel" branch of dumps.git informs me that there are no shared commits indeed.
My consfusion came from the fact that I started reading README.installation (of -test) that starts with description of -backup.
Thank you for explaining this!
//Saper
Hi Saper,
On Sat, Mar 17, 2012 at 09:22:11PM +0000, Marcin Cieslak wrote:
Christian Aistleitner christian@quelltextlich.at wrote:
[ xmldumps-{backup,test} ]
Ok so those are two separate pieces of software, [...]
Yes.
My consfusion came from the fact that I started reading README.installation (of -test) that starts with description of -backup.
You are right. Maybe this is too confusing. I'll update the documentation to make the distinction more obvious.
Kind regards, Christian
Le 16/03/12 20:02, Christian Aistleitner a écrit :
The code is up for review at https://gerrit.wikimedia.org/r/p/operations/dumps/test.git
And people can browse it using gitweb at:
https://gerrit.wikimedia.org/gitweb/operations/dumps/test.git
Anonymous access: https://gerrit.wikimedia.org/r/p/operations/dumps/test.git
Authenticated ssh: ssh://gerrit.wikimedia.org:29418/operations/dumps/test.git
wikitech-l@lists.wikimedia.org