Switching dump format from SQL to XML offers a posibility of very easy incremental dumps.

I did some tests on Polish Wikiquote database dumps (most wikis have only a single xml dump, pl.q has two).

=== Raw data ===

Full dumps, compressed: 20050713_pages_current.xml.gz 1 105 966 20050814_pages_current.xml.gz 1 187 754 20050713_pages_full.xml.gz 3 796 056 20050814_pages_full.xml.gz 4 213 036

Full dumps, uncompressed: 20050713_pages_current.xml 3 874 225 20050814_pages_current.xml 4 158 544 20050713_pages_full.xml 30 882 758 20050814_pages_full.xml 33 708 595

Diffs -slow, hardly compresses anything: current.diff 524 191 current.diff.gz 134 817 (11.5%) full.diff 28 371 190 full.diff.gz 3 859 489 (91.6%) - I suspect the horrible result is due to the data being reordered

Xdelta (xdeltas are automatically gzipped): current.xdelta 92 074 (8.3%) full.xdelta 163 110 (4.2%)

gzip and xdelta were called with zlib compression level parameter -9

$ time xdelta delta -9 20050713_pages_current.xml.gz 20050814_pages_current.xml.gz current.delta real 0m1.023s user 0m0.548s sys 0m0.124s $ time xdelta delta -9 20050713_pages_full.xml 20050814_pages_full.xml full.delta real 0m3.749s user 0m1.731s sys 0m0.217s

=== Conclusion ===

We should simply generate xdeltas every time a dump is generated. It cuts the amount of data that needs to be transferred, easy to add to the backup script, extremely fast, available everywhere, and standard enough.

I doubt other compression methods (DTD-aware XML-deltas or whatever) will be significantly better than that, and if they are, it's certainly going to cost a lot more effort than simply using xdelta.