In my continued quest to Make Imports Suck Less (tm), I've written a
little perl script to shovel data from a tab-delimited escaped file to a
fifo in pieces while forking off mysql to LOAD DATA INFILE from the fifo
for each chunk. It's only been tested on linux, specifically my laptop,
but I did run it using current article content dumps and all the
auxiliary tables for a wiki of a few hundred thousand articles, and it
You can find it in the xmlfileutils directory of my branch of the git
You'll notice that all my tools are linux + mysql, and that's because
that is what I use. If folks want similar tools for other platforms
they'll have to write them, I don't have the expertise for that.
Ah also the docs on Meta about dumps have been reorganized and
rewritten, not that they are either error-free or complete but they
should be in much better shape now:
And lastly, the uncompressed en wp meta history dumps are now over 10T.
As always, feedback, edits, patches welcome.
P.S. Sorry Platonides but if you were going to rework a script of yours
you were too slow ;-) (However if you have such a script with
different/better features I'll still take it.)