Hello folks,
Today I shot the full history en wiki dumps that claimed they would take
three months to complete. I started a new en wiki run which tests out
two new features: running jobs step by step (in arbitrary order,
assuming any dependencies have been run) and breaking up the xml file
creation into chunks that run in parallel.
Because of this, you may notice some funkiness with the status pages
over the next little while; things like the progress line are going to
be out of whack, and I am sure we will find new and exciting bugs
(though hopefully not in the dump file output).
We have some bizarre behavior around the index pages which all seem to
claim that the given date is its own previous dump. I'll be looking
into it, but in the meantime, the old dumps are available at:
http://dumps.wikimedia.org/enwiki/20100817/
http://dumps.wikimedia.org/enwiki/20100730
http://dumps.wikimedia.org/enwiki/20100130
http://dumps.wikimedia.org/enwiki/20100116
http://dumps.wikimedia.org/enwiki/20091103
The parallelizing scheme just does dumps of n pages in sequence (where n
is arbitrarily set at 2 million right now), including all revisions or
not, depending on the dump. This shouldn't screw with anyone's code
that depends on the page IDs to be in order.
This is only being tested out on the en wiki dumps at present; all other
jobs will run just as they used to.
Ah, I wonder if anyone out there would be interested in working on
dbzip2 or seeing if it is still needed; this is a parallelizing bzip2
with some features that pbzip2 doesn't have. See
http://www.mediawiki.org/wiki/Dbzip2
This could potentially save us time in the recombine phase of the bzip2
history dumps, if people want to have those available as one file nad
not just as separate pieces.
We don't have even a start on that for 7zip, so that's another thing for
someone to look into... any takers?
Ariel Glenn