Magnus Manske wrote:
Knowing little about the current dump generation
process, but some
about terabyte-scale data handling (actually, we here are well into
the petabyte range by now;-), how about this:
* Set up the usual MySQL replication slave
* At one point in time, disconnect it from the MySQL master, but leave
it running in read-only mode
* Use that as the dump base
This should result in a single-point-in-time snapshot.
Why? As i already said, the revision status is a snapshot, it's done in
a transaction.
Also, it will reduce load to the rest of the system.
Not sure if
IDs will change internally, though.
IDs won't change, but you don't need
the disconnected slave. Once you
have the revisions, you will be querying external storage. That's where
the load goes.
Independent of that,
* Run several parallel processes on several servers (assuming we have several)
* Each process generates the complete history dump of a single
article, or a small group of them, bz2ipped to save intermittent disk
space
* Success/faliure is checked, so each process can be rerun if needed
* At the end, all these files are appended into a single bzip2/7zip file
The system we use is not exactly that. It's writing compressed data
from the compressed reading of last dump and the revision snapshot. It's
never using uncompressed data.
The little processes would need to know where in the last file is the
section they're doing.
However, if you knew in which part it was in the old dump... it's
worthwhile considering.
This will need more diskspace while the entire thing
is running, as
small text files compress less well than larger ones. Also, it eats
more CPU cycles, for starting all these processes, and then for
re-bzip2ing the intermediate files.
Not neccessarily. If the number of files per bzip2 group is large
enough, there is almost no difference.
But, it is a lot less error-prone (if a process or a
bunch of them
fail, just restart them), and it scales better (just throw more
machines at it to make it faster; or use apaches during low-traffic
hours). Individual processes should be less memory-intensive, so
several of them can run on the same machine.
My 2c
Magnus
We are talking very happily here, but what is slowing the dump process?
Brion, Tim, there's some profiling information about that? Is it I/O
waiting for the revisions fetched for external storage? Is it disk speed
when reading/writing? Is it CPU for decompressing previous dump? Is it
CPU for compressing? How is dbzip2 helping with it?*
*I thought you were using dbzip2, but i now see mw:Dbzip2 says "dbzip2
is not ready for public use yet" Has it been indefinitely postponed?