On Sat, Aug 2, 2008 at 9:49 PM, Platonides <Platonides(a)gmail.com> wrote:
Magnus Manske wrote:
Independent of that,
* Run several parallel processes on several servers (assuming we have several)
* Each process generates the complete history dump of a single
article, or a small group of them, bz2ipped to save intermittent disk
space
* Success/faliure is checked, so each process can be rerun if needed
* At the end, all these files are appended into a single bzip2/7zip file
The system we use is not exactly that. It's writing compressed data
from the compressed reading of last dump and the revision snapshot. It's
never using uncompressed data.
The little processes would need to know where in the last file is the
section they're doing.
However, if you knew in which part it was in the old dump... it's
worthwhile considering.
Why is it using the old dump instead of the "real" storage? For
performance reasons?
Does that mean that if there's an error in an old dump, it will stay
there forever?
How does this cope with deleted revisions?
This will need
more diskspace while the entire thing is running, as
small text files compress less well than larger ones. Also, it eats
more CPU cycles, for starting all these processes, and then for
re-bzip2ing the intermediate files.
Not neccessarily. If the number of files per bzip2 group is large
enough, there is almost no difference.
Yes. we'd have to find a balance between many fast processes with lots
of overhead and few slow ones that, when failing, will set back the
dump for weeks.
At work, I'm using a computing farm with several thousand cores, and
the suggested time per process is < 2h. May be worth contemplating,
even though the technical situation for Wikimedia is very much
different.
Magnus