The problem, as I understand it (and Brion may come by to correct me)
is essentially that the current dump process is designed in a way that
can't be sustained given the size of enwiki. It really needs to be
re-engineered, which means that developer time is needed to create a
new approach to dumping.
The main target for improvement is almost certainly parallelizing the
process so that wouldn't be a single monolithic dump process, but
rather a lot of little processes working in parallel. That would also
ensure that if a single process gets stuck and dies, the entire dump
doesn't need to start over.
By way of observation, the dewiki's full history dumps in 26 hours
with 96% prefetched (i.e. loaded from previous dumps). That suggests
that even starting from scratch (prefetch = 0%) it should dump in ~25
days under the current process. enwiki is perhaps 3-6 times larger
than dewiki depending on how you do the accounting, which implies
dumping the whole thing from scratch would take ~5 months if the
process scaled linearly. Of course it doesn't scale linearly, and we
end up with a prediction for completion that is currently 10 months
away (which amounts to a 13 month total execution). And of course, if
there is any serious error in the next ten months the entire process
could die with no result.
Whether we want to let the current process continue to try and finish
or not, I would seriously suggest someone look into redumping the rest
of the enwiki files (i.e. logs, current pages, etc.). I am also among
the people that care about having reasonably fresh dumps and it really
is a problem that the other dumps (e.g. stubs-meta-history) are frozen
while we wait to see if the full history dump can run to completion.
On Tue, Jan 27, 2009 at 11:24 AM, Christian Storm <storm(a)iparadigms.com> wrote:
On 1/4/09 6:20 AM, yegg at alum.mit.edu
The current enwiki database dump (http://download.wikimedia.org/enwiki/20081008/
) has been crawling along since 10/15/2008.
The current dump system is not
sustainable on very large wikis and
is being replaced. You'll hear about it when we have the new one in
Following up on this thread:
Can you offer any general timeline estimates (weeks, months, 1/2
year)? Are there any alternatives to retrieving the article data
beyond directly crawling
the site? I know this is verboten but we are in dire need of
retrieving this data and don't know of any alternatives. The current
estimate of end of year is
too long for us to wait. Unfortunately, wikipedia is a favored source
for students to plagiarize from which makes out of date content a real
Is there any way to help this process along? We can donate disk
drives, developer time, ...? There is another possibility
that we could offer but I would need to talk with someone at the
wikimedia foundation offline. Is there anyone I could
Thanks for any information and/or direction you can give.
Wikitech-l mailing list