Very slow import of XML dumps - Xmldatadumps-l

29 Aug 2009

Greetings,
I am trying to import the French wiki (full history xml) on a Ubuntu machine
with quad-core trendy CPU and 16 GB RAM. The import query is the following:

java -Xmn256M -Xms396M -Xmx512M -XX:+DisableExplicitGC -verbose:gc
-XX:NewSize=32m -XX:MaxNewSize=64m -XX:SurvivorRatio=6 -XX:+UseParallelGC
-XX:GCTimeRatio=9 -XX:AdaptiveSizeDecrementScaleFactor=1 -jar mwdumper.jar
--format=sql:1.5 frwiki-20090810-pages-meta-history.xml.bz2 | mysql -u wiki
-p frwikiLatest

I have disabled the autocommit for mysql, disabled foreign key checks and
unique checks. I have set the pool size, buffer log size, and the buffer
size to large values as recommended for mysql good performance.

After around 3 minutes of running the above command, I have got:

6 pages (0.083/sec), 1,000 revs (13.889/sec)
8 pages (0.038/sec), 2,000 revs (9.378/sec)
13 pages (0.041/sec), 3,000 revs (9.458/sec)

The source file is on its own physical disk and the mysql data folder is on
another physical disks. Both disks are very fast.

*Any suggestions on how to improve the speed?
*
another issue is that the InnoDB (page, revision, text) do not show the
number of the records although the size of the table is non-zero. I think
the might be related to disable keys query.
*Is that correct?
*
bilal