Hi
On Sat, Apr 19, 2008 at 2:45 AM, Brion Vibber <brion(a)wikimedia.org> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Felipe Ortega wrote:
Yesterday, I was moving around mysqldump files of
our processed
databases from parsed Wikipedia dumps, and this simple question came
to my mind.
Is there any special reason to use an "ad-hoc" XML schema for
Wikipedia dumps?
1) The format is relatively stable, unlike our database schema.
2) Our databases are spread over dozens of servers, in mixes of internal
binary compression formats whose interpretation is dependent on our
configuration and custom code.
3) Our internal databases mix public and private information, which we
have to separate for external dumps. Thus only completely public tables
are dumped with mysqldump.
Thus, we use a stable, safe data schema for public page dumps. Dumping
raw SQL of these tables would be unstable, insecure, and useless for
most people.
I agree dump to SQL statements is a little bit useless, but how about CSV ?
mysqldump allow you to dump to CSV file instead of raw sql statements
(you can specify the fieds your want), they are pretty safe, and
storage efficient for download.
Even better, mysqlimport can import those CSV at a very high speed.
Issue 1, 2 and 3 that apply to SQL also apply to any other form of dump
done via MySQL, including CVS. There is no feasible way of providing a
CVS dump for the same reasons that a SQL one cannot be. The problem here
is not the format, but the process it is created via.
MinuteElectron.