howard chen wrote:
Hi
On Sat, Apr 19, 2008 at 2:45 AM, Brion Vibber brion@wikimedia.org wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Felipe Ortega wrote:
Yesterday, I was moving around mysqldump files of our processed databases from parsed Wikipedia dumps, and this simple question came to my mind.
Is there any special reason to use an "ad-hoc" XML schema for Wikipedia dumps?
The format is relatively stable, unlike our database schema.
Our databases are spread over dozens of servers, in mixes of internal
binary compression formats whose interpretation is dependent on our configuration and custom code.
- Our internal databases mix public and private information, which we
have to separate for external dumps. Thus only completely public tables are dumped with mysqldump.
Thus, we use a stable, safe data schema for public page dumps. Dumping raw SQL of these tables would be unstable, insecure, and useless for most people.
I agree dump to SQL statements is a little bit useless, but how about CSV ?
mysqldump allow you to dump to CSV file instead of raw sql statements (you can specify the fieds your want), they are pretty safe, and storage efficient for download.
Even better, mysqlimport can import those CSV at a very high speed.
Issue 1, 2 and 3 that apply to SQL also apply to any other form of dump done via MySQL, including CVS. There is no feasible way of providing a CVS dump for the same reasons that a SQL one cannot be. The problem here is not the format, but the process it is created via.
MinuteElectron.