Are you looking for enwiki-20070908-stub-meta-history.xml.gz, this is 5.4GB. Try downloading this one. fawad@cyprus:~/wiki$ ls -lh -r-------- 1 fawad fawad 5.4G 2007-10-21 09:29 enwiki-20070908-stub-meta-history.xml.gz -- Fawad Nazir http://www.geocities.com/nazir_fawad/

On Mon, 2007-10-29 at 12:47 +0200, Osnat Etgar wrote: You don't want the history, but you want all of the discussion and user pages? Are you sure? I'm testing a download of the -meta-current.xml.bz2 right now to see if it does indeed work, but it will take 1/2 day to get it all. I'll post back and let you know what happens. The server claims to have the right amount of bytes, so let's see what happens when my download completes: Server: Wikimedia dump service 20050523 (lighttpd) Content-Length: 5780471837 -- David A. Desrosiers desrod(a)gnu-designs.com setuid(a)gmail.com http://projects.plkr.org/ Skype...: 860-967-3820

For your first question .. "How do I read .. " look at this link http://www.mediawiki.org/wiki/Manual:Database_layout posted by Paul Coghlan just this morning in answer to a similar question. DSig David Tod Sigafoos | SANMAR Corporation PICK Guy 206-770-5585 davesigafoos(a)sanmar.com -----Original Message----- From: wikitech-l-bounces(a)lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of Osnat Etgar Sent: Monday, October 29, 2007 6:48 To: David A. Desrosiers; Wikimedia developers Subject: Re: [Wikitech-l] Dump is small Hi, Thanks for your replay. I am a newbie in this field. 1. I only want the articles. No history, no user information, no discussions. I do want articles, lists and disambiguation. Maybe I understood it wrong and I don't need the page-meta-current? Only the pages-articles? 2. I have some data from page-meta-current in my database 2.1. I got an error in the middle, after over a million pages where extracted. Exception in thread "main" java.io.IOException: An invalid XML character (Unicode: 0x2) was found in the element content of the document. at org.mediawiki.importer.XmlDumpReader.readDump(Unknown Source) at org.mediawiki.dumper.Dumper.main(Unknown Source) While we are here, I'd like to ask some more questions, if you don't mind: 1. How do I read the data from MySQL? I don't understand how entries are connected to one another and how I should read it. 2. Do I have to clean up MySQL tables every time I want to insert another dump? Either an update file or totally different one? 3. Is there a way to get only the delta file instead of the whole dump again? 4. How do I add .sql.gz files to MySQL? Thanks a lot for your answers Osnat P Please consider the environment before printing this e-mail -----Original Message----- From: wikitech-l-bounces(a)lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of David A. Desrosiers Sent: Monday, October 29, 2007 3:03 PM To: Wikimedia developers Subject: Re: [Wikitech-l] Dump is small On Mon, 2007-10-29 at 12:47 +0200, Osnat Etgar wrote: You don't want the history, but you want all of the discussion and user pages? Are you sure? I'm testing a download of the -meta-current.xml.bz2 right now to see if it does indeed work, but it will take 1/2 day to get it all. I'll post back and let you know what happens. The server claims to have the right amount of bytes, so let's see what happens when my download completes: Server: Wikimedia dump service 20050523 (lighttpd) Content-Length: 5780471837 -- David A. Desrosiers desrod(a)gnu-designs.com setuid(a)gmail.com http://projects.plkr.org/ Skype...: 860-967-3820 _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l

On 10/29/07, Fawad Nazir <fawad.nazir(a)gmail.com> wrote: Or mysql -u username -p db-name | gunzip --to-stdout filename.sql Should be faster and eating up less disk space. Bryan

On Mon, 2007-10-29 at 22:59 +0900, Fawad Nazir wrote: How does it eat up a lot of your time? You just set it to import the data, and walk away. I have mine automated, so that I dump and reload the top 9 languages of each project into MySQL. The whole process, for all languages, takes just about an hour.. with enwiki taking the bulk of that time. My machine is a lowly dual-core AMD64 machine with 4gb of RAM. -- David A. Desrosiers desrod(a)gnu-designs.com setuid(a)gmail.com http://projects.plkr.org/ Skype...: 860-967-3820

Hi David, The problem is not with downloading. The problem is with loading the data into mysql. -Fawad. On 10/30/07, David A. Desrosiers <desrod(a)gnu-designs.com> wrote: -- Fawad Nazir http://www.geocities.com/nazir_fawad/

On Tue, 2007-10-30 at 17:16 -0400, David A. Desrosiers wrote: The full import of page, revision and text for Wikipedia, Wikiquote and Wiktionary in 9 languages (en fr de pl it pt sv es nl) took: real 91m19.910s user 15m16.073s sys 1m43.810s The number of records found for the Wikipedia versions of these languages are: for lang in en fr de pl it pt sv es nl; do mysql mediawiki -e "select count(*) from ep_${lang}_text"; done; +----------+ | count(*) | +----------+ <- English Wikipedia | 5836166 | +----------+ +----------+ | count(*) | +----------+ <- French | 1147662 | +----------+ +----------+ | count(*) | +----------+ <- German | 1344476 | +----------+ +----------+ | count(*) | +----------+ <- Polish | 628225 | +----------+ +----------+ | count(*) | +----------+ <- Italian | 688577 | +----------+ +----------+ | count(*) | +----------+ <- Portuguese | 726593 | +----------+ +----------+ | count(*) | +----------+ <- Swedish | 445601 | +----------+ +----------+ | count(*) | +----------+ <- Spanish | 556088 | +----------+ +----------+ | count(*) | +----------+ <- Dutch | 616810 | +----------+ Everything looks to be working fine for me. -- David A. Desrosiers desrod(a)gnu-designs.com setuid(a)gmail.com http://projects.plkr.org/ Skype...: 860-967-3820

[..snip..] Care to post a link to the script? Might be useful to others and save reinventing the wheel, especially if you can say "I want languages X, Y, Z, these are my db names, my db hostname, my username, and my password", and then it can be added to /etc/cron.daily/ , and will do a daily poll to see if there are updated successful dumps in any of those langs, and if so download, test archive validity, check MD5sum, if both look okay then unpack, import, etc - or even if it does most of the above it could be a useful starting point for people who want to have a semi-recent local copy of some Wikipedia sites that will auto-update. -- All the best, Nick.