Xmldatadumps-l July 2015

xmldatadumps-l@lists.wikimedia.org

12 participants
9 discussions

pbzip2 proposal
by Richard Jelinek 16 Jan '16

16 Jan '16

Hi, don't know if this issue came up already - in case it did and has been dismissed, I beg your pardon. In case it didn't... I hereby propose, that pbzip2 (https://launchpad.net/pbzip2) is used to compress the xml dumps instead of bzip2. Why? Because its sibling (pbunzip2) has a bug bunzip2 hasn't. :-) Strange? Read on. A few hours ago, I filed a bug report for pbzip2 (see https://bugs.launchpad.net/pbzip2/+bug/922804) together with some test results done even some few hours before that. The results indicate that: bzip2 and pbzip2 are vice-versa compatible each one can create archives, the other one can read. But if it is for uncomressing, only pbzip2 compressed archives are good for pbunzip2. I propose compressing the archives with pbzip2 for the following reasons: 1) If your archiving machines are SMP systems this could lead to a better usage of system ressources (i.e. faster compression). 2) Compression with pbzip2 is harmless for regular users of bunzip2, so everything should run for these people as usual. 3) pbzip2-compressed archives can be uncompressed with pbunzip2 with a speedup that scales nearly linearly with the number of CPUs in the host. So to sum up: It's a no loose and two win situation if you migrate to pbzip2. And that just because pbunzip2 is slightly buggy. Isn't that interesting? :-) cheers, -- Dipl.-Inf. Univ. Richard C. Jelinek PetaMem GmbH - www.petamem.com Geschäftsführer: Richard Jelinek Human Language Technology Experts Sitz der Gesellschaft: Fürth 69216618 Mind Units Registergericht: AG Fürth, HRB-9201

6 11

(no subject)
by Samina Akter Jhara 22 Jul '15

22 Jul '15

1 0

Latest articles URL 404ing for some languages
by Devesh Parekh 15 Jul '15

15 Jul '15

Hi data dumpers, Starting today, some of the URLs I've been using to find the latest dumps for current article revisions have begun 404ing. Japanese (failing): $ curl -I " http://dumps.wikimedia.org/enwiki/latest/jawiki-latest-pages-articles.xml.b… " HTTP/1.1 404 Not Found Server: nginx/1.1.19 Date: Tue, 07 Jul 2015 23:40:40 GMT Content-Type: text/html; charset=utf-8 Content-Length: 169 Connection: keep-alive English (working): $ curl -I " http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.b… " HTTP/1.1 200 OK Server: nginx/1.1.19 Date: Tue, 07 Jul 2015 23:39:36 GMT Content-Type: application/octet-stream Content-Length: 11984805689 Last-Modified: Fri, 05 Jun 2015 23:45:33 GMT Connection: keep-alive Accept-Ranges: bytes Are these particular dump files going away, or is the "latest" symlink being updated before all dumps have completed? -- Devesh

2 4

Flow, LiquidThreads, and XML dumps
by wp mirror 11 Jul '15

11 Jul '15

Dear Sir of Madam, I would like to see what impact Flow will have on dump files. I see that LiquidThreads still appear in the XML dumps of enwikinews (as of 20150602). Will the tables containing Flow records be dumped in SQL or XML format? Are there now any SQL or XML dumps that contain Flow records? If not, can you forecast when Flow will appear in the dumps? SIncerely Yours, Kent

2 1

Dump processing has died
by Bryan White 09 Jul '15

09 Jul '15

All the stubs have been done, but no other process has started up in the past ~36 hours. Rest of the dump aren't being done. Bryan

3 2

Occasional duplicate page records in XML data dumps
by gnosygnu 07 Jul '15

07 Jul '15

Hi. I've noticed that some June XML data dumps have duplicate <page> records, usually at the end of the dump. Anyone know if this is intentional? One or two duplicate records is benign, but I'm slightly concerned that it may be a symptom of a larger problem. I've been working with the XML data dumps for over 3 years, and haven't seen this before.[1] I list some examples below. They're only from the Swedish wikis and Spanish Wikipedia (which is what I started looking at this week) Let me know if you need any other info, and I'll be happy to provide. Finally, for questions like these, is it best to email the mailing list, create a task in Phabricator or do both? Thanks. [1]: It may have started as recently as 2015 April. I stopped looking at dumps shortly before the May problems with the dump server. ---- Example 1: URL: http://dumps.wikimedia.org/svwikiversity/20150602/svwikiversity-20150602-pa… Title: Audi m8 ID: 18942 SHA1: gd16v3qkmjr2w2j35zhqitjfg97igjt) Note: Last article in dump. Repeated twice Example 2: URL: http://dumps.wikimedia.org/svwikiquote/20150602/svwikiquote-20150602-pages-… Title: Sommarens tolv månader ID: 6209 SHA1: 9yibnev7pn3atxicayjoay0ave7pcu6 Note: Last article in dump. Repeated twice Example 3: URL: http://dumps.wikimedia.org/svwikibooks/20150602/svwikibooks-20150602-pages-… Title: Topologi/Metriska rum ID: 10001 SHA1: 5zdkpxflzdxhy7gxclludnlasvl6tw3 Note: Last article in dump. Repeated twice Example 4: URL: http://dumps.wikimedia.org/svwikisource/20150602/svwikisource-20150602-page… Title: Afhandling om svenska stafsÃ¤ttet/4 ID: 88768 SHA1: 7zyj208ur4vit0t41z7xlftlyl69bo7 Note: Last article in dump. Repeated twice Example 5: URL: http://dumps.wikimedia.org/eswiki/20150602/eswiki-20150602-pages-articles.x… Title (1): Veguer Title (2): Promo Note: duplicates are earlier in the dump (Veguer at the 9% mark and Promo at the 23% mark). There doesn't seem to be a dupe at the end of the article. Unaffected: * http://dumps.wikimedia.org/svwiki/20150602/svwiki-20150602-pages-articles.x… * http://dumps.wikimedia.org/svwiktionary/20150603/svwiktionary-20150603-page… * http://dumps.wikimedia.org/svwikinews/20150602/svwikinews-20150602-pages-ar…

2 2

added phabricator links to database schemata web page
by wp mirror 06 Jul '15

06 Jul '15

Dear Ariel, For most of the issues described in < https://www.mediawiki.org/wiki/Wikimedia_MediaWiki_Core_Team/Backlog/Improv… > I have added links to phabricator tasks. Sincerely Yours, Kent

2 1

Need an old dump
by Saurabh Sarda 05 Jul '15

05 Jul '15

Hi All, Sorry for the spam! I need an old wikipedia dump (Oct 2008) for some research work. I can't seem to find it anywhere. Can someone please suggest how can I procure the dump? If someone has the dump lying around on their machine(s), it would be great if they can share the dump. The dump file is named enwiki-20081008-pages-articles.xml.bz2 Thanks and Regards, Saurabh

2 1

Download an old wikipedia dump for research use
by Saurabh Sarda 02 Jul '15

02 Jul '15

Hi, I would like to know if there is a way to obtain dumps that are not listed on the website https://dumps.wikimedia.org/. In particular, I am looking for enwiki-20081008-pages-articles.xml.bz2 . Thanks & Regards, Saurabh

3 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l July 2015