Xmldatadumps-l August 2012

xmldatadumps-l@lists.wikimedia.org

3 participants
3 discussions

by Richard Jelinek

Hi, don't know if this issue came up already - in case it did and has been dismissed, I beg your pardon. In case it didn't... I hereby propose, that pbzip2 (https://launchpad.net/pbzip2) is used to compress the xml dumps instead of bzip2. Why? Because its sibling (pbunzip2) has a bug bunzip2 hasn't. :-) Strange? Read on. A few hours ago, I filed a bug report for pbzip2 (see https://bugs.launchpad.net/pbzip2/+bug/922804) together with some test results done even some few hours before that. The results indicate that: bzip2 and pbzip2 are vice-versa compatible each one can create archives, the other one can read. But if it is for uncomressing, only pbzip2 compressed archives are good for pbunzip2. I propose compressing the archives with pbzip2 for the following reasons: 1) If your archiving machines are SMP systems this could lead to a better usage of system ressources (i.e. faster compression). 2) Compression with pbzip2 is harmless for regular users of bunzip2, so everything should run for these people as usual. 3) pbzip2-compressed archives can be uncompressed with pbunzip2 with a speedup that scales nearly linearly with the number of CPUs in the host. So to sum up: It's a no loose and two win situation if you migrate to pbzip2. And that just because pbunzip2 is slightly buggy. Isn't that interesting? :-) cheers, -- Dipl.-Inf. Univ. Richard C. Jelinek PetaMem GmbH - www.petamem.com Geschäftsführer: Richard Jelinek Human Language Technology Experts Sitz der Gesellschaft: Fürth 69216618 Mind Units Registergericht: AG Fürth, HRB-9201

8 years, 3 months

Dumb file of a certain flattened category

by tzali maimon

Hello, As I forwarded below, I am looking to produce a list of IDs of all articles in wikipedia under a certain category (including its sub-categories and sub-sub-categories etc.) For example: a list of IDs of all articles under Mathematics. I wodered weather or not you have a dump file that has that list per category. I know you have a dump file for ALL categories which weighs 8GB. I am looking for partial lists and wondered if you produce them. I know Semantic MediaWiki can faltten a CategoryTree but how can I get the info of the list in one textual file? Thanks for the answer and your help Tzali Maimon -- צאלי

11 years, 8 months

Citizendium full XML dump: 168, 262 pages and 753, 651 revisions

by emijrp

Hi; I think this is the first time a full XML dump of Citizendium is publicly available[1] (CZ offers dumps but only the last revision for each article[2], and our previously efforts generated corrupted and incomplete dumps). It contains 168,262 pages and 753,651 revisions (9 GB, 99 MB in 7z). I think it may be useful for researchers, including quality analysis. It was generated using WikiTeam tools.[3] This is part of our task force to make backups of thousands of wikis around the Internet.[4] Regards, emijrp [1] http://archive.org/details/wiki-encitizendiumorg [2] http://en.citizendium.org/wiki/CZ:Downloads [3] http://code.google.com/p/wikiteam/ [4] http://code.google.com/p/wikiteam/wiki/AvailableBackups -- Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com Pre-doctoral student at the University of Cádiz (Spain) Projects: AVBOT <http://code.google.com/p/avbot/> | StatMediaWiki<http://statmediawiki.forja.rediris.es> | WikiEvidens <http://code.google.com/p/wikievidens/> | WikiPapers<http://wikipapers.referata.com> | WikiTeam <http://code.google.com/p/wikiteam/> Personal website: https://sites.google.com/site/emijrp/

11 years, 8 months

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l August 2012