Xmldatadumps-l December 2010

xmldatadumps-l@lists.wikimedia.org

9 participants
5 discussions

dataset1, xml dumps
by Ariel T. Glenn 11 Jan '11

11 Jan '11

For folks who have not been following the saga on http://wikitech.wikimedia.org/view/Dataset1 we were able to get the raid array back in service last night on the XML data dumps server, and we are now busily copying data off of it to another host. There's about 11T of dumps to copy over; once that's done we will start serving these dumps read-only to the public again. Because the state of the server hardware is still uncertain, we don't want to do anything that might put the data at risk until that copy has been made. The replacement server is on order and we are watching that closely. We have also been working on deploying a server to run one round of dumps in the interrim. Thanks for your patience (which is a way of saying, I know you are all out of patience, as am I, but hang on just a little longer). Ariel

7 15

Warning: dump file formats don't always match their version numbers!
by Andrew Dunbar 25 Dec '10

25 Dec '10

I've just discovered this and thought other people parsing MediaWiki dump files could also benefit. Dump files contain a version number in the root element eg: <mediawiki xmlns="http://www.mediawiki.org/xml/export-0.3/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.3/ http://www.mediawiki.org/xml/export-0.3.xsd" version="0.3" xml:lang="vo"> A description of the changes to the format can be found at http://www.mediawiki.org/xml/export-0.4.xsd : Version 0.2 adds optional basic file upload info support, which is used by our OAI export/import submodule. Version 0.3 adds some site configuration information such as a list of defined namespaces. Version 0.4 adds per-revision delete flags, log exports, discussion threading data, and a per-page redirect flag. Notice that per-page redirect flags are documented to to begin with version 0.4 dump files. In fact the per-page redirect flag seems to be used from 28 July 2009 and can be found in dump files marked as version 0.3 Given this there are surely other features which occur in versions earlier than documented so it would be wise to allow for this when parsing dump files rather than relying on the version declaration. Andrew Dunbar (hippietrail)

1 0

nice 20TB raid 6 array
by Jamie Morken 24 Dec '10

24 Dec '10

Hi all, I came across this video of a nice budget raid array from NCIX http://ncix.com/ http://www.youtube.com/watch?v=8ixfto_1Nl8 cheers, Jamie

1 0

Importing the categorylinks table - getting "Repair with keycache"
by Shaun Waksman 23 Dec '10

23 Dec '10

I'm trying to import the categorylinks table. After MySQL finishes the inserts, it reaches the line: ALTER TABLE `categorylinks` ENABLE KEYS By doing a "show full processlist" I can see that MySQL is running this doing "Repair with keycache". From what I've read about this online, it takes 20-30 times longer to build the indexes this way than using "Repair by sorting". Any ideas how to get around this?

1 0

Dumps
by Amgine 08 Dec '10

08 Dec '10

I've spoken with folx at Linterweb, who maintain a deposit of zeno-formatted[1] dumps at http://www.okawix.com/zenos/ Files ending in .okawix are tar files. They plan to increase the frequency, and are moving FR, DE, EN, ES, and PT to another server for particular support. This information was via Nytux at irc://irc.freenode.net/Linterweb Amgine [1] Zeno format information I've only been able to find in German at http://www.openzim.org/Zeno_File_Format and http://www.digitale-bibliothek.de/scripts/ts.dll?s=1&id=57107E74&mp=/pi/6/&…. This was found via the ZIM format discussion at http://www.openzim.org/ZIM_File_Format

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l December 2010