Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

dataset1, xml dumps
by Ariel T. Glenn 11 Jan '11

11 Jan '11

For folks who have not been following the saga on http://wikitech.wikimedia.org/view/Dataset1 we were able to get the raid array back in service last night on the XML data dumps server, and we are now busily copying data off of it to another host. There's about 11T of dumps to copy over; once that's done we will start serving these dumps read-only to the public again. Because the state of the server hardware is still uncertain, we don't want to do anything that might put the data at risk until that copy has been made. The replacement server is on order and we are watching that closely. We have also been working on deploying a server to run one round of dumps in the interrim. Thanks for your patience (which is a way of saying, I know you are all out of patience, as am I, but hang on just a little longer). Ariel

7 15

Dumps
by Root Framework 09 Jan '11

09 Jan '11

Hello I've some questions for you and I think it won't be so hard for you to answer. What do you know about new wikipedia dumps? When they will be refreshed and available to be downloaded (new dumps) I've heard that they can give you an opportunity to order you own dump but I don't now how to do it. Can you help me? And the last: http://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_XML_dumps I think they are in searching for volunteers and mirrores for their projects. Can you support them and start mirroring for some languages? Thank you.

1 0

New Dumps
by Andreas Meier 06 Jan '11

06 Jan '11

Hello, first a happy new year to everybody. Are there any news about generating new dumps? Best regards Andreas

1 0

Warning: dump file formats don't always match their version numbers!
by Andrew Dunbar 26 Dec '10

26 Dec '10

I've just discovered this and thought other people parsing MediaWiki dump files could also benefit. Dump files contain a version number in the root element eg: <mediawiki xmlns="http://www.mediawiki.org/xml/export-0.3/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.3/ http://www.mediawiki.org/xml/export-0.3.xsd" version="0.3" xml:lang="vo"> A description of the changes to the format can be found at http://www.mediawiki.org/xml/export-0.4.xsd : Version 0.2 adds optional basic file upload info support, which is used by our OAI export/import submodule. Version 0.3 adds some site configuration information such as a list of defined namespaces. Version 0.4 adds per-revision delete flags, log exports, discussion threading data, and a per-page redirect flag. Notice that per-page redirect flags are documented to to begin with version 0.4 dump files. In fact the per-page redirect flag seems to be used from 28 July 2009 and can be found in dump files marked as version 0.3 Given this there are surely other features which occur in versions earlier than documented so it would be wise to allow for this when parsing dump files rather than relying on the version declaration. Andrew Dunbar (hippietrail)

1 0

nice 20TB raid 6 array
by Jamie Morken 24 Dec '10

24 Dec '10

Hi all, I came across this video of a nice budget raid array from NCIX http://ncix.com/ http://www.youtube.com/watch?v=8ixfto_1Nl8 cheers, Jamie

1 0

Importing the categorylinks table - getting "Repair with keycache"
by Shaun Waksman 23 Dec '10

23 Dec '10

I'm trying to import the categorylinks table. After MySQL finishes the inserts, it reaches the line: ALTER TABLE `categorylinks` ENABLE KEYS By doing a "show full processlist" I can see that MySQL is running this doing "Repair with keycache". From what I've read about this online, it takes 20-30 times longer to build the indexes this way than using "Repair by sorting". Any ideas how to get around this?

1 0

Dumps
by Amgine 08 Dec '10

08 Dec '10

I've spoken with folx at Linterweb, who maintain a deposit of zeno-formatted[1] dumps at http://www.okawix.com/zenos/ Files ending in .okawix are tar files. They plan to increase the frequency, and are moving FR, DE, EN, ES, and PT to another server for particular support. This information was via Nytux at irc://irc.freenode.net/Linterweb Amgine [1] Zeno format information I've only been able to find in German at http://www.openzim.org/Zeno_File_Format and http://www.digitale-bibliothek.de/scripts/ts.dll?s=1&id=57107E74&mp=/pi/6/&…. This was found via the ZIM format discussion at http://www.openzim.org/ZIM_File_Format

1 0

Re: [Xmldatadumps-l] Mirroring Wikimedia project XML dumps
by emijrp 16 Nov '10

16 Nov '10

Thanks, Paolo. I hadn't added archive.org because it requires us to upload the files, and the files cannot be deleted in the future, so, upload the latest dumps (updated every month or so) could be a waste of resources for them. I was thinking about mirrors who runs wget to slurp all the files from download.wikimedia.org, and the next month delete the previous ones. Of course, we can contact to archive.org to ask them about the wget idea. Regards, emijrp 2010/11/16 paolo massa <paolo(a)gnuband.org> > I've added archive.org ;) > > On Tue, Nov 16, 2010 at 12:05 PM, emijrp <emijrp(a)gmail.com> wrote: > > Hi all; > > > > I have started a new page in meta: for coordinating the efforts in > mirroring > > Wikimedia project XML dumps. I asked some days ago to iBiblio if they > were > > interested in this, but they replied: "Unfortunately, we do not have the > > resources to provide a mirror of wikipedia. Best of luck!" > > > > I think that we must work on this, so, all the help is welcome. If you > know > > about universities, archives, etc, that could be interested in get a copy > of > > the XML files, for backup or research purposes, please, add them to the > list > > and we can send them a letter. > > > > We are compiling all the human knowledge! That deserves being mirroring > ad > > nauseam! > > > > Regards, > > emijrp > > > > [1] > > > https://secure.wikimedia.org/wikipedia/meta/wiki/Mirroring_Wikimedia_projec… > > > > _______________________________________________ > > Xmldatadumps-l mailing list > > Xmldatadumps-l(a)lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l > > > > > > > > -- > -- > Paolo Massa > Email: paolo AT gnuband DOT org > Blog: http://gnuband.org >

2 1

Mirroring Wikimedia project XML dumps
by emijrp 16 Nov '10

16 Nov '10

Hi all; I have started a new page in meta: for coordinating the efforts in mirroring Wikimedia project XML dumps. I asked some days ago to iBiblio if they were interested in this, but they replied: *"Unfortunately, we do not have the resources to provide a mirror of wikipedia. Best of luck!" *I think that we must work on this, so, all the help is welcome. If you know about universities, archives, etc, that could be interested in get a copy of the XML files, for backup or research purposes, please, add them to the list and we can send them a letter. We are compiling all the human knowledge! That deserves being mirroring ad nauseam! Regards, emijrp [1] https://secure.wikimedia.org/wikipedia/meta/wiki/Mirroring_Wikimedia_projec…

1 0

Re: [Xmldatadumps-l] Temporary dump mirror locations?
by Diederik van Liere 15 Nov '10

15 Nov '10

I was thinking of the last 3 - 4 month, thus July - October 2010. On 2010-11-15, at 10:30 AM, emijrp wrote: > Define recent. > > 2010/11/12 Diederik van Liere <dvanliere(a)gmail.com> > Hi, > > As we all know, download.wikimedia.org is temporarily offline. Does > somebody have a recent stub-meta-history.xml available (any language > is okay)? > > Best regards, > > > Diederik > > _______________________________________________ > Xmldatadumps-l mailing list > Xmldatadumps-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l >

3 5

← Newer
1
...
62
63
64
65
66
67
68
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l