Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

Mediawiki: Issue with maintenance-script rebuildImages.php
by WOHLFAHRT Roland (Gemeindeinformatikzentrum Kärnten GI Z-K GmbH) 11 Jan '12

11 Jan '12

Hi everybody I have a migration-issue and tried your maintenance-Script rebuildImages.php. See http://meta.wikimedia.org/wiki/Talk:Data_dumps#rebuildImages.php_fails Environment: - Windows 2008 r2 with xampp - Mediawiki 1.18.0 I did the migration this way because I wanted a "fresh" mediawiki installation. I found no good place to ask for support, because the script has no article on mediawiki.org. Perhaps you can give me a small hint or workaround to get my wiki up&running. Thx a lot. Cheers, Roland

1 0

List of all words of a wiktionary
by Sébastien Druon 10 Jan '12

10 Jan '12

Hello, How is it possible to get the list of all the entries (words) of a wiktionary? For example, for the russian wiktionary, I want to get the list of all the russian entries (no other languages) Thanks a lot in advance, Sebastien

4 5

Best way to get html/parsed version of a wiktionary
by Sébastien Druon 09 Jan '12

09 Jan '12

Hi! What is the best/easiest way to get a parsed version (including template resolution) of all entries of a wiktionary (separate html files for each entry for example). Thanks in advance Sebastien

2 1

Import of an XML dump of the RU Wiktionary with mwdumper / problem
by Sébastien Druon 08 Jan '12

08 Jan '12

Hello! I imported the ruwiktionary dump with mwdumper, but I cannot access any word. For each of the I get a similar page: For example for the page http://localhost/mediawiki/index.php/Ребёнок I get the following result: Ребёнок (Redirected from Ребёнок<http://localhost/mediawiki/index.php?title=%D0%A0%D0%B5%D0%B1%D1%91%D0%BD%D…> ) [image: #REDIRECT]Ребёнок<http://localhost/mediawiki/index.php/%D0%A0%D0%B5%D0%B1%D1%91%D0%BD%D0%BE%D…> The templates do not seem to be correctly rendered either. Any idea what is missing? Thnaks in advance, Sebastien Druon

4 6

Pages-logging dump - more information?
by Katja Mueller 04 Jan '12

04 Jan '12

Dear list, do you provide more information about the pages-logging dump somewhere? While parsing it we came across some questions that we are trying to clarify: * Why are there only ~40 million logs (at over 450 million revisions)? Which logs does the pages-logging dump / the "public" logging table (not) contain? (We double-checked the number of logs on the database, using our Toolserver-Account (Logging table).) * Can you give us more information about the TextElement that is defined in the XML Schema? Definition: <element name="text" type="mw:TextType"/> In pages-logging it sometimes occurs as: <text deleted="deleted" /> How is this being used? Kind regards, Katja Mueller do you provide more information about the pages-logging dump somewhere? While parsing it we came across some questions that we are trying to clarify: * Why are there only ~40 million logs (at over 450 million revisions)? Which logs does the pages-logging dump / the "public" logging table contain? We double-checked the number of logs on the database, using our Toolserver-Account (logging-table). * Can you give us more information about the TextElement that is defined in the XML Schema? Definition: <element name="text" type="mw:TextType"/> In pages-logging it sometimes occurs as: <text deleted="deleted" /> How is this used? Kind regards, Katja Mueller

1 0

Something new for the new year
by Ariel T. Glenn 04 Jan '12

04 Jan '12

Hello and happy new year to all fellow xml-crunchers out there. Anyone who uses the bz2 file format of the page dumps, either for hadoop or for some other reason might be interested in the following: (For en wikipedia only for right now) I've written a little program that will write a second copy of the pages-articles bz2 file, as concatenated multiple bz2 streams with 100 pages per stream, with a separate index bz2 file which contains a list of offsets/page ids/page titles where each offset is to the start of the particular bz2 stream in the file. The nice thing about multiple streams is that essentially these behave like separate bz2 files, so you can just seek to that point in the file, pass the data starting from that byte directly into the bz2 decompresser of your choice, and work with it. No need to monkey around with bit-aligned crap, nor with fudging together a bz2 header to fool the library into thinking it's looking at a full file, nor with tossing away the crc at the end. The 100-pages-per stream makes the output somewhat bigger than the regular pages-articles file but not excessively so. I'm hoping this format will be useful to folks working with offline readers, hadoop or other analysis tools. Let me know. A first run of these, for the December en wp pages-articles file, is available at http://dumps.wikimedia.org/other/multistream/ and the second run (assuming this works, I just deployed the changes to the python scripts) will be generated in the usual way as part of the regular dumps and found with them at the normal location. If the job runs ok I expect to enable it on the other projects soon afterwards. Ariel

1 0

Ogg files& wiktionary dump
by Sébastien Druon 22 Dec '11

22 Dec '11

Hi! I would like to work with a local copy of a wiktionary and I have the following question: - the dumps do not provide any files like images or sound; - is there any way to get them separately? I would need the whole ogg file for specific wiktionaries - if not, is it possible to get them from the public wiktionary server? in that case, I would have to send many thousands of requests to the server: is it allowed? Thanks a lot in advance for the answers. Best regards, Sebastien Druon

2 1

Stopped mirroring?
by Hydriz Wikipedia 12 Dec '11

12 Dec '11

Hi guys, While I was trying to rsync all the dumps available on the mirror site, I noticed that the latest dump available was only made in early October, what happened to the newer dumps made on the Brazilian mirror? Has it stopped following updates from dumps.wikimedia.org? BTW if anyone of you were following the mirroring XML dumps talk page, I have just sent an email to ask for AARNet to mirror our dumps too. They have about 84TB of space available and is located in Australia. Cross your fingers! Regards, Hydriz

2 1

"add/changes" dumps... highly experimental, you have been warned
by Ariel T. Glenn 24 Nov '11

24 Nov '11

Hello folks, So this has been running all of one day now, and I expect it to break in wild and crazy ways over the next period while we get the bugs out. But, throwing caution to the winds... I'm generating dumps each day for each non-closed non-private project, of revisions added since the previous day. It uses the standard xml format, writing out stubs and history files. This is a sort of poor person's incremental dump. What do I mean by that? Well... It doesn't contain a list of deletions, page moves, undeletes. It just dumps the metadata and text for every revision between X1 (last revision dumped the day before) and X2 (last revision in db as of the time it's dumped). The reason for that? Dumping a range of revisions is relatively easy. Accounting for page deletions, moves etc. since the previous dump is hard, so that is an exercise left to the reader :-P Even with these limitations I'm hoping the data will be useful to folks. These are specifically *not* intended to be kept around forever; we'll keep some reasonable number, 20-30 of them, and then start tossing old ones after that. A note about the timing of the dumps: they run once a day, there's no progress reporting. An updated index file is published near the end of the day. Also, we dump content with a delay of 12 hours, to allow admins to delete things that might contain sensitive information. This was less of a concern for dumps generated once a week, but daily runs increase the odds of something bad getting dumped. And speaking of the index file, it's here: http://dumps.wikimedia.org/other/incr/ Guess I'll add some documentation on wikitech too. The code is in my branch in svn, see http://svn.wikimedia.org/viewvc/mediawiki/branches/ariel/xmldumps-backup/in… I may well be patching things tomorrow at this time for jobs that failed to run, so feel free to point out issues, but also don't be surprised by frequent outages. Happy trails, Ariel

4 3

Re: [Xmldatadumps-l] Xmldatadumps-l Digest, Vol 21, Issue 2 - proposals for categories of Commons
by burslem 23 Nov '11

23 Nov '11

> Ariel: >> Providing multiple terabyte sized files for download doesn't make any kind of sense to me. >> However, if we get concrete proposals for categories of Commons images people really want >> and would use, we can put those together. I think this has been said before on wikitech-l if not here. The Picture of the Year (POTY) collections are truly stunning! I am not very interested in having terabytes of random snapshots on my computer, instead I find smaller collections of "best of the best" much more suiting for the public. This way, it will be accessible to those with smaller amounts of diskspace and they'll be equally impressed. I doubt there are many people interested in the image tarballs at all, they're just going for the principle of accessibility. Presumably wikipedia has plenty of back-up capabilities and there are enough gurus doing everything to prevent possible data loss. Offering public back-ups has no additional value in this perspective. Most of us probably use the wikipedia XML's for offline usage or research. I have not yet come across image research projects requiring tens of terabytes of images to be successful! I say, if the POTY downloads are popular according to statistics, why not compile a couple more years? The thing I'm talking about is hosted at http://dumps.wikimedia.org/other/poty/ .

3 5

← Newer
1
...
56
57
58
59
60
61
62
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l