Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

compression tests
by Bryan White 07 Feb '16

07 Feb '16

Changing compression programs was mentioned on the list last month. Doing Google searches for one thing can bring up something totally different... https://github.com/powturbo/TurboBench This compares over 50 different compression programs with "pretty" graphs and tables as output. I think the test results on the web page uses part of an enwiki dump file. See http://mattmahoney.net/dc/text.html for information on the file. Note: The developer might be same person who did LZTurbo. A closed source fork of GPL Tornado compression program. If wondering, Tornado leaves BZ2 in the dust in terms of speed with same compression size, It is available for Linux and Windows, but is not in any Linux repository I could find. There is also Squash Compression Benchmark https://quixdb.github.io/squash-benchmark/ Bryan

1 1

Could somebody start the dumps
by Bryan White 02 Feb '16

02 Feb '16

Could somebody start the dumps. It would be nice to get two rounds of dumps in a month for a change. Also, shouldn't enwiki (dewiki? frwiki?) go first as they take the longest? I think the second round won't start till these are finished. Bryan

4 4

Extracting featured article meta-history dumps
by Anmol Dalmia 31 Jan '16

31 Jan '16

Hello. I wish to extract the meta- history dumps of the articles currently enlisted under various quality categories like Featured, Good, Stub, etc. Is there is list of such articles and their article ids or any such tool that can crawl through to produce these lists? Any help is highly appreciated. I have already been months on it. -- With Regards ANMOL DALMIA M Tech (Dual) Information Security, 2017 National Institute of Technology, Rourkela, India

2 1

Old Dumps
by Christian Morbidoni 27 Jan '16

27 Jan '16

Hi all, My name is Christian Morbidoni, I am a researcher at the University of Ancona, Italy, and I'm new to this list. We are using wikimedia pageviews and dumps in our research experiments in temporal mining and we are looking for a old dump (summer 2014) to mach our testing period. In particular what we would like to have is a complete list of all wikipedia titles as up to summer 2014. I see the archived dumps ( https://dumps.wikimedia.org/enwiki/) do not reach that date. Is there some way to reach the old dumps? may be upon request? Any help is appreciated. best, Christian

1 0

Re: [Xmldatadumps-l] [Wikitech-l] Wikipedia dumps
by Tilman Bayer 23 Jan '16

23 Jan '16

On Sun, Jan 10, 2016 at 4:05 PM, Bernardo Sulzbach < mafagafogigante(a)gmail.com> wrote: > On Sun, Jan 10, 2016 at 9:55 PM, Neil Harris <neil(a)tonal.clara.co.uk> > wrote: > > Hello! I've noticed that no enwiki dump seems to have been generated so > far > > this month. Is this by design, or has there been some sort of dump > failure? > > Does anyone know when the next enwiki dump might happen? > > > > I would also be interested. > > -- > Bernardo Sulzbach > > _______________________________________________ > Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > CCing the Xmldatadumps mailing list <https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l>, where someone has already posted <https://lists.wikimedia.org/pipermail/xmldatadumps-l/2016-January/001214.ht…> about what might be the same issue. -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB

13 22

pbzip2 proposal
by Richard Jelinek 16 Jan '16

16 Jan '16

Hi, don't know if this issue came up already - in case it did and has been dismissed, I beg your pardon. In case it didn't... I hereby propose, that pbzip2 (https://launchpad.net/pbzip2) is used to compress the xml dumps instead of bzip2. Why? Because its sibling (pbunzip2) has a bug bunzip2 hasn't. :-) Strange? Read on. A few hours ago, I filed a bug report for pbzip2 (see https://bugs.launchpad.net/pbzip2/+bug/922804) together with some test results done even some few hours before that. The results indicate that: bzip2 and pbzip2 are vice-versa compatible each one can create archives, the other one can read. But if it is for uncomressing, only pbzip2 compressed archives are good for pbunzip2. I propose compressing the archives with pbzip2 for the following reasons: 1) If your archiving machines are SMP systems this could lead to a better usage of system ressources (i.e. faster compression). 2) Compression with pbzip2 is harmless for regular users of bunzip2, so everything should run for these people as usual. 3) pbzip2-compressed archives can be uncompressed with pbunzip2 with a speedup that scales nearly linearly with the number of CPUs in the host. So to sum up: It's a no loose and two win situation if you migrate to pbzip2. And that just because pbunzip2 is slightly buggy. Isn't that interesting? :-) cheers, -- Dipl.-Inf. Univ. Richard C. Jelinek PetaMem GmbH - www.petamem.com Geschäftsführer: Richard Jelinek Human Language Technology Experts Sitz der Gesellschaft: Fürth 69216618 Mind Units Registergericht: AG Fürth, HRB-9201

6 11

2016-01 dumps halted?
by gnosygnu 11 Jan '16

11 Jan '16

Hi all. Sorry if this is a known issue, but dumps look like they've stopped after 2016-01-04. See https://dumps.wikimedia.org/backup-index.html Is there an ETA on resumption? Or is something else at stake? Let me know if you need more info. Thanks.

3 2

corrupted files english december
by Luigi Assom 28 Dec '15

28 Dec '15

Hello Wikiteam! just in time to say have a good vacation and happy 2016 :) well I am also here about corrupted files :) I downloaded three times from different wifi networks and using download managers from Firefox: enwiki-20151201-pages-articles-multistream.xml.bz2 and two times: enwiki-latest-pages-articles-multistream.xml.bz2 MD5 checksum is correct (*-latest-* have checksum of -*20151201-*) but file is corrupted. Cannot use bzip2recover for file is too large and I should recompile it because it will have more than maxlimit of generated tokens... and I think it is way better to take a fixed file :D Could you please check if I am the only one having this issue? dumps from other languages had worked fine for me, en-* is problematic.

3 3

dump problem?
by Bryan White 19 Dec '15

19 Dec '15

I see where almost all the dumps have "Dump complete" next to them and the data has been transferred to labs. Problem is, the dumps are not complete. Is this the new paradigm?... After each stage of the dump, label them done and then transfer what files were generated? Wash, rinse and repeat? Bryan

4 3

Request for input on de fr wiki breaking change to dumps!
by Ariel T. Glenn 02 Dec '15

02 Dec '15

I assume you have all seen https://phabricator.wikimedia.org/T116907 "Explore the possibility of splitting dewiki and frwiki into smaller chunks" If not, and you ever use frwiki or dewiki page content dumps, go read it now. Or if you know of anyone who uses them, please nag them to go read it. The upshot is that we will most likely on January 1st 2016 do all further dump runs of frwiki and dewiki with so-called 'checkpointing'. This change is being made so that if one of these jobs is interrupted for whatever reason, it can be rerun with only the missing page ranges dumped on the second run, saving quite a lot of time. A second reason is to ease the burden on downloaders, who generally prefer downloading several smaller files rather than one large 90gb file (example taken from dewiki history dumps). WHat does this mean in practice for you, users of the dumps? It means that filenames for the page content (article, meta-current and meta -history) dumps will have pXXXXpYYYY in the names, where XXXX is the first page id in the file and YYY is the last pageid in the file. For examples of this you can look at the enwiki page content dumps, which have been running that way for a few years now. This notice should give you plenty of time to convert your tools to use the new nameing scheme. I encourage you to forward this message to other appropriate people or groups. Thanks, Ariel

2 3

← Newer
1
...
28
29
30
31
32
33
34
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l