Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 03 Mar '19

03 Mar '19

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20190201 full revision history content run. We are currently dumping 917 projects in total. --------------------- Stats for angwikibooks on date 20190201 Total size of page content dump files for articles, current content only: 1377805 Total size of page content dump files for all pages, current content only: 1929910 Total size of page content dump files for all pages, all revisions: 6492585 --------------------- Stats for enwiki on date 20190201 Total size of page content dump files for articles, current content only: 70922565540 Total size of page content dump files for all pages, current content only: 158336597712 Total size of page content dump files for all pages, all revisions: 18607622666061 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

question about wikidata entity dumps usage (please forward to interested parties)
by Ariel Glenn WMF 19 Feb '19

19 Feb '19

Hey folks, We've had a request to reschedule the way the various wikidata entity dumps are run. Right now they go once a week on set days of the week; we've been asked about pegging them to specific days of the month, rather as the xml/sql dumps are run. See https://phabricator.wikimedia.org/T216160 for more info. Is this going to cause problems for anyone? Do you ingest these dumps on a schedule, and what works for you? Please weigh in here or on the phabricator task; thanks! Ariel

2 1

New dumps mirror: The Free Mirror Project
by Ariel Glenn WMF 06 Feb '19

06 Feb '19

I am happy to announce a new mirror site, located in Canada, which is hosting the last two good dumps of all projects. Please welcome and put to good use https://dumps.wikimedia.freemirror.org/ ! I want to thank Adam for volunteering bandwidth and space and for getting everything set up. More information about the project can be found at http://freemirror.org/ Enjoy! Ariel

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 30 Jan '19

30 Jan '19

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20190101 full revision history content run. We are currently dumping 917 projects in total. --------------------- Stats for rmywiki on date 20190101 Total size of page content dump files for articles, current content only: 2372738 Total size of page content dump files for all pages, current content only: 5012147 Total size of page content dump files for all pages, all revisions: 225293841 --------------------- Stats for enwiki on date 20190101 Total size of page content dump files for articles, current content only: 70514635946 Total size of page content dump files for all pages, current content only: 157464548890 Total size of page content dump files for all pages, all revisions: 18472280381222 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

incorrect links for pages articles multistream files for big wikis
by Ariel Glenn WMF 23 Jan '19

23 Jan '19

Folks may have noticed already that the links presented for downlod of pages-articles-multistream dumps are incorrect on the web pages for big wikis. The files exist for download but the wrong links were created. I'll be looking into that and fixing it up over the next days, but in the meantime you can manually download the files by specifying the right name. Apologies for the inconvenience. Ariel

1 0

no name listed for file contained in a 7z file?
by Albretch Mueller 21 Jan '19

21 Jan '19

I downloaded: http://dumps.wikimedia.your.org/other/static_html_dumps/2008-06/en/wikipedi… using wget and it seems to be fine: $ _IFL="wikipedia-en-html.tar.7z" $ ls -l "${_IFL}" -rw-r--r-- 1 niggahme niggahme 15363543213 Jun 21 2008 wikipedia-en-html.tar.7z $ file "${_IFL}" wikipedia-en-html.tar.7z: 7-zip archive data, version 0.2 $ md5sum -b "${_IFL}" 03ce695cbf32a3f8636fa8d3f9c7d12e *wikipedia-en-html.tar.7z $ sha256sum -b "${_IFL}" c2794b6371a05017f03e2a345730fd763b1052872290b5c78763978a0b43c747 *wikipedia-en-html.tar.7z $ sha512sum -b "${_IFL}" d52a737ceca25ef18272ba70a4a56000a7a0bff92653fb462674333a0855f397c892b8aeb2e11206d391ba4cca48d46f5814d92db4d2096467519de38c5a189c *wikipedia-en-html.tar.7z $ 7z l "${_IFL}" 7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21 p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,2 CPUs Intel(R) Pentium(R) CPU B940 @ 2.00GHz (206A7),ASM) Scanning the drive for archives: 1 file, 15363543213 bytes (15 GiB) Listing archive: wikipedia-en-html.tar.7z -- Path = wikipedia-en-html.tar.7z Type = 7z Physical Size = 15363543213 Headers Size = 100 Method = LZMA:22 Solid = - Blocks = 1 Date Time Attr Size Compressed Name ------------------- ----- ------------ ------------ ------------------------ 2008-06-18 13:02:15 ..... 223674511360 15363543113 wikipedia-en-html.tar ------------------- ----- ------------ ------------ ------------------------ 2008-06-18 13:02:15 223674511360 15363543113 1 files $ But I ca'nt get the name of the compressed/contained file even though ark and 7z show it to you. Here is my simple piece of code: String aIFl = "wikipedia-en-html.tar.7z"; File I7ZKFl = new File(aIFl); if(I7ZKFl.exists()){ try{ SevenZFile SvnZFl = new SevenZFile(I7ZKFl); SevenZArchiveEntry entry; int iIx = 0; while((entry = SvnZFl.getNextEntry()) != null){ System.out.println("// __ [" + iIx + "]: |" + entry + "|"); System.out.println("// __ .getName() |" + entry.getName() + "|"); System.out.println("// __ .getSize() |" + entry.getSize() + "|"); System.out.println("// __ .getLastModifiedDate() |" + entry.getLastModifiedDate() + "|"); ++iIx; }// ((entry = SvnZFl.getNextEntry()) != null) }catch(IOException IOX){ IOX.printStackTrace(System.err); } } which, except for the name, its faithful output was: // __ [0]: |org.apache.commons.compress.archivers.sevenz.SevenZArchiveEntry@179d3b25| // __ .getName() |null| // __ .getSize() |223674511360| // __ .getLastModifiedDate() |Wed Jun 18 14:02:15 EDT 2008| Why is it that I can't get the file name? Also, if OO works, I should be able to access and process this file while addressing it like (using an exclamation mark): wikipedia-en-html.tar.7z!wikipedia-en-html.tar So, I this point I should be able to go: String aIFl = "wikipedia-en-html.tar.7z!wikipedia-en-html.tar" FileInputStream FISTarK = new FileInputStream(new File(aIFl)); TarArchiveInputStream tarInput = new TarArchiveInputStream(FISTarK); TarArchiveEntry tArKEnt; while((tArKEnt=tarInput.getNextTarEntry()) != null){ ... } right? lbrtchx

1 0

Change in multistream dump file production
by Ariel Glenn WMF 19 Jan '19

19 Jan '19

TL;DR: Don't panic, the single articles multistream bz2 file for big wikis will be produced shortly after the new smaller fles. Long version: For big wikis which already have split up article files, we now produce one multistream file per article file. These are now recombined into a single file later, wth a single index file in the fashion everyone is used to. This part of the speedup work mentioned in the previous email. Have a good weekend, Ariel

1 0

mwbzutils BREAKING CHANGE
by Ariel Glenn WMF 19 Jan '19

19 Jan '19

If you use recompressxml in the mwbzutils package, as of version 0.0.9 (just deployed) it no longer writes bz2 compressed data by default to stdout; instead it relies on the extension of the output file and will write either gzipped, bz2 or plain text output, accordingly. This means that if it is directed to write to stdout, this will be uncompressed data. You can work around this in your scripts by piping the text from stdout to bzip directly from recompressxml. This change came as part of some speedup work. I won't discuss that more until we see how the next couple of runs go. Thanks for your understanding. Ariel

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 30 Dec '18

30 Dec '18

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20181201 full revision history content run. We are currently dumping 916 projects in total. --------------------- Stats for euwikibooks on date 20181201 Total size of page content dump files for articles, current content only: 8729127 Total size of page content dump files for all pages, current content only: 9378881 Total size of page content dump files for all pages, all revisions: 214022959 --------------------- Stats for enwiki on date 20181201 Total size of page content dump files for articles, current content only: 70084335596 Total size of page content dump files for all pages, current content only: 156517701014 Total size of page content dump files for all pages, all revisions: 18342890924003 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Article geotags missing for some wikis/languages?
by Martin Dittus 06 Nov '18

06 Nov '18

Hallo, I noticed that for some larger wikis, the geo_tags.sql.gz dump file does not include any geotags found in articles. Is it possible to determine why this is, and for which languages this is the case? For example, the geotags dump file for Indonesian (a wiki with >400,000 articles) is only 7kb large, and all geotags in it are from user pages, file uploads, or file templates; but not from articles: https://dumps.wikimedia.org/idwiki/20181020/ Yet it doesn't take much effort to find pages that are geotagged, such as this one (see the infobox): https://id.wikipedia.org/wiki/London I realise that there are a number of alternative tagging conventions. Does idwiki possibly use a geotagging scheme that is not supported by some part of this ingestion/export process? Which other wikis may fall in this category? I tried to find the script(s) that populate the geo_tags table from page content but so far had no luck, as I'm not sufficiently familiar with WP's software architecture; if someone can point me in the right direction I'd be happy to investigate myself. Many thanks! m.

1 0

← Newer
1
...
17
18
19
20
21
22
23
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l