Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

Your help requested (testing decompression)
by Ariel Glenn WMF 08 Jan '20

08 Jan '20

if you use one of the utilities listed here: https://phabricator.wikimedia.org/T239866 I'd like you to download one of the 'multistream' dumps and see if your utility decompresses it fully or not (you can compare the md5sum of the decompressed content to the regular file's decompressed content and see if they are the same). Then note the results and the version of the utility on this task. Alternatively, if you use some other utility to work with the bz2 files, please test using that, and add that on the task too. Here are two files for download and comparison of decompressed content: https://dumps.wikimedia.org/cewiki/20191201/cewiki-20191201-pages-articles.… and https://dumps.wikimedia.org/cewiki/20191201/cewiki-20191201-pages-articles-… Both are around 50 megabytes. Thank you in advance to whomever participates! Ariel

1 1

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 01 Jan '20

01 Jan '20

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20191201 full revision history content run. We are currently dumping 910 projects in total. --------------------- Stats for bnwiktionary on date 20191201 Total size of page content dump files for articles, current content only: 22543877 Total size of page content dump files for all pages, current content only: 34990319 Total size of page content dump files for all pages, all revisions: 181821103 --------------------- Stats for enwiki on date 20191201 Total size of page content dump files for articles, current content only: 74758773669 Total size of page content dump files for all pages, current content only: 166645226713 Total size of page content dump files for all pages, all revisions: 19963955381507 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 02 Dec '19

02 Dec '19

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20191101 full revision history content run. We are currently dumping 910 projects in total. --------------------- Stats for wikimania2015wiki on date 20191101 Total size of page content dump files for articles, current content only: 15017153 Total size of page content dump files for all pages, current content only: 15907454 Total size of page content dump files for all pages, all revisions: 245100957 --------------------- Stats for enwiki on date 20191101 Total size of page content dump files for articles, current content only: 74373625381 Total size of page content dump files for all pages, current content only: 165746251934 Total size of page content dump files for all pages, all revisions: 19823027933877 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

BREAKING CHANGE: schema update, xml dumps
by Ariel Glenn WMF 27 Nov '19

27 Nov '19

We plan to move to the new schema for xml dumps for the February 1, 2020 run. Update your scripts and apps accordingly! The new schema contains an entry for each 'slot' of content. This means that, for example, the commonswiki dump will contain MediaInfo information as well as the usual wikitext. See https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/master/docs… for the schema and https://www.mediawiki.org/wiki/Requests_for_comment/Schema_update_for_multi… for further explanation and example outputs. Phabricator task for the update: https://phabricator.wikimedia.org/T238972 PLEASE FORWARD to other lists as you deem appropriate. Thanks! Ariel Glenn

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 31 Oct '19

31 Oct '19

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20191001 full revision history content run. We are currently dumping 922 projects in total. --------------------- Stats for cowikimedia on date 20191001 Total size of page content dump files for articles, current content only: 2144131 Total size of page content dump files for all pages, current content only: 3355092 Total size of page content dump files for all pages, all revisions: 80294987 --------------------- Stats for enwiki on date 20191001 Total size of page content dump files for articles, current content only: 73972175645 Total size of page content dump files for all pages, current content only: 164881705920 Total size of page content dump files for all pages, all revisions: 19682086841974 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Comments requested: produce empty abstract files for Wikidata?
by Ariel Glenn WMF 21 Oct '19

21 Oct '19

Currently, the abstracts dump for Wikidata consists of 62 million entries, all of which contain <abstract 'not-applicable' /> instead of any real abstract. Instead of this, I am considering producing abstract files that would contain only the mediawiki header and footer and the usual siteinfo contents. What do people think about this? Rationale: It takes 36 hours of time to produce these useless files. It places an extra burden on the db servers for no good reason. It requires more bandwidth to download and process these useless files than having a file with no entries. Wikidata will only ever have Q-entities or other entities in the main namespace that are not text or wikitext and so are not suitable for abstracts. Please comment here or on the task: https://phabricator.wikimedia.org/T236006 If there are no comments or blockers after a week, I'll start implementing this, and it will likely go into effect for the November 20th run. Your faithful dumps wrangler, Ariel Glenn

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 01 Oct '19

01 Oct '19

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20190901 full revision history content run. We are currently dumping 920 projects in total. --------------------- Stats for bawiki on date 20190901 Total size of page content dump files for articles, current content only: 507004601 Total size of page content dump files for all pages, current content only: 533549451 Total size of page content dump files for all pages, all revisions: 11862864063 --------------------- Stats for enwiki on date 20190901 Total size of page content dump files for articles, current content only: 73556940996 Total size of page content dump files for all pages, current content only: 164031720789 Total size of page content dump files for all pages, all revisions: 19534783920279 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

XML Dump at version 20190120
by Asuka Sumida 25 Sep '19

25 Sep '19

I had found two XML Dump because of download mistake. * Language Link SQL in Hindu Language(hi) at version 20190120 (hiwiki-20190120-langlinks.sql.gz) * Cirrus Search Dump in Greek Language(el) at version 20190120 (elwiki-20190120-cirrussearch-content.json.gz) I found these in Internet Archive(https://archive.org/) and torrent data(https://tools.wmflabs.org/dump-torrents/). But I can not find these data. If this topic is not suitable on this mailing list, please tell me appropriate something. regards, Asuka

1 0

Ngaý 13_09_2019 XSMB lô dê có
by សុន សុវណ្ណារឹទ្ធ official 12 Sep '19

12 Sep '19

Hello sir please help send the lottery or data ngay 13_09_2019 to. Thanks your

1 0

Incremental dumps
by Dario Montagnini 11 Sep '19

11 Sep '19

Hello, I would like to know if there are information about the incremental dumps. I noticed that the generation has been suspended for about six days. Thank you!

3 2

← Newer
1
...
13
14
15
16
17
18
19
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l