Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

BREAKING CHANGE: feature removal (private table dumps)
by Ariel Glenn WMF 06 Apr '20

06 Apr '20

For the past few years we have not dumped private tables at all; they would not be accessible to the public in any case, and they do not suffice as a backup in case of catastrophic failure. We are therefore removing the feature to dump private tables along with public tables in a dump run. Anyone who wishes to use the dump scripts in our python repo to dump privat tables in their wiki will need to create a separate dumps configuration file and tables yaml file describing which tables to dump and where to put them, as a separate dump run. This change will be committed by April 20, 2020, in time for the second dump run of the month. Note that this does not impact the actual output of the Wikimedia SQL/XML dumps at all, since we have not been dumping private tables since late 2016. See T249508 to follow along. Ariel

1 0

Duplicate entry in last Spanish dump
by Benjamín Valero Espinosa 06 Apr '20

06 Apr '20

Hi, Sorry if this is not the right place to report this. In the last Spanish Wikipedia dump (still in progress): https://dumps.wikimedia.org/eswiki/20200401/ the "pages-articles" dump is duplicated. I guess, based on the dump from March (and also the previous ones) that they are really two different files, so the filename should reflect it as before. I am aware there had been recent modifications in the way the multistream dumps are built, so maybe there is some kind of issue there. Best Regards,

2 2

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 01 Apr '20

01 Apr '20

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20200301 full revision history content run. We are currently dumping 910 projects in total. --------------------- Stats for euwiktionary on date 20200301 Total size of page content dump files for articles, current content only: 67765999 Total size of page content dump files for all pages, current content only: 68901734 Total size of page content dump files for all pages, all revisions: 532573202 --------------------- Stats for enwiki on date 20200301 Total size of page content dump files for articles, current content only: 76154026008 Total size of page content dump files for all pages, current content only: 169370554542 Total size of page content dump files for all pages, all revisions: 20379079466354 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

No second dump run this month
by Ariel Glenn WMF 19 Mar '20

19 Mar '20

As mentioned earlier on the xmldatadumps-l, the dumps are running very slow this month, ince the vslow db hosts they use are also serving live traffic during a tables migration. Even manual runs of partial jobs would not help the situation any, so there will be NO SECOND DUMP RUN THIS MONTH. The March 1 Wikidata run is still in process but it should complete in the next several days. With any luck everything will be back to normal in April and we'll be able to conduct two runs as usual from then on. Ariel

2 1

Monthly dumps for March; possible no second run
by Ariel Glenn WMF 17 Mar '20

17 Mar '20

Hello everybody, Those of you who follow the dumps closely may have notice that they are running slower than usual this month. That is because the db servers on which they run are also serving live traffic, so that a wikidata-related migration can complete before the end of the month. I will try to do extra manual runs to see if I can get the wikidata page content dumps to complete by the 20th, but if it turns out not to be feasible, we may delay or skip the March 20th run. In any case the two April runs should happen on schedule. Thanks in advance for your understanding. Ariel P.S. Please stay safe and look after each other.

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 03 Mar '20

03 Mar '20

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20200201 full revision history content run. We are currently dumping 910 projects in total. --------------------- Stats for elwikinews on date 20200201 Total size of page content dump files for articles, current content only: 17648606 Total size of page content dump files for all pages, current content only: 23700081 Total size of page content dump files for all pages, all revisions: 572589887 --------------------- Stats for enwiki on date 20200201 Total size of page content dump files for articles, current content only: 75698075778 Total size of page content dump files for all pages, current content only: 168456378158 Total size of page content dump files for all pages, all revisions: 20244384797754 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

kowiki joins the ranks of the 'big wikis'
by Ariel Glenn WMF 28 Feb '20

28 Feb '20

Happy almost March, everyone! Kowiki dumps jobs now take long enough to run for certain steps that the wiki has been moved to the 'big wikis' list. This means that 6 parallel jobs will produce output for stubs and page content dumps, similarly to frwiki, dewiki and so on. See [1] for more. This will take effect with the next dump run, starting tomorrow. Please adjust your scripts accordingly. Ariel [1] https://phabricator.wikimedia.org/T245721

1 0

Was format change plan postponed?
by Itsuki Toyota 12 Feb '20

12 Feb '20

Hi, Xmldatadumps team As you know, the format of xml dumps should change after February 1, 2020: https://lists.wikimedia.org/pipermail/xmldatadumps-l/2019-November/001508.h… However, I cannot find any changes on the Japanese dumps such as jawiki-20200201-pages-articles.xml.bz2. If the format change plan was postponed, could you tell me the date when this change will occur? Cheers,

2 2

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 01 Feb '20

01 Feb '20

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20200101 full revision history content run. We are currently dumping 910 projects in total. --------------------- Stats for mgwikibooks on date 20200101 Total size of page content dump files for articles, current content only: 329426 Total size of page content dump files for all pages, current content only: 874963 Total size of page content dump files for all pages, all revisions: 4294486 --------------------- Stats for enwiki on date 20200101 Total size of page content dump files for articles, current content only: 75163254305 Total size of page content dump files for all pages, current content only: 167460765690 Total size of page content dump files for all pages, all revisions: 20097373129222 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Ordering of revisions
by Christopher Wolfram 18 Jan '20

18 Jan '20

Hi, Perhaps there is documentation about this, but I have looked for the past hour and haven’t found anything. I was wondering if it is guaranteed that all revisions given in the enwiki-latest-pages-meta-history files are in order of parent->child->grandchild->… In a few examples, it looks like they follow this pattern. I ask because I need them in order and it would be nice if I didn’t have to do that with the <parentid> field. Thank you, Christopher

3 3

← Newer
1
...
12
13
14
15
16
17
18
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l