Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

1 participants
728 discussions

Start a nNew thread

reply
by 李成 03 Dec '24

03 Dec '24

reply Sent with Spark

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 29 Nov '24

29 Nov '24

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20241101 full revision history content run. We are currently dumping 1008 projects in total. --------------------- Stats for simplewiki on date 20241101 Total size of page content dump files for articles, current content only: 1,440,017,120 Total size of page content dump files for all pages, current content only: 2,270,831,092 Total size of page content dump files for all pages, all revisions: 70,225,134,087 --------------------- Stats for enwiki on date 20241101 Total size of page content dump files for articles, current content only: 104,825,619,918 Total size of page content dump files for all pages, current content only: 215,523,184,051 Total size of page content dump files for all pages, all revisions: 29,837,076,901,944 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Dumps failed delivery October dumps (frwiki, wikidata), pausing dumps generation
by ahoelzl＠wikimedia.org 14 Nov '24

14 Nov '24

Greetings XML Dump users, TL;DR: We are pausing the XML Dumps effective from now to correct runtime errors that we suspect are causing bad data dumps. We are working on a fix. Longer: Over the past couple months, we have noticed a growing amount of runtime errors coming from the process that generate the XML content Dumps. It has always been the case that this process may have transient issues and may miss some revisions some months, but the current situation has become such a recurring problem that we now suspect data corruption in recent dumps. We typically start a full dump (i.e. all revisions for all pages) on the 1st of the month for all wikis, and then we start a partial dump (i.e all current revisions for all pages) on the 20th of the month. Most of the October 1 2024 full runs are complete, except the French wiki and Wikidata wiki, which have failed for this month. The last successful copies of the French and Wikidata wikis are from September. All of the partial runs for the 20th of November are complete as well. However, any of these recent dumps may have underlying data quality issues. In the interest of not dumping potentially bad data, we have decided to pause the XML Dumps, effective for all future dumps from the date of this communication, until we find and fix the root cause of these errors. We acknowledge that many folks and downstream processes will be impacted and apologize for any inconvenience that this may cause you. We are prioritizing this work, and if interested, you can follow updates at https://phabricator.wikimedia.org/T377594. Feel free to open additional tickets if your use cases are affected, and do please link them to the main ticket. Further, if you have the ability, we welcome data quality analysis of recent dumps that you may have noticed in your use cases.

3 4

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 29 Oct '24

29 Oct '24

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20241001 full revision history content run. We are currently dumping 1006 projects in total. --------------------- Stats for bewiktionary on date 20241001 Total size of page content dump files for articles, current content only: 16,301,465 Total size of page content dump files for all pages, current content only: 17,378,260 Total size of page content dump files for all pages, all revisions: 109,221,230 --------------------- Stats for enwiki on date 20241001 Total size of page content dump files for articles, current content only: 104,233,480,802 Total size of page content dump files for all pages, current content only: 214,815,867,775 Total size of page content dump files for all pages, all revisions: 29,647,340,985,733 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 29 Sep '24

29 Sep '24

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20240901 full revision history content run. We are currently dumping 999 projects in total. --------------------- Stats for rwwiki on date 20240901 Total size of page content dump files for articles, current content only: 45,709,422 Total size of page content dump files for all pages, current content only: 48,929,791 Total size of page content dump files for all pages, all revisions: 575,109,772 --------------------- Stats for enwiki on date 20240901 Total size of page content dump files for articles, current content only: 103,620,371,971 Total size of page content dump files for all pages, current content only: 214,059,219,507 Total size of page content dump files for all pages, all revisions: 29,462,194,175,181 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Seeding XML dumps on Academic Torrents
by Will Beason 08 Sep '24

08 Sep '24

Hey, I'm maintaining some torrents for the past couple months of Wikipedia monthly XML dumps on Academic Torrents in this collection <https://academictorrents.com/collection/english-wikipedia>. I'm currently doing time-series research on Wikipedia so I'll be maintaining the collection for at least the next couple years as I work on my dissertation for my PhD (and do related work). -- Will Beason (he/him) PhD Student in Information Studies University of Texas at Austin beason(a)utexas.edu willbeason.com

2 3

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 29 Aug '24

29 Aug '24

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20240801 full revision history content run. We are currently dumping 994 projects in total. --------------------- Stats for hywikibooks on date 20240801 Total size of page content dump files for articles, current content only: 2,975,648 Total size of page content dump files for all pages, current content only: 4,155,230 Total size of page content dump files for all pages, all revisions: 97,729,331 --------------------- Stats for enwiki on date 20240801 Total size of page content dump files for articles, current content only: 103,090,295,026 Total size of page content dump files for all pages, current content only: 213,140,153,970 Total size of page content dump files for all pages, all revisions: 29,287,015,922,315 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 29 Jul '24

29 Jul '24

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20240701 full revision history content run. We are currently dumping 993 projects in total. --------------------- Stats for yowiktionary on date 20240701 Total size of page content dump files for articles, current content only: 515,878 Total size of page content dump files for all pages, current content only: 595,182 Total size of page content dump files for all pages, all revisions: 685,677 --------------------- Stats for enwiki on date 20240701 Total size of page content dump files for articles, current content only: 102,557,302,337 Total size of page content dump files for all pages, current content only: 212,187,264,815 Total size of page content dump files for all pages, all revisions: 29,112,412,667,891 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 29 Jun '24

29 Jun '24

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20240601 full revision history content run. We are currently dumping 991 projects in total. --------------------- Stats for kawiktionary on date 20240601 Total size of page content dump files for articles, current content only: 52,079,883 Total size of page content dump files for all pages, current content only: 55,753,024 Total size of page content dump files for all pages, all revisions: 415,227,440 --------------------- Stats for enwiki on date 20240601 Total size of page content dump files for articles, current content only: 102,189,100,980 Total size of page content dump files for all pages, current content only: 211,455,859,206 Total size of page content dump files for all pages, all revisions: 28,978,618,857,960 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Empty text elements in dump dewiki-20240520-pages-articles.xml
by Sven Hartrumpf 19 Jun '24

19 Jun '24

Hello. The dump dewiki-20240520-pages-articles.xml contains many (96069 for ns 0) empty articles. The first one is for <id>15</id>, the last one for <id>13102212</id> For ns=0, this is a new phenomenon (introduced after 2024-03-01). For all articles, the number of affected articles grew a lot: # grep -c ' <text bytes="[0-9]*" />' dewiki-20240520-pages-articles.xml 101259 # grep -c ' <text bytes="[0-9]*" />' dewiki-20240301-pages-articles.xml 129 Greetings Sven

2 3

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l