Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

4 participants
738 discussions

Start a nNew thread

TeleAI
by morgoook＠gmail.com 13 Mar '25

13 Mar '25

AI dataset

1 0

Pagelinks.sql.gz file
by Abraham Israeli 03 Mar '25

03 Mar '25

Hi, I am trying to work with the Pagelinks file in order to extract the number of articles that link to a specific Wiki page (the same idea as can be seen in the "What-links-here" tool: https://en.wikipedia.org/wiki/Help:What_links_here). However, when I loop over this SQL-like file, I find very weird cases of links that don't seem to exist in Wikipedia. For example, the first line in the file indicates that there is a link between page ids 1939 and 2. PageID 2 doesn't even seem to exist. Even when I look at pages that do exist, the link indicated in the file does not exist on the actual wiki page. Am I missing anything?

3 4

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 01 Mar '25

01 Mar '25

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20250201 full revision history content run. We are currently dumping 1013 projects in total. --------------------- Stats for fonwiki on date 20250201 Total size of page content dump files for articles, current content only: 5,476,690 Total size of page content dump files for all pages, current content only: 5,599,766 Total size of page content dump files for all pages, all revisions: 51,571,009 --------------------- Stats for enwiki on date 20250201 Total size of page content dump files for articles, current content only: 107,670,254,474 Total size of page content dump files for all pages, current content only: 219,549,060,294 Total size of page content dump files for all pages, all revisions: 30,374,642,194,249 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

latest-all.json.bz2 no more available
by Guillaume Karcher 24 Feb '25

24 Feb '25

Hi there, I'm used to consolidate some wikidata knowledge, based on the file latest-all.json.bz2 <https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.bz2>. And when looking at the usual URL at https://dumps.wikimedia.org/wikidatawiki/entities/, I can see that the file is giving a very small size since February, 16th 2025 (39). Is there an issue at your side that could help understanding why the file is giving this small size, or is the file no more available ? Thanks for your answer ! Best, Guillaume

2 1

dataset
by said gölleroğlu 19 Feb '25

19 Feb '25

hello i am studying for a AI project i need big datas

2 1

2025-01-22 Dumps failed delivery January dumps (enwiki, wikidatawiki)
by ahoelzl＠wikimedia.org 31 Jan '25

31 Jan '25

Greetings XML Dump users, TL;DR: This is a note to inform you that enwiki and wikidatawiki dumps have not been produced for January 2025. enwiki Dumps were explicitly paused due to causing issues with their production interference. The wikidatawiki dump has failed to run successfully at the beginning of January. We are working re-running the enwiki January data and resuming full dump operations as of February 1st. Longer: Enwiki and wikidatawiki dumps have not been produced for January 2025 https://dumps.wikimedia.org/enwiki/20250103/ https://dumps.wikimedia.org/wikidatawiki/20241201/ The latest respective dump versions are of December 2024. Respective dumps are skipped in the current partial run, see https://dumps.wikimedia.org/backup-index.html Enwiki data has been explicitly paused on December 22nd to not interfere with production wiki instances: https://phabricator.wikimedia.org/T368098#10421298 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1106019 While the failed processing of wikidatawiki in January is yet not clear: https://phabricator.wikimedia.org/T383568 On January 6th we started working on switching dumps from production MariaDB to the analytics replicas (dbstore100[7-9]) to resolve the production dependency: https://phabricator.wikimedia.org/T382947 Patches for the change were deployed on Jan 20th https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1109109 An initial test was successfully run on the eswiki. We are further working on switching all wikis over to the analytics replicas: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1113475 Recommendation: Pause dependent pipeline until data will become available. This should be the case for the February dump runs.

1 2

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 26 Jan '25

26 Jan '25

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20241201 full revision history content run. We are currently dumping 1010 projects in total. --------------------- Stats for amwikiquote on date 20250101 Total size of page content dump files for articles, current content only: 361,850 Total size of page content dump files for all pages, current content only: 856,843 Total size of page content dump files for all pages, all revisions: 1,945,415 --------------------- Stats for enwiki on date 20241201 Total size of page content dump files for articles, current content only: 105,373,313,207 Total size of page content dump files for all pages, current content only: 216,495,087,582 Total size of page content dump files for all pages, all revisions: No info available (no files found) --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

2 1

Temporary pause of enwiki dumps until the New Year
by Ben Tullis 22 Jan '25

22 Jan '25

Hello XML dumps users and contributors, Unfortunately, we have had to interrupt the enwiki dump for 20241220 <https://dumps.wikimedia.org/enwiki/20241220/> and delay the start of the enwiki dump on January 1st. This is due to some performance issues in the database layer, which have been identified in https://phabricator.wikimedia.org/T368098 and have been causing significant impact to on-wiki workflows, as well as out-of-hours notifications to our SRE teams. We do not do this lightly and we apologise for any inconvenience that this may cause you, as we understand how important these data sets are. Please be assured that we at the WMF are working hard on improving the dumps architecture at the moment, on multiple fronts. We will follow up with further information early in the New Year, when we re-enable the enwiki dump. Kind regards, Ben -- *Ben Tullis*(he/him) Staff Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

2 1

Proposal to sunset "Page abstract for Yahoo" dumps
by Amir Sarabadani 02 Jan '25

02 Jan '25

Hi, I'm proposing to sunset "Extracted page abstracts for Yahoo" dumps. We have to maintain a dedicated extension called ActiveAbstract <https://www.mediawiki.org/wiki/Extension:ActiveAbstract> just to produce this type of dumps while It's rather easy to extract the same values from the main dump. There are also other reasons outlined in the ticket: T382069 <https://phabricator.wikimedia.org/T382069> If you rely on this type of dumps and can't easily switch or if I missed something obvious about page abstract dumps or ActiveAbstract extension, please make a comment in the ticket. Thank you -- Amir (he/him)

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 29 Dec '24

29 Dec '24

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20241201 full revision history content run. We are currently dumping 1010 projects in total. --------------------- Stats for frwikiquote on date 20241201 Total size of page content dump files for articles, current content only: 82,583,841 Total size of page content dump files for all pages, current content only: 117,149,292 Total size of page content dump files for all pages, all revisions: 3,886,165,819 --------------------- Stats for enwiki on date 20241201 Total size of page content dump files for articles, current content only: 105,373,313,207 Total size of page content dump files for all pages, current content only: 216,495,087,582 Total size of page content dump files for all pages, all revisions: 30,002,460,924,971 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l