Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

755 discussions

Question about "Page abstract for Yahoo" dumps
by Bernd Fehling 11 May '26

11 May '26

Dear list, we have included parts of wikibooks (languages with most content) in our academic search engine. The pages are generated by getting title, url and abstract from wikibooks abstract.xml This is combined with pages-arcticles.xml to build metadata pages for indexing. Unfortunately, the building of wikibooks-latest-abstract.xml.gz has been discontinued last year. Currently we have 389.429 records indexed with 24 languages but they are now over a year old. Is there any replacement for "Page abstract for Yahoo" dumps? If not, is there any easy solution to fetch or generate these data somehow? Kind regards Bernd -- ************************************************************* Bernd Fehling Bielefeld University Library Dipl.-Inform. (FH) LibTec - Library Technology Universitätsstr. 25 and Knowledge Management 33615 Bielefeld Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de https://www.ub.uni-bielefeld.de/~befehl/ BASE - Bielefeld Academic Search Engine - www.base-search.net *************************************************************

2 1

abstractwiki dump questions
by Nizo Priskorn 06 Apr '26

06 Apr '26

Hi I searched for a complete dump of AW today and found none. It's been live for over a week so I expected a full dump. When can I expect a complete dump? Are there other ways to download the complete database if the dumps are broken? Since the wiki is amall with only 400 articles I can also fallback to scraping but I'm unsure if that is allowed and I would much prefer a full recent dump. What is the interval for full xml dumps these days? Are dumps of raw json from wikifunctions and/or abstractwiki planned? Is there a SLA or similar for the full dump services? Can I replicate AW in real time easily like I can with OSM and minutely diffs? How is that done? Download full dump + follow change events and scrape articles as they are changed? Is reliable minutely diffs planned? Sorry for all the questions, if you have a FAQ that answer them please link! Med vänlig hälsning Nizo Priskorn

2 2

categorylinks files changed
by Sven Hartrumpf 13 Mar '26

13 Mar '26

Hi. Somewhere between 2025-07-01 and 2025-10-20 the format of categorylinks files like dewiki-YYYYMMDD-categorylinks.sql.gz was reduced, mainly dropping the category label itself. A line like 12005616,'DFB-Pokal-Sieger','GERRESHEIM, URSULA\nURSULA GERRESHEIM','2025-02-10 13:08:35','Gerresheim, Ursula','uppercase','page',1,10608637 (with category 'DFB-Pokal-Sieger') became 12005616,'GERRESHEIM, URSULA\nURSULA GERRESHEIM','2025-02-10 13:08:35','Gerresheim, Ursula','page',1,10608637 Sizes of categorylinks files shrunk by over 40 %. Is this change intended? If so, how can the category label be retrieved? Greetings Sven

2 2

New MediaWiki Content File Exports Now Available
by Xabriel Collazo Mojica 30 Jan '26

30 Jan '26

Hi everyone, We're pleased to announce the availability of the MediaWiki Content File Exports <https://wikitech.wikimedia.org/wiki/MediaWiki_Content_File_Exports>, a new way to access the unparsed content from Wikimedia’s public wikis in XML format. *What’s Available:* The exports are provided in two datasets, updated monthly starting generation on the 1st: - mediawiki_content_history <https://dumps.wikimedia.org/other/mediawiki_content_history/> - Full revision history for all pages - mediawiki_content_current <https://dumps.wikimedia.org/other/mediawiki_content_current/> - Latest revision only for each page Both are available per wiki in compressed XML format compatible with MediaWiki’s Special:Export <https://www.mediawiki.org/wiki/Manual:Parameters_to_Special:Export>and the legacy XML dumps. *Why the Change:* The legacy dump infrastructure at https://dumps.wikimedia.org/backup-index.html has struggled to reliably produce XML exports for larger wikis. The Data Engineering team has reimplemented this process to ensure this data is accessible long term. *How to Access:* Files are available at https://dumps.wikimedia.org/other/mediawiki_content_history/ and https://dumps.wikimedia.org/other/mediawiki_content_current/. For any specific monthly export, say, for https://dumps.wikimedia.org/other/mediawiki_content_current/simplewiki/2026…, check first for the SHA256SUMS file to confirm the export is complete before downloading. Full instructions are available at: https://wikitech.wikimedia.org/wiki/MediaWiki_Content_File_Exports *Other Related Changes:* While we’ll continue attempting legacy XML generation for the time being, that path is now deprecated. Note that this deprecation affects only the XML content artifacts. All other SQL dumps of various database tables will continue. Additionally, the publication of artifacts on the legacy infrastructure will be reduced from the current twice per month cadence to once per month. The incremental dumps at https://dumps.wikimedia.org/other/incr/, which were experimental, will be sunset. *Questions?* Please review the FAQ <https://wikitech.wikimedia.org/wiki/MediaWiki_Content_File_Exports#FAQ> on the documentation page. You can also reply here if you have any questions, or follow up at https://phabricator.wikimedia.org/T414389. Best regards, The Data Engineering Team

1 0

The CirrusSearch index dumps are moving
by Erik Bernhardson 07 Nov '25

07 Nov '25

The WMF makes available a dump of the search indices from CirrusSearch (the MediaWiki extension that provides search functionality on WMF wikis) on a weekly basis. These have been running for many years, but sadly have been getting slower and slower over time as the relevant datasets have grown. A few months ago we got to the point where sometimes a weekly dump takes more than a week to generate. As such we've reworked these dumps to generate in a slightly different manner. You can reach out to us if there are difficulties migrating to the replacement dumps. The best place to provide feedback will be in https://phabricator.wikimedia.org/T366248. Changes: * Old dumps location: https://dumps.wikimedia.org/other/cirrussearch/ * New dumps location: https://dumps.wikimedia.org/other/cirrus_search_index/ * The new dumps are bzip2 compressed, while the old ones were gzip. * Old dumps were one file per search index. New dumps are one directory per search index. A directory may have one or more files. * The content of the files is exactly the same. It's just split across multiple files for ease of generation. * The old files can be recreated locally, if needed, by concatenating the decompressed versions. Something like `bzcat *.json.bz2 > full-dump.json` We will continue producing the old dumps through November, expecting to shut them off before the end of the year. Erik Bernhardson Search Platform

1 0

Approx. number of pages in enwiki-latest-pages-articles.xml
by Sigbert Klinke 30 Oct '25

30 Oct '25

Hi, I would be interested to know how many pages in enwiki-latest-pages-articles.xml . My own count gives 19,4 Mio. pages. Can this be, at least roughly, confirmed? In the internet I just find these numbers: 5,861,178 - I guess this are all namespace 0 pages 47,826,337 - this are all pages in all namespaces ‎ Sigbert -- https://hu.berlin/sk https://hu.berlin/mmstat3

3 2

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 29 Aug '25

29 Aug '25

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20250601 full revision history content run. We are currently dumping 1015 projects in total. --------------------- Stats for thwikisource on date 20250601 Total size of page content dump files for articles, current content only: 197,613,486 Total size of page content dump files for all pages, current content only: 217,943,655 Total size of page content dump files for all pages, all revisions: 2,055,501,582 --------------------- Stats for enwiki on date 20250601 Total size of page content dump files for articles, current content only: 109,753,004,895 Total size of page content dump files for all pages, current content only: 223,135,978,699 Total size of page content dump files for all pages, all revisions: 31,065,577,207,178 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Problems with recent dumps
by Danny B. 11 Aug '25

11 Aug '25

Hello, I found at least two problems with recent dumps (20250801): * https://dumps.wikimedia.org/cswiki/20250801/ - some dumps including the most important "pages-articles" failed. * https://dumps.wikimedia.org/skwiktionary/20250801/ (or without trailing slash) reports 403 Forbidden, same do direct links to dumps inside the dir Wondering if at least the latter can be fixed soon? Thanks. Kind regards Danny B.

2 1

Help offer for restoring archive dumps uploads
by Leo Max 11 Aug '25

11 Aug '25

*Hi Xabriel and Hayley,* I saw your discussion regarding the missing incremental dumps on archive.org and the project previously maintained by Hydriz. I’d be glad to help restore and automate the dumps uploading workflow. Here are some things I could help with: - Writing or maintaining upload scripts (Python, Bash) - Setting up a lightweight CI/CD or cron-based solution on Cloud Services or other servers - Monitoring uploads and sending alerts if something breaks - Documenting the process so it’s easier to maintain in the future If this sounds useful, please let me know how I can coordinate with you or the team, and what access or info would be needed to start. *Best regards, * MAX

2 1

Re: Incremental Dumps
by Xabriel Collazo Mojica 30 Jul '25

30 Jul '25

I dug around and it seems that those dumps were not uploaded by WMF staff, but by a volunteer with the handle 'hydriz'. Here is their phabricator profile: https://phabricator.wikimedia.org/p/Hydriz/ Unfortunately, it seems like they have not been active recently. If you are interested, feel free to contact them <https://wikitech.wikimedia.org/wiki/User_talk:Hydriz>, perhaps they just need a bit of help to get those contributions back on track. -xabriel On Tue, Jul 22, 2025 at 5:07 PM Hayley <hayleybale2(a)gmail.com> wrote: > > Regarding archive.org, we have no control over what they do or do not > archive. > > No, I am referring to these > <https://archive.org/search?query=incremental+dumps>, which are all > uploaded by WMF staff and abruptly stop in November 2022. > > > > On Tue, Jul 22, 2025 at 5:03 PM Xabriel Collazo Mojica < > xcollazo(a)wikimedia.org> wrote: > >> Hayley: >> >> The incremental dumps that we keep at >> https://dumps.wikimedia.org/other/incr/ continue to be there, but they >> continue to be experimental (so do please the disclaimer over there), and >> we continue to only keep the last ~40 days or so. >> >> Regarding archive.org, we have no control over what they do or do not >> archive. >> >> If you share a bit about your use case, we can help point you in the >> right direction. Let us know. >> >> Best, >> -xabriel >> >> On Tue, Jul 15, 2025 at 1:03 AM Hayley <hayleybale2(a)gmail.com> wrote: >> >>> What happened to the incremental dumps on archive.org? Why do they stop >>> in late 2023? Are they saved elsewhere, or are they lost forever? >>> _______________________________________________ >>> Xmldatadumps-l mailing list -- xmldatadumps-l(a)lists.wikimedia.org >>> To unsubscribe send an email to xmldatadumps-l-leave(a)lists.wikimedia.org >>> >> >> >> -- >> Xabriel J. Collazo Mojica (he/him, pronunciation >> <https://commons.wikimedia.org/wiki/File:Xabriel_Collazo_Mojica_-_pronunciat…> >> ) >> Sr Software Engineer >> Wikimedia Foundation >> > -- Xabriel J. Collazo Mojica (he/him, pronunciation <https://commons.wikimedia.org/wiki/File:Xabriel_Collazo_Mojica_-_pronunciat…> ) Sr Software Engineer Wikimedia Foundation

2 1

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l