Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

760 discussions

The first letter of every entry in page title dump is capitalized
by Celeste "Coelacanthus" Liu 21 Sep '26

21 Sep '26

Hi, Recently I try to improve fcitx5-pinyin-zhwiki project, which turn Wikimedia title dump into usable pinyin dictionary. But when I introduce Greek-Chinese-mixed word support[1], I find the title dump https://dumps.wikimedia.org/zhwiki/zhwiki-$DATE-all-titles-in-ns0 has an issue: the first letter of every entry in the title dump provided by Wikipedia is capitalized, so "α粒子" becomes "Α粒子". It's wrong typing, here alpha shall be lower case. Simply applying `tolower` to the starting Greek letters won't work either, because there are also entries like "Γ函数". Recovery all infomation from corrupted data is impossible. So I want to know, how can I get the title dump where no first latter uncoditionally capitalized? [1]: https://github.com/felixonmars/fcitx5-pinyin-zhwiki/pull/56 Best regards, Celeste Liu

3 4

Re: 7z Dumps
by Xabriel Collazo Mojica 09 Sep '26

09 Sep '26

Hayley: We have opened a task for this: T437454 Implement 7z compression for v2 history dumps <https://phabricator.wikimedia.org/T437454>. It looks like we will go ahead with supporting it now that we have numbers (see task), but do please chime in to further make the case. Thanks, -xabriel On Tue, Aug 18, 2026 at 10:40 AM Hayley <hayleybale2(a)gmail.com> wrote: > > We went with bzip2 for compatibility with existing dumps > > Well the old dumps provided 7z as well as bz2, and I was using 7z. So this > is an incompatibility issue as well. > > On Mon, Aug 17, 2026 at 2:35 PM Xabriel Collazo Mojica < > xcollazo(a)wikimedia.org> wrote: > >> Hayley: >> >> Apologies for the delay, I was on leave. >> >> The new infrastructure is built to support multiple compressions, and >> hopefully better containers other than XML, but as you pointed out, today >> we only support XML compressed via bzip2. >> >> We went with bzip2 for compatibility with existing dumps, and because it >> is easy to split in systems like Apache Spark, but I hear you on the >> decompression cost. >> >> We currently have no plans to reintroduce 7z, but feel free to open a >> ticket on phabricator to make your case. There is a related ticket ( T324954 >> Provide zstd compression algorithm for dumps >> <https://phabricator.wikimedia.org/T324954> ), which makes a similar >> case on decompression cost. >> >> -xabriel >> >> On Sat, Jul 25, 2026 at 1:12 PM Hayley via Xmldatadumps-l < >> xmldatadumps-l(a)lists.wikimedia.org> wrote: >> >>> Hello, I am wondering with the transition to the new data dumps >>> infrastructure, will 7z-compressed dumps be provided at some point? The old >>> dumps pipeline provided both bz2 and 7z. 7z is much preferable to bz2, as >>> bz2 is several times larger, and at the same time, also several times more >>> computationally expensive to decompress. >>> _______________________________________________ >>> Xmldatadumps-l mailing list -- xmldatadumps-l(a)lists.wikimedia.org >>> To unsubscribe send an email to xmldatadumps-l-leave(a)lists.wikimedia.org >>> >> >> >> -- >> Xabriel J. Collazo Mojica (he/him, pronunciation >> <https://commons.wikimedia.org/wiki/File:Xabriel_Collazo_Mojica_-_pronunciat…> >> ) >> Staff Data Engineer >> Wikimedia Foundation >> > -- Xabriel J. Collazo Mojica (he/him, pronunciation <https://commons.wikimedia.org/wiki/File:Xabriel_Collazo_Mojica_-_pronunciat…> ) Staff Data Engineer Wikimedia Foundation

1 0

July data dumps
by Jan Berkel 17 Aug '26

17 Aug '26

Are there problems/delays with the July XML dumps? There are no updates on https://dumps.wikimedia.org/enwiktionary/20260701/, but some other wikis seem to have been generated successfully.

2 1

7z Dumps
by Hayley 17 Aug '26

17 Aug '26

Hello, I am wondering with the transition to the new data dumps infrastructure, will 7z-compressed dumps be provided at some point? The old dumps pipeline provided both bz2 and 7z. 7z is much preferable to bz2, as bz2 is several times larger, and at the same time, also several times more computationally expensive to decompress.

2 1

Fwd: User-agent enforcement for downloads from dumps.wikimedia.org
by Halley Coplin 18 Jun '26

18 Jun '26

Forwarding the message below -- a mistake in the to: line prevented it from being sent to this distro. *Halley Coplin* (she/her) Sr. Product Manager, MediaWiki Interfaces Wikimedia Foundation <https://wikimediafoundation.org/> ---------- Forwarded message --------- From: Halley Coplin <hcoplin(a)wikimedia.org> Date: Mon, Jun 15, 2026 at 12:54 PM Subject: User-agent enforcement for downloads from dumps.wikimedia.org To: Wikitech-l <wikitech-l(a)lists.wikimedia.org>, < wiki-research-l(a)lists.wikimedia.org>, <xmldatadumps-l(a)wikimedia.org> Howdy dumps users, Starting by the end of June, we will begin enforcing our user-agent policy <https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agen…> on dumps published to dumps.wikimedia.org. This means that a valid user-agent will be required on all automated requests to download dumps files from the public facing dumps website. Requests that are missing or provide a generic user agent (such as a basic library name) will be blocked. Why will valid user-agents be required? In August 2025, we began enforcing <https://phabricator.wikimedia.org/T400119>our long-standing user-agent policy on other forms of programmatic access, including APIs and automated crawlers. We are now following up with dumps <https://phabricator.wikimedia.org/T427836> to ensure consistent policy enforcement. Today, more than 70% of database_backup download traffic originates from automated processes, yet only 29% of that traffic is compliant with our user-agent policy. Enforcing the existing user-agent policy ensures access continues trending more towards responsible reuse. WMCS Access Users who access dumps data through the Wikimedia Cloud Services fileshare <https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Dumps> will be unaffected. Additionally, requests to the Dumps website that originate through WMCS services (Toolforge, CloudVPS) will be allowed access regardless of whether or not the user-agent is compliant. Please feel free to reach out directly through email or leave a comment in phabricator <https://phabricator.wikimedia.org/T427836> if you have any questions, comments, or concerns. Cheers, Halley *Halley Coplin* (she/her) Sr. Product Manager, MediaWiki Interfaces Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

Question about "Page abstract for Yahoo" dumps
by Bernd Fehling 11 May '26

11 May '26

Dear list, we have included parts of wikibooks (languages with most content) in our academic search engine. The pages are generated by getting title, url and abstract from wikibooks abstract.xml This is combined with pages-arcticles.xml to build metadata pages for indexing. Unfortunately, the building of wikibooks-latest-abstract.xml.gz has been discontinued last year. Currently we have 389.429 records indexed with 24 languages but they are now over a year old. Is there any replacement for "Page abstract for Yahoo" dumps? If not, is there any easy solution to fetch or generate these data somehow? Kind regards Bernd -- ************************************************************* Bernd Fehling Bielefeld University Library Dipl.-Inform. (FH) LibTec - Library Technology Universitätsstr. 25 and Knowledge Management 33615 Bielefeld Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de https://www.ub.uni-bielefeld.de/~befehl/ BASE - Bielefeld Academic Search Engine - www.base-search.net *************************************************************

2 1

abstractwiki dump questions
by Nizo Priskorn 06 Apr '26

06 Apr '26

Hi I searched for a complete dump of AW today and found none. It's been live for over a week so I expected a full dump. When can I expect a complete dump? Are there other ways to download the complete database if the dumps are broken? Since the wiki is amall with only 400 articles I can also fallback to scraping but I'm unsure if that is allowed and I would much prefer a full recent dump. What is the interval for full xml dumps these days? Are dumps of raw json from wikifunctions and/or abstractwiki planned? Is there a SLA or similar for the full dump services? Can I replicate AW in real time easily like I can with OSM and minutely diffs? How is that done? Download full dump + follow change events and scrape articles as they are changed? Is reliable minutely diffs planned? Sorry for all the questions, if you have a FAQ that answer them please link! Med vänlig hälsning Nizo Priskorn

2 2

categorylinks files changed
by Sven Hartrumpf 13 Mar '26

13 Mar '26

Hi. Somewhere between 2025-07-01 and 2025-10-20 the format of categorylinks files like dewiki-YYYYMMDD-categorylinks.sql.gz was reduced, mainly dropping the category label itself. A line like 12005616,'DFB-Pokal-Sieger','GERRESHEIM, URSULA\nURSULA GERRESHEIM','2025-02-10 13:08:35','Gerresheim, Ursula','uppercase','page',1,10608637 (with category 'DFB-Pokal-Sieger') became 12005616,'GERRESHEIM, URSULA\nURSULA GERRESHEIM','2025-02-10 13:08:35','Gerresheim, Ursula','page',1,10608637 Sizes of categorylinks files shrunk by over 40 %. Is this change intended? If so, how can the category label be retrieved? Greetings Sven

2 2

New MediaWiki Content File Exports Now Available
by Xabriel Collazo Mojica 30 Jan '26

30 Jan '26

Hi everyone, We're pleased to announce the availability of the MediaWiki Content File Exports <https://wikitech.wikimedia.org/wiki/MediaWiki_Content_File_Exports>, a new way to access the unparsed content from Wikimedia’s public wikis in XML format. *What’s Available:* The exports are provided in two datasets, updated monthly starting generation on the 1st: - mediawiki_content_history <https://dumps.wikimedia.org/other/mediawiki_content_history/> - Full revision history for all pages - mediawiki_content_current <https://dumps.wikimedia.org/other/mediawiki_content_current/> - Latest revision only for each page Both are available per wiki in compressed XML format compatible with MediaWiki’s Special:Export <https://www.mediawiki.org/wiki/Manual:Parameters_to_Special:Export>and the legacy XML dumps. *Why the Change:* The legacy dump infrastructure at https://dumps.wikimedia.org/backup-index.html has struggled to reliably produce XML exports for larger wikis. The Data Engineering team has reimplemented this process to ensure this data is accessible long term. *How to Access:* Files are available at https://dumps.wikimedia.org/other/mediawiki_content_history/ and https://dumps.wikimedia.org/other/mediawiki_content_current/. For any specific monthly export, say, for https://dumps.wikimedia.org/other/mediawiki_content_current/simplewiki/2026…, check first for the SHA256SUMS file to confirm the export is complete before downloading. Full instructions are available at: https://wikitech.wikimedia.org/wiki/MediaWiki_Content_File_Exports *Other Related Changes:* While we’ll continue attempting legacy XML generation for the time being, that path is now deprecated. Note that this deprecation affects only the XML content artifacts. All other SQL dumps of various database tables will continue. Additionally, the publication of artifacts on the legacy infrastructure will be reduced from the current twice per month cadence to once per month. The incremental dumps at https://dumps.wikimedia.org/other/incr/, which were experimental, will be sunset. *Questions?* Please review the FAQ <https://wikitech.wikimedia.org/wiki/MediaWiki_Content_File_Exports#FAQ> on the documentation page. You can also reply here if you have any questions, or follow up at https://phabricator.wikimedia.org/T414389. Best regards, The Data Engineering Team

1 0

The CirrusSearch index dumps are moving
by Erik Bernhardson 07 Nov '25

07 Nov '25

The WMF makes available a dump of the search indices from CirrusSearch (the MediaWiki extension that provides search functionality on WMF wikis) on a weekly basis. These have been running for many years, but sadly have been getting slower and slower over time as the relevant datasets have grown. A few months ago we got to the point where sometimes a weekly dump takes more than a week to generate. As such we've reworked these dumps to generate in a slightly different manner. You can reach out to us if there are difficulties migrating to the replacement dumps. The best place to provide feedback will be in https://phabricator.wikimedia.org/T366248. Changes: * Old dumps location: https://dumps.wikimedia.org/other/cirrussearch/ * New dumps location: https://dumps.wikimedia.org/other/cirrus_search_index/ * The new dumps are bzip2 compressed, while the old ones were gzip. * Old dumps were one file per search index. New dumps are one directory per search index. A directory may have one or more files. * The content of the files is exactly the same. It's just split across multiple files for ease of generation. * The old files can be recreated locally, if needed, by concatenating the decompressed versions. Something like `bzcat *.json.bz2 > full-dump.json` We will continue producing the old dumps through November, expecting to shut them off before the end of the year. Erik Bernhardson Search Platform

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l