Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

1 participants
751 discussions

The CirrusSearch index dumps are moving
by Erik Bernhardson 08 Nov '25

08 Nov '25

The WMF makes available a dump of the search indices from CirrusSearch (the MediaWiki extension that provides search functionality on WMF wikis) on a weekly basis. These have been running for many years, but sadly have been getting slower and slower over time as the relevant datasets have grown. A few months ago we got to the point where sometimes a weekly dump takes more than a week to generate. As such we've reworked these dumps to generate in a slightly different manner. You can reach out to us if there are difficulties migrating to the replacement dumps. The best place to provide feedback will be in https://phabricator.wikimedia.org/T366248. Changes: * Old dumps location: https://dumps.wikimedia.org/other/cirrussearch/ * New dumps location: https://dumps.wikimedia.org/other/cirrus_search_index/ * The new dumps are bzip2 compressed, while the old ones were gzip. * Old dumps were one file per search index. New dumps are one directory per search index. A directory may have one or more files. * The content of the files is exactly the same. It's just split across multiple files for ease of generation. * The old files can be recreated locally, if needed, by concatenating the decompressed versions. Something like `bzcat *.json.bz2 > full-dump.json` We will continue producing the old dumps through November, expecting to shut them off before the end of the year. Erik Bernhardson Search Platform

1 0

Approx. number of pages in enwiki-latest-pages-articles.xml
by Sigbert Klinke 31 Oct '25

31 Oct '25

Hi, I would be interested to know how many pages in enwiki-latest-pages-articles.xml . My own count gives 19,4 Mio. pages. Can this be, at least roughly, confirmed? In the internet I just find these numbers: 5,861,178 - I guess this are all namespace 0 pages 47,826,337 - this are all pages in all namespaces ‎ Sigbert -- https://hu.berlin/sk https://hu.berlin/mmstat3

3 2

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 30 Aug '25

30 Aug '25

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20250601 full revision history content run. We are currently dumping 1015 projects in total. --------------------- Stats for thwikisource on date 20250601 Total size of page content dump files for articles, current content only: 197,613,486 Total size of page content dump files for all pages, current content only: 217,943,655 Total size of page content dump files for all pages, all revisions: 2,055,501,582 --------------------- Stats for enwiki on date 20250601 Total size of page content dump files for articles, current content only: 109,753,004,895 Total size of page content dump files for all pages, current content only: 223,135,978,699 Total size of page content dump files for all pages, all revisions: 31,065,577,207,178 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Problems with recent dumps
by Danny B. 12 Aug '25

12 Aug '25

Hello, I found at least two problems with recent dumps (20250801): * https://dumps.wikimedia.org/cswiki/20250801/ - some dumps including the most important "pages-articles" failed. * https://dumps.wikimedia.org/skwiktionary/20250801/ (or without trailing slash) reports 403 Forbidden, same do direct links to dumps inside the dir Wondering if at least the latter can be fixed soon? Thanks. Kind regards Danny B.

2 1

Help offer for restoring archive dumps uploads
by Leo Max 12 Aug '25

12 Aug '25

*Hi Xabriel and Hayley,* I saw your discussion regarding the missing incremental dumps on archive.org and the project previously maintained by Hydriz. I’d be glad to help restore and automate the dumps uploading workflow. Here are some things I could help with: - Writing or maintaining upload scripts (Python, Bash) - Setting up a lightweight CI/CD or cron-based solution on Cloud Services or other servers - Monitoring uploads and sending alerts if something breaks - Documenting the process so it’s easier to maintain in the future If this sounds useful, please let me know how I can coordinate with you or the team, and what access or info would be needed to start. *Best regards, * MAX

2 1

Re: Incremental Dumps
by Xabriel Collazo Mojica 31 Jul '25

31 Jul '25

I dug around and it seems that those dumps were not uploaded by WMF staff, but by a volunteer with the handle 'hydriz'. Here is their phabricator profile: https://phabricator.wikimedia.org/p/Hydriz/ Unfortunately, it seems like they have not been active recently. If you are interested, feel free to contact them <https://wikitech.wikimedia.org/wiki/User_talk:Hydriz>, perhaps they just need a bit of help to get those contributions back on track. -xabriel On Tue, Jul 22, 2025 at 5:07 PM Hayley <hayleybale2(a)gmail.com> wrote: > > Regarding archive.org, we have no control over what they do or do not > archive. > > No, I am referring to these > <https://archive.org/search?query=incremental+dumps>, which are all > uploaded by WMF staff and abruptly stop in November 2022. > > > > On Tue, Jul 22, 2025 at 5:03 PM Xabriel Collazo Mojica < > xcollazo(a)wikimedia.org> wrote: > >> Hayley: >> >> The incremental dumps that we keep at >> https://dumps.wikimedia.org/other/incr/ continue to be there, but they >> continue to be experimental (so do please the disclaimer over there), and >> we continue to only keep the last ~40 days or so. >> >> Regarding archive.org, we have no control over what they do or do not >> archive. >> >> If you share a bit about your use case, we can help point you in the >> right direction. Let us know. >> >> Best, >> -xabriel >> >> On Tue, Jul 15, 2025 at 1:03 AM Hayley <hayleybale2(a)gmail.com> wrote: >> >>> What happened to the incremental dumps on archive.org? Why do they stop >>> in late 2023? Are they saved elsewhere, or are they lost forever? >>> _______________________________________________ >>> Xmldatadumps-l mailing list -- xmldatadumps-l(a)lists.wikimedia.org >>> To unsubscribe send an email to xmldatadumps-l-leave(a)lists.wikimedia.org >>> >> >> >> -- >> Xabriel J. Collazo Mojica (he/him, pronunciation >> <https://commons.wikimedia.org/wiki/File:Xabriel_Collazo_Mojica_-_pronunciat…> >> ) >> Sr Software Engineer >> Wikimedia Foundation >> > -- Xabriel J. Collazo Mojica (he/him, pronunciation <https://commons.wikimedia.org/wiki/File:Xabriel_Collazo_Mojica_-_pronunciat…> ) Sr Software Engineer Wikimedia Foundation

2 1

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 29 Jul '25

29 Jul '25

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20250601 full revision history content run. We are currently dumping 1015 projects in total. --------------------- Stats for lbwiktionary on date 20250601 Total size of page content dump files for articles, current content only: 14,485,208 Total size of page content dump files for all pages, current content only: 15,574,700 Total size of page content dump files for all pages, all revisions: 147,743,553 --------------------- Stats for enwiki on date 20250601 Total size of page content dump files for articles, current content only: 109,753,004,895 Total size of page content dump files for all pages, current content only: 223,135,978,699 Total size of page content dump files for all pages, all revisions: 31,065,577,207,178 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Incremental Dumps
by Hayley 23 Jul '25

23 Jul '25

What happened to the incremental dumps on archive.org? Why do they stop in late 2023? Are they saved elsewhere, or are they lost forever?

2 1

July 1 2025 enwiki dump unavailable
by Chandler May 23 Jul '25

23 Jul '25

Hi all, Does anyone know what the status of the 2025-07-01 enwiki dump is? The web status page shows it is in progress, but the 2025-07-20 dump shows complete. Does this mean there will not be a 2025-07-01 enwiki dump (due to an error)? Thank you! Chandler May

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 30 Jun '25

30 Jun '25

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20250601 full revision history content run. We are currently dumping 1015 projects in total. --------------------- Stats for kuwikibooks on date 20250601 Total size of page content dump files for articles, current content only: 3,228,797 Total size of page content dump files for all pages, current content only: 3,876,025 Total size of page content dump files for all pages, all revisions: 124,717,657 --------------------- Stats for enwiki on date 20250601 Total size of page content dump files for articles, current content only: 109,753,004,895 Total size of page content dump files for all pages, current content only: 223,135,978,699 Total size of page content dump files for all pages, all revisions: 31,065,577,207,178 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l