Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

1 participants
728 discussions

ANN: A Go package providing utilities for processing Wikipedia and Wikidata dumps
by Mitar 09 Jan '22

09 Jan '22

Hi! I just published the first version of a Go package which provides utilities for processing Wikidata entities JSON dumps and Wikimedia Enterprise HTML dumps. It processes them in parallel on multiple cores, so processing is rather fast. I hope it will be useful to others, too. https://gitlab.com/tozd/go/mediawiki Any feedback is welcome. Mitar -- http://mitar.tnode.com/ https://twitter.com/mitar_m

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 02 Jan '22

02 Jan '22

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20211201 full revision history content run. We are currently dumping 945 projects in total. --------------------- Stats for tlhwiktionary on date -- Total size of page content dump files for articles, current content only: No info available (no full run date found) Total size of page content dump files for all pages, current content only: No info available (no full run date found) Total size of page content dump files for all pages, all revisions: No info available (no full run date found) --------------------- Stats for enwiki on date 20211201 Total size of page content dump files for articles, current content only: 86,591,331,146 Total size of page content dump files for all pages, current content only: 190,740,317,768 Total size of page content dump files for all pages, all revisions: 23,832,535,435,128 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

QIDs in XML dumps from projects other than Wikidata
by Andy Mabbett 24 Dec '21

24 Dec '21

[my first post to this list - hello, everyone] In conversation elsewhere, it was mentioned that the XML dumps for Wikispecies don't include the Wikidata link (QID) of the corresponding entries there. Is that so? Is it the same for Wikipedia's dumps? If so, how feasible would it be to add QIDs to the dumps? -- Andy Mabbett @pigsonthewing http://pigsonthewing.org.uk

2 1

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 02 Dec '21

02 Dec '21

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20211101 full revision history content run. We are currently dumping 945 projects in total. --------------------- Stats for igwiki on date 20211101 Total size of page content dump files for articles, current content only: 14,294,974 Total size of page content dump files for all pages, current content only: 21,955,636 Total size of page content dump files for all pages, all revisions: 322,127,219 --------------------- Stats for enwiki on date 20211101 Total size of page content dump files for articles, current content only: 86,097,776,919 Total size of page content dump files for all pages, current content only: 189,730,768,977 Total size of page content dump files for all pages, all revisions: 23,676,250,697,261 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Directory listing too small/filename too long
by Wurgl 29 Nov '21

29 Nov '21

Hallo! Maybe you could change the directory listing a little bit. In https://dumps.wikimedia.org/wikidatawiki/latest/ I see a lot of file names which are too long: wikidatawiki-latest-pages-articles-multistream-..> 23-Nov-2021 20:53 329566492 wikidatawiki-latest-pages-articles-multistream-..> 24-Nov-2021 16:37 877 wikidatawiki-latest-pages-articles-multistream-..> 23-Nov-2021 07:17 1471303 ... Thanks! Wolfgang

2 1

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 01 Nov '21

01 Nov '21

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20211001 full revision history content run. We are currently dumping 942 projects in total. --------------------- Stats for frwikibooks on date 20211001 Total size of page content dump files for articles, current content only: 139,654,749 Total size of page content dump files for all pages, current content only: 193,441,591 Total size of page content dump files for all pages, all revisions: 9,942,075,801 --------------------- Stats for enwiki on date 20211001 Total size of page content dump files for articles, current content only: 85,634,741,423 Total size of page content dump files for all pages, current content only: 188,778,524,710 Total size of page content dump files for all pages, all revisions: 23,511,675,704,710 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Wikimedia Enterprise HTML dumps available for public download
by Ariel Glenn WMF 19 Oct '21

19 Oct '21

I am pleased to announce that Wikimedia Enterprise's HTML dumps [1] for October 17-18th are available for public download; see https://dumps.wikimedia.org/other/enterprise_html/ for more information. We expect to make updated versions of these files available around the 1st/2nd of the month and the 20th/21st of the month, following the cadence of the standard SQL/XML dumps. This is still an experimental service, so there may be hiccups from time to time. Please be patient and report issues as you find them. Thanks! Ariel "Dumps Wrangler" Glenn [1] See https://www.mediawiki.org/wiki/Wikimedia_Enterprise for much more about Wikimedia Enterprise and its API.

2 1

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 02 Oct '21

02 Oct '21

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20210901 full revision history content run. We are currently dumping 942 projects in total. --------------------- Stats for iswikiquote on date 20210901 Total size of page content dump files for articles, current content only: 1,406,306 Total size of page content dump files for all pages, current content only: 2,239,225 Total size of page content dump files for all pages, all revisions: 57,520,269 --------------------- Stats for enwiki on date 20210901 Total size of page content dump files for articles, current content only: 85,212,251,874 Total size of page content dump files for all pages, current content only: 187,857,277,782 Total size of page content dump files for all pages, all revisions: 23,348,582,776,634 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

still only a partial dump for 20210801 for a lot of wikis
by griffin tucker 02 Sep '21

02 Sep '21

strangely the enwiki dump is complete, but not a lot of the other wikis (such as enwiktionary) that are a lot smaller usually they're finished after a few days something going wrong with the dump script?

4 4

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 01 Sep '21

01 Sep '21

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20210801 full revision history content run. We are currently dumping 942 projects in total. --------------------- Stats for ilowiki on date 20210801 Total size of page content dump files for articles, current content only: 110,426,279 Total size of page content dump files for all pages, current content only: 120,468,480 Total size of page content dump files for all pages, all revisions: 1,661,587,793 --------------------- Stats for enwiki on date 20210801 Total size of page content dump files for articles, current content only: 84,775,627,212 Total size of page content dump files for all pages, current content only: 186,886,040,561 Total size of page content dump files for all pages, all revisions: 23,185,464,570,213 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

← Newer
1
...
7
8
9
10
11
12
13
...
73
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l