Hi!
I just published the first version of a Go package which provides
utilities for processing
Wikidata entities JSON dumps and Wikimedia Enterprise HTML dumps. It
processes them in parallel on multiple cores, so processing is rather
fast. I hope it will be useful to others, too.
https://gitlab.com/tozd/go/mediawiki
Any feedback is welcome.
Mitar
--
http://mitar.tnode.com/https://twitter.com/mitar_m
Greetings XML Dump users and contributors!
This is your automatic monthly Dumps FAQ update email. This update
contains figures for the 20211201 full revision history content run.
We are currently dumping 945 projects in total.
---------------------
Stats for tlhwiktionary on date --
Total size of page content dump files for articles, current content only:
No info available (no full run date found)
Total size of page content dump files for all pages, current content only:
No info available (no full run date found)
Total size of page content dump files for all pages, all revisions:
No info available (no full run date found)
---------------------
Stats for enwiki on date 20211201
Total size of page content dump files for articles, current content only:
86,591,331,146
Total size of page content dump files for all pages, current content only:
190,740,317,768
Total size of page content dump files for all pages, all revisions:
23,832,535,435,128
---------------------
Sincerely,
Your friendly Wikimedia Dump Info Collector
[my first post to this list - hello, everyone]
In conversation elsewhere, it was mentioned that the XML dumps for
Wikispecies don't include the Wikidata link (QID) of the corresponding
entries there. Is that so? Is it the same for Wikipedia's dumps?
If so, how feasible would it be to add QIDs to the dumps?
--
Andy Mabbett
@pigsonthewing
http://pigsonthewing.org.uk
Greetings XML Dump users and contributors!
This is your automatic monthly Dumps FAQ update email. This update
contains figures for the 20211101 full revision history content run.
We are currently dumping 945 projects in total.
---------------------
Stats for igwiki on date 20211101
Total size of page content dump files for articles, current content only:
14,294,974
Total size of page content dump files for all pages, current content only:
21,955,636
Total size of page content dump files for all pages, all revisions:
322,127,219
---------------------
Stats for enwiki on date 20211101
Total size of page content dump files for articles, current content only:
86,097,776,919
Total size of page content dump files for all pages, current content only:
189,730,768,977
Total size of page content dump files for all pages, all revisions:
23,676,250,697,261
---------------------
Sincerely,
Your friendly Wikimedia Dump Info Collector
Hallo!
Maybe you could change the directory listing a little bit.
In https://dumps.wikimedia.org/wikidatawiki/latest/ I see a lot of
file names which are too long:
wikidatawiki-latest-pages-articles-multistream-..> 23-Nov-2021 20:53
329566492
wikidatawiki-latest-pages-articles-multistream-..> 24-Nov-2021 16:37
877
wikidatawiki-latest-pages-articles-multistream-..> 23-Nov-2021 07:17
1471303
...
Thanks!
Wolfgang
Greetings XML Dump users and contributors!
This is your automatic monthly Dumps FAQ update email. This update
contains figures for the 20211001 full revision history content run.
We are currently dumping 942 projects in total.
---------------------
Stats for frwikibooks on date 20211001
Total size of page content dump files for articles, current content only:
139,654,749
Total size of page content dump files for all pages, current content only:
193,441,591
Total size of page content dump files for all pages, all revisions:
9,942,075,801
---------------------
Stats for enwiki on date 20211001
Total size of page content dump files for articles, current content only:
85,634,741,423
Total size of page content dump files for all pages, current content only:
188,778,524,710
Total size of page content dump files for all pages, all revisions:
23,511,675,704,710
---------------------
Sincerely,
Your friendly Wikimedia Dump Info Collector
I am pleased to announce that Wikimedia Enterprise's HTML dumps [1] for
October 17-18th are available for public download; see
https://dumps.wikimedia.org/other/enterprise_html/ for more information. We
expect to make updated versions of these files available around the 1st/2nd
of the month and the 20th/21st of the month, following the cadence of the
standard SQL/XML dumps.
This is still an experimental service, so there may be hiccups from time to
time. Please be patient and report issues as you find them. Thanks!
Ariel "Dumps Wrangler" Glenn
[1] See https://www.mediawiki.org/wiki/Wikimedia_Enterprise for much more
about Wikimedia Enterprise and its API.
Greetings XML Dump users and contributors!
This is your automatic monthly Dumps FAQ update email. This update
contains figures for the 20210901 full revision history content run.
We are currently dumping 942 projects in total.
---------------------
Stats for iswikiquote on date 20210901
Total size of page content dump files for articles, current content only:
1,406,306
Total size of page content dump files for all pages, current content only:
2,239,225
Total size of page content dump files for all pages, all revisions:
57,520,269
---------------------
Stats for enwiki on date 20210901
Total size of page content dump files for articles, current content only:
85,212,251,874
Total size of page content dump files for all pages, current content only:
187,857,277,782
Total size of page content dump files for all pages, all revisions:
23,348,582,776,634
---------------------
Sincerely,
Your friendly Wikimedia Dump Info Collector
strangely the enwiki dump is complete, but not a lot of the other
wikis (such as enwiktionary) that are a lot smaller
usually they're finished after a few days
something going wrong with the dump script?
Greetings XML Dump users and contributors!
This is your automatic monthly Dumps FAQ update email. This update
contains figures for the 20210801 full revision history content run.
We are currently dumping 942 projects in total.
---------------------
Stats for ilowiki on date 20210801
Total size of page content dump files for articles, current content only:
110,426,279
Total size of page content dump files for all pages, current content only:
120,468,480
Total size of page content dump files for all pages, all revisions:
1,661,587,793
---------------------
Stats for enwiki on date 20210801
Total size of page content dump files for articles, current content only:
84,775,627,212
Total size of page content dump files for all pages, current content only:
186,886,040,561
Total size of page content dump files for all pages, all revisions:
23,185,464,570,213
---------------------
Sincerely,
Your friendly Wikimedia Dump Info Collector