Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

Re: changes to Page table?
by Hannah Okwelum 14 Jun '22

14 Jun '22

Hello Carley, the page_restrictions column is no longer used within MediaWiki, see https://phabricator.wikimedia.org/T218446 The column is being removed from the page table, a change that is almost complete; you can track the progress of that here https://phabricator.wikimedia.org/T60674 if you are interested. You can use the page_restrictions table instead. The Manual page you refer to will likely be updated once the schema change is complete. Thank you, Hannah Okwelum

1 0

changes to Page table?
by Carley Zepeda 13 Jun '22

13 Jun '22

Hello, I think a column was removed from the page DDL but I'm not sure if it was intentional because I didn't see any postings and the online documentation still refers to it. Can someone confirm the column has been dropped/removed? I don't want to make changes on my end if it was done by mistake. The column is page_restrictions <https://www.mediawiki.org/wiki/Manual:Page_table#page_restrictions> I see the note about it being empty but when I download one of the latest files and review the DDL, it's no longer in the table definition -- Link to file I downloaded for yowiki-latest-page.sql.gz <https://dumps.wikimedia.your.org/yowiki/latest/yowiki-latest-page.sql.gz> I downloaded an earlier version from https://dumps.wikimedia.your.org/yowiki/20220520/ and compared them -- the DDL sections for yowiki-20220520-page.sql.gz <https://dumps.wikimedia.your.org/yowiki/20220520/yowiki-20220520-page.sql.gz> vs yowiki-20220601-page.sql.gz <https://dumps.wikimedia.your.org/yowiki/20220601/yowiki-20220601-page.sql.gz> is in the screenshot below, with page_restrictions highlighted on the left (MAY file). [image: Screen Shot 2022-06-13 at 2.25.13 PM.png] https://meta.wikimedia.org/wiki/Data_dumps/Dump_format#Format_of_the_sql_fi… Thank you, CJZee

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 02 Jun '22

02 Jun '22

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20220501 full revision history content run. We are currently dumping 948 projects in total. --------------------- Stats for srnwiki on date 20220501 Total size of page content dump files for articles, current content only: 3,819,579 Total size of page content dump files for all pages, current content only: 5,442,150 Total size of page content dump files for all pages, all revisions: 210,326,461 --------------------- Stats for enwiki on date 20220501 Total size of page content dump files for articles, current content only: 88,902,717,881 Total size of page content dump files for all pages, current content only: 195,143,033,220 Total size of page content dump files for all pages, all revisions: 24,683,031,783,592 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 02 May '22

02 May '22

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20220401 full revision history content run. We are currently dumping 947 projects in total. --------------------- Stats for liwikisource on date 20220401 Total size of page content dump files for articles, current content only: 4,200,395 Total size of page content dump files for all pages, current content only: 4,900,834 Total size of page content dump files for all pages, all revisions: 49,466,324 --------------------- Stats for enwiki on date 20220401 Total size of page content dump files for articles, current content only: 88,489,072,067 Total size of page content dump files for all pages, current content only: 194,397,759,222 Total size of page content dump files for all pages, all revisions: 24,523,894,687,224 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Missing pages/stale data in HTML dumps
by Jan Berkel 05 Apr '22

05 Apr '22

Hello, just a heads-up for anyone using HTML dumps, apart from the missing namespaces issue already mentioned on this list, there also seem to be entire pages missing, and some of the included page data is outdated and does not contain the latest changes. I have no idea how many pages are affected. phabricator ticket with more details: https://phabricator.wikimedia.org/T305407 – Jan

2 2

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 01 Apr '22

01 Apr '22

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20220301 full revision history content run. We are currently dumping 945 projects in total. --------------------- Stats for csbwiktionary on date 20220301 Total size of page content dump files for articles, current content only: 2,005,634 Total size of page content dump files for all pages, current content only: 2,900,582 Total size of page content dump files for all pages, all revisions: 46,688,480 --------------------- Stats for enwiki on date 20220301 Total size of page content dump files for articles, current content only: 88,064,759,123 Total size of page content dump files for all pages, current content only: 193,569,131,075 Total size of page content dump files for all pages, all revisions: 24,339,335,307,700 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

API for navigating and downloading from Wikimedia Data Dumps
by arithmatlic＠gmail.com 01 Apr '22

01 Apr '22

Hello data dump enthusiasts, I've been working on an API (written in Python) for downloading files from Wikimedia Data Dump files, and I've just released its earliest version here: https://github.com/jon-edward/wiki_dump. I'd love to know what you all think (how it can be improved, how you are interested in using it, positive/negative remarks) if you can spare the time. All the best, jon-edward

3 3

Part of pages missing in N0 enterprise dumps
by Erik del Toro 18 Mar '22

18 Mar '22

Hello. I am doing some converts to aarddict https://aarddict.org/ offline wikipedia and wiktionary app. I use mw2slob and the N0 files found on https://dumps.wikimedia.org/other/enterprise_html/runs/ for this conversions. But in the spanish Wikipedia for example the article https://es.wikipedia.org/wiki/Anexo:Aves_de_Canarias seems not to be part of the tar.gz file. And in the french Wiktionary the article https://fr.wiktionary.org/wiki/Conjugaison:espagnol/aumentar also is missing in the respective tar.gz file. Can they be found somewhere else? In N6 or N14? For me it seems that articles/pages that have a colon like Anexo: or Conjugaison: are not part. But why? And where could I find them? Or are they to big or what is the idea of not including them? Regards, Erik

4 5

Newbie question: Which file?
by Hugh Barnard 11 Mar '22

11 Mar '22

Hello I'm looking through the dump files and am not sure 'what contains what'. Maybe there's a descriptive page that I've missed somewhere? I'd like XML or HTML, no images, to make a crawl of UK local elections, via keywords or simulating a web crawler (or a mixture of both, some pruning, then crawling). Sorry about the question, best regards Hugh Barnard --------- https://www.hughbarnard.org Twitter: @hughbarnard

3 2

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 04 Mar '22

04 Mar '22

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20220201 full revision history content run. We are currently dumping 945 projects in total. --------------------- Stats for thwikibooks on date 20220201 Total size of page content dump files for articles, current content only: 29,982,138 Total size of page content dump files for all pages, current content only: 42,139,996 Total size of page content dump files for all pages, all revisions: 363,425,740 --------------------- Stats for enwiki on date 20220201 Total size of page content dump files for articles, current content only: 87,651,925,597 Total size of page content dump files for all pages, current content only: 192,724,767,532 Total size of page content dump files for all pages, all revisions: 24,174,134,255,220 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

← Newer
1
...
5
6
7
8
9
10
11
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l