Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

[Breaking Change Announcement] Empty Senses / Forms Lists Presentation in Lexeme Dumps
by Mohammed Sadat Abdulai 24 Jan '24

24 Jan '24

(*apologies for cross-posting*) Hello, This is a breaking change announcement relevant to those working with Lexeme dumps. In Lexeme dumps, "senses" and "forms" values, when not empty, are shown as arrays. When these lists are empty, they are currently displayed as objects. For example, values with content are displayed in array format: "senses":[{"id":"L4-S1",...] but empty values are treated as objects: "senses":{} However, empty lists should be presented as arrays as well: "senses":[] In this change, empty lists of forms and senses will be switched from objects to arrays. This adjustment makes the dumps more consistent and matches the same way non-empty values are presented. We will roll this change out on February 8th. We anticipate the impact of this change to be minimal and harmless for most use cases. Therefore, we haven't generated a test dump, as it would demand substantial resources and time. If you have any questions or concerns about this change, please don’t hesitate to reach out to us in this ticket ( T305660 <https://phabricator.wikimedia.org/T305660>). Cheers, -- Mohammed S. Abdulai *Community Communications Manager, Wikidata* Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0) 30 577 116 2466 https://wikimedia.de Grab a spot in my calendar for a chat: calendly.com/masssly. A lot is happening around Wikidata - Keep up to date! <https://www.wikidata.org/wiki/Wikidata:Status_updates> Current news and exciting stories about Wikimedia, Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now <https://www.wikimedia.de/newsletter/> . Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision! https://spenden.wikimedia.de Wikimedia Deutschland — Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Charlottenburg, VR 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. Geschäftsführende Vorstände: Franziska Heine, Dr. Christian Humborg

2 1

Is there a problem with the wikidata-dump?
by Wurgl 14 Jan '24

14 Jan '24

Hello! I am having some unexpected messages, so I tried the following: curl -s https://dumps.wikimedia.org/wikidatawiki/latest/wikidatawiki-latest-pages-a… | bzip2 -d | tail an got this: bzip2: Compressed file ends unexpectedly; perhaps it is corrupted? *Possible* reason follows. bzip2: Inappropriate ioctl for device Input file = (stdin), output file = (stdout) It is possible that the compressed file(s) have become corrupted. You can use the -tvv option to test integrity of such files. You can use the `bzip2recover' program to attempt to recover data from undamaged sections of corrupted files. <parentid>1227967782</parentid> <timestamp>2023-12-07T00:22:05Z</timestamp> <contributor> <username>Renamerr</username> <id>2883061</id> </contributor> <comment>/* wbsetdescription-add:1|uk */ бактеріальний білок, наявний у Listeria monocytogenes EGD-e, [[:toollabs:quickstatements/#/batch/218434|batch #218434]]</comment> <model>wikibase-item</model> <format>application/json</format> The first part is an error message which I could also see when running my PHP-script from within the toolserver-cloud (php 7.4 because class XMLReader with the installed php 8.2 simple core dumps, see T352886). The second part is the output from the "tail" command. Just as a crosschek: I have no such problem with curl -s https://dumps.wikimedia.org/dewiki/latest/dewiki-latest-pages-meta-current.… | bzip2 -d | tail No error and the last line is "</mediawiki>" Cheers, Wolfgang

5 14

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 29 Dec '23

29 Dec '23

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20231201 full revision history content run. We are currently dumping 982 projects in total. --------------------- Stats for eowikiquote on date 20231201 Total size of page content dump files for articles, current content only: 25,649,838 Total size of page content dump files for all pages, current content only: 26,954,003 Total size of page content dump files for all pages, all revisions: 830,995,797 --------------------- Stats for enwiki on date 20231201 Total size of page content dump files for articles, current content only: 97,431,055,664 Total size of page content dump files for all pages, current content only: 201,191,572,243 Total size of page content dump files for all pages, all revisions: 27,833,207,510,221 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

How to get wikipedia data dump?
by hrishipatel10＠gmail.com 29 Nov '23

29 Nov '23

Hello, I'm currently looking for latest Wikipedia data dumps that includes the complete history of Wikipedia edits for research purpose. I'm aware that a similar data dump exists for Wikidata edits, but I haven't been able to locate the same for Wikipedia. Despite checking https://dumps.wikimedia.org/, I couldn't find the latest dump featuring Wikipedia edits. I would greatly appreciate any help in this matter. Cheers, Hrishi

2 1

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 29 Nov '23

29 Nov '23

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20231101 full revision history content run. We are currently dumping 982 projects in total. --------------------- Stats for snwiktionary on date 20231101 Total size of page content dump files for articles, current content only: 506,341 Total size of page content dump files for all pages, current content only: 607,858 Total size of page content dump files for all pages, all revisions: 741,521 --------------------- Stats for enwiki on date 20231101 Total size of page content dump files for articles, current content only: 96,959,256,118 Total size of page content dump files for all pages, current content only: 200,144,987,436 Total size of page content dump files for all pages, all revisions: 27,663,890,580,870 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Need Wikipedia data dump
by venkat ekkuluri 22 Nov '23

22 Nov '23

Hi team, I need XML data dumps for the Wikipedia website for analysis. Can you please give me access to that. Thanks & Regards Narayana Ekkuluri

2 1

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 29 Oct '23

29 Oct '23

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20231001 full revision history content run. We are currently dumping 978 projects in total. --------------------- Stats for siwikibooks on date 20231001 Total size of page content dump files for articles, current content only: 132,780,514 Total size of page content dump files for all pages, current content only: 137,973,182 Total size of page content dump files for all pages, all revisions: 400,297,875 --------------------- Stats for enwiki on date 20231001 Total size of page content dump files for articles, current content only: 96,514,232,871 Total size of page content dump files for all pages, current content only: 199,261,948,528 Total size of page content dump files for all pages, all revisions: 27,494,135,074,915 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Data about category views or uses
by mcentellesv＠gmail.com 17 Oct '23

17 Oct '23

Hi, I'm interested in category views or uses by users. I'd found statistics from some years ago (https://es.wikipedia.org/wiki/Wikipedia:Ranking_de_visitas_(Categor%C3%ADa)). It's said that they are updated in Wiki Shark trends https://www.wikishark.com, but I could'n find them. Other sources like Massviews Analysis https://pageviews.wmcloud.org/massviews/ let us search pageviews through categories, but not directly category views. Any idea will be appreciated. Miquel Centelles

3 3

Wiki content and other dumps new ownership, feedback requested on new version!
by Ariel Glenn WMF 29 Sep '23

29 Sep '23

Hello folks! For some years now, I've been the main or only point of contact for the Wiki project sql/xml dumps semimonthly, as well as for a number of miscellaneous weekly datasets. This work is now passing to Data Platform Engineering (DPE), and your new points of contact, starting right away, will be Will Doran (email:wdoran) and Virginia Poundstone (email:vpoundstone). I'll still be lending a hand in the background for a little while but by the end of the month I'll have transitioned into a new role at the Wikimedia Foundation, working more directly on MediaWiki itself. The Data Products team, a subteam of DPE, will be managing the current dumps day-to-day, as well as working on a new dumps system intended to replace and greatly improve the current one. What formats will it produce, and what content, and in what bundles? These are all great questions, and you have a chance to help decide on the answers. The team is gathering feedback right now; follow this link [ https://docs.google.com/forms/d/e/1FAIpQLScp2KzkcTF7kE8gilCeSogzpeoVN-8yp_S…] to give your input! If you want to follow along on work being done on the new dumps system, you can check the phabricator workboard at https://phabricator.wikimedia.org/project/board/6630/ and look for items with the "Dumps 2.0" tag. Members of the Data Products team are already stepping up to manage the xmldatadumps-l mailing list, so you should not notice any changes as far as that goes. And as always, for dumps-related questions people on this list cannot answer, and which are not covered in the docs at https://meta.wikimedia.org/wiki/Data_dumps or https://wikitech.wikimedia.org/wiki/Dumps you can always email ops-dumps (at) wikimedia.org. See you on the wikis! Ariel Glenn ariel(a)wikimedia.org

3 2

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 29 Sep '23

29 Sep '23

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20230901 full revision history content run. We are currently dumping 977 projects in total. --------------------- Stats for zh_min_nanwikisource on date 20230901 Total size of page content dump files for articles, current content only: 24,387,985 Total size of page content dump files for all pages, current content only: 25,074,834 Total size of page content dump files for all pages, all revisions: 107,678,639 --------------------- Stats for enwiki on date 20230901 Total size of page content dump files for articles, current content only: 96,081,888,863 Total size of page content dump files for all pages, current content only: 198,396,512,576 Total size of page content dump files for all pages, all revisions: 27,333,035,543,183 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l