Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

dumps of short urls now available
by Ariel Glenn WMF 13 May '19

13 May '19

Folks who use the Wikimedia projects may have heard that a url shortener is now available for them [1]. We now dump a list of the urls and their shortcuts once a week. The first such dump is available for download already [2]. It's been a long time coming. Enjoy! Ariel [1] https://meta.wikimedia.org/wiki/Wikimedia_URL_Shortener [2] https://dumps.wikimedia.org/other/shorturls/

1 0

Questions regarding the index.json file
by Aron Bergman 10 May '19

10 May '19

Hi, I've recently taken interest in the Wikipedia data dumps. I'd like to download a subset of files when they are updated. On the Data dumps page[1] a monitoring file[2] is mentioned, but the file doesn't contain any data (except the "wiki" object). I did some research and found the monitor.py script and some info in the relevant README [3]. If I've understood it correctly, a server will periodically run monitor.py which will create the index.json file. Is this deployed now? Since the file exists with (very little) content I'd guess that monitor.py has been run. [1] https://meta.wikimedia.org/wiki/Data_dumps#Monitoring_dump_generation [2] https://dumps.wikimedia.org/index.json [3] https://phabricator.wikimedia.org/source/operations-dumps/browse/master/xml… Regards Aron Bergman

2 1

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 02 May '19

02 May '19

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20190401 full revision history content run. We are currently dumping 918 projects in total. --------------------- Stats for ruwiki on date 20190401 Total size of page content dump files for articles, current content only: 21223127433 Total size of page content dump files for all pages, current content only: 26717658778 Total size of page content dump files for all pages, all revisions: 2662395674167 --------------------- Stats for enwiki on date 20190401 Total size of page content dump files for articles, current content only: 71723968674 Total size of page content dump files for all pages, current content only: 160245260388 Total size of page content dump files for all pages, all revisions: 18880938139465 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 31 Mar '19

31 Mar '19

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20190301 full revision history content run. We are currently dumping 917 projects in total. --------------------- Stats for eswikivoyage on date 20190301 Total size of page content dump files for articles, current content only: 29272320 Total size of page content dump files for all pages, current content only: 37816923 Total size of page content dump files for all pages, all revisions: 1591275166 --------------------- Stats for enwiki on date 20190301 Total size of page content dump files for articles, current content only: 71290510732 Total size of page content dump files for all pages, current content only: 159230276934 Total size of page content dump files for all pages, all revisions: 18730978850438 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Re: [Xmldatadumps-l] image dumps tarballs
by Federico Leva (Nemo) 20 Mar '19

20 Mar '19

Samuel Hoover, 20/03/19 21:24: > Does that mean Commons is currently culling its content? and that it > makes most sense to wait for a post 2016 dump until after housecleaning > is complete? No, I just mean that it takes time to identify copyright violations and so on. Most deletions happen for content uploaded in the last few months, so I generally respected an "embargo" of at least six months (hte Internet Archive items are supposed to be durable). > > Even with fiber connection/torrents, downloading will take time. Does > any organization sell terabyte drives containing the Commons dump? Or > can one travel to a physical location and connect several terabyte > drives to quickly copy over? Your best chance is probably to find some machines connected to the CENIC/Internet2 network or "nearby" and download the Internet Archive torrents from there. Hopefully you get 5-10 MiB/s per item and if you do all of them concurrently you should manage in a day or two. Internet Archive also routinely provides researcher access, but I'm not sure whether that's for private items only. Wikimedia Foundation used to provide data feeds for some companies back in the day. If there is a significant need I suppose they could arrange for someone to have rsync access or something, but it's not going to happen overnight. Federico

1 0

image dumps tarballs
by Samuel Hoover 20 Mar '19

20 Mar '19

Hello everyone, Are Wikipedia or Commons image dumps available anywhere? I found this one below but it stops at 2012. /http://ftpmirror.your.org/pub/wikimedia/imagedumps/tarballs/fulls/20121201//

2 1

https://dumps.wikimedia.org/
by 泥 14 Mar '19

14 Mar '19

Hello! Do you have wiki data available for download in 2012 and 2014? There is no download link found in https://dumps.wikimedia.org/. If yes, can you provide a download link? Thank you very much.

2 1

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 10 Mar '19

10 Mar '19

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20181101 full revision history content run. We are currently dumping 916 projects in total. --------------------- Stats for bnwikivoyage on date 20181101 Total size of page content dump files for articles, current content only: 6587638 Total size of page content dump files for all pages, current content only: 6825818 Total size of page content dump files for all pages, all revisions: 99741838 --------------------- Stats for enwiki on date 20181101 Total size of page content dump files for articles, current content only: 69650333728 Total size of page content dump files for all pages, current content only: 155512399552 Total size of page content dump files for all pages, all revisions: 18210409893326 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

2 1

some dump failures today
by Ariel Glenn WMF 06 Mar '19

06 Mar '19

Those of you watching the xml/sql dumps run this month may have noticed some dump failures today. These were caused by depooling of the database server for maintenance while the dump hosts were querying it. The jobs in question should be rerun automatically over the next few days, and I'll be keeping an eye on things. Ariel

1 0

wikimedia.bytemark.co.uk mirror is not updated from 2017-11
by Mariusz "Nikow" Klinikowski 04 Mar '19

04 Mar '19

Greetings XML Dump users and contributors! Looks like https://wikimedia.bytemark.co.uk/ is not updated from 2017-11-26. I think, maybe somebody should delete it from mirror list or contact bytemark notify them? Best regards, Mariusz "Nikow" Klinikowski.

3 5

← Newer
1
...
16
17
18
19
20
21
22
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l