Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

Mirror status
by Count Count 03 Aug '20

03 Aug '20

Just checked the mirrors. - The Academic Computer Club, Umeå University mirror apparently only mirrors the last two good dumps, not the last five: https://ftp.acc.umu.se/mirror/wikimedia.org/dumps/enwiki/ - The Bytemark mirror seems to have stopped mirroring in May 2020: https://wikimedia.bytemark.co.uk/enwiki/ - The C3SL mirror seems to have stopped mirroring in November 2019: http://wikipedia.c3sl.ufpr.br/enwiki/ - The Free Mirror Project mirror seems to work fine. - The Your.org mirror seems to work fine. - https://wikimedia.mirror.us.dev/ seems to work fine. Maybe the hosters of the failing mirrors should be contacted or the list updated? Best regards, Count Count

2 3

List of dumped wikis, discrepancy with Wikidata
by Count Count 03 Aug '20

03 Aug '20

Hi! I am currently working on a dump search and download tool for all Wikimedia wikis. In order to find out which Wikimedia wikis exist I used Wikidata. While comparing the list of wikis from Wikidata with the list of dumped projects I found out that the following wikis are currently not being dumped: - alswikibooks (last dump 20180101) - alswikiquote (last dump 20180101) - alswiktionary (last dump 20180101) - ecwikimedia (never dumped, private but not marked private in Wikidata?) - fixcopyrightwiki (last dump 20200220) - labswiki (never dumped?) - labtestwiki (never dumped?) - mowiki (last dump 20180101) - mowiktionary (last dump 20180101) - ru_sibwiki (last dump 20071011) - ukwikiversity (never dumped?) Is there an uptodate machine-readable list of currently dumped wikis besides https://dumps.wikimedia.org/backup-index.html? (Off-topic) Spoiler for dump searching tool on my laptop: $ target/release/wdgrep "asdfdefased" /c/Users/xyz/wpdumps/dewiki-20200701-pages-articles-multistream.xml -v --ns 0 Searched 21437.064 MiB in 8.467969 seconds (2531.5474 MiB/s). Best regards, Count Count

4 3

Re: [Xmldatadumps-l] Xmldatadumps-l Digest, Vol 120, Issue 2
by T. Shandelman 02 Aug '20

02 Aug '20

Hi, If you don't mind, please, starting next time, insert commas into those huge counts. Without commas they are VERY difficult to read. Thanks! Sincerely, Todd Shandelman Austin, TX On Sun, Aug 2, 2020, 07:01 <xmldatadumps-l-request(a)lists.wikimedia.org> wrote: > Send Xmldatadumps-l mailing list submissions to > xmldatadumps-l(a)lists.wikimedia.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l > or, via email, send a message with subject or body 'help' to > xmldatadumps-l-request(a)lists.wikimedia.org > > You can reach the person managing the list at > xmldatadumps-l-owner(a)lists.wikimedia.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Xmldatadumps-l digest..." > > > Today's Topics: > > 1. XML Dumps FAQ monthly update (noreply.xmldatadumps(a)wikimedia.org) > 2. List of dumped wikis, discrepancy with Wikidata (Count Count) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 01 Aug 2020 16:07:36 +0000 > From: noreply.xmldatadumps(a)wikimedia.org > To: xmldatadumps-l(a)lists.wikimedia.org > Subject: [Xmldatadumps-l] XML Dumps FAQ monthly update > Message-ID: <20200801160736.AneN_%noreply.xmldatadumps(a)wikimedia.org> > > > Greetings XML Dump users and contributors! > > This is your automatic monthly Dumps FAQ update email. This update > contains figures for the 20200701 full revision history content run. > > We are currently dumping 916 projects in total. > > > --------------------- > Stats for lmowiki on date 20200701 > > Total size of page content dump files for articles, current content only: > 151410097 > > Total size of page content dump files for all pages, current content only: > 179774126 > > Total size of page content dump files for all pages, all revisions: > 3555369968 > --------------------- > Stats for enwiki on date 20200701 > > Total size of page content dump files for articles, current content only: > 78326324425 > > Total size of page content dump files for all pages, current content only: > 173926604054 > > Total size of page content dump files for all pages, all revisions: > 21045320844828 > --------------------- > > > Sincerely, > > Your friendly Wikimedia Dump Info Collector > > > > ------------------------------ > > Message: 2 > Date: Sun, 2 Aug 2020 00:04:22 +0200 > From: Count Count <countvoncount123456(a)gmail.com> > To: xmldatadumps-l(a)lists.wikimedia.org > Subject: [Xmldatadumps-l] List of dumped wikis, discrepancy with > Wikidata > Message-ID: > <CAOHwkzAk6R+W4Xj673h= > p44zxwX+22Pt+Zd3UBg_NbSUUTg+1w(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi! > > I am currently working on a dump search and download tool for all Wikimedia > wikis. In order to find out which Wikimedia wikis exist I used Wikidata. > While comparing the list of wikis from Wikidata with the list of dumped > projects I found out that the following wikis are currently not being > dumped: > > - alswikibooks (last dump 20180101) > - alswikiquote (last dump 20180101) > - alswiktionary (last dump 20180101) > - ecwikimedia (never dumped, private but not marked private in > Wikidata?) > - fixcopyrightwiki (last dump 20200220) > - labswiki (never dumped?) > - labtestwiki (never dumped?) > - mowiki (last dump 20180101) > - mowiktionary (last dump 20180101) > - ru_sibwiki (last dump 20071011) > - ukwikiversity (never dumped?) > > Is there an uptodate machine-readable list of currently dumped wikis > besides https://dumps.wikimedia.org/backup-index.html? > > (Off-topic) Spoiler for dump searching tool on my laptop: > $ target/release/wdgrep "asdfdefased" > /c/Users/xyz/wpdumps/dewiki-20200701-pages-articles-multistream.xml -v --ns > 0 > Searched 21437.064 MiB in 8.467969 seconds (2531.5474 MiB/s). > > Best regards, > > Count Count >

1 1

XML Dumps FAQ monthly update
by noreply.xmldatadumps＠wikimedia.org 01 Aug '20

01 Aug '20

Greetings XML Dump users and contributors! This is your automatic monthly Dumps FAQ update email. This update contains figures for the 20200701 full revision history content run. We are currently dumping 916 projects in total. --------------------- Stats for lmowiki on date 20200701 Total size of page content dump files for articles, current content only: 151410097 Total size of page content dump files for all pages, current content only: 179774126 Total size of page content dump files for all pages, all revisions: 3555369968 --------------------- Stats for enwiki on date 20200701 Total size of page content dump files for articles, current content only: 78326324425 Total size of page content dump files for all pages, current content only: 173926604054 Total size of page content dump files for all pages, all revisions: 21045320844828 --------------------- Sincerely, Your friendly Wikimedia Dump Info Collector

1 0

Re: [Xmldatadumps-l] Xmldatadumps-l Digest, Vol 119, Issue 7
by Sami Mourad 31 Jul '20

31 Jul '20

help i would like to unsubscribe On Tue, Jul 28, 2020 at 1:01 PM <xmldatadumps-l-request(a)lists.wikimedia.org> wrote: > Send Xmldatadumps-l mailing list submissions to > xmldatadumps-l(a)lists.wikimedia.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l > or, via email, send a message with subject or body 'help' to > xmldatadumps-l-request(a)lists.wikimedia.org > > You can reach the person managing the list at > xmldatadumps-l-owner(a)lists.wikimedia.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Xmldatadumps-l digest..." > > > Today's Topics: > > 1. Has anyone had success with data deduplication? (griffin tucker) > 2. Re: Has anyone had success with data deduplication? (Count Count) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 28 Jul 2020 01:50:09 +0000 > From: griffin tucker <gtucker4.une(a)hotmail.com> > To: "xmldatadumps-l(a)lists.wikimedia.org" > <xmldatadumps-l(a)lists.wikimedia.org> > Subject: [Xmldatadumps-l] Has anyone had success with data > deduplication? > Message-ID: > < > TY2PR03MB3997DB2177073F2871ABE5E2D2730(a)TY2PR03MB3997.apcprd03.prod.outlook.com > > > > Content-Type: text/plain; charset="utf-8" > > I've tried using freenas/truenas with a data deduplication volume to store > multiple sequential dumps, however it doesn't seem to save much space at > all - I was hoping someone could point me in the right direction so that I > can download multiple dumps and not have it take up so much room > (uncompressed). > > Has anyone tried anything similar and had success with data deduplication? > > Is there a guide? >

1 0

Has anyone had success with data deduplication?
by griffin tucker 29 Jul '20

29 Jul '20

I've tried using freenas/truenas with a data deduplication volume to store multiple sequential dumps, however it doesn't seem to save much space at all - I was hoping someone could point me in the right direction so that I can download multiple dumps and not have it take up so much room (uncompressed). Has anyone tried anything similar and had success with data deduplication? Is there a guide?

4 6

no XML header?
by Alex Voss 21 Jul '20

21 Jul '20

Hi all, I am trying to read the dump from https://dumps.wikimedia.your.org/enwiki/20200701/enwiki-20200701-pages-arti… using a Java XMLStreamReader but it complains about the format. It looks like the file does not contain an XML header (<?xml version="1.0"?> or such) and after unpacking and prepending the header all seems fine. Is there a good reason why headers are missing? Cheers, Alex

1 0

Re: [Xmldatadumps-l] Request for Wikipedia dump of February 2017
by Ariel Glenn WMF 20 Jul '20

20 Jul '20

Dear Rajakumaran Archulan, Older dumps can often be found on the Internet Archive. The February 2017 full dumps for the English language Wikipedia are here: https://archive.org/details/enwiki-20170201 A reminder for all new and older members of this list: comprehensive documentation for dumps users is available on MetaWiki: https://meta.wikimedia.org/wiki/Data_dumps In the section "Getting the dumps" there are pointers for locating older dumps that are no longer available on the Wikimedia dumps download host. Ariel Glenn ariel(a)wikimedia.org On Mon, Jul 20, 2020 at 6:54 AM Rajakumaran Archulan < archulan.16(a)cse.mrt.ac.lk> wrote: > Dear sir/madam, > > I am a final year undergrad at the department of computer science & > engineering at University of Moratuwa, Sri Lanka. We are in the process of > building an evaluator for word embeddings for our final year project. > > We need the *Wikipedia dump of February 2017* for our research purpose. > We searched across the web for several hours. But we couldn't find it. It > would be grateful if you grant us access to the above corpus to > continue our research. > > Thank you! > > -- > *Best regards,* > *R.Archulan* > *Final year undergrad (16' Batch),* > *Dept. of Computer Science & Engineering,* > *Faculty of Engineering,* > *University of Moratuwa, Sri Lanka.* > *Mobile: (+94) 771761696* > *Linkedin* <https://www.linkedin.com/in/archulan> >

2 1

sample html dumps available FOR QA ONLY
by Ariel Glenn WMF 10 Jul '20

10 Jul '20

NOTE: I did not produce the HTML dumps, they are being managed by another team. If you are interested in weighing in on the output format, what's missing, etc, here is the phabricator task: https://phabricator.wikimedia.org/T257480 Your comments and suggestions would be welcome! Ariel

1 0

Commons structured data dumps
by Ariel Glenn WMF 09 Jul '20

09 Jul '20

RDF dumps of structured data from commons are now available at https://dumps.wikimedia.org/other/wikibase/commonswiki/ They are run on a weekly basis. See https://lists.wikimedia.org/pipermail/wikidata/2020-July/014125.html for more information. Enjoy!

1 0

← Newer
1
...
10
11
12
13
14
15
16
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l