Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

no need to panic (wikidatawiki run)
by Ariel Glenn WMF 17 Dec '16

17 Dec '16

This run was taking a long time to complete (see https://phabricator.wikimedia.org/T153345). The underlying problem has been fixed and the last steps are running manually right now as we play catch-up. Expect the run to show up as completed late today or early tomorrow (UTC time). Ariel

1 0

Re: [Xmldatadumps-l] Xmldatadumps-l Digest, Vol 80, Issue 2
by Scott Roberts 09 Dec '16

09 Dec '16

Thanks for doing this. I would rather have all of the table dumps together. I parse the name so order of precedence isn’t an issue for me. --Scott On 12/9/16, 7:00 AM, "Xmldatadumps-l on behalf of xmldatadumps-l-request(a)lists.wikimedia.org" <xmldatadumps-l-bounces(a)lists.wikimedia.org on behalf of xmldatadumps-l-request(a)lists.wikimedia.org> wrote: Send Xmldatadumps-l mailing list submissions to xmldatadumps-l(a)lists.wikimedia.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l or, via email, send a message with subject or body 'help' to xmldatadumps-l-request(a)lists.wikimedia.org You can reach the person managing the list at xmldatadumps-l-owner(a)lists.wikimedia.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Xmldatadumps-l digest..." Today's Topics: 1. changing order of dump steps in status and checksum files (Ariel Glenn WMF) ---------------------------------------------------------------------- Message: 1 Date: Thu, 8 Dec 2016 17:37:29 +0200 From: Ariel Glenn WMF <ariel(a)wikimedia.org> To: Wikipedia Xmldatadumps-l <Xmldatadumps-l(a)lists.wikimedia.org> Subject: [Xmldatadumps-l] changing order of dump steps in status and checksum files Message-ID: <CALCvg_6qG1JC0Qq-biaH8zNwgK6LvatYjr=wGvazUdBivrym=Q(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Before I do this, I want to know if anyone here relies on the specific order of the contents of the md5 or sha1 sum files for the dumps, or on the order of the entries in the dumpruninfo file. The reason I want to fiddle with the order is to have all the table dumps together, rather than scattered around in these files. And the reason for that is convenience; I'm about to update the code that adds these jobs as steps to be run, and it's more readable/maintainable to add them all in one group. Anyone here who would be impacted? Please let me know; I'd like to roll this out for the second monthly run. Thanks, Ariel -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.wikimedia.org/pipermail/xmldatadumps-l/attachments/20161208/6…> ------------------------------ Subject: Digest Footer _______________________________________________ Xmldatadumps-l mailing list Xmldatadumps-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l ------------------------------ End of Xmldatadumps-l Digest, Vol 80, Issue 2 *********************************************

1 0

changing order of dump steps in status and checksum files
by Ariel Glenn WMF 08 Dec '16

08 Dec '16

Before I do this, I want to know if anyone here relies on the specific order of the contents of the md5 or sha1 sum files for the dumps, or on the order of the entries in the dumpruninfo file. The reason I want to fiddle with the order is to have all the table dumps together, rather than scattered around in these files. And the reason for that is convenience; I'm about to update the code that adds these jobs as steps to be run, and it's more readable/maintainable to add them all in one group. Anyone here who would be impacted? Please let me know; I'd like to roll this out for the second monthly run. Thanks, Ariel

1 0

Wikipedia page IDs
by Renato Stoffalette Joao 03 Dec '16

03 Dec '16

Hi all. Firstly, apologies for eventual duplicates or posting the question in the wrong mailing list. Secondly, could anybody kindly explain to me if some Wikipedia pages changed their IDs from the past ? Or if so point to me where this might be documented ? I have Wikipedia pages-articles XML dumps from the years 2006 and 2008 and when I was parsing those dumps I ran across some situations such as the following one. In the dumps from 2006 and 2008 I found that the South Africa page has the ID 68854, while in the most current Wikipedia pages-articles XML dump (i.e. 2016) the same article has the ID 17416221. I am trying to match some Wiki pages by IDs across time, but the example above is not helping. Much appreciated in advance for any help. -- Renato Stoffalette Joao - PhD Student - L3S Research Center / Leibniz Uni. 15th Floor, Room:1519 Appelstraße 9a 30167 Hannover, Germany +49.511.762-17759

3 2

9 am UTC maintenance for dataset1001 (dumps.wikimedia.org)
by Ariel Glenn WMF 14 Nov '16

14 Nov '16

On Tuesday Nov 13, at 9 am UTC, the web server for the dumps and other datasets will be unavailable due to maintenance. This should take no longer than 10 minutes. Thanks for your understanding. Ariel

1 1

Wikimedia datasets collection on the Internet Archive has surpassed 1 million items
by Hydriz Scholz 14 Nov '16

14 Nov '16

Dear all, The Wikimedia Foundation datasets collection on the Internet Archive [1] has now surpassed 1 million items (and about 50,000 full database dumps)! This marks a major milestone in our archiving efforts of Wikimedia's vast amount of data and ensures that the vital content submitted by volunteers across the moment is preserved. All these would not have been possible without the help of many people, including Nemo, Ariel and Emijrp (thanks!). We started archiving towards the end of 2011 and reached a milestone of half a million items back in June 2015. [2] We have since moved on from archiving just the main database dumps to saving research-worthy data such as the pageviews data and even attempting to keep a copy of Wikimedia Commons. Today, we are working on making the items on the Internet Archive more accessible for researchers by working on an interface for searching old dumps. Despite this feat, we are in constant need of more help. If you are a researcher, a programmer or someone with a computer, we need your help in many tasks! Have a look at WikiTeam's project [3] or Emijrp's Wikipedia Archive page [4] for more information. If you regularly work on the Wikimedia database dumps, please provide your input in the Dumps-Rewrite project [5] and the API interface [6]. As before, here's to the next million! [1]: https://archive.org/details/wikimediadownloads [2]: https://groups.google.com/forum/#!msg/wikiteam-discuss/Vj3oonpYphg/h9HE6r3v… [3]: https://github.com/WikiTeam/wikiteam [4]: https://en.wikipedia.org/wiki/User:Emijrp/Wikipedia_Archive [5]: https://phabricator.wikimedia.org/tag/dumps-rewrite/ [6]: https://phabricator.wikimedia.org/T147177 -- Hydriz Scholz

1 0

8 am UTC Oct 29, maintenance for dataset1001 (dumps.wikimedia.org)
by Ariel Glenn WMF 28 Oct '16

28 Oct '16

On Saturday Oct 29, at 8 am UTC, the web server for the dumps and other datasets will be unavailable due to maintenance. This should take no longer than 10 minutes. Thanks for your understanding. Ariel

1 0

The Wikimedia Developer Summit wants YOU!
by Srishti Sethi 19 Oct '16

19 Oct '16

Hello! The Wikimedia Developer Summit <https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit> is the annual meeting to push the evolution of MediaWiki and other technologies supporting the Wikimedia movement. The next edition will be held in San Francisco on January 9-11, 2017. We welcome all Wikimedia technical contributors, third party developers, and users of MediaWiki and the Wikimedia APIs. We specifically want to increase the participation of volunteer developers and other contributors dealing with extensions, apps, tools, bots, gadgets, and templates. Important deadlines: - Monday, October 24: This is the last day to request travel sponsorship. Applying takes less than five minutes. - Monday, October 31: This is the last day to propose an activity. Bring the topics you care about! Subscribe to weekly updates: https://www.mediawiki.org/ wiki/Topic:Td5wfd70vptn8eu4 Please feel free to forward this email to anyone who might be interested in attending! Thanks, Srishti -- Srishti Sethi ssethi(a)wikimedia.org

2 1

Suggestions wanted: api for monitoring dump runs
by Ariel Glenn WMF 04 Oct '16

04 Oct '16

The next Wikimedia Developers Summit will be in January 2017. I plan to hold an unconference session on development of an API for monitoring/stats for dumps of all sorts. Let's get the discussion going now; what do you want to see? Note that tis is for the rewrite so you need not be restricted by how the dumps run now. Add your thoughts, questions, comments to https://phabricator.wikimedia.org/T147177 and feel free to forward this where you think appropriate. Thanks! Ariel

1 0

New mirror of 'other' datasets
by Ariel Glenn WMF 27 Sep '16

27 Sep '16

I'm happy to announce a new mirror for datasets other than the XML dumps. This mirror comes to us courtesy of the Center for Research Computing, University of Notre Dame, and covers everything "other" [1] which includes such goodies as Wikidata entity dumps, pageview counts, titles of all files on each wiki (daily), titles of all articles of each wiki (daily), and the so-called "adds-changes" dumps, among other things. You can access it at http://wikimedia.crc.nd.edu/other/ so please do! Ariel [1] https://dumps.wikimedia.org/other/

2 7

← Newer
1
...
24
25
26
27
28
29
30
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l