Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

1 participants
719 discussions

Format
by John 10 Nov '12

10 Nov '12

I am looking to create a script for creating manual dumps for those wikis that either dont or wont publish their own dumps and that I dont have server access to. To that end I am writing a python dump creator, however I would like to ensure that my format is the same as the existing. I could reverse engineer it by looking at multiple different dumps but that takes a lot of time and is not fool proof, is there or can I get documentation and details on exactly how the XML dumps are formatted?

3 4

Need access to song (album song, movie song, single) information for English language
by Venkatesh Channal 07 Nov '12

07 Nov '12

Hi, I am trying to create a project that has the above mentioned information and can be used to correct the metadata information for songs. Is there a data dump available that contains the song data like song title, singer, duration, music director, genre, album/movie, language and country information? If there is no such dump available is there any tool to extract that information from the entire page-articles.xml dump of English wikipedia. If this is not the correct mailing list, could you please point me to the right mailing list to get data dump of song information. Thanks and regards, Venkatesh

2 2

multistream format; media bundle creation problems
by Ariel T. Glenn 30 Oct '12

30 Oct '12

now enabled for all wikis, even the small ones where it's completely pointless :-P (But someday they'll be big too!) Some of you may have noticed that media bundle production is behind. We are running into new performance issues and poking at it; for now please just be patient and we'll get them going again as soon as we can. Arie

2 4

Re: [Xmldatadumps-l] [Wiki-research-l] 1-year dump of English Wikipedia article ratings
by Jérémie Roquet 22 Oct '12

22 Oct '12

cc-ed xmldatadumps-l Hi, 2012/10/23 Dario Taraborelli <dtaraborelli(a)wikimedia.org>: > 2012/10/23 James Forrester <james(a)jdforrester.org>: >> On 22 October 2012 16:03, Hydriz Wikipedia <admin(a)alphacorp.tk> wrote: >>> I have long been wanting to say this, but is it possible for the team behind >>> compiling such datasets to put future (and if possible, current) datasets >>> into dumps.wikimedia.org so that it is easier for everyone to find stuff and >>> not be all over the place? Thanks for that! >> >> Many one-off and regular datasets, from query results to data dumps >> and similar, are now indexed[0] on The Data Hub (formerly CKAN) run by >> the Open Knowledge Foundation for precisely this reason - so that data >> researchers can easily find data about Wikimedia, and see when it's >> updated. >> >> [0] - http://thedatahub.org/en/group/wikimedia > > The dumps server was never meant to become a permanent open data repository, but it started being used as an ad-hoc solution to host all sort of datasets published by WMF on top of the actual XML dumps: that's the problem we're trying to fix. > > Regardless of where the data is physically hosted, your go-to point to discover WMF datasets from now on is the DataHub. Think of it as a data registry: the registry is all you need to know in order to find where the data is hosted and to extract the appropriate metadata/documentation. That's fine for me but I think more communication about this would be welcome. I've added a link to meta:Data_dumps¹ and I'll communicate about this on the French Wikipedia, but a link on the dumps' page for other downloads² would be great. Most people I've helped to find data on the Wikimedia projects now know about dumps.wikimedia.org, but AFAIK none of them is reading wiki-research-l. Best regards, ¹ https://meta.wikimedia.org/wiki/Data_dumps ² http://dumps.wikimedia.org/other/ -- Jérémie

2 1

Fwd: [Wikitech-l] "Latest" md5sums of dumps
by Jeremy Baron 11 Oct '12

11 Oct '12

---------- Forwarded message ---------- From: Zachary Harris <zacharyharris(a)hotmail.com> Date: Mon, Oct 8, 2012 at 5:49 PM Subject: [Wikitech-l] "Latest" md5sums of dumps To: wikitech-l(a)lists.wikimedia.org It is a bit confusing that http://dumps.wikimedia.org/enwiki/20121001/enwiki-20121001-md5sums.txt has newer hash values that aren't in http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-md5sums.txt. I'm guessing that the "latest" md5sums files get updated only at the end of a cycle when all dumps for the month have been completed? (In other words, the idea is that the "latest" directory contains backups that have been *completed*, and the md5sums file isn't complete until *all of the other backups* which it is going to reference are complete.) I was expecting the "latest" md5sums file to have "rolling" content matching the rolling content of the "latest" directory itself. If the latter is possible without too great of administrative hassle, it could be a nice feature. Otherwise, it would seem that "latest-md5sums" is almost always outdated relative to the other files in "latest". -Zach

4 3

Wikipedia page content dump based on category
by Venkatesh Channal 09 Oct '12

09 Oct '12

Hi, I would like to fetch all page text information of all wiki pages that belong to a movie category. Eg: http://en.wikipedia.org/wiki/Category:Hindi_songs >From the page text I would like to extract information related to song title, song length, singer, name of movie/album etc. I am not interested in extracting images just the information about the song. My questions: 1) Is there a way to download only those pages that I am interested in that belong to a particular category instead of downloading the entire dump? 2) Is it required to have PHP knowledge to install the db dump on a local machine? 3) Are there are tools that extract the information and provide the required data to be stored in MySQL database? If this is not the right forum to have my questions answered could you please redirect me to the appropriate forum. Thanks and regards, Venkatesh Channal

3 3

mail
by Abhishek prataap R 07 Oct '12

07 Oct '12

i want all the lists and images

1 0

Re: [Xmldatadumps-l] [Wikitech-l] HTML wikipedia dumps: Could you please provide them, or make public the code for interpreting templates?
by James L 04 Oct '12

04 Oct '12

£1.99 per page? :D From: Roberto Flores Sent: Thursday, October 04, 2012 3:15 AM To: Pablo N. Mendes Cc: James L ; Wikipedia Xmldatadumps-l ; Wikimedia developers Subject: Re: [Xmldatadumps-l] [Wikitech-l] HTML wikipedia dumps: Could you please provide them, or make public the code for interpreting templates? Could we have an HTML dump for X amount of money? Something like a paid feature. Include the CSS of course. Also, leave the <math> tags as they are, as those have to be processed by 3rd party libraries. 2012/9/17 Pablo N. Mendes <pablomendes(a)gmail.com> I also think the HTML dumps would be super useful! Cheers Pablo On Sep 17, 2012 8:05 PM, "James L" <james_leaver(a)hotmail.com> wrote: I’m all vote for continuing the HTML wiki dumps that were once done, 2007 was the last? Why are these discontinued? they would be more useful than the so called “XML”. There is no complete solution to processing dumps, the XML is most certainly not XML in its lowest form, and it IS DEFINITELY a moving target! Regards, From: Roberto Flores Sent: Sunday, September 09, 2012 8:07 PM To: Wikimedia developers Cc: Wikipedia Xmldatadumps-l Subject: Re: [Xmldatadumps-l] [Wikitech-l] HTML wikipedia dumps: Could you please provide them, or make public the code for interpreting templates? Allow me to reply to each point: (By the way, my offline app is called WikiGear Offline:) http://itunes.apple.com/us/app/wikigear-offline/id453614487?mt=8 > Templates are dumped just like all other pages are... Yes, but that's only a text description of what the template does. Code must be written to actually process them into HTML. There are tens of thousands of them, and some can't be even programmed by me (e.g., Wiktionary's conjugation templates) If they were already pre-processed into HTML inside the articles' contents, that would solve all of my problems. > what purpose would the dump serve? you dont want to keep the full dump > on the device. I made an indexing program that selects only content articles (namespaces included) and compresses it all to a reasonable size (e.g. about 7gb for the English Wikipedia) > How would this template API function? What does import mean? By this I mean, a set of functions written in some computer language to which I could send them the template within the wiki markup and receive HTML to display. Wikipedia does this whenever a page is requested, but I ignore the exact mechanism through which it's performed. Maybe you just need to make that code publicly available, and I'll try to make it work with my application somehow. 2012/9/9 Jeremy Baron <jeremy(a)tuxmachine.com> On Sun, Sep 9, 2012 at 6:34 PM, Roberto Flores <f.roberto.isc(a)gmail.com> wrote: > I have developed an offline Wikipedia, Wikibooks, Wiktionary, etc. app for > the iPhone, which does a somewhat decent job at interpreting the wiki > markup into HTML. > However, there are too many templates for me to program (not to mention, > it's a moving target). > Without converting these templates, many articles are simply unreadable and > useless. Templates are dumped just like all other pages are. Have you found them in the dumps? which dump are you looking at right now? > Could you please provide HTML dumps (I mean, with the templates > pre-processed into HTML, everything else the same as now) every 3 or 4 > months? 3 or 4 month frequency seems unlikely to be useful to many people. Otherwise no comment. > Or alternatively, could you make the template API available so I could > import it in my program? How would this template API function? What does import mean? -Jeremy _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ---------------------------------------------------------------------------- _______________________________________________ Xmldatadumps-l mailing list Xmldatadumps-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l _______________________________________________ Xmldatadumps-l mailing list Xmldatadumps-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l

1 0

bz2 multistreams redux
by Ariel T. Glenn 02 Oct '12

02 Oct '12

Some time ago I announced a trial of bz2 multistream files for en wikipedia on this list. The generated index files turned out to have a problem with the offsets somewhere and due to various other tasks this fell by the wayside. That bug is now fixed, the September en wikipedia bz2 multistream index file was regenerated, and a little toy offline reader is now available as a proof of concept for how one might work with these files. A brief reminder about what the format does: it allows rough random access to the XML page content. For the code, see: https://gerrit.wikimedia.org/r/gitweb?p=operations/dumps.git;a=tree;f=toys/… If I don't hear how broken things are over the next few days, I expect to enable generation of this format for all wiki projects shortly. Ariel

2 1

what's needed to fork
by Ariel T. Glenn 27 Sep '12

27 Sep '12

A discussion on the toolserver list brought up the question, once again, of what would be needed to fork the projects. Because some data is private we aren't going to be able to provide data for perfect copies, but the content can be preserved. The question is how close we can get. In particular I would like folks to think about how we can manage the user account issue. It would be very nice indeed if users could reclaim their accounts on a copy of the project, and yet we cannot provide any outside project a copy of the user table (which has email addresses and other useful bits in it). And many many users don't give an email address anyways. I'd like to hear proposals for how this could be handled. Wouldn't it be awesome if this could be done today, and Wikipedia editors could have editing privileges on copies of the project around the globe that provided different experimental features? Assuming of course that there were groups or organizations that wanted to run such copies of the site... Ariel

2 1

← Newer
1
...
50
51
52
53
54
55
56
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l