Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

7/22 enwiki dump pages-meta-history
by Napolitano, Diane 03 Aug '11

03 Aug '11

Hello, are there any plans to combine all of the pages-meta-history XML dumps from the 7/22 dump into one file? This is useful for importing into JWPL. Thanks, Diane M. Napolitano Associate Research Engineer Educational Testing Service Turnbull Hall R-239 Princeton, New Jersey 08540

5 7

happy august
by Ariel T. Glenn 01 Aug '11

01 Aug '11

A new month, another couple of en wikipedia dumps... It looks like the various upgrade issues are all straightened out. The June files that were truncated have all been rerun and are ready for download. In the meantime the July dumps are ready, for those willing to grab the bz2 files. 7z files should be available in anouther couple of days, barring any site issues. If the July files look ok to folks, I'll do the last of our OS upgrades so that all our dump servers will be up to date. Ariel

1 0

Missing templates
by Kevin Clark 22 Jul '11

22 Jul '11

I have been tasked with building an offline copy of the Wikipedia website. The main goal is to have the database and images stored locally so that we can run a Wikipedia website on a local server. Our ultimate goal is to have Mediawiki and the Wikipedia database and images stored on a single hard drive in a server in a location with no Internet access. I have already been in contact with the Wikimedia Offline list about this issue but as yet have received no feedback about how to go about solving this problem. I've made good progress configuring Mediawiki and importing the articles and page links but when I look at some pages I can see that some templates are missing from my offline Wikipedia. The missing templates are made obvious because the text "Template:abcdef" is displayed in red, and have the alt text "Template:abcdef (page does not exist)". A couple of specific examples of missing templates are 'Template:Citation/make link' and 'Template:Gaps'. My offline Wikipedia data was imported into the MySQL database using the Mwdumper program. The source data came from the enwiki-latest-pages-articles.xml file. I imported the page links into the database using the enwiki-latest-pagelinks.sql file. Can anyone give me some guidance on how to fix this problem? My sincere apologies if this is not the appropriate place to ask for assistance with this problem. Kevin

2 1

testing after OS upgrade, please share your eyeballs
by Ariel T. Glenn 18 Jul '11

18 Jul '11

I've finally moved past manual testing to running a worker process on one of our upgraded hosts. This means new versions of various pieces of software as well as the OS. I hope I've found all the issues but I would feel a whole lot better aobut it if folks would take a look at the jobs running now. Small wikis only, the large ones are still running on the old setup. Anyone notice anything weird or is it all ok? Ariel

1 0

latest enwiki dump broken
by Eric Sun 05 Jul '11

05 Jul '11

The latest enwiki pages dump of enwiki-latest-pages-articles.xml.bz2 in http://dumps.wikimedia.org/enwiki/latest/ is only 5.8 GB. Previous versions, e.g. http://dumps.wikimedia.org/enwiki/20110526/ and http://dumps.wikimedia.org/enwiki/20110405/ have been consistently around 6.7-6.8GB. I saw this after noticing that many pages are missing from the newest dump, e.g. http://en.wikipedia.org/wiki/Liar_Liar and http://en.wikipedia.org/wiki/Juan_que_re%C3%ADa. Is this a known problem? Can anything be done to prevent this in the future? Thanks, Eric

2 1

Fwd: Wikis around Europe!
by emijrp 29 Jun '11

29 Jun '11

Hi. I forward this e-mail, I hope there are people interested on this map. ---------- Forwarded message ---------- From: emijrp <emijrp(a)gmail.com> Date: 2011/6/11 Subject: Wikis around Europe! To: wikiteam-discuss(a)googlegroups.com Hi all; A friend of mine has sent me this link about wikis (locapedias) around Europe.[1] I'm very surprised about the huge amount of wikis available. Time to archive all of them.[2] I have been working on Spanish ones. If you want to help archiving one country, please, reply to this message to coordinate. If not, I will try to archive entire Europe! Regards, emijrp [1] http://maps.google.com/maps/ms?ie=UTF8&t=h&msa=0&msid=115570622864617231547… [2] http://code.google.com/p/wikiteam/

5 5

Re: [Xmldatadumps-l] [Wiki-research-l] Wikipedia dumps downloader
by emijrp 28 Jun '11

28 Jun '11

Hi; @Derrick: I don't trust Amazon. Really, I don't trust Wikimedia Foundation either. They can't and/or they don't want to provide image dumps (what is worst?). Community donates images to Commons, community donates money every year, and now community needs to develop a software to extract all the images and packed them, and of course, host them in a permanent way. Crazy, right? @Milos: Instead of spliting image dump using the first letter of filenames, I thought about spliting using the upload date (YYYY-MM-DD). So, first chunks (2005-01-01) will be tiny, and recent ones of several GB (a single day). Regards, emijrp 2011/6/28 Derrick Coetzee <dcoetzee(a)eecs.berkeley.edu> > As a Commons admin I've thought a lot about the problem of > distributing Commons dumps. As for distribution, I believe BitTorrent > is absolutely the way to go, but the Torrent will require a small > network of dedicated permaseeds (servers that seed indefinitely). > These can easily be set up at low cost on Amazon EC2 "small" instances > - the disk storage for the archives is free, since small instances > include a large (~120 GB) ephemeral storage volume at no additional > cost, and the cost of bandwidth can be controlled by configuring the > BitTorrent client with either a bandwidth throttle or a transfer cap > (or both). In fact, I think all Wikimedia dumps should be available > through such a distribution solution, just as all Linux installation > media are today. > > Additionally, it will be necessary to construct (and maintain) useful > subsets of Commons media, such as "all media used on the English > Wikipedia", or "thumbnails of all images on Wikimedia Commons", of > particular interest to certain content reusers, since the full set is > far too large to be of interest to most reusers. It's on this latter > point that I want your feedback: what useful subsets of Wikimedia > Commons does the research community want? Thanks for your feedback. > > --=20 > Derrick Coetzee > User:Dcoetzee, English Wikipedia and Wikimedia Commons administrator > http://www.eecs.berkeley.edu/~dcoetzee/ > >

2 3

Wikipedia dumps downloader
by emijrp 28 Jun '11

28 Jun '11

Hi all; Can you imagine a day when Wikipedia is added to this list?[1] WikiTeam have developed a script[2] to download all the Wikipedia dumps (and her sister projects) from dumps.wikimedia.org. It sorts in folders and checks md5sum. It only works on Linux (it uses wget). You will need about 100GB to download all the 7z files. Save our memory. Regards, emijrp [1] http://en.wikipedia.org/wiki/Destruction_of_libraries [2] http://code.google.com/p/wikiteam/source/browse/trunk/wikipediadownloader.py

5 6

Re: [Xmldatadumps-l] Wikipedia dumps downloader
by emijrp 27 Jun '11

27 Jun '11

Hi Richard; Yes, a distributed project would be probably the best solution, but it is not easy to develop, unless you use a library like bittorrent, or similar and you have many peers. Althought most of the people don't seed the files long time, so sometimes is better to depend on a few committed persons than a big but ephemeral crowd. Regards, emijrp 2011/6/26 Richard Farmbrough <richard(a)farmbrough.co.uk> > ** > It would be useful to have an archive of archives. I have to delete my > old data dumps as time passes, for space reasons, however a team could, > between them, maintain multiple copies of every data dump. This would make a > nice distributed project. > > On 26/06/2011 13:53, emijrp wrote: > > Hi all; > > Can you imagine a day when Wikipedia is added to this list?[1] > > WikiTeam have developed a script[2] to download all the Wikipedia dumps > (and her sister projects) from dumps.wikimedia.org. It sorts in folders > and checks md5sum. It only works on Linux (it uses wget). > > You will need about 100GB to download all the 7z files. > > Save our memory. > > Regards, > emijrp > > [1] http://en.wikipedia.org/wiki/Destruction_of_libraries > [2] > http://code.google.com/p/wikiteam/source/browse/trunk/wikipediadownloader.py > > > _______________________________________________ > Xmldatadumps-l mailing listXmldatadumps-l@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l > > >

1 0

zh dump has stopped
by Andreas Meier 22 Jun '11

22 Jun '11

see http://dumps.wikimedia.org/zhwiki/20110521/ Best regards, Andreas

2 3

← Newer
1
...
59
60
61
62
63
64
65
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l