2009/2/22 River Tarnell <river(a)loreley.flyingparchment.org.uk>uk>: According to Brion on wikitech-l nearly a month ago: "[...] it's a software architecture issue. We'll restart [the enwiki dump] with the new arch when it's ready to go." So I guess that means it is being worked on, but it doesn't seem to be a high enough priority. This problem has been around for years now, it should have been fixed a long time ago.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Casey Brown: the shell user in question is/was a WMF contractor, and i believe it was done on paid time at Brion's request--but anyway, the point is the WMF is clearly past the stage of running around like headless chickens trying to keep the site on-line, so will we see some progress with dumps any time soon? - river.

2009/2/23 Newyorkbrad (Wikipedia) <newyorkbrad(a)gmail.com>om>: It's broken, I don't think there is any dispute there. If you want complete statistics, you need all the information. It might be interesting to see how the ratio of edits to non-article namespaces to total edits varies over time, for instance.

Actually, I was thinking primarily of userspace. Newyorkbrad On Mon, Feb 23, 2009 at 2:44 PM, David Gerard <dgerard(a)gmail.com> wrote:

2009/2/23 Chad <innocentkiller(a)gmail.com>om>: The community is a total write-off, but the WMF seems sane enough to pay the electricity bill on time.

Copying the Commons list. I am interested in hosting (and running some scripts on) copies of the commons media dump on offline regional servers for offline-reading purposes. This is difficult without an image dump. The last time I looked, I was able to find an image dump from 2007? Now I have a hard time finding that... can someone help me out / point me to a recent torrent? My 300+G copy from 2007 was deleted in an unfortunate fileserver cleanup, by someone who said "hey, you can always download another copy later." Perhaps I waited too long... SJ On Sun, Feb 22, 2009 at 7:49 AM, River Tarnell &lt;river(a)loreley.flyingparchment.org.uk&gt; wrote: <

On 2/22/09 4:49 AM, River Tarnell wrote: Well, you could have just asked me. :) http://leuksman.com/log/2009/02/24/wikimedia-data-dump-update/ -- brion

Its not at all clear why the english wikipedia dump or other large dumps need to be compressed. It is far more absurd to spend hundreds of days compressing a file than it is to spend tens of days downloading one. On Tue, Feb 24, 2009 at 10:45 AM, Brion Vibber &lt;brion(a)wikimedia.org&gt; wrote:

I am of the understanding that the WMF's bandwidth is very cheap. If you want to consider costs, I think its appropriate to consider the costs not only to the WMF but to the user. Different compression algorithms have different encode/decode ratios but if it takes a cluster to compress a file there's a good chance you're going to want one to decompress it. It may in fact be much more user friendly to simply offer an enormous text file for download because users don't have to unpack it. Our mission is to spread knowledge. Compressing that knowledge has been in the way of spreading it for years now. Its high time we gave up! On Tue, Feb 24, 2009 at 11:18 AM, Anthony &lt;wikimail(a)inbox.org&gt; wrote:

On Tue, Feb 24, 2009 at 1:24 PM, Brian &lt;Brian.Mingus(a)colorado.edu&gt; wrote: Another point, which I forgot to mention. If you have the bandwidth to kill and just want an enormous uncompressed text file, why not just screen-scrape everything?

On Tue, Feb 24, 2009 at 2:16 PM, Thomas Dalton &lt;thomas.dalton(a)gmail.com&gt;wrote;wrote: What's the average ratio of CPU-seconds to download seconds for an article? Surely a single machine could handle thousands of simultaneous screen-scrapers doing this 24/7. I don't buy it.

On Tue, Feb 24, 2009 at 2:41 PM, Brian &lt;Brian.Mingus(a)colorado.edu&gt; wrote: No, I wasn't suggesting it as an alternative to compressed dumps at all. And Special:Export doesn't even work for historical versions any more. Providing only uncompressed dumps would be an even more awful idea.

On Tue, Feb 24, 2009 at 2:55 PM, Chad &lt;innocentkiller(a)gmail.com&gt; wrote: Yeah, I guess that bug was reintroduced. They had it fixed for a while.

2009/2/24 Anthony &lt;wikimail(a)inbox.org&gt;rg>: I don't know, but people are asked not to crawl the entire site like that, so I guess there is a reason.

On Tue, Feb 24, 2009 at 3:12 PM, Anthony &lt;wikimail(a)inbox.org&gt; wrote: Afaik, it's fully open on all the wikis except enwiki (according to the admin log). However, I've yet to find any place in CommonSettings or InitialiseSettings that changes it from the default. It's no so much a bug as it is a configuration choice :) -Chad

Anthony wrote: I did make indexed, random-access, backwards compatible, XML dumps. http://lists.wikimedia.org/pipermail/wikitech-l/2009-January/040812.html Wouldn't be hard to plug into the dump process (just replace bzip2 on a new DumpPipeOutput) but so far nobody seemed interested on it. And there's the added benefit of the offline reader I implemented using those files.

Why not make the uncompressed dump available as an Amazon Public Dataset? http://aws.amazon.com/publicdatasets/ You can already find DBPedia and FreeBase there. Its true that the uncompressed dump won't fit on a commercial drive (the largest is a 4-platter 500GB = 2TB drive). Cloud computing seems to be the most economically feasible alternative for all parties involved. "Typically the data sets in the repository are between 1 GB to 1 TB in size (based on the Amazon EBS volume limit), but we can work with you to host larger data sets as well. You must have the right to make the data freely available." Clearly a boon! On Sun, Feb 22, 2009 at 5:49 AM, River Tarnell &lt;river(a)loreley.flyingparchment.org.uk&gt; wrote:

A dump with just the article namespace would be grossly incomplete. Much important information about the validity of the contents is on the discussion pages. But not only there. Other discussions on articles have been held on user talk pages. Missing these out would greatly hamper any judgement on the validity of the articles. teun spaans On Mon, Feb 23, 2009 at 8:35 PM, Newyorkbrad (Wikipedia) < newyorkbrad(a)gmail.com&gt; wrote:

Excellent -- following up offlist. SJ On Tue, Feb 24, 2009 at 12:45 PM, Brion Vibber &lt;brion(a)wikimedia.org&gt; wrote:

On Tue, Feb 24, 2009 at 11:26 PM, Brian &lt;Brian.Mingus(a)colorado.edu&gt; wrote: Which uncompressed dump? The full history English Wikipedia dump doesn't exist, and there doesn't seem to be any demand for this anyway. You can already find DBPedia and FreeBase there. Its true that the "Cloud computing" might be a good alternative for some reusers, but if so it'd be most economical to just host the cloud at the source, i.e. API/live feed access open to everyone (for free or for a cost). For certain uses there's the toolserver, but access to it is handed out with special permission. For small amounts of traffic there's an API, and there's the live feed which seems to be limited to major corporations with special permission. The WMF hasn't put any real resources into this for the small time commercial user (big players have the live feed and non-commercial users can probably get toolserver access). Of course, there isn't all that much demand either. If there was, a third party would have set it up by now (I'd personally be willing to set up a pay-for-access toolserver and custom dump service if I could get a commitment from one or more people for a couple hundred a month in funding). "Typically the data sets in the repository are between 1 GB to 1 TB in Yeah, if there was any demand for this, nothing's stopping someone from setting it up on their own.

2009/2/25 Brian &lt;Brian.Mingus(a)colorado.edu&gt;du>: He didn't say there was no demand, he said there was no demand for having it on Amazon.

Ahh ok. Anyone who wants to do processing on the full history (and there are a lot of these people who exist!) by definition *has* to be willing to throw some money at it. It simply doesn't fit on commercial drives. In fact, it would hardly fit on either of the two raid clusters I have access to. Making it available on Amazon means that, for a fair market rate, you don't have to download or uncompress the data. You can just start your data crunching. I can only speak for academics but there is generally funding available for Amazon EC2 etc... for specific projects. Professors are even known to pay for a fixed amount of processing for ambitious student projects, and these kinds of earmarks are easily fit into grants. The claim that there is no demand for having it on amazon is some kind of fallacy that I don't know the name for. Its never been available on Amazon, how could there be demand? Heck, it hasn't been available for several years in the first place so how could there be a demand for it? People *just want the data*. Many people would be willing to pay a fee. Thus, for an extremely reasonable price they can now create a new amazon disk image and download it to their own raid cluster if they want. The foundation doesn't have to foot the bill. Or they can find funding for their specific project, or whatever. I have a rare copy of the last available full text dump. Perhaps I should initiate the process myself. On Wed, Feb 25, 2009 at 2:20 PM, Thomas Dalton &lt;thomas.dalton(a)gmail.com&gt;wrote;wrote:

--- El jue, 26/2/09, Brian &lt;Brian.Mingus(a)colorado.edu&gt; escribió: Not necessarily. For instance, WikiXRay is capable of parsing the dump file on the fly, so you don't need to uncompress the whole file if you don't want to, and the result tipically fits in a 6-8 GB DB (depending on the amount of data your recover), which fits perfectly in commodity hw. On the other hand, I completely agree with you in that working with the huge XML file requires specific hw (we bought a couple of servers for that). Probably, but anyway, I would like to avoid paying a fee to access what should be publicly available (at least, until the dump process broke, it was). Some universities (including ourselves) has offered storage capacity and some bandwith to distribute mirrors and improve the dump availability, at no cost at all :). Nothing prevents you to do that (I think) and it could be a stimulus for thinking on subsequent solutions. Best, F.

I went ahead and submitted it to Amazon. I'll leave the file up for a week or so if anyone else wants it (18GB): http://mist.colorado.edu/enwiki-20080103-pages-meta-history.xml.7z Just to emphasize my point - I have never been able to unpack this file. I've got no place to put it!! On Wed, Feb 25, 2009 at 4:47 PM, Felipe Ortega &lt;glimmer_phoenix(a)yahoo.es&gt;wrote;wrote:

Brian wrote: I've personally never found much of a compelling reason to actually uncompress the dump, rather than working on the stream as it's being decompressed. 7zip decompression is pretty fast, and can use multiple cores on multi-core machines, so it never seems to be a bottleneck, for me at least--- I get somewhere around 30-40 MB/s typically. From what I can tell, the top-end EC2 instances do perform rather better than that, topping out at around 200 MB/s for sequential reads. But I don't personally run anything that can't run 5x slower in return for being free, and I suspect lots of analysis is of that "just let it run for a week, who cares" variety. I'm not going to argue that nobody could benefit from using EC2 to do their analysis instead, but it's hardly the case that it's impossible to do full-history analysis on commodity hardware. -Mark

On Wed, Feb 25, 2009 at 15:48, Brian &lt;Brian.Mingus(a)colorado.edu&gt; wrote: Using 1TB hard drives and a bit of creativity, you can build a 9TB storage server for that $1000. Disk space is getting cheaper all the time, and it's one of those cases where you can save a small fortune by building the computer yourself. -- Mark

Hoi, People have publicly said that they were willing to provide hardware or developers to work on this. The thing is that such offers have not been accepted. Thanks, GerardM 2009/2/26 Anthony &lt;wikimail(a)inbox.org&gt;

On Thu, Feb 26, 2009 at 9:49 AM, Anthony &lt;wikimail(a)inbox.org&gt; wrote: It would also take even longer than the Wikimedia dumps. Say 500ms average to request a single revision. Multiply by 250,000,000 revisions. I get a figure of four years. You're going to need a lot more than a single thread to get a remotely recent dump. You probably couldn't even keep up with the rate of new revision creation with a single thread blocking on each HTTP request.

On Thu, Feb 26, 2009 at 10:53 AM, Aryeh Gregor < Simetrical+wikilist(a)gmail.com &lt;Simetrical%2Bwikilist(a)gmail.com&gt;&gt; wrote: What's your estimate of how long it's going to take to get the next full history English Wikipedia dump? Say 500ms average to request a single revision. Why say that? 500ms is a long time. Besides, through the API, you can get multiple revisions at once. You'd only need to get revisions since the last successful dump - maybe 150,000,000. I get a figure of four years. You're going to need a lot I just got 50 revisions of [[Georgia]] in 6.389 seconds using the API and my slow internet connection. Even at that rate all the revisions since the last dump could be downloaded in seven months, which is much less than the time since the last successful full history dump. Ongoing it'd take 7 hours to download a new day's edits, but more realistically a live feed could be set up at that point. And this is a worst case scenario. It assumes the WMF doesn't help *at all* aside from allowing a single thread to access its servers.

Brian wrote: People downloading it would like to verify they downloaded it correctly. md5(enwiki-20080103-pages-meta-history.xml.7z) is 20a201afc05a4e5f2f6c3b9b7afa225c