[Labs-l] Dumps project storage

[Ryan Lane]

On Tue, May 21, 2013 at 12:05 PM, Federico Leva (Nemo)
<nemowiki at gmail.com>wrote:

> Ryan Lane, 21/05/2013 19:41:
>
>> Hydriz told me in the past that the dumps project is just uploading the
>> dumps to archive.org <http://archive.org>. I'm confused as to why it's
>>
>> necessary to write them anywhere at all. They are accessible directly at
>> /public/data. Can they not be directly uploaded from that location?
>>
>> Is dumps being used for an additional purpose?
>>
>
> Yes, closely related to those, for instance Commons media (not progressing
> much lately). https://archive.org/details/**wikimediacommons<https://archive.org/details/wikimediacommons>
> Anyway, if it's a big deal we can just use my home PC as usual.
>
>
It's not that I'm opposed to it, but it's a massive waste of resources to
download from something in the network to a network fileserver, then to
upload it to archive.org.

Why is it necessary to write hundreds of GB to the fileserver before they
are uploaded?

- Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/labs-l/attachments/20130521/df1933da/attachment.html>