Thanks for the info Hydriz!<span></span><br><br>On Wednesday, May 22, 2013, Hydriz Scholz wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>
Yes, this has been discussed in the past that we will be reducing the amount of resources used for the dumps project. I am currently coming up with a few scripts and libraries to actually make this process of uploading/downloading much less resource intensive and hell more efficient. So Ryan, you don't have to worry too much about this one :)</div>
<div><br></div><div>However, the Wikimedia Commons grab is something that was undertaken by a team not directly related to Wikimedia. We download from upload.wm.o (yes), but at a rather slow speed to avoid overloading the servers. It has stopped since quite a while ago in the process of trying to optimize bandwidth and resource usage.</div>
<div><br></div><div>I am not exactly sure what Nemo wished to do in the original request, but I believe the team is still discussing better ways to handle this (like using the mirrors).</div><div><br></div>
<div>So, don't worry about the resource usage, we are currently still testing only, so not much usage of precious resources.</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, May 22, 2013 at 1:36 PM, Ryan Lane <span dir="ltr"><<a href="javascript:_e({}, 'cvml', 'rlane32@gmail.com');" target="_blank">rlane32@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>On Tue, May 21, 2013 at 10:28 PM, Federico Leva (Nemo) <span dir="ltr"><<a href="javascript:_e({}, 'cvml', 'nemowiki@gmail.com');" target="_blank">nemowiki@gmail.com</a>></span> wrote:<br>
</div><div class="gmail_extra"><div class="gmail_quote"><div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Ryan Lane, 21/05/2013 22:27:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
It's not that I'm opposed to it, but it's a massive waste of resources<br>
to download from something in the network to a network fileserver, then<br></div>
to upload it to <a href="http://archive.org" target="_blank">archive.org</a> <<a href="http://archive.org" target="_blank">http://archive.org</a>>.<div><br>
<br>
Why is it necessary to write hundreds of GB to the fileserver before<br>
they are uploaded?<br>
</div></blockquote>
<br>
Sorry, I don't understand the question. Consider the request withdrawn, thanks for answering.<span><font color="#888888"><br>
<br></font></span></blockquote></div><div><br>I'd like to make sure your need is handled, but I'd like to understand the need too. We've had quite a bit of discussion with Hydriz in the past about this project. It's resource intensive for us, so we try to make sure it's being done efficiently. We made the dumps available at /public/data so that it wouldn't be necessary to download them from download.wm.o, then upload them to <a href="http://archive.org" target="_blank">archive.org</a> (it's possible to upload them directly from the read-only dumps filesystem).<br>
<br></div><div>What I'm trying to understand is what is being written to /data/project and why it's larger than 200GB. Based on what I've been told so far, the project uploads dumps to <a href="http://archive.org" target="_blank">archive.org</a>. This is the first I'm hearing about uploading commons images. Are you downloading large amounts of images from upload.wm.o, writing them to /data/project, uploading them to <a href="http://archive.org" target="_blank">archive.org</a>, then deleting them from /data/project?<span><font color="#888888"><br>
</font></span></div><span><font color="#888888"><div><br></div><div>- Ryan<br></div></font></span></div></div></div>
<br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="javascript:_e({}, 'cvml', 'Labs-l@lists.wikimedia.org');" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Regards,</div><div>Hydriz</div><div><br></div><div>Be social, follow/add me:</div>Facebook: <a href="http://tinyurl.com/hydrizfb" target="_blank">http://tinyurl.com/hydrizfb</a><div>
Google+: <a href="http://tinyurl.com/hydrizgl" target="_blank">http://tinyurl.com/hydrizgl</a></div><div>Twitter: @hydrizwiki</div>
</div>
</blockquote>