This is phase one of a plan to make uploaded media from WMF projects accessible for download in bulk. It, like many other things lately, is experimental and subject to breakage, change, etc.
First, a big thanks to Kevin Day from Your.org who offered us the space and worked with us many hours to sort out networking issues, try different NAS setups, and generally do what was needed to get this going.
Rsync url: ftpmirror.your.org::wikimedia-images/projectname/languagecode
For example:
rsync -a ftpmirror.yours.org::wikimedia-images/wikipedia/commons /my/dir
would get you all of commons including archived versions (no deleted images of course).
Folks who are trying to download media for a specific project should bear in mind that they will need the files not only from that project but also those which are hosted on commons and used on the local project. I'm looking into producing lists of those files for easy use by rsyncers.
I would suggest rather than everyone downloading a zillion copies of commons at once, that folks coordinate a little bit, or just get the pieces they need :-D
The data that is there now is probably about 15-20 days old. It will likely be a little while before I get the media rsync going on a regular basis, I'm juggling a lot of pieces right now.
Ariel
P.S. This is not an April fools joke, it's April 2 here already :-P
Hi,
2012/4/2 Ariel T. Glenn ariel@wikimedia.org:
This is phase one of a plan to make uploaded media from WMF projects accessible for download in bulk. It, like many other things lately, is experimental and subject to breakage, change, etc.
First, a big thanks to Kevin Day from Your.org who offered us the space and worked with us many hours to sort out networking issues, try different NAS setups, and generally do what was needed to get this going.
Rsync url: ftpmirror.your.org::wikimedia-images/projectname/languagecode
For example:
rsync -a ftpmirror.yours.org::wikimedia-images/wikipedia/commons /my/dir
would get you all of commons including archived versions (no deleted images of course).
That's awesome news! Thank you all.
Best regards,
On Mon, Apr 2, 2012 at 1:08 AM, Ariel T. Glenn ariel@wikimedia.org wrote:
rsync -a ftpmirror.yours.org::wikimedia-images/wikipedia/commons /my/dir
(Typo: your.org not yours.org - bet you did that intentionally to throttle us ;-)
Great work, thanks for making this happen. :-) Are we talking to archive.org as well about setting up a copy there?
All best, Erik
Στις 02-04-2012, ημέρα Δευ, και ώρα 10:07 -0700, ο/η Erik Moeller έγραψε:
On Mon, Apr 2, 2012 at 1:08 AM, Ariel T. Glenn ariel@wikimedia.org wrote:
rsync -a ftpmirror.yours.org::wikimedia-images/wikipedia/commons /my/dir
(Typo: your.org not yours.org - bet you did that intentionally to throttle us ;-)
Great work, thanks for making this happen. :-) Are we talking to archive.org as well about setting up a copy there?
And I proofread that three times and took an s out of the name somewhere else in that email too, rats! :-P
No, I'm not talking to archive.org at the moment. I have too many irons in the fire at this point to open that discussion (including the half-started project of automated uploads of dumps to archive.org at 6 month intervals).
This is really phase 0.1, there's a lot more to be done to make this generally usable and to keep it regularly updated.
Ariel
Στις 02-04-2012, ημέρα Δευ, και ώρα 20:19 +0300, ο/η Ariel T. Glenn έγραψε:
No, I'm not talking to archive.org at the moment. I have too many irons in the fire at this point to open that discussion (including the half-started project of automated uploads of dumps to archive.org at 6 month intervals).
And already I'm a liar... someone just passed me a name at IA and a rumor, so I've left a message for my contacts there to see what's up.
A.
xmldatadumps-l@lists.wikimedia.org