Hi;<br><br>@Derrick: I don&#39;t trust Amazon. Really, I don&#39;t trust Wikimedia Foundation either. They can&#39;t and/or they don&#39;t want to provide image dumps (what is worst?). Community donates images to Commons, community donates money every year, and now community needs to develop a software to extract all the images and packed them, and of course, host them in a permanent way. Crazy, right?<br>

<br>@Milos: Instead of spliting image dump using the first letter of filenames, I thought about spliting using the upload date (YYYY-MM-DD). So, first chunks (2005-01-01) will be tiny, and recent ones of several GB (a single day).<br>

<br>Regards,<br>emijrp<br><br><div class="gmail_quote">2011/6/28 Derrick Coetzee <span dir="ltr">&lt;<a href="mailto:dcoetzee@eecs.berkeley.edu">dcoetzee@eecs.berkeley.edu</a>&gt;</span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">

As a Commons admin I&#39;ve thought a lot about the problem of<br>
distributing Commons dumps. As for distribution, I believe BitTorrent<br>
is absolutely the way to go, but the Torrent will require a small<br>
network of dedicated permaseeds (servers that seed indefinitely).<br>
These can easily be set up at low cost on Amazon EC2 &quot;small&quot; instances<br>
- the disk storage for the archives is free, since small instances<br>
include a  large (~120 GB) ephemeral storage volume at no additional<br>
cost, and the cost of bandwidth can be controlled by configuring the<br>
BitTorrent client with either a bandwidth throttle or a transfer cap<br>
(or both). In fact, I think all Wikimedia dumps should be available<br>
through such a distribution solution, just as all Linux installation<br>
media are today.<br>
<br>
Additionally, it will be necessary to construct (and maintain) useful<br>
subsets of Commons media, such as &quot;all media used on the English<br>
Wikipedia&quot;, or &quot;thumbnails of all images on Wikimedia Commons&quot;, of<br>
particular interest to certain content reusers, since the full set is<br>
far too large to be of interest to most reusers. It&#39;s on this latter<br>
point that I want your feedback: what useful subsets of Wikimedia<br>
Commons does the research community want? Thanks for your feedback.<br>
<br>
--=20<br>
Derrick Coetzee<br>
User:Dcoetzee, English Wikipedia and Wikimedia Commons administrator<br>
<a href="http://www.eecs.berkeley.edu/%7Edcoetzee/" target="_blank">http://www.eecs.berkeley.edu/~dcoetzee/</a><br><br></blockquote></div>