As I said below, providing multiterabyte dumps does not seem reasonable to me. Monthly incrementals don't provide a workaround, unless you are suggesting that we put dumps online for every month since the beginning of the project. I think that a much more workable way to jump-start a mirror is to copy directly to disks in the datacenter, for an organization which will provide public access to its copy. This requires three things: 1) an organization that wants to host such a mirror, 2) them sending us disks, 3) me clearing it with Rob and with our datacenter tech, but he's agreed to this in principle in the past.
Ariel
Στις 17-11-2011, ημέρα Πεμ, και ώρα 14:11 +0100, ο/η emijrp έγραψε:
People can't mirror Commons if there is no public image dump. As there is no public image dump, people don't care about mirror. And so on...
You can offer monthly incremental image dumps.[1] Until mid-2008, month uploads are lower than 100 GB. Recently, it is on the 200-300GB rage. People is mirroring Domas visit logs at Internet Archive, ok, Commons monthly size in this case is about 10x, but it is not impossible. Arcnhive Team has mirrored GeoCities (0.9TB), Yahoo! Videos (20TB), Jamendo (2.5TB) and other huge sites. So, if you put that image dumps online, they are going to rage-download all.
You can start offering full resolution monthly dumps until 2007 or similar. But, man, we have to restart this soon or later.
[1] http://archiveteam.org/index.php?title=Wikimedia_Commons#Size_stats
2011/11/17 Ariel T. Glenn ariel@wikimedia.org I had a quick look and it turns out that the English language Wikipedia uses over 2.8 million images today. So, as you point out, an off line reader that just used thumbnails would still have to be selective about its image use.
In any case, putting together collections of thumbs doesn't resolve the need for a mirror of the originals, which I would really like to see happen. Ariel Στις 17-11-2011, ημέρα Πεμ, και ώρα 01:46 +0100, ο/η Erik Zachte έγραψε: > Ariel: > > Providing multiple terabyte sized files for download doesn't make any kind of sense to me. However, if we get concrete proposals for categories of Commons images people really want and would use, we can put those together. I think this has been said before on wikitech-l if not here. > > There is another way to cut down on download size, which would serve a whole class of content re-users, e.g. offline readers. > For offline readers it is not so important to have pictures of 20 Mb each, rather to have pictures at all, preferably 10's Kb's in size. > A download of all images, scaled down to say 600x600 max would be quite appropriate for many uses. > Map and diagrams would not survive this scale down (illegible text), but are very compact already. > In fact the compress ratio of each image is very reliable predictor of the type of content. > > In 2005 I distributed a DVD [1] with all unabridged texts for English Wikipedia and all 320,000 images on one DVD, to be loaded on 4Gb CF card for handheld. > Now we have 10 million images on Commons, so even scaled down images would need some filtering, but any collection would still be 100-1000 times smaller in size. > > Erik Zachte > > [1] http://www.infodisiac.com/Wikipedia/ > > > > _______________________________________________ > Xmldatadumps-l mailing list > Xmldatadumps-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l _______________________________________________ Xmldatadumps-l mailing list Xmldatadumps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l