Samuel Hoover, 20/03/19 21:24:

Does that mean Commons is currently culling its content? and that it makes most sense to wait for a post 2016 dump until after housecleaning is complete?

No, I just mean that it takes time to identify copyright violations and so on. Most deletions happen for content uploaded in the last few months, so I generally respected an "embargo" of at least six months (hte Internet Archive items are supposed to be durable).

Even with fiber connection/torrents, downloading will take time. Does any organization sell terabyte drives containing the Commons dump? Or can one travel to a physical location and connect several terabyte drives to quickly copy over?

Your best chance is probably to find some machines connected to the CENIC/Internet2 network or "nearby" and download the Internet Archive torrents from there. Hopefully you get 5-10 MiB/s per item and if you do all of them concurrently you should manage in a day or two. Internet Archive also routinely provides researcher access, but I'm not sure whether that's for private items only.

Wikimedia Foundation used to provide data feeds for some companies back in the day. If there is a significant need I suppose they could arrange for someone to have rsync access or something, but it's not going to happen overnight.

Federico