As well as exact duplicates, there may often also be different versions of the same painting with different lighting, or scans of slightly different reproductions of the same work. I don't know whether the algorithm is permissive enough to pick all of these up, but as many as can be picked up would be good to tag as "other versions" of the same underlying image. In general, we probably wouldn't *remove* duplicate images, but we would want to identify them as versions of each other.

Careful here - algorithms that spot almost-duplicates will happily flag different shots from the same shoot. Definitely not something to act upon without close human inspection.

They're very similar, though the smaller image is in fact sharper, a little darker, and slightly differently framed.

Hi Federico, and others, I'm still running the comparison, but I made a first list of ~500 duplicate works available here: http://belar.coyote.org/~jonas/wmcdups.html It would be very useful to get some feedback on that. Looking through some of those will give an idea of the kind of "duplicates" we find. Sincerely, Jonas _______________________________________________ Commons-l mailing list Commons-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/commons-l

That's something probably best done by WMF staff themselves, I think a simple md5 comparison would give quite a few matches. Doing on the WMF side would alleviate the need to transfer large amounts of data.

On 4 December 2014 at 18:39, Jonas Öberg <jonas(a)commonsmachinery.se> wrote: Volunteers can do this using simple database queries, which is a lot more efficient than pulling data out of the API. For example while writing this email I knocked out a query to show all non-trivial images (>2 pixels wide) on Commons with at least *3* files having the same SHA1 checksum and showing each image just once. The matching files are listed at the bottom of each image page on Commons. Interestingly, this shows that most of the 226 files have been from an upload of Gospel illustrations. The low number seems reassuring considering the size of Commons. The files are reported in descending order by image resolution. Report: http://commons.wikimedia.org/w/index.php?title=User:F%C3%A6/sandbox&old… On its own this is an interesting list to use as a backlog for fixes. Listing identical duplicates with 2 or more files matching would be simpler but longer; at the moment I count 3,279 files like this on Commons which took over 9 minutes to run. :-) Fae -- faewik(a)gmail.com https://commons.wikimedia.org/wiki/User:Fae _______________________________________________ Commons-l mailing list Commons-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/commons-l

select A.img_name, A.img_sha1, B.img_name from image as A join image as B on A.img_sha1 = B.img_sha1 and A.img_name < B.img_name;

On Thu, Dec 4, 2014 at 1:49 PM, Daniel Kinzler wrote: This is happening automatically: the SHA1 hash of every file is computed on upload, and placed in the img_sha1 field on the database. I believe this is used to warn users who try to upload an exact duplicate, but I'm not sure this is true. Indeed; so chances are exact duplicates have been uploaded intentionally. (It's not certain because they might have been uploaded before this warning existed, or uploaded by a bot configured to ignore warnings.)