On 12/4/06, Alexandre NOUVEL alexandre.nouvel@alnoprods.net wrote:
Hi all,
---Selon Magnus Manske magnusmanske@googlemail.com:
Well, cross-checking one million commons images against a few hundred thousand on one of the larger wikipedias might kill the toolserver quite efficiently ;-)
Well, I agree that image processing is a very CPU-consuming task, and cross-checking adds to the difficulty.
However, I think that it may be possible to build a kind of hash signature for each file and sort them to find duplicates. The process itself of hashing would require some time but may be splitted amongst some servers. The resulting hash lists may then be sorted, so that matching signatures would lead to further checking of their initial images.
There was a discussion somewhere (maybe on this list? I don't remember) to store MD5-hashes of image data in the table with the other image information (size etc.). Nothing came of it, I'm afraid. Too bad.
One drawback for this solution is to maintain a huge index of all the signatures (each one associated with the image name and the originating wiki).
With images being replaced, deleted, undeleted, etc. the only practical place is indeed the image table on the respective wiki. An outside solution (i.e. toolserver) is out of the question IMHO.
Or perhaps I'm just writing bullshit :)
Nope :-)
Magnus