We got an automated notice from one of the big search engines that both http://ftpmirror.your.org and http://dumps.wikimedia.your.org were hosting some unspecified malware. I've verified nothing on the mirror box itself is compromised from the best I can tell, which leaves them being unhappy with something that we're mirroring.
I've started ClamAV scanning the whole public volume, but that's going to take quite a while (+20 million files, 80TB of data). The only thing it's complained about so far is:
http://ftpmirror.your.org/pub/wikimedia/images/wiktionary/fj/c/c4/citibank-c...
which was making the scanner crash. I don't see anything wrong with the file itself though.
Is it possible someone could have uploaded something at one point that was malicious and it's still floating around in the archives that got pushed to us?
-- Kevin
As far as I know, the chances are rather slim, because the MediaWiki software has a malware checker (I think).
Perhaps we shall see what outputs from the ClamAV checking, before we can know what is happening.
On Mon, Jul 2, 2012 at 10:13 AM, Kevin Day kevin@your.org wrote:
We got an automated notice from one of the big search engines that both http://ftpmirror.your.org and http://dumps.wikimedia.your.org were hosting some unspecified malware. I've verified nothing on the mirror box itself is compromised from the best I can tell, which leaves them being unhappy with something that we're mirroring.
I've started ClamAV scanning the whole public volume, but that's going to take quite a while (+20 million files, 80TB of data). The only thing it's complained about so far is:
http://ftpmirror.your.org/pub/wikimedia/images/wiktionary/fj/c/c4/citibank-c...
which was making the scanner crash. I don't see anything wrong with the file itself though.
Is it possible someone could have uploaded something at one point that was malicious and it's still floating around in the archives that got pushed to us?
-- Kevin
Xmldatadumps-l mailing list Xmldatadumps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l
On Jul 1, 2012, at 10:13 PM, Hydriz Wikipedia wrote:
As far as I know, the chances are rather slim, because the MediaWiki software has a malware checker (I think).
Perhaps we shall see what outputs from the ClamAV checking, before we can know what is happening.
I've been having a lot of problems with ClamAV crashing, so I've temporarily switched to F-Prot which *did* find something wrong with the earlier mentioned file, as well as two others:
[Found trojan] <JS/Redir.HY (exact, not disinfectable)> /z/public/pub/wikimedia/images/wiktionary/fj/c/c4/citibank-car-loan.pdf [Found exploit] <CVE-2009-0658 (not disinfectable)> /z/public/pub/wikimedia/images/wikisource/ar/7/7d/الحراب_في_صدر_البهاء_والباب.pdf [Found exploit] <CVE-2009-0658 (not disinfectable)> /z/public/pub/wikimedia/images/wikisource/ar/b/be/السنة_لابن_حنبل.pdf
At the rate it's going, it's going to take several days to finish, even with several running in parallel. I'll let it finish, but it's looking like at minimum there are some old PDFs that have some exploit code in them.
-- Kevin
Kevin Day, 02/07/2012 05:27:
[Found trojan] <JS/Redir.HY (exact, not disinfectable)> /z/public/pub/wikimedia/images/wiktionary/fj/c/c4/citibank-car-loan.pdf [Found exploit] <CVE-2009-0658 (not disinfectable)> /z/public/pub/wikimedia/images/wikisource/ar/7/7d/الحراب_في_صدر_البهاء_والباب.pdf [Found exploit] <CVE-2009-0658 (not disinfectable)> /z/public/pub/wikimedia/images/wikisource/ar/b/be/السنة_لابن_حنبل.pdf
At the rate it's going, it's going to take several days to finish, even with several running in parallel. I'll let it finish, but it's looking like at minimum there are some old PDFs that have some exploit code in them.
I've asked to delete those, and for the sake of tracking also filed https://bugzilla.wikimedia.org/show_bug.cgi?id=38113
Nemo
My final list of possibly naughty things uploaded. I know some of these are pretty harmless (html being appended to jpegs), and most are just encrypted RARs appended to images or encrypted PDF files. I don't know if there's a policy on barring encrypted files but I can't really think of a good reason to have them anywhere in commons.
/z/public/pub/wikimedia/images/wikipedia/commons/0/0a/Joseon-Kang_Huian-Gosagwansudo.jpg: HTML.Spy.IMG-1 FOUND /z/public/pub/wikimedia/images/wikipedia/commons/7/7c/Silvana_Suárez_7.jpg: HTML.Spy.IMG FOUND /z/public/pub/wikimedia/images/wikipedia/commons/c/c0/The_Qing_Dynasty_Cixi_Imperial_Dowager_Empress_of_China_On_Throne_5.JPG: HTML.Spy.IMG-1 FOUND [Found exploit] <CVE-2009-0658 (not disinfectable)> /z/public/pub/wikimedia/images/wikisource/ar/7/7d/الحراب_في_صدر_البهاء_والباب.pdf [Found exploit] <CVE-2009-0658 (not disinfectable)> /z/public/pub/wikimedia/images/wikisource/ar/b/be/السنة_لابن_حنبل.pdf [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/0d/PAY_SLIP_078322_Aug_2011_Tony.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/0d/IPhone31-511-1.part07.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/0d/IPod31-511.part01.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/03/IPod31-511.part13.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/03/IPhone31-511-1.part25.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/07/IPod41-511-1.part20.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/02/IPod41-511-1.part12.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/02/IPod41-511.part11.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/08/IPhone31-511-1.part26.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/05/IPod41-511-1.part08.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/05/IPod41-511-1.part04.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/0e/11013739714-ASKxxxxx0M-G4_ITR-V.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/0a/IPod41-511.part16.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/0a/ICICI_MAY2011.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/06/IPhone31-511-1.part04.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/7/7a/IPhone31-511.part12.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/7/76/IPhone31-511.part07.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/7/7d/IPhone31-511-1.part16.jpg->(appended) [Found exploit] <IFrame.gen (exact, not disinfectable)> /z/public/pub/wikimedia/images/wikipedia/commons/7/7c/Silvana_Suárez_7.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/7/77/IPod41-511-1.part22.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/7/77/IPod41-511.part02.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/c7/IPhone31-511.part11.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/c7/HDFC_BANK-_310711_(1).pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/c7/IPhone31-511-1.part32.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/c9/IPod41-511.part13.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/cc/Ch1A.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/cc/IPhone31-511-1.part30.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/c4/IPod41-511-1.part15.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/c4/Test1.rar.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/c6/IPod41-511-1.part19.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/ca/IPod41-511-1.part28.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/ce/IPod41-511-1.part14.jpg->(appended) [Found exploit] <IFrame.gen (exact, not disinfectable)> /z/public/pub/wikimedia/images/wikipedia/commons/c/ce/Silvana_Suárez_6.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/cb/احراز_هويت_مشتریان_در_خدمات_بانک_ملت.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/cb/IPod41-511-1.part02.jpg->(appended) [Found exploit] <IFrame.gen (exact, not disinfectable)> /z/public/pub/wikimedia/images/wikipedia/commons/c/c2/Votantes-1924.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/c8/IPhone31-511-1.part17.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/3/3c/IPhone31-511.part16.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/3/39/IPod41-511-1.part01.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/3/30/IPod41-511-1.part06.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/3/32/IPod41-511.part06.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/3/3b/IPhone31-511-1.part20.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/3/3e/IPhone31-511-1.part28.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/3/3a/IPod41-511-1.part21.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/3/3a/IPod41-511.part12.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/3/3f/IPhone31-511-1.part19.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/9/9e/IPhone31-511-1.part33.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/9/95/IPhone31-511-1.part23.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/9/9b/VADOFONE_DEC.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/9/9a/Farsinameh-Final_Draft.pdf->OBJ002 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/9/99/IPhone31-511-1.part09.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/9/99/IPhone31-511.part14.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/9/94/IPod41-511.part10.jpg->(appended) [Found exploit] <CVE-2004-0200 (not disinfectable)> /z/public/pub/wikimedia/images/wikipedia/commons/9/9d/Exploit-MS04-028.proof.jpg [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/9/90/IPod41-511-1.part07.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/9/97/IPod41-511-1.part24.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/4/44/IPod41-511.part14.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/4/4c/IPhone31-511.part06.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/4/49/ICICI_JUN2011.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/4/47/IPhone31-511-1.part10.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/4/47/IPod41-511-1.part23.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/4/48/IPhone31-511-1.part02.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/4/4f/IPhone31-511-1.part08.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/4/46/IPhone31-511-1.part29.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/d/d8/IPod41-511-1.part18.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/d/d4/IPod41-511-1.part16.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/d/dd/IPod41-511-1.part31.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/d/dc/IPod41-511-1.part10.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/d/d3/Dev26.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/d/d7/IPhone31-511.part04.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/d/d7/Citibank_Account_Statement-20110501_TO_20110705.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/f/f6/IPhone31-511-1.part03.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/f/fb/PAY_SLIP_078470_Aug_2011.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/f/fb/IPod31-511.part08.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/f/f2/IPod41-511-1.part05.jpg->(appended) [Found exploit] <HTML/IFrame (exact, not disinfectable)> /z/public/pub/wikimedia/images/wikipedia/commons/f/f8/Old_Jinan_Station_04.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/f/f5/IPhone31-511.part10.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/f/f5/Ch3Q.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/f/fe/IPhone31-511-1.part06.jpg->(appended) [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/f/fd/ICICI_JUL2011.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/f/f9/Farsinameh-abridged_English_version.pdf->OBJ002 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/6/63/IPhone31-511-1.part27.jpg->(appended)
On 11/07/12 23:50, Kevin Day wrote:
My final list of possibly naughty things uploaded. I know some of these are pretty harmless (html being appended to jpegs), and most are just encrypted RARs appended to images or encrypted PDF files. I don't know if there's a policy on barring encrypted files but I can't really think of a good reason to have them anywhere in commons.
[Found exploit] <CVE-2009-0658 (not disinfectable)> /z/public/pub/wikimedia/images/wikisource/ar/7/7d/الحراب_في_صدر_البهاء_والباب.pdf [Found exploit] <CVE-2009-0658 (not disinfectable)> /z/public/pub/wikimedia/images/wikisource/ar/b/be/السنة_لابن_حنبل.pdf
Already checked.
IPhone31-* and IPod41-* files, plus Ifaithipsw.jpg and Snowbreeze295.jpg were all uploaded by IcisTececoy user. (all but one were already deleted). I have just banned him.
[Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/c4/Test1.rar.jpg->(appended)
Uploaded by Danielito132, which seems a puppet of IcisTececoy. Also note by this user Test2.part01.rar.jpg, Test2.part02.rar.jpg, Thus_contumely.jpg, IThus_contumely.jpg all of them with embedded rar files. Deleted and blocked.
[Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/0d/PAY_SLIP_078322_Aug_2011_Tony.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/f/fb/PAY_SLIP_078470_Aug_2011.pdf->OBJ001
Already deleted. Both by the same user.
[Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/0e/11013739714-ASKxxxxx0M-G4_ITR-V.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/0/0a/ICICI_MAY2011.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/c7/HDFC_BANK-_310711_(1).pdf->OBJ001
[Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/cc/Ch1A.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/f/f5/Ch3Q.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/d/d3/Dev26.pdf->OBJ001
Short-lived files uploaded by an admin "This upload is part of a speed and endurance test for an application and bot platform I've been developing."
[Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/c/cb/احراز_هويت_مشتریان_در_خدمات_بانک_ملت.pdf->OBJ001
Deleted
[Found exploit] <IFrame.gen (exact, not disinfectable)> /z/public/pub/wikimedia/images/wikipedia/commons/c/c2/Votantes-1924.jpg->(appended)
Already deleted. Looks like the hosting iframe.
[Found exploit] <IFrame.gen (exact, not disinfectable)> /z/public/pub/wikimedia/images/wikipedia/commons/c/ce/Silvana_Suárez_6.jpg->(appended) [Found exploit] <IFrame.gen (exact, not disinfectable)> /z/public/pub/wikimedia/images/wikipedia/commons/7/7c/Silvana_Suárez_7.jpg->(appended)
More instances of the web-hosting iframe. The AV is being a bit paranoid here.
[Found exploit] <HTML/IFrame (exact, not disinfectable)> /z/public/pub/wikimedia/images/wikipedia/commons/f/f8/Old_Jinan_Station_04.jpg->(appended)
A slightly different iframe here.
/z/public/pub/wikimedia/images/wikipedia/commons/0/0a/Joseon-Kang_Huian-Gosagwansudo.jpg: HTML.Spy.IMG-1 FOUND /z/public/pub/wikimedia/images/wikipedia/commons/c/c0/The_Qing_Dynasty_Cixi_Imperial_Dowager_Empress_of_China_On_Throne_5.JPG: HTML.Spy.IMG-1 FOUND
More web-hosting iframes.
[Found exploit] <CVE-2004-0200 (not disinfectable)> /z/public/pub/wikimedia/images/wikipedia/commons/9/9d/Exploit-MS04-028.proof.jpg
MS04-028 proof of code. Not sure why it was uploaded...
[Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/9/9b/VADOFONE_DEC.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/d/d7/Citibank_Account_Statement-20110501_TO_20110705.pdf->OBJ001
By the same user. Deleted.
[Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/4/49/ICICI_JUN2011.pdf->OBJ001 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/f/fd/ICICI_JUL2011.pdf->OBJ001
By the same user. Deleted.
[Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/9/9a/Farsinameh-Final_Draft.pdf->OBJ002 [Unscannable] <File is encrypted> /z/public/pub/wikimedia/images/wikipedia/commons/f/f9/Farsinameh-abridged_English_version.pdf->OBJ002
These don't seem to be encrypted. They are displayed fine.
On Jul 1, 2012, at 10:13 PM, Hydriz Wikipedia wrote:
As far as I know, the chances are rather slim, because the MediaWiki software has a malware checker (I think).
Perhaps we shall see what outputs from the ClamAV checking, before we can know what is happening.
MediaWiki supports running ClamAV on upload, but WMF isn't running one. I used to run multiple checks on uploads to Wikimedia Commons, until the server where it ran had a disk failure. AFAIK, there's no extra check being done at all.
On 02/07/12 05:27, Kevin Day wrote:
I've been having a lot of problems with ClamAV crashing, so I've temporarily switched to F-Prot which *did* find something wrong with the earlier mentioned file, as well as two others:
[Found trojan] <JS/Redir.HY (exact, not disinfectable)> /z/public/pub/wikimedia/images/wiktionary/fj/c/c4/citibank-car-loan.pdf [Found exploit] <CVE-2009-0658 (not disinfectable)> /z/public/pub/wikimedia/images/wikisource/ar/7/7d/الحراب_في_صدر_البهاء_والباب.pdf [Found exploit] <CVE-2009-0658 (not disinfectable)> /z/public/pub/wikimedia/images/wikisource/ar/b/be/السنة_لابن_حنبل.pdf
At the rate it's going, it's going to take several days to finish, even with several running in parallel. I'll let it finish, but it's looking like at minimum there are some old PDFs that have some exploit code in them.
-- Kevin
Interesting. I had a really simple code in that project for detecting CVE-2009-0658. Turns out it would have caught it: الحراب_في_صدر_البهاء_والباب.pdf PDF uses JBIG2. Beware of the 0-day! السنة_لابن_حنبل.pdf PDF uses JBIG2. Beware of the 0-day!
Although looking to http://ar.wikisource.org/wiki/%D9%85%D9%84%D9%81:%D8%A7%D9%84%D8%B3%D9%86%D8... and http://ar.wikisource.org/wiki/%D9%85%D9%84%D9%81:%D8%A7%D9%84%D8%B3%D9%86%D8..., they could be false positives.
These other files are also malware: http://ftpmirror.your.org/pub/wikimedia/images/wiktionary/fj/4/4a/quick-mone... http://ftpmirror.your.org/pub/wikimedia/images/wiktionary/fj/0/01/loan-perso... those three files were uploaded by the same user, and are the only files ever uploaded to fjwiktionary.
You may also find some executables. I remember that someone uploaded once as proof of concept a wine cmd.exe binary. There were also people uploading embedded files and other nasty bits, but they *should* be deleted. Clearly we failed here.
As for Nemo bug for detecting wrong pdfs, it's a daunting task. You would need a -quite complete- pdf parser (and it's not a simple format!). Even worse, CVE-2009-0658 was an Adobe vulnerability parsing JBIG2 images, so you would need to verify if the file is consistent or not (OTOH, detecting usage of JBIG2 is simpler).
Thanks, Kevin!
On Jul 2, 2012, at 4:14 PM, Platonides wrote:
On Jul 1, 2012, at 10:13 PM, Hydriz Wikipedia wrote:
As far as I know, the chances are rather slim, because the MediaWiki software has a malware checker (I think).
Perhaps we shall see what outputs from the ClamAV checking, before we can know what is happening.
MediaWiki supports running ClamAV on upload, but WMF isn't running one. I used to run multiple checks on uploads to Wikimedia Commons, until the server where it ran had a disk failure. AFAIK, there's no extra check being done at all.
Even temporarily forgetting about the complexity of scanning PDFs, there's a lot of weirdness in a lot of files that even ClamAV doesn't find. For example: (replacing < and > with [ and ] so this doesn't trigger anyone's mail spam filters)
strings images/wikipedia/commons/7/7c/Silvana_Suárez_7.jpg | tail -9 [!-- INICIO - PUBLICIDAD POP-UP UNDER --] [IFRAME SRC="http://www.ciudad.com.ar/ar/popunder/p_submit.asp?site=personales.ciudad.com..." width=1 height=1][/IFRAME] [SCRIPT LANGUAGE="JavaScript"] //[!-- for (var i=1; i<15; i++){ setTimeout('self.focus();',i*30); //--] [/SCRIPT] [!-- FIN - PUBLICIDAD POP-UP UNDER --]
There are dozens of jpeg files that are valid jpegs that have encrypted rar files appended to the end of the jpeg data. It might be a worthwhile idea to take any uploaded jpg/png/gif/etc and completely rewrite it before using it. Tools like jpegoptim / pngcrush / etc are pretty good at taking "wild" images and completely rewriting them to remove any oddities.
-- Kevin
On 03/07/12 18:47, Kevin Day wrote:
Even temporarily forgetting about the complexity of scanning PDFs, there's a lot of weirdness in a lot of files that even ClamAV doesn't find. For example: (replacing < and > with [ and ] so this doesn't trigger anyone's mail spam filters)
strings images/wikipedia/commons/7/7c/Silvana_Suárez_7.jpg | tail -9 [!-- INICIO - PUBLICIDAD POP-UP UNDER --] [IFRAME SRC="http://www.ciudad.com.ar/ar/popunder/p_submit.asp?site=personales.ciudad.com..." width=1 height=1][/IFRAME] [SCRIPT LANGUAGE="JavaScript"] //[!-- for (var i=1; i<15; i++){ setTimeout('self.focus();',i*30); //--] [/SCRIPT] [!-- FIN - PUBLICIDAD POP-UP UNDER --]
This looks like the image was stored in a free hosting web server configured to append that content to the served files... and not filtering out for the images. Then it got uploaded to commons.
There are dozens of jpeg files that are valid jpegs that have encrypted rar files appended to the end of the jpeg data. It might be a worthwhile idea to take any uploaded jpg/png/gif/etc and completely rewrite it before using it. Tools like jpegoptim / pngcrush / etc are pretty good at taking "wild" images and completely rewriting them to remove any oddities.
-- Kevin
Appended Rar files is one of the things my tool detected. If you send me a list of the images I can go trying to kill them.
Modifying the original images would be a bad idea. It'd be better to forbid uploading of such files (rars are hard to block, since you need to scan the full file...).
xmldatadumps-l@lists.wikimedia.org