Hello, everyone. While trying to add images to our Hebrew Wikipedia ZIM file, we ran into memory-consumption problems with the Kiwix mirroring scripts.
Despite using a quad-core Linux machine with 4GB of physical memory, both * listAllImages.pl* and *mirrorMediawikiPages.pl* ran for a very long time but eventually crashed the perl process with an *out of memory* error, and indeed, while running, they consumed ever more memory, until they actually consumed the entire physical memory.
Emmanuel said this shouldn't happen, but it does, and we'd like your advice on whether there's any point in attempting this again with a larger machine (perhaps an Amazon EC2 machine with more physical memory and faster CPUs) or whether we should try to find the bug in the scripts.
Thanks in advance,
Asaf Bartov Wikimedia Israel
-- Asaf Bartov asaf@forum2.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Asaf Bartov a écrit :
While trying to add images to our Hebrew Wikipedia ZIM file, we ran into memory-consumption problems with the Kiwix mirroring scripts.
Despite using a quad-core Linux machine with 4GB of physical memory, both * listAllImages.pl* and *mirrorMediawikiPages.pl* ran for a very long time but eventually crashed the perl process with an *out of memory* error, and indeed, while running, they consumed ever more memory, until they actually consumed the entire physical memory.
Emmanuel said this shouldn't happen, but it does, and we'd like your advice on whether there's any point in attempting this again with a larger machine (perhaps an Amazon EC2 machine with more physical memory and faster CPUs) or whether we should try to find the bug in the scripts.
Please open a bug at http://bugs.kiwix.org I will try to reproduce it.
Regards Emmanuel