Hello, everyone.
While trying to add images to our Hebrew Wikipedia ZIM file, we ran into memory-consumption problems with the Kiwix mirroring scripts.
Despite using a quad-core Linux machine with 4GB of physical memory, both listAllImages.pl and mirrorMediawikiPages.pl ran for a very long time but eventually crashed the perl process with an out of memory error, and indeed, while running, they consumed ever more memory, until they actually consumed the entire physical memory.
Emmanuel said this shouldn't happen, but it does, and we'd like your advice on whether there's any point in attempting this again with a larger machine (perhaps an Amazon EC2 machine with more physical memory and faster CPUs) or whether we should try to find the bug in the scripts.
Thanks in advance,
Asaf Bartov
Wikimedia Israel