Hey,
I ported OpenZIM and it's dependencies to OpenWrt (not committed yet) - mainly, to get it running on the embedded device "Ben NanoNote" of qi-hardware. Unfortunately OpenZIM seems to consume an amount of memory the Ben isn't able to serve (it has 32MB of RAM). Even with a really minimalistic system (lightweight kernelconfig + busybox-based rootfs, having just a shell) and accessing the zimreader via a webbrowser running on another machine, OpenZIM is getting killed by the kernels out-of-memory killer after a few clicks.
Is there a way of configuring / adjusting the zimreader to reduce memory consumption to get it running smoothly and stable on device such the ben NanoNote?
Thanks a lot in advance - great work!
Have a nice weekend,
mirko
Hey Mirko,
Am Freitag, 25. September 2009 01:22:41 schrieb Mirko Vogt:
Unfortunately OpenZIM seems to consume an amount of memory the Ben isn't able to serve (it has 32MB of RAM). Even with a really minimalistic system (lightweight kernelconfig + busybox-based rootfs, having just a shell) and accessing the zimreader via a webbrowser running on another machine, OpenZIM is getting killed by the kernels out-of-memory killer after a few clicks.
I had a discussion with Mirko Lindner about this issue a few days ago.
My guess was that the caches inside zimreader are to aggressive.
Zimreader tries to optimize access time by caching the extracted data and by pre-fetching articles which are linked in the currently viewed article.
Another issue may be that the cluster size (number of articles which are bundled together for better compression) might be too high. For this I filed a bug: http://bugs.openzim.org/show_bug.cgi?id=9.
Is there a way of configuring / adjusting the zimreader to reduce memory consumption to get it running smoothly and stable on device such the ben NanoNote?
Concerning the caches I suggest that the zimreader gets a configure option "--low-memory" which disables pre-fetching and limits the number of cached articles. I asked Mirko to file a bug for this.
Regards,
Manuel
Hey,
Concerning the caches I suggest that the zimreader gets a configure option "--low-memory" which disables pre-fetching and limits the number of cached articles. I asked Mirko to file a bug for this.
So done here[1].
Regards,
/mirko
On Freitag, 25. September 2009 09:38:06 Mirko Lindner wrote:
Hey,
Concerning the caches I suggest that the zimreader gets a configure option "--low-memory" which disables pre-fetching and limits the number of cached articles. I asked Mirko to file a bug for this.
So done here[1].
Regards,
/mirko
Hi,
thank you for your report.
I know, that zimlib uses quite some memory. It has a decompress cache for articles and a cache for directory entries. It caches up to 16 chunks, which are normally 1M and 512 directory entries (some few bytes each). To reduce memory consumption the directory cache is not so important but the decompress cache is. Currently this 16 is a fixed size (defined in src/fileimpl.cpp).
It is a quite bad solution to hard code that value but to add a "--low- memory"-flag and compile a smaller value into the lib is not really a very good option. It would be better to either let the user of the library decide or to use e.g. a environment variable to tune it. I'm not sure, what is the best solution.
What do you think? The library may read the value from the env variable ZIM_CLUSTERCACHE and ZIM_DIRENTCACHE and default to the values above. Then the size is configurable without recompilation.
Tommi
Hi Tommi,
I know, that zimlib uses quite some memory. It has a decompress cache for
articles and a cache for directory entries. It caches up to 16 chunks, which are normally 1M and 512 directory entries (some few bytes each). To reduce memory consumption the directory cache is not so important but the decompress cache is. Currently this 16 is a fixed size (defined in src/fileimpl.cpp).
It is a quite bad solution to hard code that value but to add a "--low- memory"-flag and compile a smaller value into the lib is not really a very good option. It would be better to either let the user of the library decide or to use e.g. a environment variable to tune it. I'm not sure, what is the best solution.
What do you think? The library may read the value from the env variable ZIM_CLUSTERCACHE and ZIM_DIRENTCACHE and default to the values above. Then the size is configurable without recompilation.
As I am not really a coder I won't go into details about how to implement a low memory version :) I'll leave that to the other Mirko.
I was more wondering if you had a device with limited resources that you use for testing. If not, would it be helpful if I'd send you a NanoNote with a pre-flashed image that you could use for testing? (Getting an environment set up that allows you to rebuild the openZIM programs should be an easy fix)
Or are you uber busy with your day job at the moment?
Regards, /mirko