Hi,
I talked with the xz developer in IRC and he helped me to reduce memory usage.
The problem was, that compressing with lzma with 9 uncompressing needs about 65MB of RAM. The needed memory is in fact dependend of used compression level. Using level 3 reduced the memory usage to about 1,5MB, that it works now on the NanoNote. There is a additional flag LZMA_PRESET_EXTREME, which makes compressing slower but compression ratio better. Using 3 + extreme results in a file, which works on Nanonote while the size is almost identical with 9 or bzip2.
Uncompressing bzip2 chunks take about 2 seconds while lzma needs 0,5 seconds on the nanonote. So we keep the size while improving speed with the factor 4!!!
Have a good Year.
Tommi
What should I say?
Great! Very good work done, Tommi!
So does this mean we can dismiss all other compression algorithms?
What about the streaming mode when uncompressing? You talked about that as you said we could improve memory usage on small devices.
Greetings and a good year to all of us!
/Manuel
Am 31.12.2009 16:03, schrieb Tommi Mäkitalo:
Hi,
I talked with the xz developer in IRC and he helped me to reduce memory usage.
The problem was, that compressing with lzma with 9 uncompressing needs about 65MB of RAM. The needed memory is in fact dependend of used compression level. Using level 3 reduced the memory usage to about 1,5MB, that it works now on the NanoNote. There is a additional flag LZMA_PRESET_EXTREME, which makes compressing slower but compression ratio better. Using 3 + extreme results in a file, which works on Nanonote while the size is almost identical with 9 or bzip2.
Uncompressing bzip2 chunks take about 2 seconds while lzma needs 0,5 seconds on the nanonote. So we keep the size while improving speed with the factor 4!!!
Have a good Year.
Tommi _______________________________________________ dev-l mailing list dev-l@openzim.org https://intern.openzim.org/mailman/listinfo/dev-l
Am Donnerstag, 31. Dezember 2009 16:32:35 schrieb Manuel Schneider:
What should I say?
Great! Very good work done, Tommi!
So does this mean we can dismiss all other compression algorithms?
What about the streaming mode when uncompressing? You talked about that as you said we could improve memory usage on small devices.
Greetings and a good year to all of us!
/Manuel
I can implement the streaming mode, but I do we really need that? Do we need to support even smaller devices than Nanonote? Take into account that we still need 1,5MB for uncompressing and so the 1MB used for caching the uncompressed chunk is not that significant any more.
The only problem that remains is the support of xz, which is not that wide spread yet. You already saw, that xz is not yet decided stable on gentoo. As emmanuel said, it works with windows, so that should not be the problem. If building on windows is difficult, thats not really a big problem because we (or someone) can offer precompiled dlls for use (and it is no problem, because I don't use windows anyway ;-) ).
I moved the zim5 to trunk now.
Emmanuel: it would be great, if you can check it on windows soon. The zimwriter uses bzip2 by default but you can pass it a switch --lzma. It uses lzma with level 3 + extreme then.
If you create a zim file, you can check the used compression with "zimDump -liv yourZimFile.zim". It shows the compression algorithm for the clusters. 3 is bzip2 and 4 is lzma.
Tommi
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Tommi Mäkitalo a écrit :
Am Donnerstag, 31. Dezember 2009 16:32:35 schrieb Manuel Schneider:
What should I say?
Great! Very good work done, Tommi!
So does this mean we can dismiss all other compression algorithms?
What about the streaming mode when uncompressing? You talked about that as you said we could improve memory usage on small devices.
Greetings and a good year to all of us!
/Manuel
I can implement the streaming mode, but I do we really need that?
I'm not sure to fully understand the streaming challenge, but I try. We will have in a near future HTML5 videos... Do we have here a relationship between both (HTML5 videos and streaming mode)?
Do we need to support even smaller devices than Nanonote?
I do not thing so.
I moved the zim5 to trunk now.
Great, will try to build a first "new" ZIM.
Emmanuel: it would be great, if you can check it on windows soon.
OK. I will do it ASAP.
If you create a zim file, you can check the used compression with "zimDump -liv yourZimFile.zim". It shows the compression algorithm for the clusters. 3 is bzip2 and 4 is lzma.
Thank you for the advice. Is there anywhere a document about how to deal by the writing process with new features (Metadata, new lang header, mime-types, Categories) ?
Best regards Emmanuel