Re: [Offline-l] Kiwix/openZIM is moving to Zstandard compression

Hi The ZIM format relies on third party compression algorithms to provide the best user experience. This means, high compression level, quick decompression, high portability, etc. Overtime the ZIM has used gzip, bzip2 and since almost 10 years the lzma2 version of the xz library. Time to time new modern compression algorithms are created and we use to evaluate them to see if this would bring a significant improvement for our users. Usually it would not be worth the effort of switching, but this has been different in the case of Zstandard (zstd). The key advantage is that zstd offers a higher decompression speed for a similar (to lzma) compression ratio. That means articles are displayed even quicker than before and the energy consumption is a lot less. You can read the whole discussion at https://github.com/openzim/libzim/issues/84. The decision of switching from lzma to xz has been made earlier this year and actually most of our Kiwix ports have been able to read zstd based ZIM files since more than a quarter. We are approaching now the end of the transition stage for Kiwix readers and therefore we will soon (during October) start to publish ZIM files using the new compression zstd. For 99% of the users this move will be totally transparent and they should only notice the reading/searching speed-up. For the 1% left a few things have to be considered: * If you can not read a new ZIM file, please update to the latest version of your (Kiwix) reader (this is always a good advice). * For all non Kiwix readers, but still libzim based, these readers should be re-compiled/re-released with the latest libzim. * For users relying on ZIM files but not on the libzim/Kiwix software stack, the software needs to be fixed. We don't plan so far to discontinued the support of lzma compression algorithm even if we discouraged to use it as zstd is simply always better. Older ZIM files are still perfectly readable, and this won't change in the next couple of years. Regards Emmanuel -- Kiwix - Wikipedia Offline & more * Web: https://kiwix.org/ * Twitter: https://twitter.com/KiwixOffline * Wiki: https://wiki.kiwix.org/ _______________________________________________ Offline-l mailing list Offline-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/offline-l