Hi
The ZIM format relies on third party compression algorithms to provide
the best user experience. This means, high compression level, quick
decompression, high portability, etc. Overtime the ZIM has used gzip,
bzip2 and since almost 10 years the lzma2 version of the xz library.
Time to time new modern compression algorithms are created and we use to
evaluate them to see if this would bring a significant improvement for
our users. Usually it would not be worth the effort of switching, but
this has been different in the case of Zstandard (zstd).
The key advantage is that zstd offers a higher decompression speed for a
similar (to lzma) compression ratio. That means articles are displayed
even quicker than before and the energy consumption is a lot less. You
can read the whole discussion at
https://github.com/openzim/libzim/issues/84.
The decision of switching from lzma to xz has been made earlier this
year and actually most of our Kiwix ports have been able to read zstd
based ZIM files since more than a quarter. We are approaching now the
end of the transition stage for Kiwix readers and therefore we will soon
(during October) start to publish ZIM files using the new compression zstd.
For 99% of the users this move will be totally transparent and they
should only notice the reading/searching speed-up. For the 1% left a few
things have to be considered:
* If you can not read a new ZIM file, please update to the latest
version of your (Kiwix) reader (this is always a good advice).
* For all non Kiwix readers, but still libzim based, these readers
should be re-compiled/re-released with the latest libzim.
* For users relying on ZIM files but not on the libzim/Kiwix software
stack, the software needs to be fixed.
We don't plan so far to discontinued the support of lzma compression
algorithm even if we discouraged to use it as zstd is simply always
better. Older ZIM files are still perfectly readable, and this won't
change in the next couple of years.
Regards
Emmanuel
--
Kiwix - Wikipedia Offline & more
* Web:
https://kiwix.org/
* Twitter:
https://twitter.com/KiwixOffline
* Wiki:
https://wiki.kiwix.org/