Hi,
merry chrismas to you all.
I would like to move development of the new file format (zim5) from the zim2- branch to trunk. I would tag the current trunk under svn+ssh://openzim.org/svnroot/tags/zim4 and then merge the zim2-branch to trunk. The trunk code won't be able to read the old zim files any more. You have to recreate your files.
Any objections?
The next step is to remove support for zlib and bzip2.
Currently I'm working on porting the new file format to the nano note. Unfortunately I'm stuck with porting xz (the lzma-library) to openwrt. There are some difficulties there, but I'm sure, they can be resolved. I have already asked for help at the openwrt developer list but have not yet received any answers.
Tommi
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Tommi Mäkitalo a écrit :
Hi,
merry chrismas to you all.
I would like to move development of the new file format (zim5) from the zim2- branch to trunk. I would tag the current trunk under svn+ssh://openzim.org/svnroot/tags/zim4 and then merge the zim2-branch to trunk. The trunk code won't be able to read the old zim files any more. You have to recreate your files.
Any objections?
OK for me.
The next step is to remove support for zlib and bzip2.
Currently I'm working on porting the new file format to the nano note. Unfortunately I'm stuck with porting xz (the lzma-library) to openwrt. There are some difficulties there, but I'm sure, they can be resolved. I have already asked for help at the openwrt developer list but have not yet received any answers.
That's why I asked to be careful by removing bzip2 and/or zlib. It's essential to have at least at any moment a working solution for every arch/system. I have for example not checked if xz works good with Windows.
Emmanuel
Am Samstag, 26. Dezember 2009 14:19:26 schrieb Emmanuel Engelhart:
Tommi Mäkitalo a écrit :
...
The next step is to remove support for zlib and bzip2.
Currently I'm working on porting the new file format to the nano note. Unfortunately I'm stuck with porting xz (the lzma-library) to openwrt. There are some difficulties there, but I'm sure, they can be resolved. I have already asked for help at the openwrt developer list but have not yet received any answers.
That's why I asked to be careful by removing bzip2 and/or zlib. It's essential to have at least at any moment a working solution for every arch/system. I have for example not checked if xz works good with Windows.
Emmanuel
We want to define a standard format. And if you create zim files with lzma compression we need to make sure, we can read the file on every system, we support. It makes no sense to have some zim files with bzip2 compression, which work on windows and other zim files with lzma compresion which don't. We have to commit us to one compression method to prevent fragmentation. Or do you want to offer zim files for ubuntu and zim files for windows and zim files for fedora and zim files for freebsd?
Tommi
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Tommi Mäkitalo a écrit :
Am Samstag, 26. Dezember 2009 14:19:26 schrieb Emmanuel Engelhart:
Tommi Mäkitalo a écrit :
...
The next step is to remove support for zlib and bzip2.
Currently I'm working on porting the new file format to the nano note. Unfortunately I'm stuck with porting xz (the lzma-library) to openwrt. There are some difficulties there, but I'm sure, they can be resolved. I have already asked for help at the openwrt developer list but have not yet received any answers.
That's why I asked to be careful by removing bzip2 and/or zlib. It's essential to have at least at any moment a working solution for every arch/system. I have for example not checked if xz works good with Windows.
Emmanuel
We want to define a standard format. And if you create zim files with lzma compression we need to make sure, we can read the file on every system, we support. It makes no sense to have some zim files with bzip2 compression, which work on windows and other zim files with lzma compresion which don't. We have to commit us to one compression method to prevent fragmentation. Or do you want to offer zim files for ubuntu and zim files for windows and zim files for fedora and zim files for freebsd?
I prefer having *now* a solution with 3 compression algorithms and knowing one of them does not always work than having a format which is totally unusable in a few use cases.
So, we will try to have lzma working everywhere... but as long as this is not sure, it would be preferable IMO to be able to say "look you can use bzip2" than "sorry zimlib is not for you, it does not work currently for your use case".
We have to choice our disadvantage: * Remove bzip2 and having the risk to discover portability issues * Keeping the bzip2/gzip dependence a little bit longer
... in any case this is not so critical... in the worth case we will have to work hard to fix the portability issues... and I hope we will because we want the zimlib being portable.
Emmanuel
Am Samstag, 26. Dezember 2009 18:49:20 schrieb Emmanuel Engelhart:
Tommi Mäkitalo a écrit :
Am Samstag, 26. Dezember 2009 14:19:26 schrieb Emmanuel Engelhart:
Tommi Mäkitalo a écrit :
...
The next step is to remove support for zlib and bzip2.
Currently I'm working on porting the new file format to the nano note. Unfortunately I'm stuck with porting xz (the lzma-library) to openwrt. There are some difficulties there, but I'm sure, they can be resolved. I have already asked for help at the openwrt developer list but have not yet received any answers.
That's why I asked to be careful by removing bzip2 and/or zlib. It's essential to have at least at any moment a working solution for every arch/system. I have for example not checked if xz works good with Windows.
Emmanuel
We want to define a standard format. And if you create zim files with lzma compression we need to make sure, we can read the file on every system, we support. It makes no sense to have some zim files with bzip2 compression, which work on windows and other zim files with lzma compresion which don't. We have to commit us to one compression method to prevent fragmentation. Or do you want to offer zim files for ubuntu and zim files for windows and zim files for fedora and zim files for freebsd?
I prefer having *now* a solution with 3 compression algorithms and knowing one of them does not always work than having a format which is totally unusable in a few use cases.
So, we will try to have lzma working everywhere... but as long as this is not sure, it would be preferable IMO to be able to say "look you can use bzip2" than "sorry zimlib is not for you, it does not work currently for your use case".
We have to choice our disadvantage:
- Remove bzip2 and having the risk to discover portability issues
- Keeping the bzip2/gzip dependence a little bit longer
... in any case this is not so critical... in the worth case we will have to work hard to fix the portability issues... and I hope we will because we want the zimlib being portable.
Emmanuel
How do you want to create your zim files? With lzma? With bzip2? With zlib? If you sometimes use one and sometimes the other your zim files will work sometimes on one platform sometimes on the other. Either we use lzma or we don't. Otherwise we create lzma compresed zim files, which do not work everywhere.
If you offer lzma compressed zim files for download, not dropping bzip2 support do not solve any portability issue. We just get a zimlib which does not work with your zim files. And that's really bad.
We should work on the portability of xz. As I read xz works on windows. It can be compiled using mingw. So this should not really be an issue.
Tommi
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Tommi Mäkitalo a écrit :
Am Samstag, 26. Dezember 2009 18:49:20 schrieb Emmanuel Engelhart:
Tommi Mäkitalo a écrit :
Am Samstag, 26. Dezember 2009 14:19:26 schrieb Emmanuel Engelhart:
Tommi Mäkitalo a écrit :
...
The next step is to remove support for zlib and bzip2.
Currently I'm working on porting the new file format to the nano note. Unfortunately I'm stuck with porting xz (the lzma-library) to openwrt. There are some difficulties there, but I'm sure, they can be resolved. I have already asked for help at the openwrt developer list but have not yet received any answers.
That's why I asked to be careful by removing bzip2 and/or zlib. It's essential to have at least at any moment a working solution for every arch/system. I have for example not checked if xz works good with Windows.
Emmanuel
We want to define a standard format. And if you create zim files with lzma compression we need to make sure, we can read the file on every system, we support. It makes no sense to have some zim files with bzip2 compression, which work on windows and other zim files with lzma compresion which don't. We have to commit us to one compression method to prevent fragmentation. Or do you want to offer zim files for ubuntu and zim files for windows and zim files for fedora and zim files for freebsd?
I prefer having *now* a solution with 3 compression algorithms and knowing one of them does not always work than having a format which is totally unusable in a few use cases.
So, we will try to have lzma working everywhere... but as long as this is not sure, it would be preferable IMO to be able to say "look you can use bzip2" than "sorry zimlib is not for you, it does not work currently for your use case".
We have to choice our disadvantage:
- Remove bzip2 and having the risk to discover portability issues
- Keeping the bzip2/gzip dependence a little bit longer
... in any case this is not so critical... in the worth case we will have to work hard to fix the portability issues... and I hope we will because we want the zimlib being portable.
Emmanuel
How do you want to create your zim files? With lzma? With bzip2? With zlib? If you sometimes use one and sometimes the other your zim files will work sometimes on one platform sometimes on the other. Either we use lzma or we don't. Otherwise we create lzma compresed zim files, which do not work everywhere.
If you offer lzma compressed zim files for download, not dropping bzip2 support do not solve any portability issue. We just get a zimlib which does not work with your zim files. And that's really bad.
We should work on the portability of xz. As I read xz works on windows. It can be compiled using mingw. So this should not really be an issue.
For Kiwix it has to compile with MS cl.exe compiler... because the Mozilla frameworks only compiles with this compiler (on Windows) and there is no possible binary compatibility.
I will work on that and give feedback.
Emmanuel