Hi.

Note that the bug exists in GNU/Linux as well -- it's just better hidden... :)

UTF8 uses a _variable_ amount of bytes to encode a code point. Often a single byte is enough. But if your filename includes very special characters, such as an "em-dash" (–) or an IPA charachter such as ʧ -- then the character would take up two bytes, and for some obscure characters it can be up to _four_ bytes.

So French accents fit in one byte, but some other characters do not. If I had a ZIM file with such a character on GNU/Linux, the code would fail too.

We do need a portable solution. I don't know the right way to do it off the top of my head, so perhaps someone else on the list can offer advice. If no one can, I'm willing to figure it out myself.

Asaf

On Wed, Nov 17, 2010 at 6:30 PM, <emmanuel@engelhart.org> wrote:

Hi,

I have discovered an new bug on Kiwix for Windows:
Kiwix is not able to open filenames containing accents, something containing "wikipédia" for example.

This is a Windows specific issue, if I do the same on GNU/Linux, it works.

I think the explanation is that ext4 uses UTF8 as charset and NTFS, UTF16... and a code point in UTF16 needs 2 bytes although in UTF8, it needs only one.
As the method to open a ZIM file takes a char* as argument (so base on 1 byte) it works with UTF8.

This is a pretty new problematic for me and I simply try to understand what need to be fixed.
So feel free to give here a feedback.

For example, this is not clear if we need absolutely an (additional) method accepting wide chars to open such files in the zimlib?
This is also not clear for me if this is possible to build a generic and portable solution here?

Regards
Emmanuel

_______________________________________________
dev-l mailing list
dev-l@openzim.org
https://intern.openzim.org/mailman/listinfo/dev-l

--
Asaf Bartov <asaf.bartov@gmail.com>