Awesome work Christian. Where can we find your code to test out?
If you don't have a place then I'll carve one out in our depots. On Sep 24, 2011 5:57 AM, "Christian Pühringer" christian@puehringer.net wrote:
Hi Emanuel,
Makes sense that images are not compressed in zim file again, thanks for
the
clarification. In particular for windows mobile this increases the probability that with reducing the cluster size a sufficient performance is achieved (i.p. if there was a better way to
display
images than in android, but I haven't looked into this). For android I still expect that it it will be better to use a native
library.
Christian Am 24.09.2011 14:30, schrieb Emmanuel Engelhart:
On 24/09/2011 14:24, Christian Pühringer wrote:
The JAVA liblzma performance is pretty bad: To increase efficiency of compression in the zim-format articles (and also all other data like images) are stored in clusters. Cluster size is
apparently about
1 MB. This implies that loading an article which is stored at the end of a cluster involves decompressing the
complete
cluster.
Images should not be compressed in ZIM files for the obvious reasons they mainly are already compressed. This is the case for all ZIM files I made. As far as I know this is also the case for Mediawiki:Collection build ZIM files.
Emmanuel
dev-l mailing list dev-l@openzim.org https://intern.openzim.org/mailman/listinfo/dev-l
Hi Tomasz,
I've pushed the code to https://github.com/cip/zimgap-android. Note that the only functionality is currently to load an article from the zim file and display it. In particular there is no search functionality, so you have to know the exact article name to open it. The reason is simply that I am currently focusing on the feasibility, and I expect that the main issues are related to loading an article and and not related to searching the index.
Best regards, Christian
Am 24.09.2011 23:14, schrieb Tomasz Finc:
Awesome work Christian. Where can we find your code to test out?
If you don't have a place then I'll carve one out in our depots.
On Sep 24, 2011 5:57 AM, "Christian Pühringer" <christian@puehringer.net mailto:christian@puehringer.net> wrote:
Hi Emanuel,
Makes sense that images are not compressed in zim file again, thanks for the clarification. In particular for windows mobile this increases the probability that with reducing the cluster size a sufficient performance is achieved (i.p. if there was a better way to display images than in android, but I haven't looked into this). For android I still expect that it it will be better to use a native library.
Christian Am 24.09.2011 14:30, schrieb Emmanuel Engelhart:
On 24/09/2011 14:24, Christian Pühringer wrote:
The JAVA liblzma performance is pretty bad: To increase efficiency of compression in the zim-format articles (and also all other data like images) are stored in clusters. Cluster size is apparently
about
1 MB. This implies that loading an article which is stored at the end of a cluster involves decompressing the complete cluster.
Images should not be compressed in ZIM files for the obvious reasons they mainly are already compressed. This is the case for all ZIM files I made. As far as I know this is also the case for Mediawiki:Collection build ZIM files.
Emmanuel
dev-l mailing list dev-l@openzim.org mailto:dev-l@openzim.org https://intern.openzim.org/mailman/listinfo/dev-l
dev-l mailing list dev-l@openzim.org https://intern.openzim.org/mailman/listinfo/dev-l
Awesome .. this will be fun to play with and a great addition to the Wikipedia app community.
--tomasz
On Tue, Sep 27, 2011 at 1:51 PM, Christian Pühringer cip@gmx.at wrote:
Hi Tomasz,
I've pushed the code to https://github.com/cip/zimgap-android. Note that the only functionality is currently to load an article from the zim file and display it. In particular there is no search functionality, so you have to know the exact article name to open it. The reason is simply that I am currently focusing on the feasibility, and I expect that the main issues are related to loading an article and and not related to searching the index.
Best regards, Christian
Am 24.09.2011 23:14, schrieb Tomasz Finc:
Awesome work Christian. Where can we find your code to test out?
If you don't have a place then I'll carve one out in our depots.
On Sep 24, 2011 5:57 AM, "Christian Pühringer" christian@puehringer.net wrote:
Hi Emanuel,
Makes sense that images are not compressed in zim file again, thanks for the clarification. In particular for windows mobile this increases the probability that with reducing the cluster size a sufficient performance is achieved (i.p. if there was a better way to display images than in android, but I haven't looked into this). For android I still expect that it it will be better to use a native library.
Christian Am 24.09.2011 14:30, schrieb Emmanuel Engelhart:
On 24/09/2011 14:24, Christian Pühringer wrote:
The JAVA liblzma performance is pretty bad: To increase efficiency of compression in the zim-format articles (and also all other data like images) are stored in clusters. Cluster size is apparently about 1 MB. This implies that loading an article which is stored at the end of a cluster involves decompressing the complete cluster.
Images should not be compressed in ZIM files for the obvious reasons they mainly are already compressed. This is the case for all ZIM files I made. As far as I know this is also the case for Mediawiki:Collection build ZIM files.
Emmanuel
dev-l mailing list dev-l@openzim.org https://intern.openzim.org/mailman/listinfo/dev-l
dev-l mailing list dev-l@openzim.org https://intern.openzim.org/mailman/listinfo/dev-l
On Tue, Sep 27, 2011 at 1:51 PM, Christian Pühringer cip@gmx.at wrote:
I've pushed the code to https://github.com/cip/zimgap-android. Note that the only functionality is currently to load an article from the zim file and display it. In particular there is no search functionality, so you have to know the exact article name to open it. The reason is simply that I am currently focusing on the feasibility, and I expect that the main issues are related to loading an article and and not related to searching the index.
Just to warn: run 'git submodule init && git submodule update' after cloning the repository, or Eclipse will whinge about being unable to find the xz-java sources. :)
I did a quick test on my Nexus One (running Android 2.3.6, so with JIT). There are some faster processors out there on higher-end phones, but there's also a lot of cheaper phones that aren't going to be any faster than this.
The wikipedia-de.zim test file hasn't downloaded here yet so I started with a smaller file generated from: http://en.wikipedia.org/wiki/User:Brion_VIBBER/Books/SmallZimTest
Fastest of three runs each:
'Barack Obama' (a fairly long article with a few hundred KB of HTML) Load time 5298ms Render time 439ms
'Husiatyn Raion' (a shorter article) Load time 1789ms Render time 23ms
Better than 20 seconds, but still pretty slow; I'm not sure offhand how the articles get placed in their blocks so there may well be worse worst cases. :(
Definitely worth trying the native-code library at least for the LZMA -- though in a worst case, a few seconds to load isn't worse than you'll have on a slow mobile data network. :) Make sure it runs on a background thread and shows a spinner or progress bar if we do end up having to run fetches this slowly on some platforms...
-- brion
On Tue, Sep 27, 2011 at 6:01 PM, Brion Vibber brion@pobox.com wrote:
I did a quick test on my Nexus One (running Android 2.3.6, so with JIT). There are some faster processors out there on higher-end phones, but there's also a lot of cheaper phones that aren't going to be any faster than this.
The wikipedia-de.zim test file hasn't downloaded here yet so I started with a smaller file
Ok, 'Graz' article in wikipedia-de.zim (same one Christian tested at 20s on an Eclair-based phone, no JIT) I get:
Load time: 5088ms Render time: 131ms
That's on the Nexus One running 2.3.6 (with JIT). About the same load time as I saw with a longer article on a smaller file.
-- brion
Hi Brion,
Thanks for trying this out. Performance gain is basically as expected, but looks like that we should switch to native xz support. Alternatively we could reduce the cluster size, although this would impact compression ratio. Anybody has an idea whether cluster size reduction is feasible? (Reduction would need to be about 1:10, so pretty significant). Regarding image support after some more trials I believe that the replace-imgsrc-withbase64data in javascript may work sufficiently well for first zim support. It may be pretty slow if a lot of images need to be loaded, but on mobile phones anyway typically zimfiles with no or few images are the main use case (because with all images the zim files are extremely large). In the future (=when ice cream sandwich devices available) it makes sense to switch to/add support of the WebViewClient.shouldInterceptRequest-approach, as this is supposed to be much faster. We should talk to the phonegap team, whether they could add support for this, so that plugins can use it.
Best regards, Christian
Am 28.09.2011 21:24, schrieb Brion Vibber:
On Tue, Sep 27, 2011 at 6:01 PM, Brion Vibberbrion@pobox.com wrote:
I did a quick test on my Nexus One (running Android 2.3.6, so with JIT). There are some faster processors out there on higher-end phones, but there's also a lot of cheaper phones that aren't going to be any faster than this.
The wikipedia-de.zim test file hasn't downloaded here yet so I started with a smaller file
Ok, 'Graz' article in wikipedia-de.zim (same one Christian tested at 20s on an Eclair-based phone, no JIT) I get:
Load time: 5088ms Render time: 131ms
That's on the Nexus One running 2.3.6 (with JIT). About the same load time as I saw with a longer article on a smaller file.
-- brion _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org