On 24/09/2011 14:24, Christian Pühringer wrote:
> The JAVA liblzma performance is pretty bad: To increase efficiency of
> compression in the zim-format articles (and also all
> other data like images) are stored in clusters. Cluster size is apparently about
> 1 MB. This implies that loading an article
> which is stored at the end of a cluster involves decompressing the complete
> cluster.
Images should not be compressed in ZIM files for the obvious reasons
they mainly are already compressed. This is the case for all ZIM files I
made. As far as I know this is also the case for Mediawiki:Collection
build ZIM files.
Emmanuel
On Tue, Sep 27, 2011 at 1:51 PM, Christian Pühringer <cip(a)gmx.at> wrote:
> I've pushed the code to https://github.com/cip/zimgap-android.
> Note that the only functionality is currently to load an article from the
> zim
> file and display it.
> In particular there is no search functionality, so you have to know the
> exact
> article name to open it.
> The reason is simply that I am currently focusing on the feasibility, and I
> expect that the main issues are related to loading
> an article and and not related to searching the index.
>
Just to warn: run 'git submodule init && git submodule update' after cloning
the repository, or Eclipse will whinge about being unable to find the
xz-java sources. :)
I did a quick test on my Nexus One (running Android 2.3.6, so with JIT).
There are some faster processors out there on higher-end phones, but there's
also a lot of cheaper phones that aren't going to be any faster than this.
The wikipedia-de.zim test file hasn't downloaded here yet so I started with
a smaller file generated from:
http://en.wikipedia.org/wiki/User:Brion_VIBBER/Books/SmallZimTest
Fastest of three runs each:
'Barack Obama' (a fairly long article with a few hundred KB of HTML)
Load time 5298ms
Render time 439ms
'Husiatyn Raion' (a shorter article)
Load time 1789ms
Render time 23ms
Better than 20 seconds, but still pretty slow; I'm not sure offhand how the
articles get placed in their blocks so there may well be worse worst cases.
:(
Definitely worth trying the native-code library at least for the LZMA --
though in a worst case, a few seconds to load isn't worse than you'll have
on a slow mobile data network. :) Make sure it runs on a background thread
and shows a spinner or progress bar if we do end up having to run fetches
this slowly on some platforms...
-- brion
Awesome .. this will be fun to play with and a great addition to the
Wikipedia app community.
--tomasz
On Tue, Sep 27, 2011 at 1:51 PM, Christian Pühringer <cip(a)gmx.at> wrote:
> Hi Tomasz,
>
> I've pushed the code to https://github.com/cip/zimgap-android.
> Note that the only functionality is currently to load an article from the
> zim file and display it.
> In particular there is no search functionality, so you have to know the
> exact article name to open it.
> The reason is simply that I am currently focusing on the feasibility, and I
> expect that the main issues are related to loading
> an article and and not related to searching the index.
>
> Best regards,
> Christian
>
> Am 24.09.2011 23:14, schrieb Tomasz Finc:
>
> Awesome work Christian. Where can we find your code to test out?
>
> If you don't have a place then I'll carve one out in our depots.
>
> On Sep 24, 2011 5:57 AM, "Christian Pühringer" <christian(a)puehringer.net>
> wrote:
>> Hi Emanuel,
>>
>> Makes sense that images are not compressed in zim file again, thanks for
>> the
>> clarification.
>> In particular for windows mobile this increases the probability that with
>> reducing the cluster size
>> a sufficient performance is achieved (i.p. if there was a better way to
>> display
>> images than in android, but I haven't looked into this).
>> For android I still expect that it it will be better to use a native
>> library.
>>
>> Christian
>> Am 24.09.2011 14:30, schrieb Emmanuel Engelhart:
>>> On 24/09/2011 14:24, Christian Pühringer wrote:
>>>> The JAVA liblzma performance is pretty bad: To increase efficiency of
>>>> compression in the zim-format articles (and also all
>>>> other data like images) are stored in clusters. Cluster size is
>>>> apparently about
>>>> 1 MB. This implies that loading an article
>>>> which is stored at the end of a cluster involves decompressing the
>>>> complete
>>>> cluster.
>>> Images should not be compressed in ZIM files for the obvious reasons
>>> they mainly are already compressed. This is the case for all ZIM files I
>>> made. As far as I know this is also the case for Mediawiki:Collection
>>> build ZIM files.
>>>
>>> Emmanuel
>>>
>>
>> _______________________________________________
>> dev-l mailing list
>> dev-l(a)openzim.org
>> https://intern.openzim.org/mailman/listinfo/dev-l
>
>
> _______________________________________________
> dev-l mailing list
> dev-l(a)openzim.org
> https://intern.openzim.org/mailman/listinfo/dev-l
>
>
On Sat, Sep 24, 2011 at 5:24 AM, Christian Pühringer <cip(a)gmx.at> wrote:
> a). Extract images from zim to local file system and provide them over a
> ContentProvider.
> b) Replace image src with base64 encoded image data. (Either directly on
> loading
> of article or by using java script)
>
I'd be inclined to try the base64 data URIs for simplicity: you won't have
to deal with cleaning up and maintaining a cache of extracted files.
On the other hand, data URIs will probably take up more memory, and you have
to spend a little extra time on base64 encoding/decoding. Try it and see! :)
> While this is not a problem with C code
> even on embedded devices, it seems to be a problem with Java: Reading an
> article
> at the end of a cluster takes close to 20 s on
> my test android phone. As the phone is pretty low end (Orange
> Boston ) and uses an old android version (Eclair) without a just-in-timer
> compiler I expect that other models
> are significantly faster. However, I doubt that the performance gain will
> be
> sufficient to bring article load time in a range of << 1 s.
> I am going to try it out, but I'd expect that we probably have to switch to
> native code for zimlib support. (At least for liblzma).
>
Eek! Definitely should test on newer devices/OSs as well, at least to get a
baseline to compare the native code against.
(Note that not all Android systems are necessarily ARM -- for best
portability, being able to use the Java LZMA code if the native library
can't be loaded might be nice.)
> An other approach is to reduce cluster size of zim files. I am not sure
> right
> now whether this would be sufficiently fast, but it is
> worth considering it as an option: While for android being able to use the
> java
> implementation is a benefit, it is also not a big thing
> if native code has to be used. However, more concerning is that it may not
> be
> possible to support for Windows Mobile at all with the
> current cluster size. (Because AFAIK not native code is supported)
>
Windows Mobile / Windows Phone should have a decent JIT compiler for .NET
code, so it's at least worth testing...
-- brion
Hi,
is this maybe also useful for ZIM - to make ZIM readers which are
working cross-platform?
As far as I understood phonegap is mainly a framework to create mobile
apps based on HTML 5. At least the display of ZIM contents should be
simple then as we just need a HTML widget for that.
But what about libraries needed to read file contents, such as zimlib? I
couldn't find out if Phonegap itself supports native file access (so we
could re-implement ZIM features with that) or if it allows the use of
native libraries.
/Manuel
Am 27.08.2011 02:44, schrieb Tomasz Finc:
> Thanks for the super detailed write up Brion. I've been actively
> talking with the PhoneGap guys after doing some more research on this
> and it seems like a really good fit to have a consistent experience
> across a whole host of devices.
>
> What were looking at is not necessarily a lot of depth in every single
> platform but a lot of horizontal range. Phonegap platform support
> beats out Titanium pretty easily there.
>
> We'll be working a lot closer with the PhoneGap team going forward to
> quickly have something in the android store to start.
>
> If anyone is interested in helping then we'll have plenty of
> opportunities to join in. Over the next weeks we'll be adding bugs and
> sending out more calls to get involved.
>
> --tomasz
>
>
>
> On Tue, Aug 16, 2011 at 1:50 PM, Brion Vibber <brion(a)pobox.com> wrote:
>> On Tue, Aug 16, 2011 at 1:14 PM, Tomasz Finc <tfinc(a)wikimedia.org> wrote:
>>
>>> I've been asking around on IRC but thought it would be good to open up
>>> to a larger audience.
>>>
>>> Has anyone here used PhoneGap (http://www.phonegap.com/) for mobile
>>> app development? I'm eager to get your thoughts and potentially
>>> brainstorm some new ideas.
>>>
>>
>> I haven't used PhoneGap except for some brief testing, but I have used
>> Titanium Appcelerator, which is another framework in that space, in working
>> on StatusNet's iPhone & Android app.
>>
>> Between the two I'd recommend PhoneGap for our usage as preferable over
>> Titanium, but would appreciate more feedback from people who've done fuller
>> PhoneGap work.
>>
>> A few key differences:
>>
>> PhoneGap models around extending a full-screen web view with additional
>> JavaScript-accessible APIs to use device & OS capabilities (camera, address
>> book, notifications, etc). This gives you few/no "native widgets" for your
>> primary screens, but can make it relatively easy to create an HTML/JS-based
>> web application that's extended with native abilities and can be shipped
>> into native app stores.
>>
>> Titanium was originally based on a similar model, but switched to a native
>> widget bridging system, where your JavaScript code instantiates and
>> manipulates objects which are bridged to native UI components and such. This
>> can make your widgets look & feel more native, and can make some UI bits
>> faster. But it also makes behavior less consistent between platforms; many
>> widgets or features simply aren't available on all platforms, and last I
>> checked there was basically *no* working support other than iOS and Android.
>> (An early BlackBerry demo came out, was insufficient to do anything we
>> needed, and never got updated that we saw.)
>>
>> Since the Wikipedia app is mostly a webview and ...... maybe a menu?
>> PhoneGap is probably a good choice. Titanium can also embed a webview, but
>> it's a lot more work to deal with two levels of JS! PhoneGap has much
>> broader device support, but be warned -- it'll use the native webview on
>> each system, so JS and HTML/CSS support will still vary across platforms.
>>
>>
>> Debugging in PhoneGap basically devolves to being able to debug a web
>> application; various tools like http://phonegap.github.com/weinre/ can help
>> with this (or if you code carefully you may get away debugging your app in
>> your favorite desktop browser directly ;)
>>
>>
>> Titanium was always a bear to debug things in and basically came down to
>> 'watch the system log output in Android, that's the only place you'll
>> actually see low-level errors'; this may be better now with their IDE
>> support.
>>
>> Titanium also pretty aggressively pushes their support & training services
>> which I find offputting; their project build tool wants you to login to
>> their 'cloud' stuff to let you hook up to their remote build & analytics
>> services, which we didn't ever really use.
>>
>> Support seemed to center on getting people to take training webinars or
>> pointing people at the documentation and examples when they ask how to do
>> something; I didn't find them very responsive about platform bugs or missing
>> documentation except by contacting their couple of Android developers
>> one-on-one in IRC to ask for merges -- which was usually a pretty good
>> experience! Getting fixes for iOS merged was very difficult; I could never
>> get ahold of their iOS developers directly, and they didn't seem to be any
>> more responsive to low-level bugs we filed through their customer support
>> system.
>>
>> We had to build with a patched version of the iOS and Android runtimes for
>> quite some time as there were serious bugs. On the plus side, maintaining a
>> patched branch in git was very easy -- a lot of 'git pull origin master' and
>> occasionally tidying up conflicts. Their source is all on github and is easy
>> to fork and not too awful to build, at least for the mobile runtime.
>>
>>
>> Note that both PhoneGap and Titanium frameworks are open source & hosted on
>> github, though both require a CLA to submit code upstream. (I have signed
>> the Titanium CLA to submit patches to them last year; haven't done for
>> PhoneGap yet.)
>>
>> -- brion
>> _______________________________________________
>> Wikitech-l mailing list
>> Wikitech-l(a)lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l(a)lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
--
Regards
Manuel Schneider
Wikimedia CH - Verein zur Förderung Freien Wissens
Wikimedia CH - Association for the advancement of free knowledge
www.wikimedia.ch
Dear Chapter Representative(s) and Wikipedians,
Please treat this invitation as a personal
invite from the Indian WikiCommunity to yourselves and your community to attend
the first annual national WikiConference India scheduled to take place
between 18 and 20 November at the University of Mumbai’s Fort Campus.
The event is being co-hosted by the
community and the chapter and this would be a wonderful opportunity for Indian
Wikipedians to interact with all of you. Further, we would appreciate any help
that could be extended by yourselves to make this conference a success.
Do let us know of any requirements you may
have. Kindly confirm the above at the earliest to allow us to schedule
everything in.
Meta Page:
http://meta.wikimedia.org/wiki/WikiConference_India_2011
Kind
Regards,
Dear Wikipedians Worldwide,
The Mumbai Community in association with Wikimedia India is hosting
the first annual national WikiConference India 2011 later this year.
Kindly find attached herewith a formal invite from the Indian
Wikimedia Community to attend WikiConference India 2011.
We hope some if not all of you can make it.
Kind Regards,
Pranav Curumsey
Chairperson - Organising Committee
WCI 2011
http://meta.wikimedia.org/wiki/WikiConference_India_2011
(copied from wikitech-l)
What: Collection extension triage bug triage
When: Wednesday, August 24, 17:00UTC
Time zone conversion: http://hexm.de/65
Where: #wikimedia-dev on freenode
Use http://webchat.freenode.net/ if you don't have an IRC
client
This week I'll be focusing on the Collection extension In week 2 of "The
Bugmeister and Tomasz Finc". If you've ever tried to create PDFs or
OpenZIM files using the Book Creator or wanted to try to adapt these
tools for your own site, this is the bug triage for you.
Following are the bugs I really want to focus on. But if you don't see
your bug here, then check out the etherpad: http://hexm.de/5l
If it isn't listed there, send me an email and I'll try to make sure
that it gets attention.
(FWIW, I hope to have a list of bugs ready for a sprint this weekend
based, mostly, on the Collection extension.)
http://bugzilla.wikimedia.org/30326 -- PDF export extension doesn't
support some characters in Arabic script
http://bugzilla.wikimedia.org/19830 -- PDF prints don't join Arabic
letters properly
http://bugzilla.wikimedia.org/28206 -- PDF generation does not support
Complex Script Wikis
http://bugzilla.wikimedia.org/30437 -- change the Hebrew default font to
Taamey Frank CLM
http://bugzilla.wikimedia.org/27462 -- <noinclude> showing in PDF
http://bugzilla.wikimedia.org/28060 -- Collection extension should not
add chapters in reverse order
http://bugzilla.wikimedia.org/30503 -- template exception for book maker
(pdf export)
http://bugzilla.wikimedia.org/26330 -- collection contents lost when
only loading js via https
http://bugzilla.wikimedia.org/24512 -- Collection uses curl_*()
functions instead of Http::*() functions
http://bugzilla.wikimedia.org/28118 -- The path to images
https://bugzilla.wikimedia.org/30511 -- Collection extention should
place time stamp of revision extracted into the offline file
http://bugzilla.wikimedia.org/30199 -- ZIM external links should be
always marked as external.... or removed
Happy hacking!
Mark.
--
Mark A. Hershberger
Bugmeister
Wikimedia Foundation
mhershberger(a)wikimedia.org
717.271.1084
Dear Jessie,
I am sorry for responding almost more than a week late as i had been
mostly traveling and attending a series of meetings. I wish I could
responded little earlier.
We distribute free computers (all based on GNU/Linux, currently on
Ubuntu), and from last two years, laptops, to meritorious students of the
State of Assam (India) who secure >=60% marks in the 10th standard
examination. Every year about 15-20,000 systems are distributed. So far,
we have distributed 92,000+ system, and this year about 19000+ students
are about to get benefited.
The current year's scheme is scheduled for launch on the 23rd of
September, 2011. Along with Ubuntu we bundle a host of application
software and content for the benefit of the students. Mostly, so far we
have ben very successful in providing rich application level content for
science students. There has been carry home stuff for students who are
aspiring to become economists, historians, psychologists, anthropologists,
English literature etc.. Further, broadband penetration being poor in the
remote areas of the state (a survey tells that 58% of the recipient
students belong to remote areas and come from poor families), we have been
thinking how we could provide wiki kind of content to the students offline
on the laptops to be distributed.
We shot an email several months back, and it has generated lot of
enthusiasm. However, my programme is to start from the 23rd of September,
2011. I am not sure whether, this year also we shall be able to provide
some offline content. As a language, ENGLISH is okay. We can think of
translation at a later date.
Pl tell us how to proceed further on the issue. We have now very little
time to organize...
Yours,
MK Yadava IFS,
MD, AMTRON
---------------------------
Please exchange editable Office documents only in ODF Format. No other
format is acceptable. Sending such information in other formats is at your
own risk. Please be assured that non ODF formats will not be opened.
To get a free editor supporting ODF, please visit http://www.openoffice.org/
---------------------------------------
Visit my blogs at http://letsbetteroursociety.blogspot.com/
------------------------------------------------
> Hi MK Yadava -
>
> Thanks for passing along this email!
>
> I am the project manager for the Offline Wikipedia work. I'm very
> interested
> in supporting your efforts in content circulation, and I am eager to learn
> more about your project referenced below. Who are you distributing your
> computers to? What sort of software are you looking for? Where exactly are
> you hoping to distribute?
>
> Thanks so much; I'd love to help out with this as much as possible!
> Jessie
>
> On Fri, Aug 5, 2011 at 2:26 AM, M K Yadava IFS <in(a)amtron.in> wrote:
>
>> Dear All,
>> Sounds great! If we have sizeable content for school children from
>> standard V-XII in humanities, science and mathematics in English alone,
>> it
>> will be a great service for bridging the digital divide. We shall be
>> glad
>> to put such a content in our PC distribution program and make it
>> operational by middle of September, 2011.
>>
>> Regards,
>> Yours,
>> MK Yadava
>> ---------------------------
>> Please exchange editable Office documents only in ODF Format. No other
>> format is acceptable. Sending such information in other formats is at
>> your
>> own risk. Please be assured that non ODF formats will not be opened.
>>
>> To get a free editor supporting ODF, please visit
>> http://www.openoffice.org/
>> ---------------------------------------
>> Visit my blogs at http://letsbetteroursociety.blogspot.com/
>> ------------------------------------------------
>>
>> > Thanks Manuel!
>> >
>> > I think this will be a good group. I just put together a brief
>> > presentation
>> > with kind of the general overview of Offline and how all the pieces
>> fit
>> > together, but I can go through it quickly and allow the different
>> pieces
>> > to
>> > fill in the blanks throughout?
>> >
>> > I'm attaching it in case others want to take a look. I put it together
>> > quickly so if there are errors please forgive me (and correct me!) :)
>> >
>> > Jessie
>> >
>> > On Fri, Aug 5, 2011 at 1:45 AM, Manuel Schneider <
>> > manuel.schneider(a)wikimedia.ch> wrote:
>> >
>> >> Hi offline folks,
>> >>
>> >> we will have our Wikimedia Offline session today at 15:15 in room
>> Tavor
>> >> (in Beit Hecht, 1st floor)
>> >>
>> >> http://wikimania2011.wikimedia.org/wiki/Submissions/Wikimedia_Offline
>> >>
>> >> The session was planned as an interactive workshop for all people
>> >> interested in the offline issues. This means that YOU are invited to
>> >> present your project, your ideas and ask your questions.
>> >> Interesting topics are selection of content, publishing, storage,
>> reader
>> >> applications... and what you think is.
>> >>
>> >> There will be at least three people talking today:
>> >> * Jessie Wild from Wikimedia Foundation
>> >> * Shiju Alex from Malayallam Wikipedia
>> >> * me from openZIM
>> >>
>> >> See you there! If you have questions or want to coordinate in advance
>> -
>> >> please contact me.
>> >>
>> >>
>> >> /Manuel
>> >> --
>> >> Regards
>> >> Manuel Schneider
>> >>
>> >> Wikimedia CH - Verein zur Förderung Freien Wissens
>> >> Wikimedia CH - Association for the advancement of free knowledge
>> >> www.wikimedia.ch
>> >>
>> >> _______________________________________________
>> >> Offline-l mailing list
>> >> Offline-l(a)lists.wikimedia.org
>> >> https://lists.wikimedia.org/mailman/listinfo/offline-l
>> >>
>> >
>> >
>> >
>> > --
>> > *Jessie Wild
>> > Global Development, Manager
>> > Wikimedia Foundation
>> > *
>> > _______________________________________________
>> > Offline-l mailing list
>> > Offline-l(a)lists.wikimedia.org
>> > https://lists.wikimedia.org/mailman/listinfo/offline-l
>> >
>>
>>
>> _______________________________________________
>> Offline-l mailing list
>> Offline-l(a)lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/offline-l
>>
>
>
>
> --
> *Jessie Wild
> Global Development, Manager
> Wikimedia Foundation
> *
>
Good day!
My name is Oleg Novoselov. I work as English teacher at the rural school and
I want to give a lesson about your organization and its activity to my
students. I will be very thankful if you send some information and souvenirs
with symbols of the organization. I plan to present them to my students
during school activities to raise their interest to learn foreign language.
I wish you good health and success in your work. Look forward to hearing
from you as soon as possible. Thank you in advance!
Address:
Oleg Novoselov
Rudneva St 63-7
Tula
Russia
300026