Offline-l July 2009

offline-l@lists.wikimedia.org

7 participants
15 discussions

Re: [openZIM dev-l] OpenZim
by Tommi Mäkitalo 07 Aug '09

07 Aug '09

Hi, sorry for my late answer. I was on holiday. we are very interested in adopting openzim wherever possible. A nice handheld device or phone would be a great showcase and of. The ZimReader actually was once compiled on a openmoko device and it worked. How can we help? Tommi On Mittwoch, 22. Juli 2009 17:49:42 you wrote: > Hi Tommi, > > Sorry for using your tntnet address for an OpenZim related mail :) > > I am Mirko and part of the Qi Hardware Team. > > This Monday we launched our company [1] and announced our first product > "Ben NanoNote", a fully opened multifunction device [3]. > > As you can see on the product page the Ben NanoNote does not have a > build-in RF chip. We therefore think it would be ideal for a project such > as OpenZim and as a device for an offline version of Wikipedia. > > We are very interested to hear what you thoughts are. We have a mailing > list dedicated for development [3]. > > We see an OpenZim client as a possibility for one of the standard > applications shipping in our image, but would need guidance. If you think > this would be doable and something you would be interested in, let me know. > > Regards, > /Mirko > > > [1] > http://linux.com/news/embedded-mobile/mids/29263-openmoko-layoffs-lead-to-n >ew-open-hardware-venture [2] > http://www.qi-hardware.com/products/ben-nanonote/ > [3] http://lists.qi-hardware.com/cgi-bin/mailman/listinfo/developer

3 4

[openZIM dev-l] ZimReader for Openmoko
by Marc Bantle 04 Aug '09

04 Aug '09

Hi all! Since an offline encyclopedia is most valuable on mobile devices, I worked on building ZimReader for the Openmoko Neo. As a start I did a native build on a Debian (hackable:1 [3]) driven Openmoko Freerunner. After fixing some minor build issue ([1], [2]), I can now browse Wikipedia offline on the phone. ZimReader performs amazingly well on the limited resources of the device. I use it in conjunction with woosh, a browser that comes with hackable:1. The content resides on SD-card. A full-text search takes around 5s, mostly even less. Since I'm not too much into packaging, I wrapped up a tar ball for those interested in binaries [4]. Please see the Readme for installation instructions. Note: The binaries will most likely _not_ run on OE-based distributions (OM2009, SHR). They have only been tested on hackable1. I'll try to supply packages for other distros, as soon as I find the time. Thank's a lot to the openzim team for their great work! Cheers, Marc [1] http://bugs.openzim.org/show_bug.cgi?id=6 [2] http://bugs.openzim.org/show_bug.cgi?id=5 [3] http://www.hackable1.org/wiki/Main_Page [4] http://www.gut-informierte-kreise.de/openmoko/openzim/

2 2

[openZIM dev-l] Windows portability feedback
by Emmanuel Engelhart 03 Aug '09

03 Aug '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I have achieved to compile a first version of a zimlib.lib (static library under Windows). I do not really know if I can use it... but at least it compiles. I want to give you here the modifications I have done to achieve that. Tommi, it would be great to include them (or others with the same purpose) into the SVN. Index: include/zim/zim.h =================================================================== - --- include/zim/zim.h (revision 236) +++ include/zim/zim.h (working copy) @@ -22,6 +22,12 @@ #include <stdint.h> +#ifdef _WIN32 +typedef unsigned __int64 uint64_t; +typedef signed __int8 int8_t; +typedef signed __int32 int32_t; +#endif + namespace zim { typedef uint32_t size_type; Index: src/search.cpp =================================================================== - --- src/search.cpp (revision 236) +++ src/search.cpp (working copy) @@ -106,7 +106,7 @@ // weight position of words in the document if (Search::getWeightPos()) for (itp = posList.begin(); itp != posList.end(); ++itp) - - priority += Search::getWeightPos() / pow(1.01, itp->first); + priority += Search::getWeightPos() / pow((double)1.01, (double)itp->first); // here is a ambiguity between to pow() functions (pow(double, int) or pow(double, double)) if (Search::getWeightPosRel()) for (itp = posList.begin(); itp != posList.end(); ++itp) Index: src/bunzip2stream.cpp =================================================================== - --- src/bunzip2stream.cpp (revision 236) +++ src/bunzip2stream.cpp (working copy) @@ -113,7 +113,8 @@ // there is data already available // read compressed data from source into ibuffer log_debug("in_avail=" << sinksource->in_avail()); - - stream.avail_in = sinksource->sgetn(ibuffer(), std::min(sinksource->in_avail(), ibuffer_size())); + + stream.avail_in = sinksource->sgetn(ibuffer(), min(sinksource->in_avail(), ibuffer_size())); // compile genereates an error if you have the "std::", do not know exactly why. } else { Index: src/uuid.cpp =================================================================== - --- src/uuid.cpp (revision 236) +++ src/uuid.cpp (working copy) @@ -20,12 +20,31 @@ #include <zim/uuid.h> #include <iostream> #include <time.h> - -#include <sys/time.h> +#include <zim/zim.h> // necessary to have the new types #include "log.h" #ifdef WITH_CXXTOOLS #include <cxxtools/md5stream.h> #endif +#if defined(_MSC_VER) || defined(__MINGW32__) +# include <time.h> +# include <Windows.h> +#else +# include <sys/time.h> +#endif + +#if defined(_MSC_VER) || defined(__MINGW32__) +int gettimeofday(struct timeval* tp, void* tzp) { + DWORD t; + t = timeGetTime(); + tp->tv_sec = t / 1000; + tp->tv_usec = t % 1000; + /* 0 indicates that the call succeeded. */ + return 0; +} +#endif // time related function/headers are differents und win32 + + log_define("zim.uuid") namespace zim @@ -62,7 +81,12 @@ *reinterpret_cast<int32_t*>(ret.data) = reinterpret_cast<int32_t>(&ret); *reinterpret_cast<int32_t*>(ret.data + 4) = static_cast<int32_t>(tv.tv_sec); *reinterpret_cast<int32_t*>(ret.data + 8) = static_cast<int32_t>(tv.tv_usec); + + #if defined(_MSC_VER) + *reinterpret_cast<int32_t*>(ret.data + 12) = static_cast<int32_t>(GetCurrentProcessId()); + #else *reinterpret_cast<int32_t*>(ret.data + 12) = static_cast<int32_t>(getpid()); // getpid() is GetCurrentProcessId() under win32 + #endif #endif Index: src/qunicode.cpp =================================================================== - --- src/qunicode.cpp (revision 236) +++ src/qunicode.cpp (working copy) @@ -18,6 +18,7 @@ */ #include <zim/qunicode.h> +#include <zim/zim.h> // for the types compatibilty #include "log.h" #include <iostream> #include <sstream> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkpwmusACgkQn3IpJRpNWtN4YQCfaYy9rGS5nrWyYqb5Nk8bs6eo hgsAoKTPFo/75jY1GhUZM+iWGbGDP2dg =goMa -----END PGP SIGNATURE-----

3 5

[openZIM dev-l] For your attention: New Bugs
by Manuel Schneider 20 Jul '09

20 Jul '09

Dear developers team, I am happy to forward you some mails concerning new bugs we have received. It lloks like we have got some more developers looking into the software. Please see the attachments. Cheers, Manuel -- Regards Manuel Schneider Wikimedia CH - Verein zur Förderung Freien Wissens Wikimedia CH - Association for the advancement of free knowledge www.wikimedia.ch

1 0

[openZIM dev-l] [Mediawiki] New bug affecting us
by Emmanuel Engelhart 18 Jul '09

18 Jul '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, a few weeks ago, Frederico reported me a small bug in a special HTML page in a test ZIM file in Italian: https://sourceforge.net/tracker/?func=detail&aid=2798771&group_id=175508&at… The symptom is that references (<references/>) are not always rendered. After auditing a little bit the Mediawiki code, it seems that this is a Mediawiki bug. I have reported it there: https://bugzilla.wikimedia.org/show_bug.cgi?id=19807 I can not fix it, this is too complicated for me... but maybe Mediawiki hackers listening there have ideas ;) Regards Emmanuel -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkphxUEACgkQn3IpJRpNWtPX+ACgtP1QDNuRlZxz8FV2Dh/YChHK FI8An0clv19tJo8leLNQkGSBd2vAIf2V =t4UO -----END PGP SIGNATURE-----

1 0

[openZIM dev-l] Wikipedia for Schools ZIM version
by emmanuel＠engelhart.org 13 Jul '09

13 Jul '09

Hi, I have this week end prepared a ZIM file with the last Wikipedia for Schools selection (from SOS children's Villages UK). http://tmp.kiwix.org/zim/schools-wikipedia-full-20081023-rc1.zim Content should be exactly the same as before but the size is 30% smaller. After a few minutes indexing with Kiwix the whole content are available with the search engine. At this occasion, I have implemented the ignore feature for HTML content with the NOINDEX meta tag. This is important to avoid indexing index pages, it would be maybe good to do also that with the zimindexer (if not already done). I have maybe someone who is ready to pay a little bit to help to port Kiwix and libzim under Windows... nothing sure but I hope it will help to find someone to do this job quickly as we ;) I have also made a feature request to add a few fields in the header, please have a look at it: http://bugs.openzim.org/show_bug.cgi?id=4 Regards Emmanuel

1 0

Re: [openZIM dev-l] Kiwix index size
by Asaf Bartov 08 Jul '09

08 Jul '09

Clarification: This last message was by Rotem, a fellow WM-IL member helping me with the embedding of the Hebrew Wikipedia in the One Computer Per Child project. He is reporting issues with Kiwix and the ZIM file I created last week. Regarding size: Size is important, because we intend to add images (the 300MB ZIM file is the complete Hebrew Wikipedia text, but no pictures). We are hoping to have at least 5GB reserved for us in those One Computer Per Child machines we are to install on, but we may be forced to make do with 3GB. So every MB saved from the index, is another MB available for images... Asaf Bartov Wikimedia Israel On Mon, Jul 6, 2009 at 3:58 PM, Rotem Simha <hidroo(a)gmail.com> wrote: > * there are some errors in links of files and special pages > examples > קובץ:Nuvola_apps_important.svg<http://commons.wikimedia.org/wiki/File:Nuvola_apps_important.svg> link > to ויקיפדיה:מיזמי ויקיפדיה/מיזם ערכים ללא תמונות/קטגוריות/ספורטאים איטלקים(wikipedia:wikipedia projects\ articles without images\categories\Sports > people from Italy) > מיוחד:אקראי (Special:Random) > 15 במאי (may 15) > מיוחד:שינויים אחרונים (Special:RecentChanges) > 10_באוגוסט > > * size is important because we intend to add images > > 2009/7/6 <dev-l-request(a)openzim.org> > >> Send dev-l mailing list submissions to >> dev-l(a)openzim.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://intern.openzim.org/mailman/listinfo/dev-l >> or, via email, send a message with subject or body 'help' to >> dev-l-request(a)openzim.org >> >> You can reach the person managing the list at >> dev-l-owner(a)openzim.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of dev-l digest..." >> >> >> Today's Topics: >> >> 1. Kiwix index size (Asaf Bartov) >> 2. Re: Kiwix index size (Manuel Schneider) >> 3. Re: Kiwix index size (Emmanuel Engelhart) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Sun, 5 Jul 2009 19:18:57 +0300 >> From: Asaf Bartov <asaf.bartov(a)gmail.com> >> Subject: [openZIM dev-l] Kiwix index size >> To: dev-l(a)openzim.org >> Message-ID: >> <50a20d900907050918r3fcff23l275c67690ed7fc20(a)mail.gmail.com> >> Content-Type: text/plain; charset="iso-8859-1" >> >> Hi, everyone. >> >> When running Kiwix's indexer on the ZIM file I had created from the Hebrew >> Wikipedia last week, the Kiwix data directory ran up to a total of 31 >> items, >> totalling 2.3 GB. The ZIM file itself is ~300MB. Does this proportion >> make >> sense? >> >> Detailed ls output attached. >> >> Thanks in advance, >> >> Asaf Bartov >> -- >> Asaf Bartov <asaf(a)forum2.org> >>

2 2

Re: [openZIM dev-l] Kiwix index size
by Asaf Bartov 08 Jul '09

08 Jul '09

Hello, Pascal. Thanks for the pointer about OkaWix! I've downloaded it and tried it with the Hebrew Wikipedia. Initial findings: - excellent rendering, including directionality. - categories don't seem to work I'm downloading the pictures now... I note it's using Zeno files. Is it planned to use ZIM files in the future? A. On Mon, Jul 6, 2009 at 7:15 PM, Pascal Martin <pmartin(a)linterweb.fr> wrote: > So do you try okawix with the hebrew contents ? > >

2 3

Re: [openZIM dev-l] dev-l Digest, Vol 5, Issue 2
by emmanuel＠engelhart.org 06 Jul '09

06 Jul '09

About the dead links, few thinks: * Are you sure the problem is not at the source (HTML files) * the zimwriter does not check if all links in all HTML pages are OK * it seems that the libzim returns a bad content if the content does not exist (Tommi can you confirm?). Should returns nothing or an error code IMO. Emmanuel Le lun 06/07/09 14:58, "Rotem Simha" hidroo(a)gmail.com a écrit: > * there are some errors in links of files and special pages > examples > קובץ:Nuvola_apps_important.svg [1] link to > ויקיפדיה:מיזמי ויקיפדיה/מיזם ערכים > ללא תמונות/קטגוריות/ספורטאים איטלקים > (wikipedia:wikipedia projects articles without imagescategoriesSports > people from Italy) > מיוחד:אקראי (Special:Random) > 15 במאי (may 15) > מיוחד:שינויים אחרונים (Special:RecentChanges) > > 10_באוגוסט > > * size is important because we intend to add images > > 2009/7/6 > Send dev-l mailing list submissions to > dev-l(a)openzim.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://intern.openzim.org/mailman/listinfo/dev-l [2] > or, via email, send a message with subject or body help to > dev-l-request(a)openzim.org > > You can reach the person managing the list at > dev-l-owner(a)openzim.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of dev-l digest..." > > Todays Topics: > > 1. Kiwix index size (Asaf Bartov) > 2. Re: Kiwix index size (Manuel Schneider) > 3. Re: Kiwix index size (Emmanuel Engelhart) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 5 Jul 2009 19:18:57 +0300 > From: Asaf Bartov > Subject: [openZIM dev-l] Kiwix index size > To: dev-l(a)openzim.org > Message-ID: > > > Content-Type: text/plain; charset="iso-8859-1" > > Hi, everyone. > > When running Kiwixs indexer on the ZIM file I had created from the > Hebrew > Wikipedia last week, the Kiwix data directory ran up to a total of > 31 items, > totalling 2.3 GB. The ZIM file itself is ~300MB. Does this > proportion make > sense? > > Detailed ls output attached. > > Thanks in advance, > > Asaf Bartov > -- > Asaf Bartov >

2 1

Re: [openZIM dev-l] Kiwix index size
by emmanuel＠engelhart.org 06 Jul '09

06 Jul '09

I have answered to Rotem about the links. I have also open a bug on the Kiwix side: https://sourceforge.net/tracker/?func=detail&aid=2817440&group_id=175508&at… For the search engine index size, we have to search a solution with a smaller index. Starting with the openzim solution should be good. I will have a look during this week. Emmanuel Le lun 06/07/09 15:03, "Asaf Bartov" asaf.bartov(a)gmail.com a écrit: > Clarification: > > This last message was by Rotem, a fellow WM-IL member helping me with > the embedding of the Hebrew Wikipedia in the One Computer Per Child > project. > > He is reporting issues with Kiwix and the ZIM file I created last > week. > > Regarding size: Size is important, because we intend to add images > (the 300MB ZIM file is the complete Hebrew Wikipedia text, but no > pictures). We are hoping to have at least 5GB reserved for us in > those One Computer Per Child machines we are to install on, but we may > be forced to make do with 3GB. So every MB saved from the index, is > another MB available for images... > > Asaf Bartov > Wikimedia Israel > > On Mon, Jul 6, 2009 at 3:58 PM, Rotem Simha wrote: > * there are some errors in links of files and special pages > examples > קובץ:Nuvola_apps_important.svg [1] link to > ויקיפדיה:מיזמי ויקיפדיה/מיזם ערכים > ללא תמונות/קטגוריות/ספורטאים איטלקים > (wikipedia:wikipedia projects articles without imagescategoriesSports > people from Italy) > מיוחד:אקראי (Special:Random) > 15 במאי (may 15) > מיוחד:שינויים אחרונים (Special:RecentChanges) > > 10_באוגוסט > > * size is important because we intend to add images > > 2009/7/6 > Send dev-l mailing list submissions to > dev-l(a)openzim.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://intern.openzim.org/mailman/listinfo/dev-l [2] > or, via email, send a message with subject or body help to > dev-l-request(a)openzim.org > > You can reach the person managing the list at > dev-l-owner(a)openzim.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of dev-l digest..." > > Todays Topics: > > 1. Kiwix index size (Asaf Bartov) > 2. Re: Kiwix index size (Manuel Schneider) > 3. Re: Kiwix index size (Emmanuel Engelhart) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 5 Jul 2009 19:18:57 +0300 > From: Asaf Bartov > Subject: [openZIM dev-l] Kiwix index size > To: dev-l(a)openzim.org > Message-ID: > > > Content-Type: text/plain; charset="iso-8859-1" > > Hi, everyone. > > When running Kiwixs indexer on the ZIM file I had created from the > Hebrew > Wikipedia last week, the Kiwix data directory ran up to a total of > 31 items, > totalling 2.3 GB. The ZIM file itself is ~300MB. Does this > proportion make > sense? > > Detailed ls output attached. > > Thanks in advance, > > Asaf Bartov > -- > Asaf Bartov >

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Offline-l July 2009