Offline-l

offline-l@lists.wikimedia.org

7 participants
692 discussions

[openZIM dev-l] Vido
by Mirko Lindner 06 Feb '10

06 Feb '10

Hey there, I just wanted to give you a little progress report on vido [1]. After the openzim developer-meeting [2] quite a few changes were made to the format to allow the usage of zim files even on limited devices such as the Ben NanoNote. Last night, after finally finding the right file, I managed to get the full german wikipedia (without images) to run nicely. The file is on the openzim server in their download section[3]. There are quite a few neat functions in vido. Please be patient with it, responses are not instant, but it does what you ask it after 1-2s at the most. There are still a few quirks here and there. Scrolling is not optimal, the way links can be selected deserves another look etc etc. But the foundation works. One word abotu the zim files. Currently there are only german wiki files that have been tested. I will dive into the mechanism of creating other zim files as soon as I get back from FOSDEM. I have been told there is a process for that already, but it is not well documented. We are also in talks with the Wikimedia Foundation and other wikis to get them to publish wiki files themselves. If anyones feels like diving into the code to make it better, faster, nicer please go ahead. Just contact me for git access and such. Once again a big yay for the openzim project! Ok, here a small list of what vido can do: - open and display contents of a zim file (on the console): vido [PATH TO FILE] - get a random article (in vido): ctrl + r - search for an article (in vido): ctrl + s - go back to last article (in vido): ctrl + b - go to next article (if you went back before) (in vido): ctrl + f - display your history (in vido): ctrl + h - go to top (in vido): ctrl + t To move through links in an article use TAB. If you only want to scroll use the arrow keys. Note: If you scroll the cursor doesn't move, which means that if you tab after you scrolled for a while the display will go back to the last link you selected. /mirko [1] http://projects.qi-hardware.com/index.php/p/vido/ [2] http://openzim.org/Developer_Meetings/2009-2 [3] http://openzim.org/ZIM_File_Archive

2 1

Re: [openZIM dev-l] Vido
by emmanuel＠engelhart.org 03 Feb '10

03 Feb '10

Hi Mirko, this is really a good news. Hope to see a tar.gz soon to test on my computer. Regards Emmanuel Le mer 03/02/10 10:06, "Mirko Lindner" mirko(a)sharism.cc a écrit: > Hey there, > > I just wanted to give you a little progress report on vido [1]. > > After the openzim developer-meeting [2] quite a few changes were made > to the format to allow the usage of zim files even on limited devices > such as the Ben NanoNote. > > Last night, after finally finding the right file, I managed to get > the full german wikipedia (without images) to run nicely. The file is > on the openzim server in their download section[3]. > > There are quite a few neat functions in vido. Please be patient with > it, responses are not instant, but it does what you ask it after 1-2s > at the most. There are still a few quirks here and there. Scrolling is > not optimal, the way links can be selected deserves another look etc > etc. But the foundation works. > > One word abotu the zim files. Currently there are only german wiki > files that have been tested. I will dive into the mechanism of > creating other zim files as soon as I get back from FOSDEM. I have > been told there is a process for that already, but it is not well > documented. We are also in talks with the Wikimedia Foundation and > other wikis to get them to publish wiki files themselves. > > If anyones feels like diving into the code to make it better, faster, > nicer please go ahead. Just contact me for git access and such. > > Once again a big yay for the openzim project! > > Ok, here a small list of what vido can do: > > - open and display contents of a zim file (on the console): > vido [PATH TO FILE] > > - get a random article (in vido): > > ctrl + r > > - search for an article (in vido): > > ctrl + s > > - go back to last article (in vido): > > ctrl + b > > - go to next article (if you went back before) (in vido): > > ctrl + f > > - display your history (in vido): > > ctrl + h > > - go to top (in vido): > > ctrl + t > > To move through links in an article use TAB. If you only want to > scroll use the arrow keys. > > Note: If you scroll the cursor doesnt move, which means that if you > tab after you scrolled for a while the display will go back to the > last link you selected. > > /mirko > > [1] http://projects.qi-hardware.com/index.php/p/vido/ [1] > [2] http://openzim.org/Developer_Meetings/2009-2 [2] > [3] http://openzim.org/ZIM_File_Archive [3] > > > Links: > ------ > [1] http://projects.qi-hardware.com/index.php/p/vido/ > [2] http://openzim.org/Developer_Meetings/2009-2 > [3] http://openzim.org/ZIM_File_Archive > >

1 0

[openZIM dev-l] Changes in zim tools
by Tommi Mäkitalo 18 Jan '10

18 Jan '10

Hi, I implemented some of the changes Emmanuel suggested. The tools are now all lowercase (zimdump, zimsearch, zimreader, zimwriter and zimbench). The mime type is now in a separate table for the zimwriter. The mimetype-table has a column, if the mimetype should be compressed. I also changed the name of the table "zimarticles" to "zimarticle". Tablenames should be singular like "article". It has annoyed me quite some time that the naming was not consistent. There is also a new source for article data in zimwriter. I combined the full text search with the zimwriter. Using e.g.: zimwriter -S film -I wikipedia-de.zim -X wikipedia-de-x.zim film creates a zim file "film.zim" with all articles from wikipedia-de.zim containing the word "film". This is especially useful for me to create smaller subsets from the big wikipedia-de.zim. Tommi

2 1

Re: [openZIM dev-l] [ZIMREADER] remarks and questions
by emmanuel＠engelhart.org 15 Jan '10

15 Jan '10

Le jeu 14/01/10 21:18, "Tommi Mäkitalo" tommi(a)tntnet.org a écrit: > I have to think about the skin. The zimreader needs a user interface and > this is realized as html files. This is what is called "skin". I currently don't > exactly know, how to change that. I need some dynamic html in the > application. Ok, I do not really know in details your opinion about the whole topic... but my feeling is that having the possibility to choose a skin on the command line, would be an improvement. > The content-type is perfectly ok in zimreader. Your zim file is wrong. And > I know also why. The mimetype column in article is now a text instead of a > integer. You have to put the actual mime type into that column. Remeber, > that we do not use a fixed enum for mime types any more but a dynamically built > list inside the zim file. > > This is also the explanation, why your images are slower. They are indeed > clustered and compressed, since the mime type do not match the hard coded > mime type list found in zimwriter/src/zimcreator.cpp (image/jpeg, image/png, > image/tiff, image/gif, application/zip). It is normally not a good idea to > have it hard coded but I currently have no better idea. Thx, it works now and is faster. About the list, I think a "clean" solution would be to have an additional table "mimetypes" with two fields "name (varchar)" and "compressed (bool)". A default strategy (all mimetypes compressed, all not compressed, or a default configuration (like currently) would garanty that the zimwriter can work without entries in this table. Regards Emmanuel

1 0

[openZIM dev-l] [ZIMREADER] remarks and questions
by emmanuel＠engelhart.org 15 Jan '10

15 Jan '10

Hi, About the name first. It's called "ZimReader" which is not homogenous with the other ZIM binaries "zimDump" and "zimWriter"... so it should be IMO "zimWriter". But, in fact, having uppercased character in a binary seems to pretty unusual, so wouldn't be "zimwriter", "zimdump" and "zimreader" better? About the default skin, is that possible that currently, per default there is no skin anymore? I had before to tweak the skin to remove the german wikipedia UI and this seems not being necessary anymore! If that's true, great! My feeling is that the zimreader, with the new format is not able anymore to determine correctly the HTTP content-type. The HTTP header for an HTML page looks like following: (...) Keep-Alive: timeout=15000, max=999 Connection: Keep-Alive Last-Modified: Tue, 12 Jan 2010 21:01:19 GMT Content-Type: 0 Set-Cookie: tntnet.=8649805af3b641b7d9f7dbf3193b288d; Path=/;Version=1 (...) The Content-type should be different.... isn't it? Regards Emmanuel

2 1

[openZIM dev-l] Compile error on zimreader
by Andy Rabagliati 13 Jan '10

13 Jan '10

Folks, I hope it is OK if I post here. I have zimlib, zimreader, zimwriter from svn, revision 295. I am compiling from Ubuntu Karmic. I had to install quite a few dev libraries - these are the tnt ones ? aptitude show libtntnet8 libtntdb-dev tntnet-runtime libtntdb1 | egrep '^(Package|State|Version)' Package: libtntnet8 State: installed Version: 1.6.3-5 Package: libtntdb-dev State: installed Version: 1.0.1-4 Package: tntnet-runtime State: installed Version: 1.6.3-5 Package: libtntdb1 State: installed Version: 1.0.1-4 I start with "make clean ; ./autogen.sh; ./configure; make" zimlib compiles fine. zimreader has been coming up with this error the last few times I tried to compile :- ------------------------------- make[2]: Entering directory `/srv/svn/zim/zimreader/src' depbase=`echo zimcomp.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\ g++ -DHAVE_CONFIG_H -I. -I../include -g -O2 -MT zimcomp.o -MD -MP -MF $depbase.Tpo -c -o zimcomp.o zimcomp.cpp &&\ mv -f $depbase.Tpo $depbase.Po zimcomp.ecpp: In member function ‘virtual unsigned int<unnamed>::_component_zimcomp::operator()(tnt::HttpRequest&, tnt::HttpReply&, tnt::QueryParams&)’: zimcomp.ecpp:17: error: no match for ‘operator!=’ in ‘zim::Article::getUrl() const() != pathInfo’ /usr/local/include/zim/qunicode.h:87: note: candidates are: bool zim::QUnicodeString::operator!=(const zim::QUnicodeString&) const zimcomp.ecpp:20: error: no matching function for call to ‘zim::File::getArticle(char&, std::string&)’ /usr/local/include/zim/file.h:51: note: candidates are: zim::Article zim::File::getArticle(zim::size_type) const /usr/local/include/zim/file.h:52: note: zim::Article zim::File::getArticle(char, const zim::QUnicodeString&, bool) zimcomp.ecpp:49: error: no matching function for call to ‘tnt::HttpReply::redirect(zim::QUnicodeString)’ /usr/include/tnt/httpreply.h:77: note: candidates are: unsigned int tnt::HttpReply::redirect(const std::string&) zimcomp.ecpp:58: error: no match for ‘operator=’ in ‘title = zim::Article::getTitle() const()’ /usr/include/c++/4.4/bits/basic_string.h:505: note: candidates are: std::basic_string<_CharT, _Traits, _Alloc>& std::basic_string<_CharT, _Traits, _Alloc>::operator=(const std::basic_string<_CharT, _Traits, _Alloc>&) [with _CharT = char, _Traits = std::char_traits<char>, _Alloc = std::allocator<char>] /usr/include/c++/4.4/bits/basic_string.h:513: note: std::basic_string<_CharT, _Traits, _Alloc>& std::basic_string<_CharT, _Traits, _Alloc>::operator=(const _CharT*) [with _CharT = char, _Traits = std::char_traits<char>, _Alloc = std::allocator<char>] /usr/include/c++/4.4/bits/basic_string.h:524: note: std::basic_string<_CharT, _Traits, _Alloc>& std::basic_string<_CharT, _Traits, _Alloc>::operator=(_CharT) [with _CharT = char, _Traits = std::char_traits<char>, _Alloc = std::allocator<char>] make[2]: *** [zimcomp.o] Error 1 ------------------------------- What am I doing wrong ? Cheers, Andy!

3 2

[openZIM dev-l] [QUESTION] New index by title
by emmanuel＠engelhart.org 12 Jan '10

12 Jan '10

Hi, during the dev. meeting we have decided to have, in addition to the url sorted index, a title sorted index (at least optional). I can't see it in the roadmap. Is something wrong with it? I have started to play with the trunk again. Until now it works perfectly and I was also able to generate new ZIM files. I will introduce the new format in Kiwix trunk and start to look at the Windows portability. Emmanuel

4 3

Re: [openZIM dev-l] [QUESTION] New index by title
by emmanuel＠engelhart.org 07 Jan '10

07 Jan '10

OK, thank you Manuel. This was a misunderstanding by me. I have achieved to have different "title" and "url" per article in my test ZIM file, so the zimwriter deals from now by me correctly with the DB "url" and "title" fields. :) But, I was not able to check the title index: * Do we have a way with zimDump for example to get all article titles starting with "open" for example? * the zimreader has a suggest feature (able to propose articles URLs which start with the first characters you have typed). Where is the zimreader code which does that ? Is it easy to adapt it to do the same with titles? Emmanuel Le mer 06/01/10 11:56, "Manuel Schneider" manuel.schneider(a)wikimedia.ch a écrit: > Hi, > > see the first item on the Roadmap: > http://openzim.org/Roadmap > add Pointer to UrlPointerList (IndexPointerList will be named > "TitlePointerList") > > and > > add UrlPointerList (article list ordered by URL) > > both already done. > > Regards, > > Manuel > > > Am 06.01.2010 11:43, schrieb emmanu > el(a)engelhart.org: > Hi, > > > > during the dev. meeting we have decided to have, in > addition to the url sorted index, a title sorted index (at least > optional). > I can't see it in the roadmap. Is something wrong > with it? > > > I have started to play with the trunk again. Until > now it works perfectly and I was also able to generate new ZIM > files. > I will introduce the new format in Kiwix trunk and > start to look at the Windows portability. > > > Emmanuel > > > > > > > > > _______________________________________________ > dev-l mailing list > > dev-l@openz > im.org > https://intern.openzim.org/mailman/listinfo/dev-l > >> > > -- > Regards > Manuel Schneider > > Wikimedia CH - Verein zur Förderung Freien Wissens > Wikimedia CH - Association for the advancement of free knowledge > www.wikimedia.ch > >

3 2

Re: [openZIM dev-l] [QUESTION] New index by title
by emmanuel＠engelhart.org 07 Jan '10

07 Jan '10

Le mer 06/01/10 14:27, "Tommi Mäkitalo" tommi(a)tntnet.org a écrit: > There is a flag -t in zimDump, which switches to search by title. I just > realized that this is missing on the help page. I will add that. > > To find a article by title use: > zimDump -t -i -f "MyTitle" myzimfile.zim :) I wanted to try the "-t" parameter, but I now don' t understand why the second command does not return the article with idx=4 $zimDump -i -f 8P articles.zim url: 9 title: The Beatles idx: 4 namespace: A redirect: 0 mime-type: 0 article size: 177163 $ zimDump -t -i -f "The Beatles" articles.zim url: 10 title: 10 idx: 9 namespace: I redirect: 0 mime-type: 3 article size: 24152 The ZIM file can be found here: http://tmp.kiwix.org/tmp/articles.zim > To use it in your application use iterators: > > zim::File::const_iterator it = zimFile.find('A', "MyUrl"); > or > zim::File::const_iterator it = zimFile.findByTitle('A', "MyTitle"); > > This will return a iterator to the first article whose url (or title) is > equal or greater than the passed string. You have to compare the iterator against > zimFile.end() to check, if it points really to a valid article. If you look > for a range of articles starting with a specific url (or title) you have to > do that manually e.g.: > zim::File::const_iterator itFirst = zimFile.find('A', "MyUrl"); > zim::File::const_iterator itLast = zimFile.find('A', "MyUrm"); > > or by comparing yourself: > for (zim::File::const_iterator it = zimFile.find('A', "MyUrl"); > it != zimFile.end() && it->getUrl().compare(0, 5, "MyUrl") == 0) > ++it) > { > } Perfect, will test it ASAP. Emmanuel

1 0

[openZIM dev-l] support for bzip2 and zlib compression in zimlib will be dropped
by Tommi Mäkitalo 07 Jan '10

07 Jan '10

Hi, some days ago I talked to Manuel and we decided to drop bzip2 and zlib compression in zimlib. Lzma is successfully ported to the nanonote and we see no reason for supporting other compression methods any more. It will also reduce the depencency to zlib and libbzip2. I will still add a configure switch, where we can enable it but disable it by default. Tommi

1 0

← Newer
1
...
54
55
56
57
58
59
60
...
70
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Offline-l