Offline-l

offline-l@lists.wikimedia.org

687 discussions

[openZIM dev-l] Compile error on zimreader
by Andy Rabagliati 12 Jan '10

12 Jan '10

Folks, I hope it is OK if I post here. I have zimlib, zimreader, zimwriter from svn, revision 295. I am compiling from Ubuntu Karmic. I had to install quite a few dev libraries - these are the tnt ones ? aptitude show libtntnet8 libtntdb-dev tntnet-runtime libtntdb1 | egrep '^(Package|State|Version)' Package: libtntnet8 State: installed Version: 1.6.3-5 Package: libtntdb-dev State: installed Version: 1.0.1-4 Package: tntnet-runtime State: installed Version: 1.6.3-5 Package: libtntdb1 State: installed Version: 1.0.1-4 I start with "make clean ; ./autogen.sh; ./configure; make" zimlib compiles fine. zimreader has been coming up with this error the last few times I tried to compile :- ------------------------------- make[2]: Entering directory `/srv/svn/zim/zimreader/src' depbase=`echo zimcomp.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\ g++ -DHAVE_CONFIG_H -I. -I../include -g -O2 -MT zimcomp.o -MD -MP -MF $depbase.Tpo -c -o zimcomp.o zimcomp.cpp &&\ mv -f $depbase.Tpo $depbase.Po zimcomp.ecpp: In member function ‘virtual unsigned int<unnamed>::_component_zimcomp::operator()(tnt::HttpRequest&, tnt::HttpReply&, tnt::QueryParams&)’: zimcomp.ecpp:17: error: no match for ‘operator!=’ in ‘zim::Article::getUrl() const() != pathInfo’ /usr/local/include/zim/qunicode.h:87: note: candidates are: bool zim::QUnicodeString::operator!=(const zim::QUnicodeString&) const zimcomp.ecpp:20: error: no matching function for call to ‘zim::File::getArticle(char&, std::string&)’ /usr/local/include/zim/file.h:51: note: candidates are: zim::Article zim::File::getArticle(zim::size_type) const /usr/local/include/zim/file.h:52: note: zim::Article zim::File::getArticle(char, const zim::QUnicodeString&, bool) zimcomp.ecpp:49: error: no matching function for call to ‘tnt::HttpReply::redirect(zim::QUnicodeString)’ /usr/include/tnt/httpreply.h:77: note: candidates are: unsigned int tnt::HttpReply::redirect(const std::string&) zimcomp.ecpp:58: error: no match for ‘operator=’ in ‘title = zim::Article::getTitle() const()’ /usr/include/c++/4.4/bits/basic_string.h:505: note: candidates are: std::basic_string<_CharT, _Traits, _Alloc>& std::basic_string<_CharT, _Traits, _Alloc>::operator=(const std::basic_string<_CharT, _Traits, _Alloc>&) [with _CharT = char, _Traits = std::char_traits<char>, _Alloc = std::allocator<char>] /usr/include/c++/4.4/bits/basic_string.h:513: note: std::basic_string<_CharT, _Traits, _Alloc>& std::basic_string<_CharT, _Traits, _Alloc>::operator=(const _CharT*) [with _CharT = char, _Traits = std::char_traits<char>, _Alloc = std::allocator<char>] /usr/include/c++/4.4/bits/basic_string.h:524: note: std::basic_string<_CharT, _Traits, _Alloc>& std::basic_string<_CharT, _Traits, _Alloc>::operator=(_CharT) [with _CharT = char, _Traits = std::char_traits<char>, _Alloc = std::allocator<char>] make[2]: *** [zimcomp.o] Error 1 ------------------------------- What am I doing wrong ? Cheers, Andy!

3 2

[openZIM dev-l] [QUESTION] New index by title
by emmanuel＠engelhart.org 11 Jan '10

11 Jan '10

Hi, during the dev. meeting we have decided to have, in addition to the url sorted index, a title sorted index (at least optional). I can't see it in the roadmap. Is something wrong with it? I have started to play with the trunk again. Until now it works perfectly and I was also able to generate new ZIM files. I will introduce the new format in Kiwix trunk and start to look at the Windows portability. Emmanuel

4 3

Re: [openZIM dev-l] [QUESTION] New index by title
by emmanuel＠engelhart.org 06 Jan '10

06 Jan '10

OK, thank you Manuel. This was a misunderstanding by me. I have achieved to have different "title" and "url" per article in my test ZIM file, so the zimwriter deals from now by me correctly with the DB "url" and "title" fields. :) But, I was not able to check the title index: * Do we have a way with zimDump for example to get all article titles starting with "open" for example? * the zimreader has a suggest feature (able to propose articles URLs which start with the first characters you have typed). Where is the zimreader code which does that ? Is it easy to adapt it to do the same with titles? Emmanuel Le mer 06/01/10 11:56, "Manuel Schneider" manuel.schneider(a)wikimedia.ch a écrit: > Hi, > > see the first item on the Roadmap: > http://openzim.org/Roadmap > add Pointer to UrlPointerList (IndexPointerList will be named > "TitlePointerList") > > and > > add UrlPointerList (article list ordered by URL) > > both already done. > > Regards, > > Manuel > > > Am 06.01.2010 11:43, schrieb emmanu > el(a)engelhart.org: > Hi, > > > > during the dev. meeting we have decided to have, in > addition to the url sorted index, a title sorted index (at least > optional). > I can't see it in the roadmap. Is something wrong > with it? > > > I have started to play with the trunk again. Until > now it works perfectly and I was also able to generate new ZIM > files. > I will introduce the new format in Kiwix trunk and > start to look at the Windows portability. > > > Emmanuel > > > > > > > > > _______________________________________________ > dev-l mailing list > > dev-l@openz > im.org > https://intern.openzim.org/mailman/listinfo/dev-l > >> > > -- > Regards > Manuel Schneider > > Wikimedia CH - Verein zur Förderung Freien Wissens > Wikimedia CH - Association for the advancement of free knowledge > www.wikimedia.ch > >

3 2

Re: [openZIM dev-l] [QUESTION] New index by title
by emmanuel＠engelhart.org 06 Jan '10

06 Jan '10

Le mer 06/01/10 14:27, "Tommi Mäkitalo" tommi(a)tntnet.org a écrit: > There is a flag -t in zimDump, which switches to search by title. I just > realized that this is missing on the help page. I will add that. > > To find a article by title use: > zimDump -t -i -f "MyTitle" myzimfile.zim :) I wanted to try the "-t" parameter, but I now don' t understand why the second command does not return the article with idx=4 $zimDump -i -f 8P articles.zim url: 9 title: The Beatles idx: 4 namespace: A redirect: 0 mime-type: 0 article size: 177163 $ zimDump -t -i -f "The Beatles" articles.zim url: 10 title: 10 idx: 9 namespace: I redirect: 0 mime-type: 3 article size: 24152 The ZIM file can be found here: http://tmp.kiwix.org/tmp/articles.zim > To use it in your application use iterators: > > zim::File::const_iterator it = zimFile.find('A', "MyUrl"); > or > zim::File::const_iterator it = zimFile.findByTitle('A', "MyTitle"); > > This will return a iterator to the first article whose url (or title) is > equal or greater than the passed string. You have to compare the iterator against > zimFile.end() to check, if it points really to a valid article. If you look > for a range of articles starting with a specific url (or title) you have to > do that manually e.g.: > zim::File::const_iterator itFirst = zimFile.find('A', "MyUrl"); > zim::File::const_iterator itLast = zimFile.find('A', "MyUrm"); > > or by comparing yourself: > for (zim::File::const_iterator it = zimFile.find('A', "MyUrl"); > it != zimFile.end() && it->getUrl().compare(0, 5, "MyUrl") == 0) > ++it) > { > } Perfect, will test it ASAP. Emmanuel

1 0

[openZIM dev-l] support for bzip2 and zlib compression in zimlib will be dropped
by Tommi Mäkitalo 06 Jan '10

06 Jan '10

Hi, some days ago I talked to Manuel and we decided to drop bzip2 and zlib compression in zimlib. Lzma is successfully ported to the nanonote and we see no reason for supporting other compression methods any more. It will also reduce the depencency to zlib and libbzip2. I will still add a configure switch, where we can enable it but disable it by default. Tommi

1 0

[openZIM dev-l] [Bug 16] New: Version & Subversion ZIM format handling
by bugzilla-daemon＠openzim.org 06 Jan '10

06 Jan '10

http://bugs.openzim.org/show_bug.cgi?id=16 Summary: Version & Subversion ZIM format handling Product: openZIM Version: unspecified Platform: All OS/Version: All Status: NEW Severity: enhancement Priority: P1 Component: zimlib AssignedTo: tommi(a)tntnet.org ReportedBy: emmanuel(a)engelhart.org CC: dev-l(a)openzim.org Estimated Hours: 0.0 Currently, the zimlib is only able to deal with a specific version of the ZIM file format and otherwise generates an error. We want: * to be able to improve as often as possible the format. * to have a zimlib able to deal with a maximum of ZIM files (potentially with different ZIM format versions). ... consequently, the current implementation may be improved. We discussed about that during our last meeting and if I correctly remember the best proposition was IMO: * Introduce the concept of ZIM format subversion. * Zimlib can deal with all subversions of a specific ZIM format version. * If we want to add a feature which does not need to break the format, we increment the subversion counter and if necessary add subversion conditional code. * If we want to add a feature which need to break the backward compatibility, we need to increment the version counter. With such a system, the goal is to make as much as possible subversions and no new version (but this is another topic). -- Configure bugmail: http://bugs.openzim.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

1 1

[openZIM dev-l] LZMA is working now
by Tommi Mäkitalo 31 Dec '09

31 Dec '09

Hi, I talked with the xz developer in IRC and he helped me to reduce memory usage. The problem was, that compressing with lzma with 9 uncompressing needs about 65MB of RAM. The needed memory is in fact dependend of used compression level. Using level 3 reduced the memory usage to about 1,5MB, that it works now on the NanoNote. There is a additional flag LZMA_PRESET_EXTREME, which makes compressing slower but compression ratio better. Using 3 + extreme results in a file, which works on Nanonote while the size is almost identical with 9 or bzip2. Uncompressing bzip2 chunks take about 2 seconds while lzma needs 0,5 seconds on the nanonote. So we keep the size while improving speed with the factor 4!!! Have a good Year. Tommi

3 3

Re: [openZIM dev-l] LZMA is working now
by emmanuel＠engelhart.org 31 Dec '09

31 Dec '09

Hi This is a really good news :) The xz lib can not be compiled with MS cl.exe and with every non C99 compatible C compiler. But, an already compiled library (with mingw) should be usable with cl.exe (please do not ask me how they achieve to do that). These infos are in the README Have to test, but I have no timeline for that yet. Hope have time during january for that task. Emmanuel Le jeu 31/12/09 16:32, "Manuel Schneider" manuel.schneider(a)wikimedia.ch a écrit: > What should I say? > > Great! Very good work done, Tommi! > > So does this mean we can dismiss all other compression algorithms? > > What about the streaming mode when uncompressing? You talked about that > as you said we could improve memory usage on small devices. > > Greetings and a good year to all of us! > > /Manuel > > > Am 31.12.2009 16:03, schrieb Tommi M�kitalo: > > Hi, > > > > I talked with the xz developer in IRC and he helped > me to reduce memory usage. > > > The problem was, that compressing with lzma with 9 > uncompressing needs about > 65MB of RAM. The needed memory is in fact dependend > of used compression level. > Using level 3 reduced the memory usage to about > 1,5MB, that it works now on > the NanoNote. There is a additional flag > LZMA_PRESET_EXTREME, which makes > compressing slower but compression ratio better. > Using 3 + extreme results in > a file, which works on Nanonote while the size is > almost identical with 9 or > bzip2. > > > > Uncompressing bzip2 chunks take about 2 seconds > while lzma needs 0,5 seconds > on the nanonote. So we keep the size while improving > speed with the factor > 4!!! > > > > Have a good Year. > > > > Tommi > > > _______________________________________________ > dev-l mailing list > > dev-l@openz > im.org > https://intern.openzim.org/mailman/listinfo/dev-l > >> > > -- > Regards > Manuel Schneider > > Wikimedia CH - Verein zur F�rderung Freien Wissens > Wikimedia CH - Association for the advancement of free knowledge > www.wikimedia.ch _______________________________________________ > dev-l mailing list > dev-l@openz > im.org https://intern.openzim.org/mailman/listinfo/dev-l > >

1 0

[openZIM dev-l] Software-Entwicklung unter Symbian
by Manuel Schneider 28 Dec '09

28 Dec '09

Hallo Lukas, vielen Dank für Deine Software http://series60-remote.sourceforge.net/ - ich habe Sie auf meinem PC unter Windows 7 und auf meinem Nokia E71-1 installiert und alles hat auf Anhieb funktioniert. Du kannst dieses Gerät also auf Deiner Webseite als "OK" markieren (s. Screenshot im Anhang). Nun zu einem anderen Anliegen: Wir von openZIM würden gerne unsere zimlib nach Symbian portieren. Sie ist in C++ geschrieben und wir haben uns daher auf unserem Entwicklungsserver in eine VM mit Windows das Symbian SDK installiert. Allerdings bin ich auf der Suche nach Dokumentation, Anleitungen etc. da wir noch nie auf Symbian entwickelt haben. Da Du aktiv Software unter Symbian entwickelst denke ich, dass Du sicher über entsprechende Dokumentationen und Erfahrungen verfügst um uns entsprechende Anhaltspunkte zu geben? Kontakt zu einer entsprechenden Entwicklergemeinschaft etc. wären für uns sehr wertvoll, um in diesem für uns noch völlig unbekannten Gebiet die erste Schritte machen zu können. Vielen Dank für Deine Hilfe und Grüsse, Manuel Schneider -- Regards Manuel Schneider Wikimedia CH - Verein zur Förderung Freien Wissens Wikimedia CH - Association for the advancement of free knowledge www.wikimedia.ch

1 0

[openZIM dev-l] update on nanonote
by Tommi Mäkitalo 27 Dec '09

27 Dec '09

Hi, The xz (lzma) library is ported to nanonote. And also my zim benchmark program. Lzma seems to be more problematic than expected. Due to memory limitations zimlib is not able to decompress any data on the nanonote with xz. Even the original command line xz tool fails to run. There is a memory limit parameter in xz and I've played with it. 64M (which is far beyond the available RAM of 32M) is too less for the decompressor and 128M it starts but fails to allocate memory. I sent a mail to the author if there are other tweaks to reduce memory consumption in the decompressor. So I was not able to run any benchmarks with lzma. But I tested with bzip2. The device needs about 2 seconds to decompress the data. With the benchmark program it is no problem to increase the cluster cache size to 8, so that it caches the last 8 clusters. I think a default cluster cache size of 5 would be reasonable for that device. Tommi

1 0

← Newer
1
...
54
55
56
57
58
59
60
...
69
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Offline-l