Offline-l

offline-l@lists.wikimedia.org

5 participants
691 discussions

[openZIM dev-l] another update in zim format
by Tommi Mäkitalo 13 Dec '09

13 Dec '09

Hi, a few days ago I realized, that I was not done with the zim format update. The mime types were still hard coded. But now I'm through. I double checked that with my own protocol of our developers meeting. Now the zim file contains a list of contained mime types and the directory entry has an index to that list. Since the index is 16 bit and the mime type 0xffff specifies redirect, a zim file can now have up to 65535 distinct mime types. In zimwriter the database source is changed, so that the mime type in the article table is not a number any more but a text. Next task I have to do is to document the changes. Manuel already installed the xz-utils package on our server so that I can start implementing lzma compression then. Tommi

1 0

[openZIM dev-l] zim2 branch is working yet
by Tommi Maekitalo 11 Dec '09

11 Dec '09

Hi, the changes, we discussed at our devolpers meeting are implemented in zimlib and zimwriter. I was already able to create the wikipedia-de.zim and the full text index wikipedia-de-x.zim. Both files are slightly larger due to the additional title including the index for it. The zint compression is also changed. As announced it is similar to utf-8. The full text index data is slightly smaller. I did not use this zint compression in the directory entry since it is really not worth the trouble. I prefered simpler directory entries to make implementation of alternatives (Java, C#) simpler. As Manuel already mentioned I changed the way, zim files are opened. The zimlib read the whole pointer list into memory when the file was opened. This is not done any more. The memory usage is now reduced quite significantly, so that I can do a full text search even on the Nanonote with 32MB RAM. I implemented this also in trunk. Currently I'm working on porting zimreader to the new library. Porting is necessary since we decided to drop th qunicode feature. But it is still very simple since I just need to replace all references to qunicode strings with simple strings. Also I'm looking at the API of lzma. The documentation is quite confusing, but I'm making some progress here also. I promised to implement the file changes until end of this year and since I'm actually through, maybe I can add lzma compression also. I also decide to drop support for zlib and bzip2 compression as soon as lzma is working. This reduces the external dependencies. I see no advantage in supporting multiple compression methods. What do you think? Tommi

1 0

Re: [openZIM dev-l] [Argentina] WikiBrowse improvements
by emmanuel＠engelhart.org 08 Dec '09

08 Dec '09

Le mer 02/12/09 21:32, "Madeleine Price Ball" meprice(a)fas.harvard.edu a écrit: > > We think that the specific knowledge of the > publishers should be how to > select the content - which content goes where in > which form - and not > technical questions such as compression, storage or > retrieving the data on > the user's end. > > OK, if I shouldn't be talking to you guys, tell me who to talk to. > > Yes, selecting content is very difficult. I couldn't get Peru or SJ to > contribute meaningfully to generating a simple blacklist of articles > that should NOT be included on the OLPC activity. (Recall it is being > given to young children!) I ended up making the blacklist myself based > on my own gut feelings. If Peru's board of education or OLPC's > "director of content" couldn't get their act together for this simple > task, expecting others to do this task for you will be a huge > roadblock to getting content out. > > Traffic based content is simple and effective and it doesn't involve a > lot of opinions on what should or should not be included. Yes, compiling such stats with incoming link counter & interwiki counter, it is possible to get a pretty accurate and "neutral" selection quickly for any Wikipedia. I have scripts to do that automaticaly: http://kiwix.svn.sourceforge.net/viewvc/kiwix/selection_tools/ Emmanuel

1 0

Re: [openZIM dev-l] new branch for openzim created
by emmanuel＠engelhart.org 08 Dec '09

08 Dec '09

Le dim 29/11/09 18:31, "Tommi Mäkitalo" tommi(a)tntnet.org a écrit: > We already discussed about using it in the directory entry also to save > some bytes. I'm not sure if it worth the trouble. It makes alternative > implementations more difficult. If alternative implementations do not care > about full text index or categories, they don't need to implement the zint > compression. On the other hand the code for it is straight forward and > should ne easy to port. I have no particular opinion about that. > The branch is not working yet. It may take some time until it will since I > have to implement the changes in the reader as well as the writer before I > can even start testing (other than verify, that it compiles). It was easier > with zeno when I started, since I had already working zeno files. Ok, this is good to have done like that, so we have a svn trunk which continues to compile&run. Thx Emmanuel

1 0

[openZIM dev-l] [Fwd: [Wikimedia CH Board] Wikiwix und Linterweb fördern die aktuellen Spendenaktionen der lokalen Wikimedia-Vereine:]
by Manuel Schneider 08 Dec '09

08 Dec '09

FYI. As a member of the Wikimedia community I am very pleased to read that. And as Wikimedia CH is sponsoring openZIM I say thank you on behalf of the openZIM team. /Manuel -------- Original-Nachricht -------- Betreff: [Wikimedia CH Board] Wikiwix und Linterweb fördern die aktuellen Spendenaktionen der lokalen Wikimedia-Vereine: Datum: Fri, 4 Dec 2009 08:23:36 +0100 Von: Nando Stöcklin <nando.stoecklin(a)gmail.com> Antwort an: Board list <board(a)wikimedia.ch> An: Board list <board(a)wikimedia.ch> http://blog.wikiwix.com/de/2009/12/03/wikiwix-et-linterweb-soutiennent-les-… Gruss, Nando -- Regards Manuel Schneider Wikimedia CH - Verein zur Förderung Freien Wissens Wikimedia CH - Association for the advancement of free knowledge www.wikimedia.ch _______________________________________________ Board mailing list Board(a)wikimedia.ch http://lists.wikimedia.ch/listinfo/board

2 2

Re: [openZIM dev-l] [Argentina] WikiBrowse improvements
by Manuel Schneider 02 Dec '09

02 Dec '09

Hi Madeleine, Madeleine Price Ball schrieb: > I was curious if you include images? If not, are you considering doing > so, and what's stopping you? If so, how do you pick them? we haven't done this on our "Test DVD" this summer even though this is easily possible. Well, easily means: It is easy for the format, but the problem is to choose the images. Emmanuel Engelhart (Kiwix, he is also part of the openZIM team) has made some perl scripts for that. We didn't do it for two reasons: Lack of time, because even though the tools exist, it's a lot of work. Searching through the articles, get all the image URLs, get the images, decide in which size to resize them etc... And the openZIM project is not a publisher of offline content. We are developing a stable, efficient format allowing free interchange of contents between reader applications and devices and providing a GPL'ed sample implementation of it. > I did the work for picking which articles & images went into the XO > activity, based on traffic stats. (We only had 100MB, we got 24k > articles in 80MB and spent the other 20MB on highly compressed > images.) OLPC has other more critical things to worry about these > days, but some of the volunteers who worked on that project might be > interested in helping others. Well, we had a "Offline Meeting" at Wikimania in Buenos Aires this summer where Samuel Klein was also participating. Our goal is to contribute the right technology to make all the offline projects able to collaborate. Currently everyone is reinventing the wheel when it comes to storage of the content. We think that the specific knowledge of the publishers should be how to select the content - which content goes where in which form - and not technical questions such as compression, storage or retrieving the data on the user's end. Wikimedia Foundation is supporting us so far as they share our goal and work on a regular export of all Wikimedia wikis into ZIM format (like you already get SQL and XML dumps on download.wikimedia.org). They also have a lots of contacts whith publishers and other projects working on Wikipedia Offline and connect them with us, so we can improve ZIM to fit for all the Wikipedia Offline projects. That's why we defined a new format version at the last Developers Meeting. Greets, Manuel -- Regards Manuel Schneider Wikimedia CH - Verein zur Förderung Freien Wissens Wikimedia CH - Association for the advancement of free knowledge www.wikimedia.ch

2 2

Re: [openZIM dev-l] [Argentina] WikiBrowse improvements
by Samuel Klein 02 Dec '09

02 Dec '09

Jimbo - thanks for the spur to clean up the existing work. All - Let's start by cleaning up the mailing lists and setting a few short-term goals :-) It's a good sign that we have both charity and love converging to make something happen. * For all-platform all-purpose wikireaders, let's use offline-l(a)lists.wikimedia, as we discussed a month ago in the aftermath of Wikimania (Erik, were you going to set this up? I think we agreed to deprecate wiki-offline-reader-l and replace it with offline-l.) * For wikireaders such as WikiBrowse and Infoslicer on the XO, please continue to use wikireader(a)lists.laptop I would like to see WikiBrowse become the 'sugarized' version of a reader that combines the best of that and the openZim work. A standalone DVD or USB drive that comes with its own search tools would be another version of the same. As far as merging codebases goes, I don't think the WikiBrowse developers are invested in the name. I think we have a good first cut at selecting articles, weeding out stubs, and including thumbnail images. Maybe someone working on openZim can suggest how to merge the search processes, and that file format seems unambiguously better. Kul - perhaps part of the work you've been helping along for standalone usb-key snapshots would be useful here. Please continue to update this page with your thoughts and progress! http://meta.wikimedia.org/wiki/Offline_readers SJ 2009/10/23 Iris Fernández <irisfernandez(a)gmail.com> > On Fri, Oct 23, 2009 at 1:37 PM, Jimmy Wales <jwales(a)wikia-inc.com> wrote: > > > > My dream is quite simple: a DVD that can be shipped to millions of people > with an all-free-software solution for reading Wikipedia in Spanish. It > should have a decent search solution, doesn't have to be perfect, but it > should be full-text. It should be reasonably fast, but super-perfect is not > a consideration. > > > > Hello! I am an educator, not a programmer. I can help selecting > articles or developing categories related to school issues. > Iris - you know the main page of WikiBrowse that you see when the reader first loads? You could help with a new version of that page. Madeleine (copied here) worked on the first one, but your thoughts on improving it would be welcome.

7 8

[openZIM dev-l] new branch for openzim created
by Tommi Mäkitalo 29 Nov '09

29 Nov '09

Hi, I just created a branch for the changes in zim format, we discussed at our developer meeting. The branch is located in our repository under branches/zim2. I already checked in first changes. qunicode is removed from zimlib. Also distinqushing between url and title is started. I also decided to change the zint compression. The compression is now similar to utf-8 and saves a little space compared to the old algorithm (about 2%). The zint compression is used in the full text index currently. We already discussed about using it in the directory entry also to save some bytes. I'm not sure if it worth the trouble. It makes alternative implementations more difficult. If alternative implementations do not care about full text index or categories, they don't need to implement the zint compression. On the other hand the code for it is straight forward and should be easy to port. The branch is not working yet. It may take some time until it will since I have to implement the changes in the reader as well as the writer before I can even start testing (other than verify, that it compiles). It was easier with zeno when I started, since I had already working zeno files. Tommi

1 0

Re: [openZIM dev-l] zeno, zim formats
by emmanuel＠engelhart.org 20 Nov '09

20 Nov '09

Le ven 20/11/09 11:11, "Andy Rabagliati" andyr(a)wizzy.com a écrit: > On Mon, 16 Nov 2009, Emmanuel Engelhart wrote: > > > > These indexes http://ai.cs.utsa.edu/wikipedia0.7/ seem to have > been built > > using categories. > > > > This dump is one I have build (maybe extract from > the ZIM)... but a > little bit modified. This a pretty interesting url, > would be great to > know how the dev. behind have done exactly... maybe > you would be able to > do the same. > > This is his explanation :- > > > This collection of articles is called "beta2" because they were > extracted from wikipedia_en_wp1_0.7_30000+_05_2009_beta2.zim at > tmp.kiwix.org. My distribution has different file names for all the > articles based on their titles and has a title search capability that > only depends on Javascript. Below is some explanation of the steps I > performed, not necessarily done in this order. > > 1. I extracted the articles from the zim file by downloading, > compiling and running zimDump from openzim.org. Compiling zimDump is > nontrivial because it involves downloading and compiling other > packages in versions that will work together. > > 2. I created three lists with Perl scripts and manual cleaning > afterwards. > > A. The first list was a list of all articles: zim file name, UTF8 > title, and ASCII title. > > B. The second list was a list of all zim files (articles, images and > other files, Javascript and CSS): zim file name and target file name > in my distribution. > > C. The third list was a list for redirecting one zim file name to > another. The zim dump creates a lot of empty files in the A > subdirectory (A contains all the articles). It turns out that each > of them needs to be redirected to another article. The redirects > can be determined by downloading and running the zimReader program > for Linux, which can be found at openzim.org. > > There appear to be a few duplicate articles (none were deleted), which > I list below (in ASCII) for anyone who is interested: > > Abu Rayhan Biruni > 'Alawi > Battle of Mohacs > Beer-Lambert law > Charismatic movement > Elian Gonzalez affair > Ismail Enver > Ismet Inonu > Istiklal Marsi > Izmir Province > Wikipedia:0.7/0.7geo/Leopold > Macapa or Macapai > Maceia or Maceio > Nicole Vaidisova > PRIDE Fighting Championships > War in Afghanistan (2001-present) > > 3. I used a Perl script to copy all the files from the zim dump to a > staging area, modifying the links along the way. There are many, many > dead image links (26314 in my count); I changed those links to empty > strings. There are also some dead article links, most of them > correspond to dead image links, but a few of them should have been > redirected; they got added to my third list above. Here are all the > dead article links and any appropriate redirect for anyone who is > interested. > > A/5ISM ignore > A/35A A/D6N > A/53Z A/CWO > A/5J03 ignore > A/5J55 ignore > A/9XO A/HQO > A/APD A/A35 > A/D07 A/9PW > A/F5G A/163K > A/PRL ignore > A/TKV A/S4X > A/TR4 ignore > A/VBB ignore > A/ZM2 ignore > A/2QE6 ignore > A/T3B A/NQR > A/11B0 ignore > A/1NN2 ignore > A/5ISU A/4O > A/5JAZ ignore > A/5IV3 ignore > A/5IXU ignore > A/102Z ignore > A/1QTM ignore > A/5IOB ignore > A/5J51 ignore > A/5IP6 ignore > A/5JBO ignore > A/Y91 ignore > > 4. In addition to changing links, I made a few other changes. Each > article now has a search box for title search. I took some existing > GPLed Javascript (JSE search engine) and made extensive modifications > for this application. It only searches the titles; there is no > keyword index, and there is no text search. The motto of the code is > "Linear Search FTW". It is surprisingly snappy, though in hindsight, > searching 30000 titles is not a lot for a computer to do. The results > page is functional, but otherwise not too exciting. > > I changed the titles of the index pages to something less geeky, e.g.. > "Topical Index: Wikipedia" for the topic index page on Wikipedia. I > also fixed a number of incorrect links to the topical index page to > alphabetical index page. > > Enjoy, > > Tom Bylander Thank you very much Andy for having forwarded this email, this is really interesting. That's confirm a few things: * The wikipedia_en_wp1_0.7_30000+_05_2009_beta2 is only a beta and should be improved and I will do it soon. * I need to code a perl script to check many thing in a HTML directory (https://sourceforge.net/tracker/?func=detail&aid=2901059&group_id=175508&at…) before building the ZIM * I think we need such a tool (zim-check?) in C++ coded to be able to do the same with ZIM files (I see that pretty necessary if we want to setup a ZIM Library: nobody want that we spread bad quality ZIM) http://bugs.openzim.org/show_bug.cgi?id=14 As soon as I will have finished with that stuff I will publish a new version of the ZIM file and contact Tom. Regards Emmanuel

1 0

[openZIM dev-l] zeno, zim formats
by Andy Rabagliati 20 Nov '09

20 Nov '09

Folks, Coming here by way of the Wikipedia:Version 1.0 Editorial Team pages, I have a few questions. Disclaimer :- I am in South Africa, where bandwidth is expensive, so a lot of my 'fiddling' is done on my server in the USA, where I do not have b/w limitations. My interest is to be able to serve wikipedia content off a server in a classroom - usually a Thin Client setup, but not always, and they rarely have internet access. Weak, small clients, networked to a strong server, running linux. I downloaded a torrent - en.wikipedia.okawix - which appeared to be a tarball with these contents :- -rw-r--r-- 1 andyr andyr 571428918 Jul 6 23:24 article.index -rw-r--r-- 1 andyr andyr 108812163 Jul 6 23:24 article.map -rw-r--r-- 1 andyr andyr 5601891132 Jul 6 23:23 en.wikipedia.zeno -rw-r--r-- 1 andyr andyr 11 Jun 23 18:58 entry.url -rw-r--r-- 1 andyr andyr 307 Jul 6 23:23 licence.xml -rw-r--r-- 1 andyr andyr 596303890 Jul 6 23:25 word.index -rw-r--r-- 1 andyr andyr 108108941 Jul 6 23:25 word.map I see a zeno file - a precursor to the zim format ? The ZimReader static binary from http://openzim.org/Releases dated 2009-06-07 (same date as the files above ?) will not read the file. Am I doing this wrong ? Secondly, I downloaded wikipedia_en_wp1_0.7_30000+_05_2009_beta2.zim and pointed the ZimReader at this - hoping to browse to server:8080 and read the file. But I have no index, and no search - do I need to get the index files separately ? And the css files (?) are in German - do I need to compile my own reader from svn to get an English reader ? Can I build a deb (for Ubuntu ?) Cheers, Andy!

4 10

← Newer
1
...
56
57
58
59
60
61
62
...
70
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Offline-l