Hello,
I am working on a search engine (unlike sphinx or elastic search, more like bing or google), I was planning to use .zim files to feed the index, the problem is there is no systematic way to find the original URL of the documents.
I am wondering whether one of the following will be possible for kiwix project to do:
A) Add a <meta url="https://foobar"> in the html inside the .zim files,
A bis) Add a metadata field per document with the original url inside the .zim files,
B) Publish .warc files of wikipedia, stackoverflow dumps etc... so that people like myself can re-use those. WARC files are more useful than .zim files but still less user friendly than the following proposal...
C) ... One last alternative, is to pivot the custom .zim file storage to an okvs [0] like rocksdb or sqlite lsm extension [1]. The idea is to make it very easy to access the kiwix dumps from many programming languages unlike the current approach that is limited to C++ and Python. Also, it will be easier to extend a given dump with custom fields, unlike the current .zim which seems to be read-only.
Let me know what you think :-)
Thanks in advance!
[0] https://en.wikipedia.org/wiki/Ordered_Key-Value_Store [1] https://github.com/sqlite/sqlite/tree/master/ext/lsm1
Le jeu. 29 oct. 2020 à 12:14, Emmanuel Engelhart kelson@kiwix.org a écrit :
Hi
I'm very proud to announce the release of our new tool: warc2zim.
Warc2zim is a command line tool for GNU/Linux and macOS which allows to convert a WARC file to a ZIM file. WARC being a widely used storage format of the archive world, warc2zim offers new opportunities to reuse WARC stored data and benefit of the whole feature set of the ZIM file format and readers like Kiwix.
The tool has been achieved with the strong collaboration of the Webrecorder team. It is one milestone of a bigger project called Zimit, a project we run we the sponsoring of the Mozilla Foundation.
The ZIM created using that process works slightly differently than the traditional ones (the ZIM specification is formally respected). We are currently running an effort to update all the Kiwix readers, but it already works well with Kiwix Serve.
The tool is distributed at: https://pypi.org/project/warc2zim/
More news to come about warc2zim and Zimit in January 2020.
Happy scraping! Happy coding! Happy offline reading!
Emmanuel
-- Kiwix - Wikipedia Offline & more
- Web: https://kiwix.org/
- Twitter: https://twitter.com/KiwixOffline
- Wiki: https://wiki.kiwix.org/
Offline-l mailing list Offline-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/offline-l