Dear Wikimedians,
I'm sharing with all of you a digest email from the "offline-l list" to which I subscribe since I believe it has two items of wider interest than the usual ones on that list.
The first one concerns a new facility called Open ZIMfarm which automates the curation of offline archives of web material (not just wikis). I believe this is bringing a step-change in the possibilities of decentralised content hosting.
The second describes the recent deployment of the Kiwix-serve application in a commercial telecoms network in West Africa (Kiwix stores and presents the very same ZIM files created by ZIMfarm). Given that the Wikimedia Foundation no longer funds zero-rating of its products, this represents a new way to bring content to people free of charge. Not only cellular networks but local-government-supported WiFi providers such as Project Isizwe as well as community-owned and -operated networks can do this.
I will be interested to know the feelings of the community on these.
Regards,
---------- Forwarded message --------- From: offline-l-request@lists.wikimedia.org Date: Wed, Jun 24, 2020 at 6:01 PM Subject: Offline-l Digest, Vol 100, Issue 10 To: offline-l@lists.wikimedia.org
Send Offline-l mailing list submissions to offline-l@lists.wikimedia.org
To subscribe or unsubscribe via the World Wide Web, visit https://lists.wikimedia.org/mailman/listinfo/offline-l or, via email, send a message with subject or body 'help' to offline-l-request@lists.wikimedia.org
You can reach the person managing the list at offline-l-owner@lists.wikimedia.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Offline-l digest..."
Today's Topics:
1. Re: [KIWIX] Our openZIM farm... (Samuel Klein) 2. Re: [KIWIX] Our openZIM farm... (Emmanuel Engelhart) 3. [AAR] Interesting online/offline use case across West Africa (Stephane Coillet-Matillon) 4. Re: [AAR] Interesting online/offline use case across West Africa (Federico Leva (Nemo))
----------------------------------------------------------------------
Message: 1 Date: Wed, 24 Jun 2020 10:09:04 -0400 From: Samuel Klein meta.sj@gmail.com To: Using Wikimedia projects and MediaWiki offline offline-l@lists.wikimedia.org Subject: Re: [Offline-l] [KIWIX] Our openZIM farm... Message-ID: CAAtU9WJ14yPAJqikYqownk2Nh1mBnStC5TpROU8h+iJcDfuGMQ@mail.gmail.com Content-Type: text/plain; charset="utf-8"
Wow, this is fabulous. If a new zimfarm starts up, can it coordinate with existing ones?
On Tue, Jun 23, 2020 at 3:23 AM Emmanuel Engelhart kelson@kiwix.org wrote:
Hi
There is a topic I wanted to talk about here for a long time and for which I never have achieved to take the time to write something. A few recent events have been a healthy remember that I should present one our most recent and most useful tool: Zimfarm.
The Zimfarm is the online tool which is in charge of building and publishing all our ZIM files. After years of creating ZIM files by launching scrapers more or less manually, we had to automatise the process to just be able to scale the operations, ie. publishing more and more often ZIM files.
The effort started 3 years ago with the support of the WMF but we use it only since Spring 2019 in production. The tool is now perfectly running and we fully rely on it now. If we can publish an update of all our wikis one time a month, this is thanks to this piece of software too.
The Zimfarm is a half-decentralized solution which has a central node (called "dispatcher") in charge of orchestrating the work to do and multiple decentralized nodes (called "workers") which run the scraping tasks.
The dispatcher provides an API to manage the ZIM recipes and tasks, have a look to https://api.farm.openzim.org/. We have setup a Web frontend on this API to allow easy mgmt through a Web browser. For a better transparency, even anonymous users can have a look and monitor what is going on. Look at https://farm.openzim.org/.
One important point is that, like all the rest of our infrastructure, the whole system is Dockerized. Which means, this is really easy to install a Zimfarm worker and we invite anybody having a spare server to help us to provide offline snapshots of the best of the Web. The procedure is documented and a few volunteers have already joined in. Look at https://farm.openzim.org/about for more details.
The development is fully transparent at https://github.com/openzim/zimfarm. We have a few things which are on the roadmap which would welcome volunteer Python developers. Look at the good first issues and make your first PR!
https://github.com/openzim/zimfarm/issues?q=is%3Aissue+is%3Aopen+label%3A%22...
Regards Emmanuel
-- Kiwix - Wikipedia Offline & more
- Web: https://kiwix.org/
- Twitter: https://twitter.com/KiwixOffline
- Wiki: https://wiki.kiwix.org/
Offline-l mailing list Offline-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/offline-l
I am not an expert, but from what I understand it looks like a good thing for those who cannot or will not edit. It will not help build the encyclopaedias, but at least could make what exists more accessible. Cheers, Peter
-----Original Message----- From: WikimediaZA [mailto:wikimediaza-bounces@lists.wikimedia.org] On Behalf Of Michael Graaf Sent: 24 June 2020 23:57 To: wikimediaza@lists.wikimedia.org Subject: [Wikimedia ZA] Fwd: Offline-l Digest, Vol 100, Issue 10
Dear Wikimedians,
I'm sharing with all of you a digest email from the "offline-l list" to which I subscribe since I believe it has two items of wider interest than the usual ones on that list.
The first one concerns a new facility called Open ZIMfarm which automates the curation of offline archives of web material (not just wikis). I believe this is bringing a step-change in the possibilities of decentralised content hosting.
The second describes the recent deployment of the Kiwix-serve application in a commercial telecoms network in West Africa (Kiwix stores and presents the very same ZIM files created by ZIMfarm). Given that the Wikimedia Foundation no longer funds zero-rating of its products, this represents a new way to bring content to people free of charge. Not only cellular networks but local-government-supported WiFi providers such as Project Isizwe as well as community-owned and -operated networks can do this.
I will be interested to know the feelings of the community on these.
Regards,
---------- Forwarded message --------- From: offline-l-request@lists.wikimedia.org Date: Wed, Jun 24, 2020 at 6:01 PM Subject: Offline-l Digest, Vol 100, Issue 10 To: offline-l@lists.wikimedia.org
Send Offline-l mailing list submissions to offline-l@lists.wikimedia.org
To subscribe or unsubscribe via the World Wide Web, visit https://lists.wikimedia.org/mailman/listinfo/offline-l or, via email, send a message with subject or body 'help' to offline-l-request@lists.wikimedia.org
You can reach the person managing the list at offline-l-owner@lists.wikimedia.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Offline-l digest..."
Today's Topics:
1. Re: [KIWIX] Our openZIM farm... (Samuel Klein) 2. Re: [KIWIX] Our openZIM farm... (Emmanuel Engelhart) 3. [AAR] Interesting online/offline use case across West Africa (Stephane Coillet-Matillon) 4. Re: [AAR] Interesting online/offline use case across West Africa (Federico Leva (Nemo))
----------------------------------------------------------------------
Message: 1 Date: Wed, 24 Jun 2020 10:09:04 -0400 From: Samuel Klein meta.sj@gmail.com To: Using Wikimedia projects and MediaWiki offline offline-l@lists.wikimedia.org Subject: Re: [Offline-l] [KIWIX] Our openZIM farm... Message-ID: CAAtU9WJ14yPAJqikYqownk2Nh1mBnStC5TpROU8h+iJcDfuGMQ@mail.gmail.com Content-Type: text/plain; charset="utf-8"
Wow, this is fabulous. If a new zimfarm starts up, can it coordinate with existing ones?
On Tue, Jun 23, 2020 at 3:23 AM Emmanuel Engelhart kelson@kiwix.org wrote:
Hi
There is a topic I wanted to talk about here for a long time and for which I never have achieved to take the time to write something. A few recent events have been a healthy remember that I should present one our most recent and most useful tool: Zimfarm.
The Zimfarm is the online tool which is in charge of building and publishing all our ZIM files. After years of creating ZIM files by launching scrapers more or less manually, we had to automatise the process to just be able to scale the operations, ie. publishing more and more often ZIM files.
The effort started 3 years ago with the support of the WMF but we use it only since Spring 2019 in production. The tool is now perfectly running and we fully rely on it now. If we can publish an update of all our wikis one time a month, this is thanks to this piece of software too.
The Zimfarm is a half-decentralized solution which has a central node (called "dispatcher") in charge of orchestrating the work to do and multiple decentralized nodes (called "workers") which run the scraping tasks.
The dispatcher provides an API to manage the ZIM recipes and tasks, have a look to https://api.farm.openzim.org/. We have setup a Web frontend on this API to allow easy mgmt through a Web browser. For a better transparency, even anonymous users can have a look and monitor what is going on. Look at https://farm.openzim.org/.
One important point is that, like all the rest of our infrastructure, the whole system is Dockerized. Which means, this is really easy to install a Zimfarm worker and we invite anybody having a spare server to help us to provide offline snapshots of the best of the Web. The procedure is documented and a few volunteers have already joined in. Look at https://farm.openzim.org/about for more details.
The development is fully transparent at https://github.com/openzim/zimfarm. We have a few things which are on the roadmap which would welcome volunteer Python developers. Look at the good first issues and make your first PR!
https://github.com/openzim/zimfarm/issues?q=is%3Aissue+is%3Aopen+label%3A%22...
Regards Emmanuel
-- Kiwix - Wikipedia Offline & more
- Web: https://kiwix.org/
- Twitter: https://twitter.com/KiwixOffline
- Wiki: https://wiki.kiwix.org/
Offline-l mailing list Offline-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/offline-l
wikimediaza@lists.wikimedia.org