A recent discussion around high-volume APIs led to a ticket about
generating HTML dumps on the foundation servers:
Kelson weighed in there, highlighting that it isn't clear everyone working
on this is aware of the existing kiwix pipeline. Seemed worth mentioning
Sj and I have discussed and decided the following.
1) We have decided that the offline wikimedians UG will not answer the
Main argument is purely technical. Due to the complex design of the
survey, we consider it too difficult to collect your collective opinions
and draw relevant conclusions to inject into the survey.
We could have made it simpler is filling a survey that would reflect Sj
and I opinion, but we did not feel we should do that.
So, if you want to provide your opinion and answer the WMF survey,
please do so at the individual level (we strongly suggest you do)
2) With regards to signing the COOL (Community Open Letter on renaming),
we would like to ask you to please vote by saying :
either "Yes, we should sign the letter"
or "No, we should not sign the letter"
Neutral opinion will be counted as No
If we have at least 7 members voting and at least 2/3 YES, we will sign
If we have at least 7 members voting and less than 2/3 YES, we will not
sign the letter.
If we have less than 7 members voting, we will not sign the letter.
Please vote by answering to this email (public statement)
Link to the letter :
At the moment, 34 affiliates signed it (and 335 individuals)
Flo and Sj
I support signing COOL!
On Sun, Jun 28, 2020 at 2:00 PM <offline-l-request(a)lists.wikimedia.org>
> Send Offline-l mailing list submissions to
> To subscribe or unsubscribe via the World Wide Web, visit
> or, via email, send a message with subject or body 'help' to
> You can reach the person managing the list at
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Offline-l digest..."
> Today's Topics:
> 1. Rebranding: please give your opinion NOW (Florence Devouard)
> Message: 1
> Date: Sun, 28 Jun 2020 00:59:28 +0200
> From: Florence Devouard <anthere(a)anthere.org>
> To: Using Wikimedia projects and MediaWiki offline
> Subject: [Offline-l] Rebranding: please give your opinion NOW
> Message-ID: <7275054a-ad0a-708e-fcbe-e08788db305f(a)anthere.org>
> Content-Type: text/plain; charset=utf-8; format=flowed
> Hello all
> Sj and I have discussed and decided the following.
> 1) We have decided that the offline wikimedians UG will not answer the
> WMF survey.
> Main argument is purely technical. Due to the complex design of the
> survey, we consider it too difficult to collect your collective opinions
> and draw relevant conclusions to inject into the survey.
> We could have made it simpler is filling a survey that would reflect Sj
> and I opinion, but we did not feel we should do that.
> So, if you want to provide your opinion and answer the WMF survey,
> please do so at the individual level (we strongly suggest you do)
> 2) With regards to signing the COOL (Community Open Letter on renaming),
> we would like to ask you to please vote by saying :
> either "Yes, we should sign the letter"
> or "No, we should not sign the letter"
> Neutral opinion will be counted as No
> If we have at least 7 members voting and at least 2/3 YES, we will sign
> the letter.
> If we have at least 7 members voting and less than 2/3 YES, we will not
> sign the letter.
> If we have less than 7 members voting, we will not sign the letter.
> Please vote by answering to this email (public statement)
> Deadline Wenesday
> Link to the letter :
> At the moment, 34 affiliates signed it (and 335 individuals)
> Flo and Sj
> Subject: Digest Footer
> Offline-l mailing list
> End of Offline-l Digest, Vol 100, Issue 11
Michael Graaf, M.I.T.(UCT)
Researcher, Editor & Community
We might as well keep this list rolling - it’s been an eventful couple of months and there’s plenty to tell.
Just as COVID-19 lockdowns started to roll out across much of the world, our good friends at Orange (the French Telco) reached out asking for us to roll out Kiwix directly onto their West African network. So yes, here’s the short story of us making offline available online (!)
In a nutshell, it’s easier/faster for a telco to carry data on its own, local network than it is to carry that same amount of data internationally. It does make sense in hindsight, particularly if you think of the internets as a series of tube.
They asked that we roll out Kiwix and a collection of ZIMs in Arabic, French and English onto their Ivory Coast hub: Orange customers were to be directed to a specific page and would be offered the content at zero-rating or special low rate (markets could chose their pricing model). 11 markets were selected for the operation (mostly sub-saharan Africa).
We rolled-out the whole thing in a few days using Kiwix-serve - most of the time needed was for them to secure a big-ass server and grant us root access. It’s been running smoothly ever since - up to 100,000 users/month at peak, which was nice. Contents deployed were Wikipedia, Khan Academy, Wiktionary, Vikidia and a couple of video channels we also serve as ZIMs.
So what did we learn?
- Kiwix-serve is super easy to install, and can manage large loads robustly;
- Most demanded contents: Wikipedia and Khan Academy, then Wiktionary & Gutenberg library;
- Information circulated around somehow: we’ve had users from 130 countries so far (about 20-30% of total traffic), definitely not bots. A gentleman from An-Najah university in Palestine even reached out asking that we deploy the same thing on their local network.
- The URL that Orange set up was overly long, which probably impacted adoption. We lobbied to get https://kiwix.orange (they own the TLD) but to no avail :-/ There is also a huge difference between markets that communicated on the initiative in a sustained manner (e.g. Liberia) and those who did it as a one-off.
They made a simple but sweet video - in French only but you’ll get the idea.
 https://kiwix.campusafrica.gos.orange.com/ <https://kiwix.campusafrica.gos.orange.com/>
 https://github.com/kiwix/kiwix-tools <https://github.com/kiwix/kiwix-tools>
 https://www.youtube.com/watch?v=2Ug0XEFhByc <https://www.youtube.com/watch?v=2Ug0XEFhByc>
There is a topic I wanted to talk about here for a long time and for
which I never have achieved to take the time to write something. A few
recent events have been a healthy remember that I should present one our
most recent and most useful tool: Zimfarm.
The Zimfarm is the online tool which is in charge of building and
publishing all our ZIM files. After years of creating ZIM files by
launching scrapers more or less manually, we had to automatise the
process to just be able to scale the operations, ie. publishing more and
more often ZIM files.
The effort started 3 years ago with the support of the WMF but we use it
only since Spring 2019 in production. The tool is now perfectly running
and we fully rely on it now. If we can publish an update of all our
wikis one time a month, this is thanks to this piece of software too.
The Zimfarm is a half-decentralized solution which has a central node
(called "dispatcher") in charge of orchestrating the work to do and
multiple decentralized nodes (called "workers") which run the scraping
The dispatcher provides an API to manage the ZIM recipes and tasks, have
a look to https://api.farm.openzim.org/. We have setup a Web frontend on
this API to allow easy mgmt through a Web browser. For a better
transparency, even anonymous users can have a look and monitor what is
going on. Look at https://farm.openzim.org/.
One important point is that, like all the rest of our infrastructure,
the whole system is Dockerized. Which means, this is really easy to
install a Zimfarm worker and we invite anybody having a spare server to
help us to provide offline snapshots of the best of the Web. The
procedure is documented and a few volunteers have already joined in.
Look at https://farm.openzim.org/about for more details.
The development is fully transparent at
https://github.com/openzim/zimfarm. We have a few things which are on
the roadmap which would welcome volunteer Python developers. Look at the
good first issues and make your first PR!
Kiwix - Wikipedia Offline & more
* Web: https://kiwix.org/
* Twitter: https://twitter.com/KiwixOffline
* Wiki: https://wiki.kiwix.org/
1. The online meetings planned last week-end to discuss how to move
forward with regards to the survey were successfully held.
Essie (WMF staff in charge of collecting the survey) attended the first
part of the meeting to answer questions participants could have
Second part of the meeting was "without WMF staff". Long discussions etc.
Main point is that a team (Andrew Lih, Phoebe an Richard) *proposed to
write a Letter to the Board*. A draft was produced and will be made
public tomorrow (I will share the link, it is currently still private,
but Sj and I have access to current version).
*Essentially this Letter asks for a "pause" in the process to allow
further discussions.* Keep in mind that when the meeting was held, the
deadline to answer the survey was June 30th
This letter will be proposed tomorrow for signature for Affiliates.
<------ will have to be discussed from tomorrow on after link publicly
2. In the meanwhile, *the board issued a statement*. That you can read
Summary and key points
- it was published by Nataliia, and though a "board statement", mostly
read in the "I'. Apparently due to the urgency of the situation and not
all board members being available over the week-end.
- outline that, contrariwise to what Heather said in the exec statement,
the final decision has not been made yet. Likely to be made in August
2020 during the next board meeting
- and note that no decision has been made regarding the naming of the
- Natalia mentions the future Wikipedia 20th Anniversary and the former
wish of the Board to get everything fixed before that date (which
actually came a bit oddly in the discussion because very few community
members discussed that in relation to the rebranding process)
3. This morning, Samir sent us an email and said t*he deadline to answer
the survey is extended till July 7th*. He also says
There are 3 office hours this weekwhere the Brand Project Team will
continue to answer questions. All links available here in the news
If you are interested in joining and asking questions, this is the right
time to do it. If you are short of time and/or already made it your time
4. *There is a community feedback and straw poll* here *In light of
recent events, including the publication of survey text and naming
proposals, it may benefit the WMF to see how the community feels about
certain naming-related issues, transparently and on-wiki. Therefore, the
poll. This is an informal poll, and does not replace theWMF survey. At
least, the results of this one are publicly visible ;)*
_*My personal take on this is*_ * that the board already decided a LONG
time ago to rename Wikimedia Foundation into Wikipedia something; and
that they will do that, no matter what. * that the two
communication/brand companies were only hired to facilitate the process,
and take the heat, being blamed for failing to provide good suggestions
or supposingly pushing the board to adopt a new name. Whatever, they are
just the safety valve. They will not take all the heat, but part of it
making the pill easier to go down the throat of the community * that the
process is being "pushed" with a feeling of urgency, which surprises
many because usually we benefit from longer timeframe and covid19
oblige, everything is slowed. But the truth is... the main benefit of
the renaming is likely to be financial, with an easier and better way to
fundraise. With the current crisis, it is likely the future fundraising
season will be bad. Fundraising season start around September. So the
name change should benefit to be done before this year fundraising
season. Additionnaly, the Wikipedia 20th birthday could be an excellent
communication opportunity to promote the new name of the WMF. Hence the
urgency and the unlikeliness that the process significantly slows down *
in comparison, the renaming (or not) of affiliates is perceived as non
urgent and non essential, which is quite logical in the WMF perspective.
So it is possible to cut some slack here to cool down spirits * Natalia
is possibly being the other sacrificed piece in the process.
_*Your suggested todo list*_
1) Quickly read this page :
2) read the executive statement if you have not done so. Always
3) read the board statement. Always interesting.
4) reflect on the implications of the WMF rename on your own activities
5) decide whether you will answer the *individual survey* (before July
07th) : https://wikimedia.qualtrics.com/jfe/form/SV_9G2dN7P0T7gPqpD
6) decide whether you want to take the *community poll *:
7) tomorrow, read the open letter when I sent the link, and tell Sj and
I whether you think the Offline UG should sign it or not
8) tell Sj and I if you have a strong opinion on what we should answer
in the *Affiliate Survey*
Le 21/06/2020 à 19:40, Florence Devouard a écrit :
> It is a fair question Emmanuel
> Well, what you say is true. In short, if I summarize super briefly
> 1) According to Heather, the brand redefinition was a request from the
> board back in 2015. But there is no mention in board meeting minutes
> and two former board members do not remember this decision. Note: this
> was in Lila time.
> However, it seems indeed that the board confirmed its non-opposition
> to the communication team to work on that topic in 2018:
> Note that this does not appear to be a request from the board to the
> staff, but rather a request from the staff to be allowed to explore.
> 2) Brand awareness survey done in 7 countries in 2017 showed poor
> visibility and understanding of the wikimedia brand
> 2) When a survey was done a bit later, the statistical results were
> displayed in such a way that the case was made from the brand team
> that there was very little opposition from the community
> Evidence was made that the statistical presentation was broken and
> Arguments from opponents to the change include the fact the board
> members might have been mislead in believing there was no opposition
> from the community, and thus approved a rebranding without correct
> 3) Following that situation, a RFC was launched by the community, and
> show an overwelming opposition to replace Wikimedia with Wikipedia in
> our orgs and projects name.
> Note that RFC is opt-in only, so might over represent those who oppose
> the rebranding. Hence the case made for the final survey to poll
> community members about their position on the matter.
> Those who want to further explore:
> 4) The Brand team continued its work. Extensive discussions followed,
> with face to face brainstorming events to try to identify "good
> ideas". And key argument to opponants was that it was still in
> discussion phase etc.
> Brand network was created to better inform etc., give arguments in
> favor of the change etc. (I joined it as representant of offline UG to
> keep track of what was going on)
> There was further information provided about a month ago during a
> public meeting, revealing a collection of "words/directions"
> There were repeated requests from the people following this topic, for
> the final survey to include the "no change please" option. But this
> has been dismissed repeatedly.
> 5) Then finally a new survey (the one I mentionned earlier) was
> proposed a few days ago with a short list of options. The "no option"
> is not proposed, and the three options include replacing wikimedia by
> This is creating social unrest. Best person to know more about that is
> Andrew Lih.
> 6) An executive statement was published 2 days ago, stating that a)
> this rebranding was done per board request, and 2) the rename will happen
> Quote: *"We should have been clearer: a rebrand will happen. This has
> already been decided by the Board. The place where we seek
> consultation and input is on what an optimal rebrand looks like, and
> what the path to get there will be."*
> To read full statement :
> 7) There is a boiling discussion on whether to set up a central banner
> to invite participants to respond the survey, with community
> opposition to set up the banner.
> I have actually been contacted by some staff about this, who were
> apparently trying to evaluate the level of risk of WMF staff to be
> unsysoped if they decided the get over the community and activate the
> banner anyway
> I am not sure the banner is live yet. At least, I see no banner
> myself. It should have gone live on the 16th
> 8) Thus followed much discussion after the executive statement, on
> telegram and on meta.
> Probably central place is here :
> APPARENTLY, a statement from the board is expected. Unless wrong, it
> has not been published yet.
> 9) There is a meeting TONIGHT (21h UTC+2), community organized, on the
> I'll attend and will try to summarize
I am new to the mailing list. I used to work on sotoki.
My question is somewhat related to my failed attempt to store
stackoverflow dumps inside wiredtiger. Eventually, I figured that
wiredtiger could not keep up with the write load and that it is a pain
point at least with wiredtiger (but also with sqlite lsm extension).
The workaround is to have as much as RAM as the data (which is in my
opinion not acceptable), and fine-tune eviction triggers et al.
My questions are about libzim, zimwriterfs and how full-text search is
1) Why zimwriterfs or libzim succeed at putting together all the html
dumps of wikipedia? Is it because they're a lot of RAM? Or is it a
2) Follow up question, how the full-text search is put together? Is
the index written document by document then packed into the zim file?
I am working on my free time on a search engine , my goal is to
have my own search engine that I can use locally. That is why, I was
thinking about kiwix, because kiwix via .zim files provide readily
available dumps of many useful resources. The last question is:
3) how can I read the content of .zim from C code? Are there C
bindings of libzim?
 It does not work anymore but the code is at
I hope all of you are safe. I am forwarding this email from Ryan from
wikimedia-l, about the next steps of the implementation process of the
Strategy. The team is looking for a set of diverse people to prepare the
virtual transition discussions.
The placeholder page on meta : https://meta.wikimedia.org/wiki/Strategy/2030
There is an office meeting with the Movement Strategy core team on
Thursday. June 11 @ 08:00 UTC (Google Meet
<http://meet.google.com/rva-yqaq-zdk>)  to share any comments and ask
I think that if you are interested in joining this group, attending the
office meeting or getting in touch with a regional group might be a good
-------- Message transféré --------
Sujet : [Wikimedia-l] Movement Strategy: transition to implementation
Date : Tue, 9 Jun 2020 17:27:30 -0400
De : Ryan Merkley <rmerkley(a)wikimedia.org>
Répondre à : Wikimedia Mailing List <wikimedia-l(a)lists.wikimedia.org>
Pour : Wikimedia Mailing List <wikimedia-l(a)lists.wikimedia.org>
It’s my pleasure to be writing to you about the Movement Strategy. We
have come together over the past 3 years to develop our Strategic
Direction  and then the recommendations  to guide our
collaborations and future. So many of you contributed to the process —
some for only a part, and others throughout the entire process. With the
recommendations complete, our focus now shifts to implementing the
recommendations in a collaborative, open, and transparent way.
To make the transition from the publication of the recommendations to
their implementation, we need to do some planning. It’s a 10-year
strategy, with dozens of initiatives, dependencies, and connected goals,
to be delivered in a more distributed, deliberative, and open
collaborative model than ever before. We’ll need to work together to
define how we prioritize, sequence, resource, and support each initiative.
So much of our work is done online, but a lot of strategic work also
happens in person. We can’t do that now, and so we’ve had to adapt to
engage broadly, and in inclusive ways. To create this plan,
movement-wide virtual events will kick off in September. We will use the
Movement Strategy principles  as a guiding framework to ensure the
planning will be inclusive and empowering for our diverse range of
communities, without leaving anyone behind.
As a result of the pandemic, we lost the chance to work together
in-person on the transition to implementation at the Wikimedia Summit in
Berlin . Yet we gained an opportunity to include a higher number and
a more diverse profile of participants. Engaging with online
contributors, technical developer communities, and smaller user groups
throughout the process will be a key priority.
Successful virtual engagements with a high number of diverse
participants are difficult to do well. Therefore, a Design Group will
collaborate to prepare for the virtual transition discussions. This
group will consist of community members reflecting different parts and
perspectives of the movement, including representatives of regional
collaboratives (CEE, ESEAP, Indaba, Iberocoop, North America, South
Asia, WikiArabia, WikiFranca), the EDs and chairpersons groups, and WMF
Anyone who is interested can contribute. Regular summaries of the
preparation work and design discussions will be published on meta so
that anyone interested will be able to share insights and help improve
the process, even if not part of the Design Group itself.
I look forward to: Working with many of you at the virtual transition
Ways to participate and the schedule of events will be determined by the
Design Group. The current plan is to start the virtual transition
discussions with the movement in September.
The virtual events is where major discussions will take place on
sequencing, prioritizing, and resourcing the recommendations across the
Seeing those of you interested participate in the open review of the
The task will be to review the work of the Design Group and share your
perspective, enriching the thinking to improve the events.
Open review will happen in parallel to the work of the Design Group from
the end of June to the end of July / beginning of August.
Having nominations from different movement groups and collaboratives
(mentioned above) for the Design Group.
The task will be to design as a group how the transition process of
online events will be set up.
Want to know more?
We have put together a placeholder meta page  and will keep updating
it as more information becomes available.
Join office hours with the Movement Strategy core team on Wednesday.
June 10 @ 17:00 UTC (Google Meet <http://meet.google.com/uun-pzmb-kti>)
 or Thursday. June 11 @ 08:00 UTC (Google Meet
<http://meet.google.com/rva-yqaq-zdk>)  to share any comments and ask
Our email channel is always open: strategy2030(a)wikimedia.org
Chief of Staff, Wikimedia Foundation
Ryan Merkley (he/him)
Chief of Staff, Wikimedia Foundation <https://wikimediafoundation.org/>
Wikimedia-l mailing list, guidelines at:
New messages to: Wikimedia-l(a)lists.wikimedia.org