...with apologies for cross-posting.
--
deb tankersley
Program Manager, Engineering
Wikimedia Foundation
---------- Forwarded message ----------
From: Joe Matazzoni <jmatazzoni(a)wikimedia.org>
Date: Wed, Mar 28, 2018 at 7:41 PM
Subject: Turn mapframe for English Wikipedia? We’re ready if you are
To: wikitech-l(a)lists.wikimedia.org
Cc: Deborah Tankersley <dtankersley(a)wikimedia.org>
In October 2017, English Wikipedia community members approved an RfC
requesting implementation of mapframe
<https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)/Archive_15…>
[1]—a feature that enables contributors to easily display dynamic maps on
article pages. At the time, there was some question about whether
technical support for the maps service would be available long term, and
the Foundation asked that the community stand by. I’m writing now to say
that those questions have been answered, and a small team will be assigned
for routine maps maintenance.
So, unless we hear from the English Wikipedia community that the consensus
on adding mapframe has changed since last year’s RfC, we plan to move
forward in April 2018. You can read more about this plan and leave comments
on Village Pump [2]. If you foresee any issues, let us hear from you.
Otherwise,
happy mapping!
[1] https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(
technical)/Archive_159#Should_we_ask_for_mapframe_to_be_turned_on?
[2] https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(
technical)#Turn_on_mapframe?_We%E2%80%99re_ready_if_you_are
*Joe Matazzoni*
Product Manager, Collaboration
Wikimedia Foundation
*SR:* Стижу важне промене у претраживању на српском језику у вики
пројектима. Претрага коришћењем ћирилице или латинице проналази оба писма,
а ради и са падежима речи. Прочитајте више на МедијаВикију
<https://www.mediawiki.org/wiki/Wikimedia_Search_Platform/Serbian_Search_Upd…>
.
*EN:* An important update to search is coming soon to wiki projects in
Serbian. Searching for Cyrillic or Latin variants of a word will find the
other variant, and different grammatical forms of a word will also be able
to find each other. Read more on MediaWiki
<https://www.mediawiki.org/wiki/Wikimedia_Search_Platform/Serbian_Search_Upd…>
.
—Trey
Trey Jones
Sr. Software Engineer, Search Platform
Wikimedia Foundation
...with apologies for cross-posting. :)
Cheers,
Deb
--
deb tankersley
Program Manager, Engineering
Wikimedia Foundation
---------- Forwarded message ----------
From: Jeff Meyer <jeff(a)gwhat.org>
Date: Fri, Mar 16, 2018 at 3:15 PM
Subject: [Maps-l] Wikimaps User Group Newsletter March 2018
To: "maps-l(a)lists.wikimedia.org" <maps-l(a)lists.wikimedia.org>
*We’re back! ------------------------------This is the first Wikimaps user
group newsletter in a while, so hopefully, we’ll get this back up and
running on a more regular basis. All inputs for future content are welcome.
Please check the Wikimaps page
<https://meta.wikimedia.org/wiki/Wikimaps_User_Group#Next_newsletter> for a
link to the draft content for the next newsletter.Getting together
------------------------------Next User Group Hangout
<https://plus.google.com/hangouts/_/...>: 22 March 2018, 1600 UTCPlease
join in the discussion, this is an informal group to discuss what people
are working on, problems they're encountering with our mapping tools,
accomplishments we've had along the way, and upcoming events.Pattypan 18.02
includes the {{Map}} template ------------------------------Pattypan, a
tool for mass uploading media files to Wikimedia Commons now includes the
Map template predefined and ready to use without writing any
Wikitext.https://github.com/yarl/pattypan/releases/tag/v18.02
<https://github.com/yarl/pattypan/releases/tag/v18.02>Map Improvements 2018
------------------------------Be sure to check out the Wikimedia Foundation
Collaboration Team’s update on 2018 plans
<https://www.mediawiki.org/wiki/Map_improvements_2018> to improve the
Kartographer extension
<https://www.mediawiki.org/wiki/Extension:Kartographer>. Their work will be
done by the end of June and you can see the many exciting updates
<https://phabricator.wikimedia.org/project/board/2828/query/rSWQ9zrVUVS0/>
here. QGIS 3.0 Release ------------------------------For those of you who
are interested in using GIS tools as part of your map creation workflow,
please be sure to take a look at QGIS, QGIS 3.0
<https://www.qgis.org/en/site/>, which has launched a new version with many
new upgrades. If you’ve been working with .svg’s, this is a great tool to
use to start to create vector shapefiles of historical boundaries,
etc.Historical map vector data campaign ------------------------------For
anyone who’s interested, I’m trying to work on a bit of a scavenger hunt,
to find & collect as many historical shapefiles as possible. I’m still
working on how to best host / share them with the world, with a baseline
assumption that the Commons is the best place to start. If anyone is
interested, please let me know!Last, all suggestions for future newsletters
are welcome, for format, content, etc. We're just getting this started, so
I'm trying to use a fairly agile approach. Something > nothing!Regards,Jeff*
--
Jeff Meyer
206-676-2347 <(206)%20676-2347>
m: jeff(a)gwhat.org
t: @OpenHistMap
_______________________________________________
Maps-l mailing list
Maps-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/maps-l
This is a reminder to submit items for the Discovery weekly status
updates. Please add a few short descriptions of what you've been
working on during the last week. Interesting links to related
phabricator tasks, videos, reports, essays, and interesting tech are
welcome.
https://www.mediawiki.org/wiki/Discovery/Status_updates/Next
To view past updates, take a look at the archive:
https://www.mediawiki.org/wiki/Discovery/Status_updates#Archives
I'll send out the summary of items on the public discovery-l and
wikitech-l mailing lists at the end of business today (22 UTC 3pm
Pacific).
Yours,
Chris Koerner
Community Liaison
Wikimedia Foundation
Hear ye, hear ye!
This is the weekly update from the Search Platform team for the week
starting 2018-03-05. If you have feedback or questions, please let me
know.
== Discussions ==
=== Search ===
* Trey (with a *lot* of help from Guillaume and David) finished
wrapping an open-source stemmer for Serbian into an Elasticsearch
plugin[0] for use on Serbian wikis. It will also support cross-script
searching! It will still be a few weeks before the Serbian language
projects are re-indexed, but we are moving toward that goal. We're
also planning to investigate using the same stemmer on Bosnian,
Croatian, and Serbo-Croatian wikis. [1]
* Stas fixed a minor issue with using Blazegraph for deep category
searching / returning of results [2]
* Stas implemented quicker indexing for newly created Wikidata items [3]
== Did you know? ==
* The sound associated with the letter <s> in English is called a
"voiceless alveolar sibilant" by linguists, and appears in many
languages. It also comes in at least three varieties, which sound more
or less the same, but are articulated differently. You can see the
Wikipedia page for technical details, [4] but the most obvious
difference is that some people have the tip of their tongue behind
their *lower* front teeth when they say /s/, while others have the tip
of their tongue behind their *upper* front teeth or even a little
further back along the roof of their mouth.
* Another sound with multiple articulations—which isn't found in any
language but *is* found in many cultures—is an "unvoiced linguolabial
trill", also called a "raspberry". [5] Some people vibrate their lower
lip against their tongue, others use their upper lip. Trying to do it
the other way around can be very difficult, and a bit messy. Also of
note, the name "raspberry" comes from Cockney rhyming slang [6]—see
the Wikipedia page on "blowing a raspberry" for the full etymology.
[0] https://phabricator.wikimedia.org/T183015
[1] https://en.wikipedia.org/wiki/Serbo-Croatian
[2] https://phabricator.wikimedia.org/T165982
[3] https://phabricator.wikimedia.org/T183053
[4] https://en.wikipedia.org/wiki/Voiceless_alveolar_fricative#Features
[5] https://en.wikipedia.org/wiki/Blowing_a_raspberry
[6] https://en.wikipedia.org/wiki/Rhyming_slang
---
Subscribe to receive on-wiki (or opt-in email) notifications of the
Discovery weekly update.
https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly
The archive of all past updates can be found on MediaWiki.org:
https://www.mediawiki.org/wiki/Discovery/Status_updates
Interested in getting involved? See tasks marked as "Easy" or
"Volunteer needed" in Phabricator.
[1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R
[2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R
Yours,
Chris Koerner
Community Liaison
Wikimedia Foundation
Hey ya'll,
I'm not feeling well today, but I wanted to still remind you to add
your items to the weekly update before I return to bed. :/ I'll send
out the update when I hopefully return on Monday so you have a little
extra time. :)
https://www.mediawiki.org/wiki/Discovery/Status_updates/Next
To view past updates, take a look at the archive:
https://www.mediawiki.org/wiki/Discovery/Status_updates#Archives
I'll send out the summary of items on the public discovery-l and
wikitech-l mailing lists at the end of business Monday (23 UTC 3pm
Pacific).
Yours,
Chris Koerner
Community Liaison
Wikimedia Foundation
Hello all!
Usually the main problem in an elasticsearch cluster restart is that
it is a long and boring operation. This week, I ran into a number of
mostly minor issues. Probably not enough for an incident report, but
at least an email is needed. So here is some info on what went wrong:
First, my apologies for the noise caused by this restart! I'd love to
tell you that this won't happen again, but sometimes things go
wrong...
1) master re-election took longer than expected (both on codfw and on eqiad)
When a master is restarted, a re-election must occur and the cluster
state must be re-synchronized across the cluster. During that period
of time, both read and write operations can fail. Usually, this is
short enough to be mostly transparent and it is not picked up by our
monitoring (this was the first time that the issue was visible). The
synchronisation of cluster state took 1.9 minute, which while not a
crazy duration is definitely too long.
It would be great to be able to preemptively trigger a smooth master
re-election, but that's not something that elasticsearch supports,
despite multiple requests.
Potential solutions:
* have dedicated masters, which do not serve data at all: this could
ensure more resources are available for master / cluster wide
operations, but would probably not reduce the time for synchronization
of cluster state
* reduce the size of the cluster state: this would imply reducing the
number of indices, probably by splitting the cluster in multiple
independent clusters, with additional routing and general maintenance
burden.
2) indexing lag had impact on Wikidata
All (well, most, there are a few exceptions) writes to elasticsearch
are asynchronous and go through the job queue. During maintenance, we
freeze writes, restart nodes, wait for at least partial recovery,
re-enable writes, wait for full recovery, move to the next nodes.
Recovering shards which have been written to means transferring the
full shard over the network, which is much slower than recovering from
local storage (note that ES6 should enable partial shard recovery
making this issue much easier to deal with).
For most wikis, having lag on the indexing is not an issue. A new page
taking some time to be searchable is OK. An update to a page is even
less of an issue, since in most case an edit will affect the ranking,
which is already a heuristic.
Wikidata has a very different workflow, where editors add statements
(or properties or items, I'm not fluent in the wikidata terminology)
in sequence, referencing the just created statements. Searching for
those statements immediately is part of the natural wikidata workflow.
Potential solutions:
* writes are always going to be asynchronous, with potential lag
* we have a patch [1] coming up to trigger synchronous updates in the
nominal case (with still the appropriate asynchronous failover if need
be), but that is not going to help in case of planned maintenance.
* a better feedback in the UI, warning the user that the lag is higher
than usual would be nice
* I should better communicate to the wikidata community when doing maintenance
3) unassigned shard check was triggered
We have an Icinga check that raises an alert when more than 10% of
shards are unassigned. When a node is down, the shards that it hosted
go unassigned, until the shards are re-assigned, either on the same
node once it is back up, or moved to another node in the cluster.
I reboot elasticsearch servers by groups of 3. On a 36 nodes cluster,
this usually means that less than 10% of the shards are lost. But the
shards are not perfectly balanced. Other considerations, like the size
of those shards are taken into account. We also had one less node in
the cluster (elastic1021 has memory issues [2]).
This check is a heuristic. It hard to know at which point we are
really in trouble. In this case, the cluster was still fine, and
enough shards were recovered to silence the check in a few minutes.
Potential solutions:
* not much... we might want to raise this check to 12% instead of 10%,
but it is likely that we will still get a false positive at some point
(note that this is the first time it the 2 years I've been here that
this check sends a false positive, not too bad).
If you are still reading, thanks a lot for your patience and interest!
Note that the cluster restart is still ongoing on eqiad. I'm not
planning on anything else going wrong, but who knows... I might have
an addendum to this email before the weekend.
Have fun!
Guillaume
[1] https://gerrit.wikimedia.org/r/#/c/413492/
[2] https://phabricator.wikimedia.org/T188595
--
Guillaume Lederrey
Operations Engineer, Discovery
Wikimedia Foundation
UTC+2 / CEST