This is the weekly update from the Search Platform team for the week
As always, feedback and questions welcome.
== Discussions ==
=== Search ===
* Trey updated TextCat with models for detecting Russian typed on an
English keyboard and vice-versa, and UTF-8 Russian text improperly
encoded as Windows-1251,  as a precursor to providing
wrong-keyboard/encoding detection and suggestion. 
* Erik and the team did a lot of work on an epic ticket (with several
sub tasks) to explore and figure out next steps in using user click
data to tune Wikidata search parameters  and . The team will
ship the newly tuned wbsearchentities profile for en soon with de, fr,
* The team also had lots of discussions and exploration on how to
transform Wikidata autocomplete click logs into a useful dataset. They
are now transformed: Relevance Forge now has a utility for taking in
the Wikidata completion search logs and tuning the parameters of
search based on those logs. 
* David fixed a minor regression where search request failures when
offset+limit is out of bounds (cirrussearch-backend-error) 
* Mathew discovered that the required metrics have been exposed by the
prometheus exporter but they are displaying and fixed the issue with
help from David and Gehel 
* David reconfigured the ElasticSearch crosscluster on production
search servers to have persistent configs 
=== WDQS ===
* Stas & Guillaume finished moving categories namespace into a
separate Blazegraph instance 
== Did you know? ==
English text, like many others, is written left-to-right (LTR), but
some languages—most notably Arabic, Hebrew, Persian, and Urdu, but
also many others —are written right-to-left (RTL). To handle
different writing directions—especially in mixed LTR and RTL
texts—Unicode classifies characters as having "strong", "weak", or
"neutral" directionality. Strong characters definitely go in a
particular direction, like ABC or אבג. Weak characters have a "vague"
directionality, but can be changed in context, mostly numbers. Neutral
characters pick up their directionality from context, like punctuation
and whitespace characters used across scripts.
Mirrored characters change the way they display based on context. For
example "A>B>C" and "א>ב>ג" both only have the greater than character
(>) in them, but, if you are reading this somewhere that follows the
Unicode bidirectional algorithm, the ones between Latin letters point
to the right and those between Hebrew letters point to the left.
The algorithms are complicated , and when they don't work, there
are explicit characters that indicate things like "text should flow
left to right from here". The explicit formatting characters have the
most potential to cause trouble for search because they are usually
invisible, and you can pick one up without realizing it. For example,
when copying an Arabic word from a page in English, or a French word
from a page in Hebrew, the word that is "the other way around" from
the main text might have one of these marks at the beginning or end of
it. Fortunately, we can usually identify them and strip them out.
Finally, there are some scripts that have been written in other
interesting directions. Vertical text includes Chinese, Japanese, and
Korean,  and Mongolian. . Hanunó'o  and Ogham  were
written bottom-to-top! My [Trey's] favorite "direction" is
"boustrophedon,"  which means "like an ox ploughs" and alternates
left-to-right and right-to-left, and was used particularly in old
manuscripts and inscriptions in may writing systems. Why jump from one
side of the page to the other when you can just curve around where you
are or flip to mirrored letters and keep going?!
Subscribe to receive on-wiki (or opt-in email) notifications of the
Discovery weekly update.
The archive of all past updates can be found on MediaWiki.org:
Interested in getting involved? See tasks marked as "Easy" or
"Volunteer needed" in Phabricator.
Chris Koerner (he/him)
Community Relations Specialist
We are having some issues with 2 of the Wikidata Query Service
servers. So far, the issue looks like data corruption, probably
related to an issue in Blazegraph itself (the database engine behind
Wikidata Query Service). The issue prevents updates to the data, but
reads are unaffected as far as we can tell.
The 2 affected servers are part of the internal WDQS cluster, so the
public wdqs endpoint  is not affected. Data is lagging on the
internal eqiad endpoint, so Mediawiki functionalities that use WDQS
are at the moment not seeing the latest updates to Wikidata.
We are reaching out to the Blazegraph team via Github  and via
private contacts that we have. We hope to identify the root cause of
the issue so that we can fix it for good, but this looks like a hard
problem. Failing that, we will reimport the full data set.
You can follow the upstream issue on Github  and on Phabricator on
our side .
Sorry for the inconvenience and thank you for your patience!
Operations Engineer, Search Platform
UTC+1 / CET
The Search Platform Team
<https://www.mediawiki.org/wiki/Wikimedia_Search_Platform> usually holds
office hours the first Wednesday of each month—but since this month that
would have been Jan 2nd, we’ve delayed for a week. Come ask us anything
about Wikimedia search!
We’re particularly interested in:
* Opportunities for collaboration—internally or externally to the Wikimedia
* Challenges you have with on-wiki search, in any of the languages we
But we're happy to talk about anything search-related. Feel free to add
your items to the Etherpad Agenda for the next meeting.
Details for our next meeting:
Date: Wednesday, December 9th, 2018
Time: 16:00-17:00 GMT / 08:00-9:00 PST / 11:00-12:00 EST / 17:00-18:00 CET
Google Meet link: https://meet.google.com/vyc-jvgq-dww
*N.B.:* Google Meet System Requirements
Sr. Software Engineer, Search Platform
The previously announced schedule for Search Platform Team office hours was
that these office hours would happen on the first Wednesday of each month.
My guess is that January 1st is the last day of WMF's end of year holidays,
but maybe WMF's holiday break extends further than the 1st. There has been
no announcement of an office hour January 2nd. Am I correct in guessing
that the office hour will occur on January 9th?
( https://meta.wikimedia.org/wiki/User:Pine )