This is the weekly update from the Search Platform team for the week
Programming note: Given the upcoming US holiday the next update will
be for the week starting 2018-11-26.
As always, feedback and questions welcome.
== Discussions ==
=== Search ===
* David and Trey have resolved the problems with 32-bit Chinese
characters (like 𨨏—), which were returning irrelevant results, and
showing lots of unicode replacement characters (�) in the results. The
highlighter fix was deployed  first so there aren't any more �
characters in the results. The re-indexing  to improve the
relevance of results is now also done for Chinese-language wikis.
== Did you know? ==
* Letters are encoded internally by computers as numbers—for example,
“A” is 65 and “a” is 97. Years ago, programs and even websites
would use different encodings to represent text, often leading to
unreadable gibberish on screen. Unicode was intended to be a single
encoding for most of the world’s writing systems. The most-used parts
of it fit into a 16-bit representation, which can handle about 65
thousand characters. But that's not enough for the very large number
of rare and historical Chinese, Japanese, and Korean (CJK) characters,
which are represented in 16-bit Unicode using “surrogate pairs”.
1,024 Unicode characters are set aside to be “high surrogates”—the
first half of a 32-bit character—and 1,024 characters are set aside to
be “low surrogates”—the second half. By themselves, the surrogates
aren’t valid and don’t represent anything, but in pairs they can
represent over a million additional characters. Since these characters
are usually rare, software can sometimes treat them incorrectly split
them up, which can result in you seeing the Unicode replacement
character �, which is used when something has gone wrong processing
Unicode text. (When the character is fine, but you don’t have a font
to show it, you sometimes get little squares instead. Since the most
common source of these squares for English speakers is unrepresented
CJK characters, a slang term for the squares is “tofu”.)
Subscribe to receive on-wiki (or opt-in email) notifications of the
Discovery weekly update.
The archive of all past updates can be found on MediaWiki.org
Interested in getting involved? See tasks marked as "Easy" or
"Volunteer needed" in Phabricator.
Community Relations Specialist