In r80443 I added a feature allowing categories to be sorted using the Unicode Collation Algorithm (UCA). I wanted to briefly talk about the potential user impact, the design choices and the caveats. Sorting was the easy part. The hard part was providing a "first letter" concept which would be reasonably sane. The idea I came up with was to compile a list of first letters, themselves sorted using the UCA. Then the "first letter" of a given string is the nearest letter in the list which sorts above the string. For instance if you have letters A, B, C, and a string Aardvark, if you sort them you get: A Aardvark B C So we know that A is the first letter of Aardvark because Aardvark sorts immediately below A. This algorithm gives us a number of nice properties: * It automatically drops accents, since accented letters sort the same as unaccented letters (at the primary level). Same with case differences, hiragana/katakana, etc. * You can work out the initial Jamo of a Hangul syllable character by just omitting the composed syllables from the "first letter" list. Previously this was done with a special-case hack in Language::firstChar(). * Vowel reordering in Thai and Lao is automatically supported. So "แก" sorts under heading "ก" and "แข" sorts under heading "ข". * The collation can be expanded to support all sorts of other crazy features, and the first letter feature will keep working in a sane way. For instance, you could have an English collation which removed "the" from the start of a title. I compiled a list of 14,742 suitable header characters, identified by processing various Unicode data files. That list probably still needs lots of tweaks. There is a down side to this scheme. The default UCA table gives all characters with a similar logical function to the digits 0-9 the same primary sort order as the corresponding ASCII digits. So a page like [[१९२०]] on the Bihari Wikipedia will sort under a heading of "1" instead of "१". There may be other instances of accidental cultural imperialism. However, this can be fixed by compiling language-dependent lists of header characters. The UCA default table is not meant to sort any language correctly, it's just a compromise collation. Support for language-specific collations can easily be added. Whether we get language-specific collations or not, I'd like to think about enabling this feature on Wikimedia. The most glaring omission from the UCA default tables is sensible sorting of the unified Han. In a Chinese context, there's an obvious way to sort characters, and that's by their order in the KangXi dictionary. The Unihan database gives such an ordering, and it's used within code blocks. But it's not used between code blocks. So if you sort by code point, all the Han characters that aren't in the U+4E00 to U+9FFF block will sort incorrectly. That's what the default UCA does, with a few minor exceptions. In a Japanese context, the way to sort ideographic characters is to convert them to phonetic hiragana and then to sort the resulting string. I don't know if there is any free software for doing this. On the Japanese Wikipedia, they achieve the same result by manually setting the sort key of every page to be the hiragana version of the title. There's lots of room here for other people to get involved, especially if you know a language other than English. -- Tim Starling