The additional information you provided was helpful. I still think the best approach is to have an extension that returns the Jyutping value for the article title. Let's say that extension introduces a new magic word called {{TITLEINJYUPTING}}. That way you can add {{DEFAULTSORT:{{TITLEINJYUPTING}}}} to the bottom of the pages; and for exceptional pronunciations you can use {{DEFAULTSORT:[special Jyutping pronunciation]}} instead. Alternatively, you could make it a parser function like {{DEFAULTSORT:{{#JYUPTING:{{PAGETITLE}}}}}} or something like that.

If Juytping is as predictable as you state then making an extension should be a good idea because (a) it can be used by non-WMF wikis too, without having to set up Scrbunto, etc. and (b) it will be notably faster, because it runs directly on PHP, and not through the additional layer of Lua.

On Sun, Mar 15, 2020 at 7:03 PM Deryck Chan <deryckchan@gmail.com> wrote:

bawolff - Would you be able to point me to an example of mw.loadData?

Also, I've subscribed to https://phabricator.wikimedia.org/T46667 .

Huji - I was inspired by Japanese Wikipedia's approach to sorting - they have a {{DEFAULTSORT:[article name in hiragana]}} on all articles. Since Cantonese pronunciation is even more predictable than Japanese, we could potentially have a template that automatically adds {{DEFAULTSORT:[article title in Jyutping]}} using a Lua lookup table of all common Chinese characters. Exceptional pronunciations should then be coded individually. The Pinyin implementation of this would be equivalent, though it would depend on the zh.wp community agreeing on sorting things by Pinyin.

In terms of storing the data, Wikidata is not a good answer. First up, the Wikidata property creators community has rejected the notion of creating separate properties for each common phonetic transcription system of CJK languages, so the retrieval of the phonetic transcriptions from Jyutping will be unnecessarily complicated. Second, Wikidata items refer to concepts, not titles. We could theoretically ask the script to go to Lexemes to fetch the phonetic transcription but that'll involve untangling the multiple Lexemes that refer to the same Chinese character. In general, the way Wikidata is structured makes it a bad fit for the problem at hand.

Liangent's formulation of the problem is more general than the one I described, because T46667 aims to allow multiple ways of sorting Chinese characters within the same interface. That will be much welcome too.

On Sun, 15 Mar 2020 at 19:55, bawolff <bawolff+wn@gmail.com> wrote:

Consider using https://www.mediawiki.org/wiki/Extension:Scribunto/Lua_reference_manual#mw.loadData , keeping in mind that lua isn't really made with the usecase of huge data tables in mind, so there might be limits you run into if your data is really big.

--

Bawolff

On Sun, Mar 15, 2020 at 2:13 PM Deryck Chan <deryckchan@gmail.com> wrote:

Hello Ambassadors - This technical question may be relevant to multiple (particularly CJK) language communities so I'm asking it here.

What is the advice for writing a Lua script that needs to look up data from a big table (~10k rows at first deployment, potentially increasing in the future)? Does one hard-code the data into a Lua script, or is there a recommended data structure for storing those?

The design problem at hand is that the Cantonese Wikipedia wants to re-sort articles by Jyutping rather than Unicode. This will probably involve automating the generation of Jyutping phonetic guides by looking up the Jyutping transcription of common Chinese characters using a Lua module. Where do we store the data?

If another wiki has done similar things, we'd be interested in sharing the infrastructure.

Deryck

On behalf of the Cantonese Wikipedia community

_______________________________________________
Wikitech-ambassadors mailing list
Wikitech-ambassadors@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-ambassadors

_______________________________________________
Wikitech-ambassadors mailing list
Wikitech-ambassadors@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-ambassadors

_______________________________________________
Wikitech-ambassadors mailing list
Wikitech-ambassadors@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-ambassadors