On Saturday 26. March 2005. 22:52, Brion Vibber wrote:

zhengzhu wrote:

...

If the conversion between Cyrillics and Latin is strictly one-to-one, then it should be fairly simple to support that, since most related code is already written. All we need is probably a mapping table between the Cyrillics and the Latin alphabet. If you can put the table somewhere, I can probably implement it and put up a test site fairly quickly. But since I have no knowledge with those languages, I can be wrong.

The last time this was brought up, I seem to recall that the conversion is _not_ perfect; foreign words and names of either Latin or Cyrillic-spelling origin may not use the generic conversions. Also if you're quoting eg foreign-language text, math, or programming code (all likely to come up in an encyclopedia) it's going to be harder; it'd be necessary to be able to specify non-convertible text.

You remember correctly :) Basically, conversion from Cyrillc to Latin alphabet is possible and easy to do; but conversion from Latin to Cyrillic is impossible task. It would thus be possible to create a static Wikipedia mirror in Latin alphabet, if there would be any need for it; but making it editable and importing results back into Cyrillic would not work and only create problems.

At http://zh.wikipedia.org/ you'll notice three additional tabs at the right-hand of the top tab bar: "不转换" for unconverted display, "简体" for simplified conversion, and "繁体" for traditional conversion. These conversions don't change the source text or alter editing; it's for rendered page display and user interface only.

The implementation also supports a markup that allows manually specifying the conversion of a specific word/phrase in situations where the mapping table is not sufficient. For example, -{zh-cn:foo; zh-tw:bar}- will show up as "foo" when "简体"(Simplified Chinese) is selected, and "bar" when "繁体"(Traditional Chinese) is selected; -{foo}- will show up as "foo" no matter what language variant is selected (this should be sufficient for things like quotes, math, etc). There is also a way of customizing the mapping table through a page in the MediaWiki: namespace.

Kaixo!

On Mon, Apr 11, 2005 at 09:02:24AM -0400, zhengzhu wrote:

With Monk's help, I have implemented a preliminary system for conversion between cyrillics and lation for BE. There is a test site at

http://s87257573.onlinehome.us/be/

very nice indeed, a lot of languages should benefit of such feature.

However, languages that have latin script in their list show that there are various exceptions to implement:

- html entities (starting with '&' and ending with ';') must not be converted. - interwiki links should be handled as exceptions too (as the list of valid interwiki domains is known (eg, the possible xx in [[xx:foo]]) it should be easy to implement - it would be nice also to detect urls and not convert them

then, it would be need to have a way to force conversion even for things otherwise being exceptions (that is, the opposite of -{ }- ); and the very nice thing would be a way to suggest the appropriate conversion for a given string (for example, foreign people names could be written differently in cyrillic and latin, maybe something like "blablabla ={Latn:Saratxaga|Cyrl:Сарачага}= blabla", that would be displayed as "blablabla Saratxaga blaba" or "блаблаблабла Сарачага блаблабла" but not as "блаблабла Саратхага блаблабла" maybe the Latn:/Cyrl: could be removed, as the script can be found from the strings, syntax will then be easier for the editors: "blablabla ={Saratxaga|Сарачага}= blabla" or, if they write in cyrillic: "блаблаблабла ={Сарачага|Saratxaga}= блаблабла"

Maybe we can start to create the transliteration rules for some languages and put them somewhere (on meta?)

Kaixo!

On Mon, Apr 11, 2005 at 07:06:19PM -0400, zhengzhu wrote:

...

• interwiki links should be handled as exceptions too (as the list of valid interwiki domains is known (eg, the possible xx in [[xx:foo]]) it should be easy to implement
• it would be nice also to detect urls and not convert them

The conversion happens at a rather late stage of the wiki parser at which the input should be largely (x)html. I have attempted to avoid converting them but may have missed something. Can you provide an example at the test site when this is not done correctly?

For interwiki links, the test site doesn't recognize them as interwiki links, so I don't know if the code handles it correctly or not...

For urls, I didn't tested enclosed urls (eg: [http://wa.wikipedia.org/ foo]) but plain text ones (eg: http://wa.wikipedia.org/ or pablo@walon.org in the article, without square brackets around them).

I did some more tests, in order to have an url not translitered, I have to write:

[http://wa.wikipedia.org/ -{http://wa.wikipedia.org/%7D-]

and for an email (which hasn't any special meaning in wiki syntax, unlike the http url): -{pablo@walon.org}-

well, we can live with it, indeed.

...

maybe something like "blablabla ={Latn:Saratxaga|Cyrl:Сарачага}= blabla", that would be displayed as "blablabla Saratxaga blaba" or "блаблаблабла Сарачага блаблабла" but not as "блаблабла Саратхага блаблабла" maybe the Latn:/Cyrl: could be removed, as the script can be found from the strings, syntax will then be easier for the editors: "blablabla ={Saratxaga|Сарачага}= blabla" or, if they write in cyrillic: "блаблаблабла ={Сарачага|Saratxaga}= блаблабла"

This function is built in and is running at ZH. At the BE test site, you can use the following syntax (note how close it is to your suggestion:)

-{be-cyrillics: Foo; be-latin: Bar}- This will show "Foo" in cyrillics mode, and "Bar" in latin mode.

It would be better imho to standardize on ISO 15924 script codes (with possible site local aliases, same way as "Talk:" etc can be translated, but "Talk:" always work in all wikipedias)

and I also think that "|" would be a better separator (as it is used in a lot of other wikisyntax), and it would allow to have ";" included in the -{ }- block.

an auto-detection of the script, so that there is no need to explicitely tell it, will be the candy on the top (it is not always possible for Hans/Hant, indeed; but for most other cases of multi-sript needs there is no ambiguity at all), that is also why use of "|" would be better: in -{Foo;Bar}- you can't be sure that the ";" is used as a separator, while in -{Foo|Bar}- the likelihood od "|" being part of the text is much, much smaller (and in such odd cases, <nowiki>|</nowiki> could be used).

Is it possible to add a test case for a latin<->arabic site? (Kurdish or Azeri are two likely candidates), due to the right-to-left nature of arabic script, it could show some more problems that aren't seen in a cyrillic<->latin case.

Thanks

Milos Rancic wrote:

And if you want to transliterate from Latin to Cyrillic, you'll have a lot of English words transliterated in Cyrillic. Also, what about referencing in Cyrillic? If you add some Russian bibliography, you'll have Russian text transliterated into Latin.

Good bibliographic practice would have it that references remain in their original language and script. Any transliterated title would only be an addition to what is already there, and is mostly for the purpose of being able to put mixed script references in the same alphabetical order. Listing references in their own language warns the potential user that about the language that he must read if he wants to go there. If you're creating links a Roman link to a Cyrillic text won't work.

Serbs have a history of being able to go back and forth between scripts, but that habit is not shared in many other cultures. I haven't been following the proposals to change scripts for Belorussian, but I anticipate that it would take at least a full generation before people start to be comfortable with the change.

Ec

Yes, problems can be solved after some time of work. I just think that we are far a way of live transliteration (like in Chinese). But static transliterations can be test for future live transliteration.

Markup language is a good idea, but I am not sure how often would be used. Some people would not use because they don't know how to use, some people would not use because they don't like other alphabet (the second problem is very real in Serbian culture :) ). However, it would be useful if MediaWiki would support something like "don't touch marking" (for example "---"; or something like that). For the beginning it would be used for the bot parsing; but when algorithm become stable, it should be used by MediaWiki engine.

At the page [[:sr:Корисник:Millbot/извештаји]] (http://sr.wikipedia.org/wiki/%D0%9A%D0%BE%D1%80%D0%B8%D1%81%D0%BD%D0%B8%D0%B...) can be found bot's reports during it's work one and half month ago. (I stopped with transliteration because I have to make some other parts of bot's infrastructure.)

You can find that part of bot's source code at http://millosh.org/software/ltafos/pos/ (use version 0.3.2; I have some newer on my laptop, but it is not stable). Unpack pywikipediabot (http://sourceforge.net/projects/pywikipediabot/, unpack pos, go to the pos directory and link all of pywikipediabot files into pos directory. Note that pos is in something like alpha stage and you should be careful when use it. There are some documentation in English, but Serbian documentation is better (as you are Belorussian, I think you can understand documentation in Serbian).

In this moment I am working on some other bot's tasks, but I'll finish it in month or two and I'll continue to work on transliteration. Bot Millbot should be the system with IRC and pywikipediabot parts. It should monitor channels like #srrc.wikipedia and work according to informations which it gets there: When someone make some change in Cyrillic text, it should immediately implement changes into Latin text and vice versa.

At the first stage of transliteration Millbot would make changes and synchronization only according to Cyrillic texts. When it become more clever, it should work with Latin texts, too. But it is a long time task.

Also, Bosnian Wikipedia has almost the same situation as Serbian. They starts from Latin and, according to their culture (and their Main Page), they should have Cyrillic pages, too. When I implement Latin->Cyrillic transliteration for Serbian, it would be Latin->Cyrillic transliteration implementation for Bosnian, too.

And, one more thing: I think we should move our talk to pywikipediabot list.

On Apr 12, 2005 2:06 PM, zhengzhu zhengzhu@gmail.com wrote:

The Chinese wikipedia has similar problems, although for different reasons. For example, sometimes people's names shouldn't be converted at all no matter what variant is in use, sometimes different variants translates foreign words differently. So there is a user customizable dictionary for each language variant that can be used to define such special conversion rules. There is also a special markup that can be used in the text to define specific conversion rules just for that piece of text.

In the case of converting Latin to Cyrillics, I think the same thing can be used. The conversion table can be augmented with words and phrases that should not be converted to Cyrillics under any condition. Those words that can both be English and Serbian (or Belarussian) can be manually marked up in the text.

1. I can just guess what is written in Chinese interface, so how did you cover article names? Do you have both names: in Simplified and Traditional Chinese?

2. I think MediaWiki should have one general module for transliteration with extensions for specific languages. General module should be based on Chinese module. Is it possible to start to work in such way?

3. Also, we should try to make system clever: Some formal and some statistic methods can help in recognizing should we transliterate something or not (i.e.: if system find some non-Serbian Cyrillic letters, it should not transliterate it into Latin and vice versa).

In most cases titles are converted using the same conversion system, i.e. using the conversion table and do a strtr(). For wierd situations, there is also support for manually specifying title conversion inside the article body, using this syntax: -{T|zh-cn:foo; zh-tw: bar}-

I am thinking about more general solution: To make database table with exceptions. And, more general, to make some kind of interaction with Wiktionary.

Here is an example: Let's say that in the conversion table, "foo" in zh-cn is converted to "bar" in zh-tw and vice-versa. Now someone writing in zh-cn wrote an article titled "foo". When someone with zh-tw preferred sees the article, "bar" will be shown as the article title. Further, say someone using zh-tw edited some article which has a link [[bar]]. The system will identify that the article "foo" should be used for linking, if "bar" is not already created as a redirect.

What do you keep in database? Simplified, traditional or both?

btw, you should be able to change the interface at zh after you register an account;)

I remember that I was looking few minutes at left up corner of MS Excel when I tried to find position of "File" in Hebrew MS Office :) (it is at right up corner). The situation with Chinese interface is similar :)

I saw a couple of days ago that if I click on Traditional Chinese, I'll get "ugly" link with parameter "variant=zh-tw". Is it possible that Simplified Chinese has URL in Simplified and Traditional Chinese in Traditional Chinese? Or mod_rewrite redirection:

http://zh.wikipedia.org/wiki/<something in Traditional Chinese> is shown, but http://zh.wikipedia.org/wiki/<something in Simplified Chinese>...?variant=zh-tw is read?

Indeed. I have always anticipated that the Chinese system can be generalized to other languages. Most of the code for the Chinese system is not specifically tied to the Chinese language, and some code refactoring can be done to provide better support for different languages. Please watch CVS HEAD for the next couple weeks for this to happen.

I knew that Chinese have two alphabets, but I didn't have in my mind that problems are similar to Serbian :) Of course, I found it a couple of months ago...

...

3. Also, we should try to make system clever: Some formal and some

statistic methods can help in recognizing should we transliterate something or not (i.e.: if system find some non-Serbian Cyrillic letters, it should not transliterate it into Latin and vice versa).

That's certainly doable within the current system framework, but will require more specialized algorithms.

Inside of my extension to pywikipedia bot (http://millosh.org/software/ltafos/pos/), I have statistic guesser: algorithm gueses distance between two texts (something like so called edit distances, but stochastic, so it can compare texts in real time, not only words and phrases). I am using it to guess if page is in Serbian or not. However, it can be used (in future forms) for other kinds of stochastic guessing.

On Apr 12, 2005 11:51 PM, Nikola Smolenski smolensk@eunet.yu wrote:

Frankly, to me this entire conversation sounds like much ado about nothing. Even on Serbian Wikipedia noone actually said that he actually needs Latin alphabet, and a few people said that they are for Latin because maybe someone would need it. So I don't see why introducing it.

This is not true. We voted about it last summer. One person explicitly asked for Latin alphabet [[sr:Корисник:Sergivs]], two said that it should be used, six voted against (seventh person voted after the conclusion of voting). I asked about consequences of voting at wikimeda-l list and answer was clear: if Latin alphabet wouldn't be included into sr.wikipedia, it would be tyranny of majority. After all, we can vote about it again and situation in this moment would be more pro-Latin.

Nikola, your political reasons for obstruction of introduction of Latin alphabet into Serbian Wikipedia are clear. But, Serbian Wikipedia should be used by all Serbian speaking people, not only by one political faction.

Kaixo!

On Tue, Apr 12, 2005 at 10:51:57PM +0100, Nikola Smolenski wrote:

Frankly, to me this entire conversation sounds like much ado about nothing. Even on Serbian Wikipedia noone actually said that he actually needs Latin

For Serbian maybe it isn't needed, for Belarussian I didn't even knew there was anyone using latin alphabet; but there are languages for wich such a feature is absolutely needed, like Kurdish, Azeri, Uzbek, Amazigh, Punjabi, where there are very siezable communities with different writting systems; possibly also Uighur, Uzbek.

On 4/13/05, Pablo Saratxaga pablo@mandriva.com wrote:

For Serbian maybe it isn't needed,

Huh... Maybe I can say that I can't belive. Around 50% of Serbs are using Latin alphabet as their primary alphabet! I am using Cyrillic, but my girlfriend and a lot of my friends use Latin as their primary alphabet. Two of my bank cards are written in Latin alphabet and one is written in Cyrillic. Almost all companies print their visit-cards in Latin alphabet (in Serbian). A lot of people don't contribute to Serbian Wikipedia because we are waiting for Latin implementation... Come to Belgrade and see the situation.

In formal way, Cyrillic is official alphabet. So, if we have to chose between that two alphabets, Cyrillic is "more equal". But, if we can use both, we should use both.

I didn't say that it wasn't needed for Serbian, just that I didn't knew if sr: users needed it (which is different of wanted; yes it would be a welcome feature, but it isn't absolutely needed, as anyone litterate in Serbian language can read cyrillic; the situation is different for other languages where different communities use different alphabets and may not be able to read the other).

...

(btw, I wasn't aware that the proportion of latin alphabet users was so high; yes I knew that anyone could write it, and probably used it in thinks like sms or email, but thought than when it came to writting with a pen on a paper most people used cyrillic)

Thank you. Now, I understand consequences of Nikola's xenophobic propaganda. He made equal SMS and email communication with poetry and scientific papers which is written (and which is writing) in Latin alphabet.

So, the question is do we need it. The answer is: No, we can use ASCII characterset and English language for communication and we don't need even Serbian Wikipedia.

Look, I am sick of two confronted main stream (sick) factions: One is xenophobic and rejects all of Western influences into Serbian culture and another is exponent of Western cultural imperialism and rejects all of traditional parts of Serbian culture. The first doesn't want to see Latin alphabet see Serbian alphabet, the another doesn't want to see Cyrillic alphabet as Serbian alphabet. Xenophobic faction is better organized, but exponents of Western cultural imperialism have very strong "unofficial" centers. In the substance, both of factions are the same: they are aggressive and they are making pressure to others to be like them. After the first half of 20th century we have word for people like them: they are fascists. And I hope Wiki(p|m)edia is not project which support fascists.

It seems that I have to explain cultural situation in Serbia to Wiki(p|m)edia's mailing lists, from time to time. So, let me try again.

Children in Serbia, Montenegro and Republic of Srpska (the part of Bosnia and Herzegovina) learns Cyrillic during the first year of primary school (at the age of 6-7 years). All of them (I am not sure for Republic of Srpska, but I think so) learn Latin alphabet during the second year of primary school (at the age of 7-8 years). If someone don't know Latin alphabet, (s)he didn't finish the second year of primary school. And this situation is during around 50 years in Serbian culture.

Introduction of Latin alphabet into Serbian culture made Serbian philologist Djuro Danicic, who was working as secretary of Croatian Academy of Sciences and Arts (the name was "Yugoslavian Academy", not "Croatian") in Zagreb during the 19th century. But, Serbs started to use widely Latin alphabet between the end of WWII and 1960s.

In 1970s and 1980s Latin alphabet was dominant in Serbia. Latin is treated as "fancy", "modern", as well as other bullshits. When nationalists came to power with Milosevic, Cyrillic alphabet started to be used with characterization of Latin alphabet as "anti-Serb", as "conspiracy against Serbs", as well as other bullshits. After Milosevic, both of factions are strong enough.

As Internet had it's own, ASCII based rules, during 1990s we had a lot of problems with introduction of Cyrillic alphabet into computers. Also, computer technology came from the West with Western cultural imperialism: During 1990s there were no a lot of computer workers which had any idea about using Cyrillic alphabet (nor even Serbian Latin alphabet; they used only ASCII).

We have deeply divided society. In this situation we are talking about introduction of Latin alphabet. A few years ago analogue situation was with introduction of Cyrillic alphabet into KDE and Microsoft products.

And, during 1990s, opposition to Cyrillic inside of computer circles was very very strong. Again, anyone who was working on Cyrillic was characterized as "pro-Milosevic", as "nazi" (with, of course, other kinds of such bullshits).

My conclusion is:

If you (I mean, all of people involved into Wikimedia) respect Serb culture, you have to understand that Serbian culture has two alphabets.

If there is no possibility for two alphabets (but, I don't see that MediaWiki doesn't have that possibility), I think that it should be Cyrillic (I think that, but not all of people from Serbia, some people think that it should be Latin!). Cyrillic is better choice because of transliteration, for sure. But, if we have possibility to use two alphabets (we are not working on ASCII or 8-bit terminals anymore, Unicode became standard), we should use it.

It is the matter of culture, not the matter of understanding. Take care about: a lot of people don't want (or don't like, or don't know) to use Cyrillic alphabet at computers. And some of them want to become contributors to Serbian Wikipedia, but they don't want (don't like, don't know) to do that in Cyrillic.

With only Cyrillic alphabet half of Serbs are excommunicated from Serbian Wikipedia. Nikola wants that. I am wandering if others want that?

...

But, if we can use both, we should use both.

I fully agree (and there is no need to have a 50%-50% situation); btw in my work (I'm responsible of localization for a software company) I have long ago decided to provide the choice of both writtings (by requiring translators to use cyrillic, then converting from cyrillic to latin, as that is easy to do, while the conversion from latin to cyrillic is painfull, due to the high amount of things that must remain unchanged (urls, command names, file and path names, email addresses, etc.))

I am glad to know that there are computer people outside of Serbia who take care about Serb culture when there are no so much computer people in Serbia who take care about it.

Yes, conversion from Latin to Cyrillic is painful. But, we (or I) have to make it in the future. A lot of texts in Serbian are written in Latin alphabet and we need it in Cyrillic, too.

On 4/13/05, Lars Aronsson lars@aronsson.se wrote:

Milos Rancic wrote:

...

If you (I mean, all of people involved into Wikimedia) respect Serb culture, you have to understand that Serbian culture has two alphabets.

If there is no possibility for two alphabets (but, I don't see that MediaWiki doesn't have that possibility), I think that it should be

If you (I mean, all Serbs) respect wikitech-l culture, you have to understand that there can only be one language: PHP.

:) I am working on it according to Zhengzhu's principles. (It sound's like some kind of Chinese philosophy :) )

On Wednesday 13 April 2005 14:50, Milos Rancic wrote:

...

I didn't say that it wasn't needed for Serbian, just that I didn't knew if sr: users needed it (which is different of wanted; yes it would be a welcome feature, but it isn't absolutely needed, as anyone litterate in Serbian language can read cyrillic; the situation is different for other languages where different communities use different alphabets and may not be able to read the other).

...

(btw, I wasn't aware that the proportion of latin alphabet users was so high; yes I knew that anyone could write it, and probably used it in thinks like sms or email, but thought than when it came to writting with a pen on a paper most people used cyrillic)

Thank you. Now, I understand consequences of Nikola's xenophobic propaganda. He made equal SMS and email communication with poetry and scientific papers which is written (and which is writing) in Latin alphabet.

The first time that words "poetry" or "scientific papers" are mentioned in this thread, is this email of yours to which I am replying to. I have never mentioned either. I believe that it is easy to see who's conducting propaganda here.

And, please, in a previous email you said to me that "your political reasons for obstruction of introduction of Latin alphabet into Serbian Wikipedia are clear". As you are the founder of anarchopedia, is it possible that you might have some political reasons too?

So, the question is do we need it. The answer is: No, we can use ASCII characterset and English language for communication and we don't need even Serbian Wikipedia.

No, we can not. Majority of people in Serbia doesn't know a word of English and so needs Serbian Wikipedia.

Look, I am sick of two confronted main stream (sick) factions: One is xenophobic and rejects all of Western influences into Serbian culture and another is exponent of Western cultural imperialism and rejects all of traditional parts of Serbian culture. The first doesn't want to see Latin alphabet see Serbian alphabet, the another doesn't want to see Cyrillic alphabet as Serbian alphabet. Xenophobic faction is better organized, but exponents of Western cultural imperialism have very strong "unofficial" centers. In the substance, both of factions

I see that you see the world in black and white. Well, the world isn't black and white, there are shades of gray inbetween, and I have never seen an organisation which "rejects all of Western influences into Serbian culture" nor one which "rejects all of traditional parts of Serbian culture" Of organisations concerned with scripts, I also don't know of any who aims to eradicate either alphabet completely. Oh, and their organisation is the opposite of what you said - pro-Western organisations are much better organised.

We have deeply divided society. In this situation we are talking about introduction of Latin alphabet. A few years ago analogue situation was with introduction of Cyrillic alphabet into KDE and Microsoft products.

On the other hand, I might add that CEO of Microsoft in Belgrade was replaced because he tried to localize Windows in Latin alphabet. So, people, be careful! ;)

And, during 1990s, opposition to Cyrillic inside of computer circles was very very strong. Again, anyone who was working on Cyrillic was characterized as "pro-Milosevic", as "nazi" (with, of course, other

This is simply a lie.

transliteration, for sure. But, if we have possibility to use two alphabets (we are not working on ASCII or 8-bit terminals anymore, Unicode became standard), we should use it.

No, we should not. Why should we?

It is the matter of culture, not the matter of understanding. Take care about: a lot of people don't want (or don't like, or don't know) to use Cyrillic alphabet at computers. And some of them want to become contributors to Serbian Wikipedia, but they don't want (don't like, don't know) to do that in Cyrillic.

With only Cyrillic alphabet half of Serbs are excommunicated from Serbian Wikipedia. Nikola wants that. I am wandering if others want that?

No, neither is true.

A lot of people don't know how to use Cyrillic alphabet on computers. However, these same people are those who don't know how to use Latin alphabet on computers. To explain: computers usually come with English keyboards preinstalled. People who don't know how to install another keyboard use it to write Serbian, and they do so in "naked" Latin: without necessary diacritics. I agree that there is a number of such people, and that they currently can't participate on Wikipedia except in a very limited way, however naked Latin is absolutely impossible to transliterate to either Cyrillic or proper Latin so they can't participate either way. However, the very moment when someone learns how to install a keyboard, he or she can use equal procedure to install either Cyrillic or Latin keyboard, or both. Wikipedia is in UTF-8 either way, so there are no problems with encoding or other problems.

I have said, and will say again: if this would be introduced it would be tyranny of the minority. If a minority needs something, and majority can give it to them, but don't want it, that is tyranny of the majority. But if there is a minority who wishes something and a majority who does not wish that thing, then having it would be a tyranny of the minority.

On Monday 11 April 2005 22:27, Milos Rancic wrote:

Chinese situation is almost clean: One set of characters should be changed into another set. If they use Latin (or Cyrillic) alphabet (for referencing) or Arabic numbers, they would not change it during transliteration.

AFAIK, similar problems can arise with Chinese: if an article has Japanese or Korean citation, it should remain unchanged...