Within search we are moving forward with cross-wiki searching.
This means if you go to de.wikipedia.org and search for ''Ревест-Сен-Мартен" we should show a message like:
No results found on the German Wikipedia
Showing results from the Russian Wikipedia
Except of course, it should be (beware machine translation):
Keine Ergebnisse auf dem deutschen Wikipedia gefunden Zeige Ergebnis aus der russischen Wikipedia
So somehow, we have to map from a wiki id (ruwiki, enwiki, dewiki, etc) to a localized (not for that wiki, for every other wiki in all languages) version of that project name. How can we go about this?
Here are two ideas from talking to a few people, I'm not sure either is the right approach though
First is to use CLDR to convert russian -> russische, and and plug that into an i18n message. This would have to be in WikimediaMessages and override a message without wikipedia. This feels awkward and will likely be wrong with with some language pairs.
Showing results from the $1 wikipedia
Idea 2 comes from observing that we are initially able to detect 50 languages and are only mapping to a primary wikipedia for each language. We could make 50 matching i18n messages in WikimediaMessages with a fallback to a generic message.
Search-results-from-enwiki: Showing results from the English Wikipedia Search-results-from-frwiki: Showing results from the French Wikipedia Etc.
This will have scaling issues as we integrate wiktionarys and other projects but is less than a thousand messages so is feasible.
On Sep 15, 2015 3:34 PM, "Erik Bernhardson" ebernhardson@wikimedia.org wrote:
Within search we are moving forward with cross-wiki searching.
This means if you go to de.wikipedia.org and search for
''Ревест-Сен-Мартен" we should show a message like:
No results found on the German Wikipedia
Showing results from the Russian Wikipedia
Except of course, it should be (beware machine translation):
Keine Ergebnisse auf dem deutschen Wikipedia gefunden Zeige Ergebnis aus der russischen Wikipedia
So somehow, we have to map from a wiki id (ruwiki, enwiki, dewiki, etc)
to a localized (not for that wiki, for every other wiki in all languages) version of that project name. How can we go about this?
How about translating them in the same way as interface language? Is it feasible? I think people would expect such a message show in same language as their interface language instead of content language. 2015/09/16 10:00 "Erik Bernhardson" ebernhardson@wikimedia.org:
Here are two ideas from talking to a few people, I'm not sure either is the right approach though
First is to use CLDR to convert russian -> russische, and and plug that into an i18n message. This would have to be in WikimediaMessages and override a message without wikipedia. This feels awkward and will likely be wrong with with some language pairs.
Showing results from the $1 wikipediaIdea 2 comes from observing that we are initially able to detect 50 languages and are only mapping to a primary wikipedia for each language. We could make 50 matching i18n messages in WikimediaMessages with a fallback to a generic message.
Search-results-from-enwiki: Showing results from the English Wikipedia Search-results-from-frwiki: Showing results from the French Wikipedia Etc.
This will have scaling issues as we integrate wiktionarys and other projects but is less than a thousand messages so is feasible.
On Sep 15, 2015 3:34 PM, "Erik Bernhardson" ebernhardson@wikimedia.org wrote:
Within search we are moving forward with cross-wiki searching.
This means if you go to de.wikipedia.org and search for
''Ревест-Сен-Мартен" we should show a message like:
No results found on the German Wikipedia
Showing results from the Russian Wikipedia
Except of course, it should be (beware machine translation):
Keine Ergebnisse auf dem deutschen Wikipedia gefunden Zeige Ergebnis aus der russischen Wikipedia
So somehow, we have to map from a wiki id (ruwiki, enwiki, dewiki, etc)
to a localized (not for that wiki, for every other wiki in all languages) version of that project name. How can we go about this?
Mediawiki-i18n mailing list Mediawiki-i18n@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-i18n
Yes we are intending to show them in the interface language, the problem is the number of messages. The config repo for mediawiki at wmf has 879 projects. There are 369 translation languages. That means if we use the standard i18n approach there will be (maximum) 324k translated strings for this one UI element. At the very least if we did go that way we should probably do that in a way that is usable in more than just this one specific UI element.
On Tue, Sep 15, 2015 at 7:44 PM, gfb hjjhjh c933103@gmail.com wrote:
How about translating them in the same way as interface language? Is it feasible? I think people would expect such a message show in same language as their interface language instead of content language. 2015/09/16 10:00 "Erik Bernhardson" ebernhardson@wikimedia.org:
Here are two ideas from talking to a few people, I'm not sure either is the right approach though
First is to use CLDR to convert russian -> russische, and and plug that into an i18n message. This would have to be in WikimediaMessages and override a message without wikipedia. This feels awkward and will likely be wrong with with some language pairs.
Showing results from the $1 wikipediaIdea 2 comes from observing that we are initially able to detect 50 languages and are only mapping to a primary wikipedia for each language. We could make 50 matching i18n messages in WikimediaMessages with a fallback to a generic message.
Search-results-from-enwiki: Showing results from the English Wikipedia Search-results-from-frwiki: Showing results from the French Wikipedia Etc.
This will have scaling issues as we integrate wiktionarys and other projects but is less than a thousand messages so is feasible.
On Sep 15, 2015 3:34 PM, "Erik Bernhardson" ebernhardson@wikimedia.org wrote:
Within search we are moving forward with cross-wiki searching.
This means if you go to de.wikipedia.org and search for
''Ревест-Сен-Мартен" we should show a message like:
No results found on the German Wikipedia
Showing results from the Russian Wikipedia
Except of course, it should be (beware machine translation):
Keine Ergebnisse auf dem deutschen Wikipedia gefunden Zeige Ergebnis aus der russischen Wikipedia
So somehow, we have to map from a wiki id (ruwiki, enwiki, dewiki, etc)
to a localized (not for that wiki, for every other wiki in all languages) version of that project name. How can we go about this?
Mediawiki-i18n mailing list Mediawiki-i18n@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-i18n
Mediawiki-i18n mailing list Mediawiki-i18n@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-i18n
A third idea that has been proposed to use the information in wikidata. https://www.wikidata.org/wiki/Q206855 contains the mapping from 'ruwiki' to a translatable label. It also has many of the other wmf projects. This labeled would then be plugged into the message:
Showing results from the $1
This is being discussed at https://phabricator.wikimedia.org/T112349 which is the phab ticket that prompted this thread.
On Tue, Sep 15, 2015 at 7:58 PM, Erik Bernhardson < ebernhardson@wikimedia.org> wrote:
Yes we are intending to show them in the interface language, the problem is the number of messages. The config repo for mediawiki at wmf has 879 projects. There are 369 translation languages. That means if we use the standard i18n approach there will be (maximum) 324k translated strings for this one UI element. At the very least if we did go that way we should probably do that in a way that is usable in more than just this one specific UI element.
On Tue, Sep 15, 2015 at 7:44 PM, gfb hjjhjh c933103@gmail.com wrote:
How about translating them in the same way as interface language? Is it feasible? I think people would expect such a message show in same language as their interface language instead of content language. 2015/09/16 10:00 "Erik Bernhardson" ebernhardson@wikimedia.org:
Here are two ideas from talking to a few people, I'm not sure either is the right approach though
First is to use CLDR to convert russian -> russische, and and plug that into an i18n message. This would have to be in WikimediaMessages and override a message without wikipedia. This feels awkward and will likely be wrong with with some language pairs.
Showing results from the $1 wikipediaIdea 2 comes from observing that we are initially able to detect 50 languages and are only mapping to a primary wikipedia for each language. We could make 50 matching i18n messages in WikimediaMessages with a fallback to a generic message.
Search-results-from-enwiki: Showing results from the English Wikipedia Search-results-from-frwiki: Showing results from the French Wikipedia Etc.
This will have scaling issues as we integrate wiktionarys and other projects but is less than a thousand messages so is feasible.
On Sep 15, 2015 3:34 PM, "Erik Bernhardson" ebernhardson@wikimedia.org wrote:
Within search we are moving forward with cross-wiki searching.
This means if you go to de.wikipedia.org and search for
''Ревест-Сен-Мартен" we should show a message like:
No results found on the German Wikipedia
Showing results from the Russian Wikipedia
Except of course, it should be (beware machine translation):
Keine Ergebnisse auf dem deutschen Wikipedia gefunden Zeige Ergebnis aus der russischen Wikipedia
So somehow, we have to map from a wiki id (ruwiki, enwiki, dewiki,
etc) to a localized (not for that wiki, for every other wiki in all languages) version of that project name. How can we go about this?
Mediawiki-i18n mailing list Mediawiki-i18n@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-i18n
Mediawiki-i18n mailing list Mediawiki-i18n@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-i18n
Hoi, Any approach that relies on messages or CLDR will fail because we will NOT have messages for any and all language pairs. However, when you turn it around and start with something that is "easy" to understand like enwiki, ruwiki etc you can always move up when you have that information.
When you consider the text involved there are two distinct elements that will only in combination result in 324k strings. When you leave them as separate, you get something more manageable. The information for the language pairs can be had by looking into Wiktionary.You will find that it is likely the richest resource.
One question, when you search for ''Ревест-Сен-Мартен", why did you not consider every language that uses the Cyrillic script? It is as likely to find something in Serbian, Macedonian, Belarusian etc ... Thanks, GerardM
On 16 September 2015 at 04:58, Erik Bernhardson ebernhardson@wikimedia.org wrote:
Yes we are intending to show them in the interface language, the problem is the number of messages. The config repo for mediawiki at wmf has 879 projects. There are 369 translation languages. That means if we use the standard i18n approach there will be (maximum) 324k translated strings for this one UI element. At the very least if we did go that way we should probably do that in a way that is usable in more than just this one specific UI element.
On Tue, Sep 15, 2015 at 7:44 PM, gfb hjjhjh c933103@gmail.com wrote:
How about translating them in the same way as interface language? Is it feasible? I think people would expect such a message show in same language as their interface language instead of content language. 2015/09/16 10:00 "Erik Bernhardson" ebernhardson@wikimedia.org:
Here are two ideas from talking to a few people, I'm not sure either is the right approach though
First is to use CLDR to convert russian -> russische, and and plug that into an i18n message. This would have to be in WikimediaMessages and override a message without wikipedia. This feels awkward and will likely be wrong with with some language pairs.
Showing results from the $1 wikipediaIdea 2 comes from observing that we are initially able to detect 50 languages and are only mapping to a primary wikipedia for each language. We could make 50 matching i18n messages in WikimediaMessages with a fallback to a generic message.
Search-results-from-enwiki: Showing results from the English Wikipedia Search-results-from-frwiki: Showing results from the French Wikipedia Etc.
This will have scaling issues as we integrate wiktionarys and other projects but is less than a thousand messages so is feasible.
On Sep 15, 2015 3:34 PM, "Erik Bernhardson" ebernhardson@wikimedia.org wrote:
Within search we are moving forward with cross-wiki searching.
This means if you go to de.wikipedia.org and search for
''Ревест-Сен-Мартен" we should show a message like:
No results found on the German Wikipedia
Showing results from the Russian Wikipedia
Except of course, it should be (beware machine translation):
Keine Ergebnisse auf dem deutschen Wikipedia gefunden Zeige Ergebnis aus der russischen Wikipedia
So somehow, we have to map from a wiki id (ruwiki, enwiki, dewiki,
etc) to a localized (not for that wiki, for every other wiki in all languages) version of that project name. How can we go about this?
Mediawiki-i18n mailing list Mediawiki-i18n@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-i18n
Mediawiki-i18n mailing list Mediawiki-i18n@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-i18n
Mediawiki-i18n mailing list Mediawiki-i18n@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-i18n
On Tue, Sep 15, 2015 at 11:13 PM, Gerard Meijssen <gerard.meijssen@gmail.com
wrote:
One question, when you search for ''Ревест-Сен-Мартен", why did you not consider every language that uses the Cyrillic script? It is as likely to find something in Serbian, Macedonian, Belarusian etc ... Thanks, GerardM
The rest of the discussion is happening on the phab ticket, but i'll answer this here. We are using a language detection algorithm that has been trained against tweets. Tweets are not, on average, as short as the searches we are detecting the language of but it does an ok job. Trey did a great job putting together an analysis[1] of this language detection algo. We will also be using his work there to evaluate other language detection methods and perhaps change what we are using in the future.
So the short of it is, we chose russian instead of serbian because the machine learning algorithm said so.
[1] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Language_Detection_Ev...
Hoi, I do not really understand the purpose of finding the articles in other languages. Arguably they are all known in Wikidata, including aliases.Magnus did already build a functional extension to search that is implemented on many Wikipedias. It will find you data when it is only available in Wikidata and it has the option to see this in an informative way by means of the "Reasonator".
No need to train the search of languages. Arguably such a training is not available for most of the languags Wikimedia supports. Thanks, GerardM
On 16 September 2015 at 17:59, Erik Bernhardson ebernhardson@wikimedia.org wrote:
On Tue, Sep 15, 2015 at 11:13 PM, Gerard Meijssen < gerard.meijssen@gmail.com> wrote:
One question, when you search for ''Ревест-Сен-Мартен", why did you not consider every language that uses the Cyrillic script? It is as likely to find something in Serbian, Macedonian, Belarusian etc ... Thanks, GerardM
The rest of the discussion is happening on the phab ticket, but i'll answer this here. We are using a language detection algorithm that has been trained against tweets. Tweets are not, on average, as short as the searches we are detecting the language of but it does an ok job. Trey did a great job putting together an analysis[1] of this language detection algo. We will also be using his work there to evaluate other language detection methods and perhaps change what we are using in the future.
So the short of it is, we chose russian instead of serbian because the machine learning algorithm said so.
[1] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Language_Detection_Ev...
Mediawiki-i18n mailing list Mediawiki-i18n@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-i18n
I'm really happy that this is brought up.
We have a similar issue in ContentTranslation, where we have to say things like "Translate this article from __________" ( https://phabricator.wikimedia.org/T105588 ). In English it doesn't look like a problem, because you can simply put a language name in the space and it will be correct, but in Russian and in many other languages every language name will have a different ending according to the Russian grammar. It's not easily predictable if you want to cover all the words in a language, but since the number of language names in the world is fairly limited (in practice - less than one thousand), a few lines of code can resolve it.
As a pet project I started implementing a solution for something like this that would allow doing this in Russian: https://gerrit.wikimedia.org/r/#/c/172501/
I also made a simplistic extension called "TestLanguageNameGrammar" that should help translators see how the different language names appear with the different grammar rules. After some cleanup it hope to get it deployed on translatewiki.net for the translators' benefit.
Once it's merged it will be possible to do it in all MediaWiki-based software, and similar patches will have to be written for other languages by people who know these languages.
Niklas suggested doing it with less code and more data (see the comments on the patch), but it's generally conceivable.
Erik, until we have a nice solution (hopefully soon, once I get that patch merged), please 1. just take the translated names from CLDR 2. create a bug similar to https://phabricator.wikimedia.org/T105588 for your project, and list the relevant message there
For names of projects like "Wikipedia", "Wikisource", etc., you should use the WikimediaMessages extension. It can also be used to write grammar rules for project names if needed (see the file WikimediaGrammarForms.php there; it is maintained according to community feedback and usually shouldn't directly bother extension developers very much, but it's useful to be familiar with it).
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2015-09-16 1:34 GMT+03:00 Erik Bernhardson ebernhardson@wikimedia.org:
Within search we are moving forward with cross-wiki searching.
This means if you go to de.wikipedia.org and search for ''Ревест-Сен-Мартен" we should show a message like:
No results found on the German Wikipedia
Showing results from the Russian Wikipedia
Except of course, it should be (beware machine translation):
Keine Ergebnisse auf dem deutschen Wikipedia gefunden Zeige Ergebnis aus der russischen Wikipedia
So somehow, we have to map from a wiki id (ruwiki, enwiki, dewiki, etc) to a localized (not for that wiki, for every other wiki in all languages) version of that project name. How can we go about this?
Mediawiki-i18n mailing list Mediawiki-i18n@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-i18n
Oh sorry, now I see that you created a bug already. I guess I'll continue there.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2015-09-16 9:55 GMT+03:00 Amir E. Aharoni amir.aharoni@mail.huji.ac.il:
I'm really happy that this is brought up.
We have a similar issue in ContentTranslation, where we have to say things like "Translate this article from __________" ( https://phabricator.wikimedia.org/T105588 ). In English it doesn't look like a problem, because you can simply put a language name in the space and it will be correct, but in Russian and in many other languages every language name will have a different ending according to the Russian grammar. It's not easily predictable if you want to cover all the words in a language, but since the number of language names in the world is fairly limited (in practice - less than one thousand), a few lines of code can resolve it.
As a pet project I started implementing a solution for something like this that would allow doing this in Russian: https://gerrit.wikimedia.org/r/#/c/172501/
I also made a simplistic extension called "TestLanguageNameGrammar" that should help translators see how the different language names appear with the different grammar rules. After some cleanup it hope to get it deployed on translatewiki.net for the translators' benefit.
Once it's merged it will be possible to do it in all MediaWiki-based software, and similar patches will have to be written for other languages by people who know these languages.
Niklas suggested doing it with less code and more data (see the comments on the patch), but it's generally conceivable.
Erik, until we have a nice solution (hopefully soon, once I get that patch merged), please
- just take the translated names from CLDR
- create a bug similar to https://phabricator.wikimedia.org/T105588 for
your project, and list the relevant message there
For names of projects like "Wikipedia", "Wikisource", etc., you should use the WikimediaMessages extension. It can also be used to write grammar rules for project names if needed (see the file WikimediaGrammarForms.php there; it is maintained according to community feedback and usually shouldn't directly bother extension developers very much, but it's useful to be familiar with it).
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2015-09-16 1:34 GMT+03:00 Erik Bernhardson ebernhardson@wikimedia.org:
Within search we are moving forward with cross-wiki searching.
This means if you go to de.wikipedia.org and search for ''Ревест-Сен-Мартен" we should show a message like:
No results found on the German Wikipedia
Showing results from the Russian Wikipedia
Except of course, it should be (beware machine translation):
Keine Ergebnisse auf dem deutschen Wikipedia gefunden Zeige Ergebnis aus der russischen Wikipedia
So somehow, we have to map from a wiki id (ruwiki, enwiki, dewiki, etc) to a localized (not for that wiki, for every other wiki in all languages) version of that project name. How can we go about this?
Mediawiki-i18n mailing list Mediawiki-i18n@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-i18n
On 16.09.2015 08:55, Amir E. Aharoni wrote:
[...] , but since the number of language names in the world is fairly limited (in practice - less than one thousand), a few lines of code can resolve it.
The number of languages listed by SIL in ISO 639-3 is (at the moment) a bit less then 1800, not 1000. BUT: given their many errors and omissions which are obvious to the knowing, one must come to the conclusion that a much better estimate is something in the range of 100 to about 1000 times that many.
Good for the subject matter we are discussion here: more often than not, we likely can get along with language groups. E.g., the translation of "Wikipedia" to many Slavic languages using the Cyrillic script is "Википедия" and declensions are identical. This advantage comes with many proper names in many languages.
As a pet project I started implementing a solution for something like this that would allow doing this in Russian: https://gerrit.wikimedia.org/r/#/c/172501/ [5]
See also https://phabricator.wikimedia.org/T50635 where there is a request fairly overlapping with your solution in many Germanic languages (not English) where adjectives and adverbs often share identical forms.
Purodha
mediawiki-i18n@lists.wikimedia.org