New subject: Toolserver Intuition - Tech spec (Toolserver goes I18N!)

29 Mar 2011

      So, as announced in the previous mailing (Agreed, I should've sent  
this one first...).
Regarding the technical implementation.
First a few points that you are likely aware of, that have stopped,  
disabled or scared off doing this untill now. ie. creating a dedicated  
solution that can scale for more than just 1 tool or group of tools –  
And is easy to use for translators as well for developers.
-- Past issues / obstacles
* Separated Toolserver SVN repositories
** For MediaWiki extensions, TranslateWiki can easily translate things  
and can sync in a single commit because it has a partial check out of  
a single SVN repository. Having to add all ts-account repos to TW's  
configuration is way too much work and pretty much not-done. Not to  
mention the fact that most tools are either not in SVN at all, or are  
maintained outside SVN and pushed towards SVN for public source  
viewing every once in a while.
* Language of choice
** Users want to set their choice once and not have to search re-do it  
for all tools independently and have to find the right place to 'do  
it' on everybody's tool. Nor do we want to click the same link again  
every visit. And developers prefer not to have userlang-parameters  
dangling in the url and have to make sure it's preserved throughout  
the app with every link.
This can (and sometimes is) solved by using cookies (one example is  
Luxo's contributions tool that sets a cookie)
* Prevent vandalism but also slow-down and other down sides of a  
regular MediaWiki page.
** When translated on one wiki-page (ie. at Meta-Wiki such as Magnus'  
implementation, which I think is the best implementation so far) there  
isn't a good translation-oriented workflow for translators or  
developers. Of course pages could be protected by sysops - and then  
have to be updated from another page on request. And then there is  
FlaggedRevs. But neither are not optimized towards translators (ie.  
there's no way to FUZZY a message, or see translation suggestions from  
services like Google Translate, or a description of the message while  
modifying it (aka "/qqq"). Nor is it not ideal (if not impossible) to  
keep track of changes to the original message without having to  
manually check it (eg. perform "FUZZY"). And no easy workflow to  
translate many messages at once.
This is all taken care of by Extension:Translate [4] on TranslateWiki.
* Fallback languages
** Aside from the management involved, fallback is also an important  
point. It shouldn't be required that a translator has to translate  
everything at once or nothing at all. Some implementations around  
would fail if a new message wasn't available in a translated language  
yet. Other implementation created a way to fallback to English if  
there was no message-key in the selected language. But I haven't seen  
any real fine grained fallback (such falling back from NDS (Low  
German) to DE (German), or from ACE (Aceh) to ID (Indonesian) for  
untranslated messages, like MediaWiki does).
* Universal messages
** Some messages like 'welcome', 'login', 'submit' etc. are generic  
and should not required to be duplicated around everywhere for each  
tool. Eventhough TranslateWiki has {{Identical}}, it save work by just  
having a group of generic messages. Tools that are mostly data or  
visually oriented may not even have to request to be added to the  
project if they only need a few of these generic messages to control  
the input form.
* Keeping up with latest versions
** Another implementation currently around (the only one that actually  
uses TranslateWiki afaik) is done through a fake extension named  
ToolserverTools [1] in Wikimedia SVN. For the translators side this  
was perfect (since they could use the TranslateWiki workflow they know  
and love). But not so for the developers. Messages all had a prefix,  
and in order to actually use them in the tools some wheel-reinvention  
took place (like getting the message from the array in the correct  
language, providing fallback, replacing variables like $1/$2, etc.).  
Also they still had to re-create a way for users to choose a language,  
store it, remember it and apply it the next visit. Lots of wasted  
time. And of course staying up2date with the latest version in  
Wikimedia SVN was sometimes forgotten and translators are known to get  
especially motivated if there is no work required from the developer  
to put the new translations into use (ie. TranslateWiki having the  
ability to push the updates and there being no extra action required).  
We could have everybody create a cronjob to update their svn-checkout  
of the messages file from the "/extensions/ToolserverTools/" directory  
in SVN, but that's not ideal either.
-- 

All of the above have been solved with my proposal. Either because of  
the fact that it's powered via TranslateWiki, or because it's taken  
care of by the central i18n system.

-- Tool developer workflow:

I'll describe how the system would work from a tool developers point  
of view. [3]

So here's what you'd do to make it work, three easy steps:
1) The toolserver tool developer includes a single php file (eg. / 
p_i18n/ToolStart.php). This makes the i18n class available.
2) A new instance of the class is created like $I18N = new  
TsIntuition( 'mygroup' );
3) Messages can now be returned with either _('message-key') [2] or  
$TSi18n->msg( 'message-key', $options ).

The msg() function can optionally be passed a text domain name (or  
'group name' if you will) as second argument to get a message from a  
different group eg. the group 'general' for messages like 'welcome',  
'login', 'submit' etc. Or an array if you need multiple options like  
escape, variable replacement etc. (more on that in a minute).

-- Other features

Although the I18N class will be able to do a lot more, this above is  
the core principle. Here's a list of items in no particular order for  
other things that it will have:

* Variable replacement ($1, $2, etc.)

    $welcome = $I18N->msg('welcomeback', array( 'variables' =>  
array( $username, $lastvisit ) )
    from [[Toolserver:Mytool-welcomeback]] which contains "Welcome back  
$1 (last visit: $2)".

* Fallback languages:
   If a message is not found in the current user language, a fallback  
will be used. And if that one isn't found English is used.

* Getting language names (eg. de -> 'Deutsch', en -> English) is built- 
in. Currently uses a copy MediaWiki's Names.php, could be made to use  
sql.toolserver.language if that is preferable but I think it's good  
this way)

* Escaping (ie. options = array( escape => html )

* Automated updates: Since the messages are file-stored in the  
messages-directory of the tool. There's no need to keep track or  
update anything.
ToolStart.php will load the appropriate class from the correct file,  
and when initializing the class and using msg(), will load needed  
message files on demand.

* Direction (Get which direction a language is, LTR or RTL). Handy  
when constructing your <html> element:
<html dir="{$I18N->getDir()}" lang="{$I18N->getLang()}">

* Automatic detection and remembering of the right user language  
(users can choose a language from a central i18n preferences page.  
This is stored as a cookie and (if no cookies available, in session).  
It can still be overridden by using the userlang GET parameter [2].  
One can also pass the desired message language to the getMsg()  
function to force a certain language for one message.

* No prefixes or collisions for MediaWiki messages:
To avoid conflicts with other tools, message-keys are automatically  
prefixed with the name of the group. So you won't have to prefix every  
key internally to avoid conflicts with messages of another Toolserver  
tool. Also (still in talks with TranslateWiki) we're planning to put  
them in a dedicated namespace and not in the MediaWiki:-namespace on  
TranslateWiki.
Example:
* A message at [[translatewiki:Toolserver:Luxocontris-usernotfound]]
* will be available through $I18N->msg( 'usernotfound' ), assuming  
$i18n = new TsIntuition( 'luxocontris' )
* otherwise $I18N->msg('usernotfound', 'luxocontris');

* Localize other fronts as well: There's several popular tools out  
there that have an additional (or only) front-end via JavaScript  
implementation on a wiki. Since the i18n system will have an API that  
has a JSON-output format with callback (JSONP) you can get the  
messages in there as well.

* API: When not in PHP (ie. JavaScript or Python) you can do queries  
(GET or POST) like api.php? 
action=getmessages&group=luxo&message=foobar|lorem|ipsum|logout|login  
&format=json&callback=myTool.initLang

* More.. (see design specification on Toolserver wiki) [5]

-- TranslateWiki

I'm currently in talks with TranslateWiki how to best set up the  
syncing system. Although initial chat with Nikerabbit didn't bring up  
any expected problem (as it's fairly similar other projects they  
translate), it still needs to be set up. I expect to have something  
going within one or two weeks.

The source files have been added to Wikimedia SVN [6] and checked out  
in the TsIntuition directory [7] at the Toolserver.

-- Documentation / design specification

The initial concept for the class has been documented at Toolserver  
Wiki [5]. Most of it has already been implemented in SVN [6] and can  
be tested. The implemention is subject to change based on feedback  
from you.

-- Already translated
The following tools have been translated already. Log in at the  
toolserver and look at their source to learn how they work:

* http://toolserver.org/~krinkle/TsIntuition/
* http://toolserver.org/~jarry/svgtranslate/
* http://toolserver.org/~krinkle/getWikiAPI.php

--
Krinkle

[1] http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/ToolserverTools/

[2] Yes, there's a way to disable the global function _() if you don't  
want it or have a function named like that already.

[3] Right now this is only for PHP tools (which are the most common),  
but I'm currently working on providing an API to make this available  
in other formats as well (ie. jsonp-callback for usage in javascript  
gadgets. Some toolserver tools interact with a wiki-side javascript  
companion. And a format that can be easily loaded into languages like  
Python (xml/json). I will focus on that as soon as the initial system  
is up and running.

[4] http://www.mediawiki.org/wiki/Extension:Translate

[5] https://wiki.toolserver.org/view/Toolserver_Intuition

[6] http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/ToolserverI18N/

[7] http://toolserver.org/~krinkle/TsIntuition/