Hi Lee,

And the others.

At 2002-07-31 18:53 -0700, lcrocker@nupedia.com wrote:

I'm not quite ready to solicit translations yet (there is still a feature or two I want to add first), but those of you who think you may be taking on the translation task might want to browse the new code base (see http://wikipedia.sourceforge.net ) and compare the files "Language.php" and "LanguageDe.php" to see what needs to be done and estimate the work (Magnus did it a couple of days).

I started a discussion about this language.php principle some days ago on the Intlwiki-L suggesting that this problem could perhaps better be solved using a translation database of some sort.

When I asked how to properly propose this to the programmers who matter, Brion VIBBER brion@pobox.com was so friendly to point out that I dad to contact a certain Lee or to propose it on this list. Well I guess that you're that Lee, (alhough your proper name misses from your message ;-).

The original thread was called: '[Intlwiki-l] Wiki community in Hebrew'.

But here is the discussion in full (edited for readibility):

I wrote:

--- start of my original message

Looking at the code of language.php, I thought about my own ideas about localizing software in a more general way.

Instead of including a file like this, you should use some sort of database for the translation process. Also instead of defining variables that hold strings, which are different for all the languages, you could just insert the English string into the source code and send it through a general translation process.

Let me clarify this.

Suppose you want the user to press a button in a popup window, addressing him in his own language, you would write something like: "alert(x('Press the button'));"

The x-function would first check if the desired language is English and if so, it would just return it's argument. Otherwise it would consult the database and with the help of the language that the user prefers it will either come up with a translation or not and if not will present the original English text. It will then also put an a new entry in the database telling people that this term needs to be translated to this particular language. It could also put the term in a general database with new terms that need to be translated to all languages.

People can translate the data in the database at their leisure doing as many terms as they like at a time, or only translate the terms that they are good at for example. They can also correct incorrect translations by others. It would work like the Wiki system itself.

Another point that concerns me is the size of the current file. I guess that this file of around 16 Kbyte is probably loaded and interpreted for every page that is loaded from Wikipedia and may perhaps account for how slow Wikipedia is at times? Assuming that say 50% of the users use English as their prefered language, the x() system would prevent the loading of this file or accessing the database for 50% of the files.

Other advantages of my system are, that the task of the PHP programmers would become much easier, because they don't have to be bothered with having to keep adding to the include file and can work with the actual English strings instead of having to work with variable names.

Perhaps the translation tables could also be used for other purposes or could be imported from or be shared with other applications.

--- end of my original message

The rest is organized by per resulting thread:

I wrote in my original message:

Instead of including a file like this, you should use some sort of database for the translation process. Also instead of defining variables that hold strings, which are different for all the languages, you could just insert the English string into the source code and send it through a general translation process

Brion replied:

That's a little more like what the old Usemod wiki code does (though with a separate include file with a big hash array rather than a database per se, and a function that checked the string against the hash, falling back on the given English string if there was no translation); I do find moving the English strings to a separate file confusing when working with the code, particularly as Lee isn't heavy on the comment usage.

Upon which I replied back: I was not (yet) talking about how to implement it, but an associative array (called hash array by some people ;-) is indeed one of the options.

One would have to make sure that the synchronisation is handled properly and most PHP implementations don't include the semaphore functions. Using an MySQL database would have the advantage that it synchronizes itself. But also that the database could be used from other locations.

It also depends on what the average number of strings is that the average page would want to translate.

Anyway, I have no clear idea what the best implementation would be. I also don't know how the rest of Wikipedia is organized.

In my original message I wrote:

People can translate the data in the database at their leisure doing as many terms as they like at a time, or only translate the terms that they are good at for example. They can also correct incorrect translations by others. It would work like the Wiki system itself.

Upon which Brion replied: Not quite as radical as making the live source code editable by wiki. :) I think we probably should do something like this...

I wrote in my origimal message:

Another point that concerns me is the size of the current file. I guess that this file of around 16 Kbyte is probably loaded and interpreted for every page that is loaded from Wikipedia and may perhaps account for how slow Wikipedia is at times? Assuming that say 50% of the users use English as their prefered language, the x() system would prevent the loading of this file or accessing the database for 50% of the files.

Brion replied: The PHP files are pre-compiled when first loaded; in theory, at least, they only need to be parsed the first time they're loaded after a software upgrade.

I replied back: Aren't you confusing PHP with Python? Python precompiles modules but PHP not, I think.

Brion replied again: We have a module installed (Alternative PHP Cache) that does this, or at least we did on the previous server. I _think_ it's still in place, but I could be wrong.

Finally I asked: Who has to decide on this? Lee? Is he around? I haven't seen him post here since I was here I think.

And Brion replied: Group consensus; we can beat Lee into submission if need be, but I suspect he'd be amenable. I'm not sure if he reads this list regularly; try cross-posting to the tech list (wikitech-l@nupedia.com) to make sure, and/or submit the idea to the feature request tracker:

http://sourceforge.net/tracker/?atid=411195&group_id=34373&func=brow... (goody! more cryptic URLs)

--- end of quotes

I just thought that it would be quite easy to generate the current language.php files from the database regurarly (every night for example). That would make the process guaranteed to be as fast (or slow) as it is now. ;-)

Let me know what y'all think...

Greetings, Jaap