hi all
many of our tools uses messages from betawiki because they are translated in many languages. But this messages must be synchronized and I think it's not very economically if every user/tool do that self. Because of that I propose to create one database for all users which will be sync periodically.
A database with one table per user? Sounds good.
What do you think? Are any other users interested for such a db?
I would be interested.
Greetings, Luxo
Pietrodn powerpdn@gmail.com
I thought not one table per user, but rather one table for all messages. On betawiki are a lot messages e.g. http://translatewiki.net/wiki/MediaWiki:Abusefilter-revert-title/en.
So I would make a table something like
+ ----------+------------------------+-----------------------------------------------------+ | LANG | MESSAGE | TEXT | + ----------+------------------------+-----------------------------------------------------+ | en | Abusefilter-revert-title | Revert all changes by filter $1 | + ----------+------------------------+-----------------------------------------------------+ | de | Abusefilter-revert-title | Alle Änderungen durch Filter $1 rückgängig machen | + ----------+------------------------+-----------------------------------------------------+ | fr | Abusefilter-revert-title | Révoquer toutes les modifications par le filtre $1 | + ----------+------------------------+-----------------------------------------------------+
the messages which will be synced should be one a list which every ts-user can edit, so that it's easy for every user to add the messages he requires.
--Luxo
it destroys my table a bit, you can see it here http://pastebin.com/m4f87d3ad .
Luxo schrieb:
hi all
many of our tools uses messages from betawiki because they are translated in many languages. But this messages must be synchronized and I think it's not very economically if every user/tool do that self. Because of that I propose to create one database for all users which will be sync periodically.
A database with one table per user? Sounds good.
What do you think? Are any other users interested for such a db?
I would be interested.
Greetings, Luxo
Pietrodn powerpdn@gmail.com
I thought not one table per user, but rather one table for all messages. On betawiki are a lot messages e.g. http://translatewiki.net/wiki/MediaWiki:Abusefilter-revert-title/en.
So I would make a table something like
----------+------------------------+-----------------------------------------------------+ | LANG | MESSAGE | TEXT |
----------+------------------------+-----------------------------------------------------+ | en | Abusefilter-revert-title | Revert all changes by filter $1 |
----------+------------------------+-----------------------------------------------------+ | de | Abusefilter-revert-title | Alle Änderungen durch Filter $1 rückgängig machen |
----------+------------------------+-----------------------------------------------------+ | fr | Abusefilter-revert-title | Révoquer toutes les modifications par le filtre $1 |
----------+------------------------+-----------------------------------------------------+
the messages which will be synced should be one a list which every ts-user can edit, so that it's easy for every user to add the messages he requires.
--Luxo
I thought not one table per user, but rather one table for all messages. On betawiki are a lot messages e.g. http://translatewiki.net/wiki/MediaWiki:Abusefilter-revert-title/en.
I also think a global table would be better, because it's easier to sync, and there's no need to create new tables all the time. It would also allow common messages to be recycled. On the other hand, if people are not careful about prefixing message keys, it could easily lead to conflicts.
So I propose a four field scheme: language, *application*, key, message.
The application field would remove the need of prefixes. To allow for common cmessages, we could have some special applications like "toolserver", "common" or something. We could also import mediawiki messages as an application called "mediawiki" - a lot of common stuff is covered there, in a lot of languages.
I think i could get such a database up pretty quickly, though i'm not quite sure yet how the sync could be done automatically. I'll talk to siebrand about it.
-- daniel
So I propose a four field scheme: language, *application*, key, message.
application would be something like the groups in the translatewiki, right? the translatewiki offers a php output for groups, e.g. for MediaWiki - Messages http://translatewiki.net/w/i.php?title=Special%3ATranslate&task=export-t... . eventually we can use that to sync?
--Luxo
Luxo schrieb:
So I propose a four field scheme: language, *application*, key, message.
application would be something like the groups in the translatewiki, right?
yes.
the translatewiki offers a php output for groups, e.g. for MediaWiki - Messages http://translatewiki.net/w/i.php?title=Special%3ATranslate&task=export-t... . eventually we can use that to sync?
I would much rather have TSV/CSV output for syncing. Should be trivial to add that.
-- daniel
Any input/output format requires 'ffs' support in Translate, whichever format would be chosen as wanted, if not already supported (see extensions/Translate/ffs/). At the moment there is support for Wiki (MessagesXx.php), WikiExtension (MediaWiki extension i18n style), gettext, Java .properties style (key=message), PHP variables ($key="message"), and XLIFF (untested). One of these supports multiple languages in one file (WikiExtension).
I think it is unavoidable to develop a new ffs - there is little to no documentation on ffs development, but I hope the current code can serve as example. We have to keep in mind that in theory messages can contain both semi-colons and tabs, so care has to be taken chosing a separator when designing a xSV format.
If there would be a Subversion or similar repo available to store the output from Betawiki, I could commit the changes almost on a daily basis back into toolserver (or even easier, add it somewhere in svn.wikimedia.org), from where it could be processed further.
CC-ing Niklas as he knows a tiny bit more about Translate development than I do.
Daniel, would you happen to be at FOSDEM this weekend? Niklas will be there, as will I, so that would be a great opportunity to close this deal.
Cheers! Siebrand
-----Oorspronkelijk bericht----- Van: toolserver-l-bounces@lists.wikimedia.org [mailto:toolserver-l-bounces@lists.wikimedia.org] Namens Daniel Kinzler Verzonden: maandag 2 februari 2009 13:40 Aan: toolserver-l@lists.wikimedia.org Onderwerp: Re: [Toolserver-l] Betawiki translations
Luxo schrieb:
So I propose a four field scheme: language, *application*, key, message.
application would be something like the groups in the translatewiki,
right?
yes.
the translatewiki offers a php output for groups, e.g. for MediaWiki - Messages http://translatewiki.net/w/i.php?title=Special%3ATranslate&task=export -to-file&group=core&language=de&limit=100 . eventually we can use that to sync?
I would much rather have TSV/CSV output for syncing. Should be trivial to add that.
-- daniel
Siebrand Mazeland schrieb:
Any input/output format requires 'ffs' support in Translate, whichever format would be chosen as wanted, if not already supported (see extensions/Translate/ffs/). At the moment there is support for Wiki (MessagesXx.php), WikiExtension (MediaWiki extension i18n style), gettext, Java .properties style (key=message), PHP variables ($key="message"), and XLIFF (untested). One of these supports multiple languages in one file (WikiExtension).
I hade a look, it seems realtively straight forward to me
I think it is unavoidable to develop a new ffs - there is little to no documentation on ffs development, but I hope the current code can serve as example. We have to keep in mind that in theory messages can contain both semi-colons and tabs, so care has to be taken chosing a separator when designing a xSV format.
I would suggest to use TSV as supported by MySQLs LOAD INFILE command. That would make importing very easy.
If there would be a Subversion or similar repo available to store the output from Betawiki, I could commit the changes almost on a daily basis back into toolserver (or even easier, add it somewhere in svn.wikimedia.org), from where it could be processed further.
My idea was to dump TSV files to a web-readable directory, and have a cron-job on the toolserver that loads that file and imports it into MySQL using LOAD INFILE.
CC-ing Niklas as he knows a tiny bit more about Translate development than I do.
Daniel, would you happen to be at FOSDEM this weekend? Niklas will be there, as will I, so that would be a great opportunity to close this deal.
Yes, I'll be there.
Looking forward to seeing you all :)
-- daniel
Daniel Kinzler wrote:
Siebrand Mazeland schrieb:
Any input/output format requires 'ffs' support in Translate, whichever format would be chosen as wanted, if not already supported (see extensions/Translate/ffs/). At the moment there is support for Wiki (MessagesXx.php), WikiExtension (MediaWiki extension i18n style), gettext, Java .properties style (key=message), PHP variables ($key="message"), and XLIFF (untested). One of these supports multiple languages in one file (WikiExtension).
I think we will be better of with flat files (for example in PHP or java properties format or maybe) versioned in some version control, similary as MediaWiki is doing.
Java editounter is using Java properties already:
https://fisheye.toolserver.org/browse/editcount/trunk/editcount/i18n/
PHP tools like WikiSense prefer PHP format.
The reason for this is that you need to be able to understand message history - you need to know which messages have been added and which messages have been changed since the translation, to flag the modified existing messages on the Wiki. Ability to make diffs is very helpful, and doing this with database is a bit cumbersome.
On the other hand, we don't really need database for this, given that amount of writes will be much less than the amount of reads.
An automated way to move commit messages from, say, SVN, to Betawiki that explain why such message has been added would be nice.
Is Fuzzybot able to import svn commit messages in the MessagesEn.php to Betawiki?
For example, with this commit:
------------------------------------------------------------------------ r46258 | siebrand | 2009-01-26 14:19:55 +0100 (pon, 26.01.2009) | 1 line
Introduce 'deletedcontributions-title' as page title for Special:DeletedContributions to avoid having to share the message with that used on Special:SpecialPages. Copy contents from 'deletedcontributions' where available. At request of Fryed-peach.
=================================================================== --- MessagesEn.php (wersja 46246) +++ MessagesEn.php (wersja 46258) @@ -2210,7 +2210,8 @@ 'special-categories-sort-abc' => 'sort alphabetically',
# Special:DeletedContributions -'deletedcontributions' => 'Deleted user contributions', +'deletedcontributions' => 'Deleted user contributions', +'deletedcontributions-title' => 'Deleted user contributions',
# Special:LinkSearch 'linksearch' => 'Search web links',
From FuzzyBot we got edit http://translatewiki.net/w/i.php?title=MediaWiki:Deletedcontributions-title/...
Maybe FuzzyBot could put a commit message on some page like http://translatewiki.net/w/i.php?title=MediaWiki:Deletedcontributions-title/... or some other page?
Many messages are missing description what are they actually for, and putting commit messages could help a bit.
The other way around - Betawiki revisions are moved in batches from the Wiki to the SVN. We might consider importing change descriptions from wiki editors, but I am not sure this is necessary. I understand that granularity of wiki edits and SVN commits will be different, unless we will get some automatic wiki->svn commit robot.
--Marcin
Marcin Cieslak schrieb:
I think we will be better of with flat files (for example in PHP or java properties format or maybe) versioned in some version control, similary as MediaWiki is doing.
The problem is that each user has his/her own svn repo. Pushing messages to each would be rather annoying, especially with respect to access management (whos account would be used?...)
Just providing the files and letting people "pull" them into their repo is going to be slow. Betawiki is *very* interested in getting translations live *fast*. Waiting a month or so before your translation goes live is demotivating (i have let translators wait in the past too. sorry guys).
Java editounter is using Java properties already:
https://fisheye.toolserver.org/browse/editcount/trunk/editcount/i18n/
And Betawiki can already export to PHP, Java, gettext, etc
PHP tools like WikiSense prefer PHP format.
Well, I could easily change that. And I would. Because I know I often have other things on my mind then updating translations. So having it automated would be very nice to me.
The reason for this is that you need to be able to understand message history - you need to know which messages have been added and which messages have been changed since the translation, to flag the modified existing messages on the Wiki. Ability to make diffs is very helpful, and doing this with database is a bit cumbersome.
Hm... I suppose the "master" copy of the messages in the "primary" language should indeed be maintained by the author, as a flat file in the repository. But the "live" messages could be pulled from a db that is updated automatically. Maybe from there they could be exported again into the tools.
Thinking about it, it seems like part of betawiki being mirrored on the toolserver -- with the live messages in the db, and the ability to export to files and check them into the repo. Perhaps we should think of it this way.
On the other hand, we don't really need database for this, given that amount of writes will be much less than the amount of reads.
The main argument for a database is the easy of syncing with betawiki.
An automated way to move commit messages from, say, SVN, to Betawiki that explain why such message has been added would be nice. Is Fuzzybot able to import svn commit messages in the MessagesEn.php to Betawiki?
Would be nice, but I'm more worried about getting new translations from betawiki into a toolserver svn.
-- daniel
Daniel Kinzler schrieb:
Marcin Cieslak schrieb:
I think we will be better of with flat files (for example in PHP or java properties format or maybe) versioned in some version control, similary as MediaWiki is doing.
The problem is that each user has his/her own svn repo. Pushing messages to each would be rather annoying, especially with respect to access management (whos account would be used?...)
The users could just add the common repo to some folder using the svn:external property. Thus whenever they svn up their local copy, the commons one would be updated. The downside is that if they don't svn up (eg. because they haven't touched their code), their copy from the common repository won't be updated either. So people who use it must able to understand what it does and what it does not.
Daniel Kinzler wrote:
Marcin Cieslak schrieb:
Hm... I suppose the "master" copy of the messages in the "primary" language should indeed be maintained by the author, as a flat file in the repository. But the "live" messages could be pulled from a db that is updated automatically. Maybe from there they could be exported again into the tools.
Are you sure you want to query your messages "live" on the database every time?
If you download them once and store them in some kind of a hash table you need to tell the application somehow to refresh the hashtable.
Not much different from need to go via "svn up".
--m
Marcin Cieslak schrieb:
Daniel Kinzler wrote:
Marcin Cieslak schrieb:
Hm... I suppose the "master" copy of the messages in the "primary" language should indeed be maintained by the author, as a flat file in the repository. But the "live" messages could be pulled from a db that is updated automatically. Maybe from there they could be exported again into the tools.
Are you sure you want to query your messages "live" on the database every time?
why not? that's relatively cheap.
If you download them once and store them in some kind of a hash table you need to tell the application somehow to refresh the hashtable.
If you have a persistent application, yes. it could simply discard the hash after an hour or so. But cgi scripts and php applications are restarted for every request anyway.
Not much different from need to go via "svn up".
Sure it's different: it happens automatically. Which is somethign i don't want to happen with code.
In any case: applications should load messages from a database or flat file, on every request or at least periodically every few hours. Messages could be pulled automatically from betawiki. The next step would then be to get such messages into the respective tool's repository. That's the last step, really.
-- daniel
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Any news?
Daniel Kinzler schrieb:
Marcin Cieslak schrieb:
Daniel Kinzler wrote:
Marcin Cieslak schrieb: Hm... I suppose the "master" copy of the messages in the "primary"
language
should indeed be maintained by the author, as a flat file in the
repository. But
the "live" messages could be pulled from a db that is updated
automatically.
Maybe from there they could be exported again into the tools.
Are you sure you want to query your messages "live" on the database every time?
why not? that's relatively cheap.
If you download them once and store them in some kind of a hash table you need to tell the application somehow to refresh the hashtable.
If you have a persistent application, yes. it could simply discard the hash after an hour or so. But cgi scripts and php applications are restarted for every request anyway.
Not much different from need to go via "svn up".
Sure it's different: it happens automatically. Which is somethign i
don't want
to happen with code.
In any case: applications should load messages from a database or flat
file, on
every request or at least periodically every few hours. Messages could
be pulled
automatically from betawiki. The next step would then be to get such
messages
into the respective tool's repository. That's the last step, really.
-- daniel
Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
well, i've implemented the php lib for use at the toolserver mostly. i'll poke nikerabbit obout the export again. basically, i'll try to use this for my own tools, and tell you about how it works out.
however, i don't have much time for this right now, i have pretty much sold out all my computer hours. so it may take a while until it gets done. if you want to go ahead and play with it yourself, I can provide the code I have so far.
-- daniel
Luxo schrieb:
Any news?
Daniel Kinzler schrieb:
Marcin Cieslak schrieb:
Daniel Kinzler wrote:
Marcin Cieslak schrieb: Hm... I suppose the "master" copy of the messages in the "primary"
language
should indeed be maintained by the author, as a flat file in the
repository. But
the "live" messages could be pulled from a db that is updated
automatically.
Maybe from there they could be exported again into the tools.
Are you sure you want to query your messages "live" on the database every time?
why not? that's relatively cheap.
If you download them once and store them in some kind of a hash table you need to tell the application somehow to refresh the hashtable.
If you have a persistent application, yes. it could simply discard the hash after an hour or so. But cgi scripts and php applications are restarted for every request anyway.
Not much different from need to go via "svn up".
Sure it's different: it happens automatically. Which is somethign i
don't want
to happen with code.
In any case: applications should load messages from a database or flat
file, on
every request or at least periodically every few hours. Messages could
be pulled
automatically from betawiki. The next step would then be to get such
messages
into the respective tool's repository. That's the last step, really.
-- daniel
Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
_______________________________________________ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Daniel Kinzler schrieb:
if you want to go ahead and play with it yourself, I can provide the
code I have so far.
Well I think I don't have the skills to do this; I have only basic php knowledge. So I think it's better if I don't go ahead and let you do that. :-)
Thanks, Luxo
2009/2/27 Daniel Kinzler daniel@brightbyte.de:
well, i've implemented the php lib for use at the toolserver mostly. i'll poke nikerabbit obout the export again. basically, i'll try to use this for my own tools, and tell you about how it works out.
however, i don't have much time for this right now, i have pretty much sold out all my computer hours. so it may take a while until it gets done. if you want to go ahead and play with it yourself, I can provide the code I have so far.
Please, feel free to share it with us :)
Michał schrieb:
2009/2/27 Daniel Kinzler daniel@brightbyte.de:
well, i've implemented the php lib for use at the toolserver mostly. i'll poke nikerabbit obout the export again. basically, i'll try to use this for my own tools, and tell you about how it works out.
however, i don't have much time for this right now, i have pretty much sold out all my computer hours. so it may take a while until it gets done. if you want to go ahead and play with it yourself, I can provide the code I have so far.
Please, feel free to share it with us :)
all right, all right. here it is: http://brightbyte.de/repos/codebin/TSMessages.php?view=1
but please take this as a *draft*. it's UNTESTED, i didn't even check the syntax.
doin't say i didn't warn you :)
-- daniel
And the database? is there any ETA when he will be created? to whom is the next step, is it up to the translatewiki to create a export possibility? Greetings, Luxo
Daniel Kinzler schrieb:
Michał schrieb:
2009/2/27 Daniel Kinzler daniel@brightbyte.de:
well, i've implemented the php lib for use at the toolserver mostly. i'll poke nikerabbit obout the export again. basically, i'll try to use this for my own tools, and tell you about how it works out.
however, i don't have much time for this right now, i have pretty much sold out all my computer hours. so it may take a while until it gets done. if you want to go ahead and play with it yourself, I can provide the code I have so far.
Please, feel free to share it with us :)
all right, all right. here it is: http://brightbyte.de/repos/codebin/TSMessages.php?view=1
but please take this as a *draft*. it's UNTESTED, i didn't even check the syntax.
doin't say i didn't warn you :)
-- daniel
Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
well until this I will occupy myself with a function to replace $1 $2 etc in text and {{PLURAL:$1|once|$1 times}} for your draft.
Luxo schrieb:
And the database? is there any ETA when he will be created? to whom is the next step, is it up to the translatewiki to create a export possibility? Greetings, Luxo
Daniel Kinzler schrieb:
Michał schrieb:
2009/2/27 Daniel Kinzler daniel@brightbyte.de:
well, i've implemented the php lib for use at the toolserver mostly. i'll poke nikerabbit obout the export again. basically, i'll try to use this for my own tools, and tell you about how it works out.
however, i don't have much time for this right now, i have pretty much sold out all my computer hours. so it may take a while until it gets done. if you want to go ahead and play with it yourself, I can provide the code I have so far.
Please, feel free to share it with us :)
all right, all right. here it is: http://brightbyte.de/repos/codebin/TSMessages.php?view=1
but please take this as a *draft*. it's UNTESTED, i didn't even check the syntax.
doin't say i didn't warn you :)
-- daniel
Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Hey all,
I was asked for my actual version of TSMessages; I have overwrought & tested TSMessages.php a bit, details about the function of it see https://wiki.toolserver.org/view/TSMessages.php. My version is added, but she is neither finished nor completely tested (e.g. multilang file I still don't have tested). So you are invited to advance it!
Greetings, Luxo
Daniel Kinzler schrieb:
Michał schrieb:
2009/2/27 Daniel Kinzler daniel@brightbyte.de:
well, i've implemented the php lib for use at the toolserver mostly. i'll poke nikerabbit obout the export again. basically, i'll try to use this for my own tools, and tell you about how it works out.
however, i don't have much time for this right now, i have pretty much sold out all my computer hours. so it may take a while until it gets done. if you want to go ahead and play with it yourself, I can provide the code I have so far.
Please, feel free to share it with us :)
all right, all right. here it is: http://brightbyte.de/repos/codebin/TSMessages.php?view=1
but please take this as a *draft*. it's UNTESTED, i didn't even check the syntax.
doin't say i didn't warn you :)
-- daniel
Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Luxo schrieb:
Hey all,
I was asked for my actual version of TSMessages; I have overwrought & tested TSMessages.php a bit, details about the function of it see https://wiki.toolserver.org/view/TSMessages.php. My version is added, but she is neither finished nor completely tested (e.g. multilang file I still don't have tested). So you are invited to advance it!
Please put it on the toolserve somewhere, svn, plain file, wiki, anything... i just want to be able to link to it :)
-- daniel
2009/4/5 Daniel Kinzler daniel@brightbyte.de:
Please put it on the toolserve somewhere, svn, plain file, wiki, anything... i just want to be able to link to it :)
There you go: https://wiki.toolserver.org/view/TSMessages.php/source :)
On Sun, Apr 5, 2009 at 7:20 PM, Luxo luxo.de@gmail.com wrote:
Hey all,
I was asked for my actual version of TSMessages; I have overwrought & tested TSMessages.php a bit, details about the function of it see https://wiki.toolserver.org/view/TSMessages.php. My version is added, but she is neither finished nor completely tested (e.g. multilang file I still don't have tested). So you are invited to advance it!
Interesting! GMail apparently interprets some dropdown box code when I view your mail o_O
Marco
Hey all,
I have now written a litte tool in php which searchs the translatewiki files of every user and checks them.
Description: https://wiki.toolserver.org/view/Translatewiki.net_interface
Source code: https://wiki.toolserver.org/view/Translatewiki.net_interface/list_builder.ph...
for yaml it uses Spyc http://code.google.com/p/spyc/ You will find all files under /home/luxo/translatewiki/ too.
improvings and comments are welcome...
Greetings, Luxo
toolserver-l@lists.wikimedia.org