jayvdb created this task. jayvdb added subscribers: pywikipedia-bugs, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup. jayvdb added projects: i18n, Pywikibot-i18n.
TASK DESCRIPTION JSON files have been added to pywikibot/i18n automatically (https://gerrit.wikimedia.org/r/#/c/164947/), which now has python and JSON files with the same messages. The JSON files are not used yet, as the code changes to enable JSON have exposed packaging problems that are the subject of RFC https://www.mediawiki.org/wiki/Requests_for_comment/pywikibot_2.0_packaging
The message files for the English message do not include the attribution; it is missing from the python message files and now also missing from the JSON messages files.
``` $ head -50 pywikibot.py # -*- coding: utf-8 -*- msg = { 'en': { 'pywikibot-enter-category-name': u'Please enter the category name:', 'pywikibot-enter-file-links-processing': u'Links to which file page should be processed?', 'pywikibot-enter-finished-browser': u'Press Enter when finished in browser.', 'pywikibot-enter-namespace-number': u'Please enter a namespace by its number:', 'pywikibot-enter-new-text': u'Please enter the new text:', 'pywikibot-enter-page-processing': u'Which page should be processed?', 'pywikibot-enter-xml-filename': u'Please enter the XML dump's filename:', }, # Author: Als-Holder # Author: TTMTT 'qqq': { 'pywikibot-enter-xml-filename': u'Message displayed to the bot owner to enter the XML dump's filename.', 'pywikibot-enter-page-processing': u'Question displayed to the bot owner which page should be processed.', 'pywikibot-enter-file-links-processing': u'Question displayed to the bot owner processing links to a given file page.', 'pywikibot-enter-namespace-number': u'Message displayed to the bot owner to enter a namespace by its number.', 'pywikibot-enter-new-text': u'Message displayed to the bot owner to enter the new text.', 'pywikibot-enter-category-name': u'Message displayed to the bot owner to enter the category name.', 'pywikibot-enter-finished-browser': u'Message displayed to the bot owner to press Enter button when browser edits are finished.', }, # Author: Als-Holder 'als': { 'pywikibot-enter-xml-filename': u'Bitte gib dr Datename vum XML-Dump yy:', 'pywikibot-enter-page-processing': u'Weli Syte soll bearbeitet wäre?', 'pywikibot-enter-file-links-processing': u'Vu wellere Dateisyte solle d Link bearbeitet wäre?', 'pywikibot-enter-namespace-number': u'Bitte gib d Nummere vum Namensruum yy:', 'pywikibot-enter-new-text': u'Bitte gib dr nei Text yy:', 'pywikibot-enter-category-name': u'Bitte gib dr Name vu dr Kategori yy:', 'pywikibot-enter-finished-browser': u'Druck noch eme Zuemache vum Browsewr uf «Enter».', }, ... ```
The metadata for als is correct
``` $ cat pywikibot/als.json { "@metadata": { "authors": [ "Als-Holder" ] }, "pywikibot-enter-xml-filename": "Bitte gib dr Datename vum XML-Dump yy:", "pywikibot-enter-page-processing": "Weli Syte soll bearbeitet wäre?", "pywikibot-enter-file-links-processing": "Vu wellere Dateisyte solle d Link bearbeitet wäre?", "pywikibot-enter-namespace-number": "Bitte gib d Nummere vum Namensruum yy:", "pywikibot-enter-new-text": "Bitte gib dr nei Text yy:", "pywikibot-enter-category-name": "Bitte gib dr Name vu dr Kategori yy:", "pywikibot-enter-finished-browser": "Druck noch eme Zuemache vum Browsewr uf «Enter»." } ```
However the metadata for 'qqq' is omitted
``` $ cat pywikibot/qqq.json { "@metadata": [], "pywikibot-enter-xml-filename": "Message displayed to the bot owner to enter the XML dump's filename.", "pywikibot-enter-page-processing": "Question displayed to the bot owner which page should be processed.", "pywikibot-enter-file-links-processing": "Question displayed to the bot owner processing links to a given file page.", "pywikibot-enter-namespace-number": "Message displayed to the bot owner to enter a namespace by its number.", "pywikibot-enter-new-text": "Message displayed to the bot owner to enter the new text.", "pywikibot-enter-category-name": "Message displayed to the bot owner to enter the category name.", "pywikibot-enter-finished-browser": "Message displayed to the bot owner to press Enter button when browser edits are finished." } ```
And the 'en' files do not include a metadata block, which means they fail the i18n JSON syntax tests (T85335). ``` $ cat pywikibot/en.json { "pywikibot-enter-xml-filename": "Please enter the XML dump's filename:", "pywikibot-enter-page-processing": "Which page should be processed?", "pywikibot-enter-file-links-processing": "Links to which file page should be processed?", "pywikibot-enter-namespace-number": "Please enter a namespace by its number:", "pywikibot-enter-new-text": "Please enter the new text:", "pywikibot-enter-category-name": "Please enter the category name:", "pywikibot-enter-finished-browser": "Press Enter when finished in browser." } ```
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
gerritbot added a project: Patch-For-Review. gerritbot added a comment.
Change 181370 had a related patch set uploaded (by John Vandenberg): Add grunt test to validate i18n JSON
https://gerrit.wikimedia.org/r/181370
https://phabricator.wikimedia.org/tag/patch-for-review/
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: gerritbot Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
jayvdb edited the task description. jayvdb set Security to none.
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
jayvdb added a project: Google-Code-in-2014. jayvdb added a comment.
We could create a https://phabricator.wikimedia.org/tag/google-code-in-2014/ task to fix this, if the #translatewiki team dont have an easy/quick fix for this. I would need a translatewiki team member to help draft the task and ideally also co-mentor the task.
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
gerritbot added a comment.
Change 181370 merged by jenkins-bot: Add grunt test to validate i18n JSON
https://gerrit.wikimedia.org/r/181370
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: gerritbot Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
Xqt closed this task as "Resolved". Xqt claimed this task.
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Xqt Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
jayvdb reopened this task as "Open". jayvdb added a comment.
And as we can see on https://gerrit.wikimedia.org/r/#/c/182691/ , there are many cases of translated message bundles where the JSON files do not have the same attribution as the Python.
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Xqt, jayvdb Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
jayvdb reassigned this task from Xqt to siebrand.
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: siebrand, jayvdb Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
Nemo_bis added a subscriber: Nemo_bis. Nemo_bis added a comment.
In https://phabricator.wikimedia.org/T85336#957064, @jayvdb wrote:
And as we can see on https://gerrit.wikimedia.org/r/#/c/182691/ , there are many cases of translated message bundles where the JSON files do not have the same attribution as the Python.
I don't see what generated the JSON files or the Python excerpts in question, but one thing to consider is that maybe the older files are wrong: the "missing" attribution might also be attribution of translations which no longer exist, e.g. tranlsations of messages deleted in the meanwhile. Worth doing some sampled checking.
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: siebrand, Nemo_bis Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Nemo_bis, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
jayvdb added a subscriber: Xqt. jayvdb added a comment.
so just going through the first file
https://gerrit.wikimedia.org/r/#/c/182691/2/add_text.py,cm
1. removes Csisc from 'qqq'
twn only has one revision, importing it
https://translatewiki.net/w/i.php?title=Pywikibot:Add_text-adding/qqq&ac...
it changed position by @siebrand in 1a89684f . http://git.wikimedia.org/blobdiff/pywikibot%2Fi18n.git/1a89684f/add_text.py
it was changed position by @siebrand in 448c4dee , from 'aeb' to 'qqq', in what looks like a case of mis-attribution.
http://git.wikimedia.org/blobdiff/pywikibot%2Fi18n.git/448c4dee/add_text.py
2. removes 'Anskar' from 'ca'
'Anskar' did improve the translation, on twn.
https://translatewiki.net/w/i.php?title=Pywikibot:Add_text-adding/ca&act...
Also, the original twn translation ("Robot afegeix %(adding)s") isnt attributed. @xqt added it in 9af00aaa . http://git.wikimedia.org/commit/pywikibot%2Fi18n.git/9af00aaa It wasnt in the list that was removed https://www.mediawiki.org/wiki/Special:Code/pywikipedia/9378 , so it had to come from somewhere else.
Ah ..., it looks like @xqt copied the translations from the interwiki-adding message, which were added by @valhallasw , also without attribution.
http://git.wikimedia.org/blob/pywikibot%2Fi18n.git/71623c02d160849155a26ad92...
Looking through the 'compat' history of the interwiki.py script, the 'ca' translation was provided by "Arkaitz Zubiaga" in 2006. https://www.mediawiki.org/wiki/Special:Code/pywikipedia/2864
3. This is the 'ckb' entry for add_text and similar to (2) above. 'Asoxor' has been removed, but the definitely contributed to the translation over time, and part of their translation still exists in the current translation.
https://translatewiki.net/w/i.php?title=Pywikibot:Add_text-adding/ckb&ac...
The ckb original translation also has the same problem as above. The commit only mentions a SF #:
https://mediawiki.org/wiki/Special:Code/pywikipedia/8146
4. 'da' - same problem, removing Christian List. There is little change since the original imported translation, so maybe neither Christian or Kaare should be attribute.
https://translatewiki.net/w/i.php?title=Pywikibot%3AAdd_text-adding%2Fda&...
(da was also taken from interwiki.py; same problem)
5. 'es', removes attribution of both TheBITLINK and Xqt. Xqt rightly removed attribution for a mechanical change, and TheBITLINK's change has been removed from the current translation, so I guess they dont need to be attributed any more???
https://translatewiki.net/w/i.php?title=Pywikibot%3AAdd_text-adding%2Fes&...
6. 'ja', removed attribution of Fryed-peach, despite them being a contributor the current translation text, and doesnt include attribution of the original translation, provided in https://mediawiki.org/wiki/Special:Code/pywikipedia/4868 with a SF #.
(there are another 7 cases of attribution removal in add_text.py alone, which I have only glanced at.)
From only a very quick look at the cases above, it appears that only the most recent twn editor is being credited.
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: siebrand, jayvdb Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Nemo_bis, Xqt, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
Nikerabbit added a comment.
Thanks for your effort investigating this. It does match the expected behavior: translatewiki.net only credits most recent translators of each translation. But usually this is offset by the fact that we retain existing credits. Due to way this is implement (trying to parse the source file) this does not work when exporting in non-native file format, which is what I think was done in this case to do the conversion.
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: siebrand, Nikerabbit Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Nemo_bis, Xqt, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
jayvdb added a comment.
In https://phabricator.wikimedia.org/T85336#958925, @Nikerabbit wrote:
Thanks for your effort investigating this. It does match the expected behavior: translatewiki.net only credits most recent translators of each translation. But usually this is offset by the fact that we retain existing credits. Due to way this is implement (trying to parse the source file) this does not work when exporting in non-native file format, which is what I think was done in this case to do the conversion.
TWN tools can't export all message contributors into a new JSON message file? (i.e. editors of the page on TWN)
Or, more bluntly, do we need to migrate the old translation metadata data from the python files to the new JSON files? It isnt a hard job, so no problems. I just want to be clear who is doing what to fix this situation so we can complete the migration to the JSON file format.
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: siebrand, jayvdb Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Nemo_bis, Xqt, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
Nemo_bis added a comment.
In https://phabricator.wikimedia.org/T85336#958980, @jayvdb wrote:
TWN tools can't export all message contributors into a new JSON message file? (i.e. editors of the page on TWN)
The full editors list is not a list of "all message contributors", but of all *potential* message contributors. Including, say, a vandal who got reverted. If https://phabricator.wikimedia.org/T4994 was fixed, then Translate could be smarter (there was some research progress, so it might happen).
Or, more bluntly, do we need to migrate the old translation metadata data from the python files to the new JSON files?
Yes. However out of 6 cases, you found 1 (or perhaps 2) where the older editor contributed some *words* of the current sentence... it's not a concrete attribution issue except for the pre-TWN translators.
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: siebrand, Nemo_bis Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Nemo_bis, Xqt, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
jayvdb added a comment.
In https://phabricator.wikimedia.org/T85336#960707, @Nemo_bis wrote:
In https://phabricator.wikimedia.org/T85336#958980, @jayvdb wrote:
TWN tools can't export all message contributors into a new JSON message file? (i.e. editors of the page on TWN)
The full editors list is not a list of "all message contributors", but of all *potential* message contributors. Including, say, a vandal who got reverted. If https://phabricator.wikimedia.org/T4994 was fixed, then Translate could be smarter (there was some research progress, so it might happen).
It is _always_ better to over-attribute than under-attribute :/
proper reverts are able to be eliminated with very minimal fuss; that isnt a research problem. And on a small wiki like TWN, and since it works on discrete units of information, that will eliminate 90+% of problematic attributions.
Add a default enabled block button checkbox "exclude user from translation attribution", and another 8% is solved. ;-) And nobody cares about the last 2% of false positives/
Or, more bluntly, do we need to migrate the old translation metadata data from the python files to the new JSON files?
Yes. However out of 6 cases, you found 1 (or perhaps 2) where the older editor contributed some *words* of the current sentence... it's not a concrete attribution issue except for the pre-TWN translators.
My count is 4 of 6 if I am being generous, or 5 of 6 if I am not. :/ I am only really excluding case (1), as that problem occurred before the history of the TWN page history. That TheBITLINK doesnt need to be attributed for the es translation doesnt feel right given their translation was used by pywikibot for a year an a quarter.
Note that the "current sentence" for this message is two words long in English and similar length in most other languages. If one word of a translators work remains in the current message, that is 50% of the translation.
Anyway, it sounds like the pywikibot team needs to manually add the list of contributors from TWN to the JSON in order for the attribution to not regress further. i.e. the attribution removals in Xqt's patch to the python need to be reversed into additions to the JSON. Or, it might be more efficient to obtain the TWN contributor list from the TWN wiki for each message page. Luckily we have a tool which can do that .. ;-) I can't help but think we'd be re-inventing the TWN tools.
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: siebrand, jayvdb Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Nemo_bis, Xqt, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
Nikerabbit added a comment.
In absence of other replies, I would appreciate if you can copy over the missing attributions to JSON files. That also takes care if there are any non-twn authors in the list.
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: siebrand, Nikerabbit Cc: Aklapper, valhallasw, Nikerabbit, siebrand, jayvdb, Ladsgroup, Nemo_bis, Xqt, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
jayvdb moved this task to pywikibot on the Google-Code-in-2014 workboard.
TASK DETAIL https://phabricator.wikimedia.org/T85336
WORKBOARD https://phabricator.wikimedia.org/project/board/74/
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: siebrand, jayvdb Cc: Xqt, siebrand, Nemo_bis, jayvdb, valhallasw, Aklapper, Ladsgroup, Nikerabbit, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
jayvdb removed a project: Google-Code-in-2014.
TASK DETAIL https://phabricator.wikimedia.org/T85336
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: siebrand, jayvdb Cc: Xqt, siebrand, Nemo_bis, jayvdb, valhallasw, Aklapper, Ladsgroup, Nikerabbit, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
gerritbot added a subscriber: gerritbot. gerritbot added a comment.
Change 212885 had a related patch set uploaded (by Ladsgroup): Add proper attribution in json files based on .py files
https://gerrit.wikimedia.org/r/212885
TASK DETAIL https://phabricator.wikimedia.org/T85336
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: siebrand, gerritbot Cc: gerritbot, Xqt, siebrand, Nemo_bis, jayvdb, valhallasw, Aklapper, Ladsgroup, Nikerabbit, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
gerritbot added a comment.
Change 212885 merged by jenkins-bot: Add proper attribution in json files based on .py files
https://gerrit.wikimedia.org/r/212885
TASK DETAIL https://phabricator.wikimedia.org/T85336
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: siebrand, gerritbot Cc: gerritbot, Xqt, siebrand, Nemo_bis, jayvdb, valhallasw, Aklapper, Ladsgroup, Nikerabbit, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
Ladsgroup closed this task as "Resolved". Ladsgroup claimed this task. Ladsgroup removed a project: Patch-For-Review.
TASK DETAIL https://phabricator.wikimedia.org/T85336
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ladsgroup Cc: gerritbot, Xqt, siebrand, Nemo_bis, jayvdb, valhallasw, Aklapper, Ladsgroup, Nikerabbit, Gryllida, Shizhao, Arrbee, pywikipedia-bugs
Ladsgroup added a project: Wikimedia-Hackathon-2015.
TASK DETAIL https://phabricator.wikimedia.org/T85336
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ladsgroup Cc: gerritbot, Xqt, siebrand, Nemo_bis, jayvdb, valhallasw, Aklapper, Ladsgroup, Nikerabbit, Sylvain_WMFr, AlexWang, Daniel_Mietchen, Gryllida, Shizhao, Arrbee, Qgil, pywikipedia-bugs
pywikipedia-bugs@lists.wikimedia.org