jenkins-bot has submitted this change and it was merged.
Change subject: [FEAT] replace: Warn about format chars in string
......................................................................
[FEAT] replace: Warn about format chars in string
When a string contains formatting characters like U+200E it's not immediately
visible and it might not match text. So this is warning about all chars in the
Cf category as defined by the Unicode standard.
Change-Id: I38ffb9e63d6827de0a42ace39073105aa6761d2e
---
M scripts/replace.py
1 file changed, 12 insertions(+), 0 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/replace.py b/scripts/replace.py
index 19a7855..7cb7764 100755
--- a/scripts/replace.py
+++ b/scripts/replace.py
@@ -137,6 +137,7 @@
import re
import time
import sys
+import unicodedata
import pywikibot
from pywikibot import i18n, textlib, pagegenerators, Bot
@@ -666,6 +667,11 @@
return pattern
+def contains_format_characters(text):
+ """Return True when there are format characters (e.g. U+200E) in text."""
+ return any(unicodedata.category(char) == 'Cf' for char in text)
+
+
def main(*args):
"""
Process command line arguments and invoke bot.
@@ -875,6 +881,12 @@
set_summary)
for replacement in fix['replacements']:
summary = None if len(replacement) < 3 else replacement[2]
+ if contains_format_characters(replacement[0]):
+ pywikibot.warning('The old string "{0}" contains formatting '
+ 'characters like U+200E'.format(replacement[0]))
+ if contains_format_characters(replacement[1]):
+ pywikibot.warning('The new string "{0}" contains formatting '
+ 'characters like U+200E'.format(replacement[1]))
replacements.append(ReplacementListEntry(
old=replacement[0],
new=replacement[1],
--
To view, visit https://gerrit.wikimedia.org/r/213273
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I38ffb9e63d6827de0a42ace39073105aa6761d2e
Gerrit-PatchSet: 2
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: Remove .py translation files
......................................................................
Remove .py translation files
We have JSON now. Thanks, Amir!
Change-Id: I9c926ae3eca5a10ccc608e7901b08f09b8f8cf5f
---
D .pep8
M __init__.py
D add_text.py
D archivebot.py
D basic.py
D blockpageschecker.py
D capitalize_redirects.py
D casechecker.py
D catall.py
D category.py
D category_redirect.py
D clean_sandbox.py
D commons.py
D commons_link.py
D commonscat.py
D cosmetic_changes.py
D deledpimage.py
D delete.py
D djvutext.py
D editarticle.py
D featured.py
D fixing_redirects.py
D followlive.py
D interwiki.py
D isbn.py
D lonelypages.py
D makecat.py
D misspelling.py
D movepages.py
D ndashredir.py
D noreferences.py
D pagefromfile.py
D piper.py
D protect.py
D pywikibot.py
D redirect.py
D reflinks.py
D replace.py
D revertbot.py
D selflink.py
D solve_disambiguation.py
D spamremove.py
D spellcheck.py
D table2wiki.py
D template.py
D thirdparty.py
D undelete.py
D unlink.py
D unprotect.py
D weblinkchecker.py
D welcome.py
D wiktionary.py
52 files changed, 4 insertions(+), 26,820 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
--
To view, visit https://gerrit.wikimedia.org/r/213289
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I9c926ae3eca5a10ccc608e7901b08f09b8f8cf5f
Gerrit-PatchSet: 3
Gerrit-Project: pywikibot/i18n
Gerrit-Branch: master
Gerrit-Owner: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: Change year to 2015 in documentation
......................................................................
Change year to 2015 in documentation
Change-Id: Icf48959be6178ee6db61a747d63697808a72e7a1
---
M docs/conf.py
1 file changed, 2 insertions(+), 2 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/docs/conf.py b/docs/conf.py
index 94e9571..84ed0a8 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -43,7 +43,7 @@
# General information about the project.
project = u'Pywikibot'
-copyright = u'2014, Pywikibot team'
+copyright = u'2015, Pywikibot team'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
@@ -52,7 +52,7 @@
# The short X.Y version.
version = '2.0'
# The full version, including alpha/beta/rc tags.
-release = '2.0b2'
+release = '2.0b3'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
--
To view, visit https://gerrit.wikimedia.org/r/213252
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Icf48959be6178ee6db61a747d63697808a72e7a1
Gerrit-PatchSet: 2
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: Use JSON for i18n files
......................................................................
Use JSON for i18n files
The JSON files were first distributed in October 2014.
The i18n module now uses those JSON files for i18n messages.
Bug: T65327
Change-Id: I6e2c63db039854dd8d009d837b2e6b9d11ba217a
---
M pywikibot/i18n.py
D tests/i18n/test.py
A tests/i18n/test/de.json
A tests/i18n/test/en.json
A tests/i18n/test/fr.json
A tests/i18n/test/fy.json
A tests/i18n/test/ja.json
A tests/i18n/test/nl.json
8 files changed, 103 insertions(+), 119 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
Ladsgroup: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/pywikibot/i18n.py b/pywikibot/i18n.py
index 77e57c6..327abf5 100644
--- a/pywikibot/i18n.py
+++ b/pywikibot/i18n.py
@@ -29,7 +29,11 @@
import sys
import re
import locale
-import warnings
+import json
+import os
+import pkgutil
+
+from collections import defaultdict
from pywikibot import Error
from .plural import plural_rules
@@ -43,10 +47,16 @@
PLURAL_PATTERN = r'{{PLURAL:(?:%\()?([^\)]*?)(?:\)d)?\|(.*?)}}'
-# Package name for the translation messages
+# Package name for the translation messages. The messages data must loaded
+# relative to that package name. In the top of this package should be
+# directories named after for each script/message bundle, and each directory
+# should contain JSON files called <lang>.json
_messages_package_name = 'scripts.i18n'
# Flag to indicate whether translation messages are available
_messages_available = None
+
+# Cache of translated messages
+_cache = defaultdict(dict)
def set_messages_package(package_name):
@@ -70,20 +80,9 @@
global _messages_available
if _messages_available is not None:
return _messages_available
- with warnings.catch_warnings():
- # Ignore 'missing __init__.py' as import looks at the JSON
- # directories before loading the python file.
- try:
- warnings.simplefilter("ignore", ImportWarning)
- # it's explicitly using str() to bypass unicode_literals in Python 2
- module = __import__(_messages_package_name, fromlist=[str('pywikibot')])
- except ImportError:
- _messages_available = False
- return False
-
try:
- getattr(module, 'pywikibot').msg
- except AttributeError:
+ __import__(_messages_package_name)
+ except ImportError:
_messages_available = False
return False
@@ -301,29 +300,29 @@
pass
-def _get_messages_bundle(name):
- """Load all translation messages for a bundle name."""
- exception_message = 'Unknown problem'
- transdict = {}
+def _get_translation(lang, twtitle):
+ """
+ Return message of certain twtitle if exists.
- with warnings.catch_warnings():
- # Ignore 'missing __init__.py' as import looks at the JSON
- # directories before loading the python file.
- warnings.simplefilter("ignore", ImportWarning)
- try:
- # it's explicitly using str() to bypass unicode_literals in Python 2
- transdict = getattr(__import__(_messages_package_name,
- fromlist=[str(name)]),
- name).msg
- except ImportError as e:
- exception_message = str(e)
-
- if not transdict:
- raise TranslationError(
- 'Could not load bundle %s from message package %s: %s'
- % (name, _messages_package_name, exception_message))
-
- return transdict
+ For internal use, don't use it directly.
+ """
+ if twtitle in _cache[lang]:
+ return _cache[lang][twtitle]
+ message_bundle = twtitle.split('-')[0]
+ trans_text = None
+ filename = '%s/%s.json' % (message_bundle, lang)
+ try:
+ trans_text = pkgutil.get_data(
+ _messages_package_name, filename).decode('utf-8')
+ except (OSError, IOError): # file open can cause several exceptions
+ _cache[lang][twtitle] = None
+ return
+ transdict = json.loads(trans_text)
+ _cache[lang].update(transdict)
+ try:
+ return transdict[twtitle]
+ except KeyError:
+ return
def _extract_plural(code, message, parameters):
@@ -454,8 +453,7 @@
"""
Translate a message.
- The translations are retrieved from i18n.<package>, based on the callers
- import table.
+ The translations are retrieved from json files in messages_package_name.
fallback parameter must be True for i18n and False for L10N or testing
purposes.
@@ -469,10 +467,10 @@
if not messages_available():
raise TranslationError(
'Unable to load messages package %s for bundle %s'
+ '\nIt can happen due to lack of i18n submodule or files. '
+ 'Read https://mediawiki.org/wiki/PWB/i18n'
% (_messages_package_name, twtitle))
- package = twtitle.split("-")[0]
- transdict = _get_messages_bundle(package)
code_needed = False
# If a site is given instead of a code, use its language
if hasattr(code, 'lang'):
@@ -487,25 +485,18 @@
# There are two possible failure modes: the translation dict might not have
# the language altogether, or a specific key could be untranslated. Both
# modes are caught with the KeyError.
-
- trans = None
- try:
- trans = transdict[lang][twtitle]
- except KeyError:
- # try alternative languages and English
- if fallback:
- for alt in _altlang(lang) + ['en']:
- try:
- trans = transdict[alt][twtitle]
- if code_needed:
- lang = alt
- break
- except KeyError:
- continue
- if trans is None:
- raise TranslationError(
- "No English translation has been defined "
- "for TranslateWiki key %r" % twtitle)
+ langs = [lang]
+ if fallback:
+ langs += _altlang(lang) + ['en']
+ for alt in langs:
+ trans = _get_translation(alt, twtitle)
+ if trans:
+ break
+ else:
+ raise TranslationError(
+ 'No English translation has been defined for TranslateWiki key'
+ ' %r\nIt can happen due to lack of i18n submodule or files. '
+ 'Read https://mediawiki.org/wiki/PWB/i18n' % twtitle)
# send the language code back via the given list
if code_needed:
code.append(lang)
@@ -527,23 +518,16 @@
it takes that variant calculated by the plural_rules depending on the number
value. Multiple plurals are allowed.
- As an examples, if we had a test dictionary in test.py like::
-
- msg = {
- 'en': {
- # number value as format string is allowed
- 'test-plural': u'Bot: Changing %(num)s {{PLURAL:%(num)d|page|pages}}.',
- },
- 'nl': {
- # format string inside PLURAL tag is allowed
- 'test-plural': u'Bot: Pas {{PLURAL:num|1 pagina|%(num)d pagina\'s}} aan.',
- },
- 'fr': {
- # additional string inside or outside PLURAL tag is allowed
- 'test-plural': u'Robot: Changer %(descr)s {{PLURAL:num|une page|quelques pages}}.',
- },
- }
-
+ As an examples, if we had several json dictionaries in test folder like:
+ en.json:
+ {
+ "test-plural": "Bot: Changing %(num)s {{PLURAL:%(num)d|page|pages}}.",
+ }
+ fr.json
+ {
+ "test-plural": "Robot: Changer %(descr)s {{PLURAL:num|une page|quelques pages}}.",
+ }
+ and so on.
>>> from pywikibot import i18n
>>> i18n.set_messages_package('tests.i18n')
>>> # use a number
@@ -602,17 +586,13 @@
@param code: The language code
@param twtitle: The TranslateWiki string title, in <package>-<key> format
"""
- package = twtitle.split("-")[0]
- transdict = _get_messages_bundle(package)
- if not transdict:
- pywikibot.warning('twhas_key: Could not load message bundle %s.%s'
- % (_messages_package_name, package))
- return False
-
# If a site is given instead of a code, use its language
if hasattr(code, 'lang'):
code = code.lang
- return code in transdict and twtitle in transdict[code]
+ transdict = _get_translation(code, twtitle)
+ if transdict is None:
+ return False
+ return True
def twget_keys(twtitle):
@@ -621,9 +601,21 @@
@param twtitle: The TranslateWiki string title, in <package>-<key> format
"""
+ # obtain the directory containing all the json files for this package
package = twtitle.split("-")[0]
- transdict = _get_messages_bundle(package)
- return (lang for lang in sorted(transdict.keys()) if lang != 'qqq')
+ mod = __import__(_messages_package_name, fromlist=[str('__file__')])
+ pathname = os.path.join(os.path.dirname(mod.__file__), package)
+
+ # build a list of languages in that directory
+ langs = [filename.partition('.')[0]
+ for filename in sorted(os.listdir(pathname))
+ if filename.endswith('.json')]
+
+ # exclude languages does not have this specific message in that package
+ # i.e. an incomplete set of translated messages.
+ return [lang for lang in langs
+ if lang != 'qqq' and
+ _get_translation(lang, twtitle)]
def input(twtitle, parameters=None, password=False, fallback_prompt=None):
diff --git a/tests/i18n/test.py b/tests/i18n/test.py
deleted file mode 100644
index 315224a..0000000
--- a/tests/i18n/test.py
+++ /dev/null
@@ -1,32 +0,0 @@
-# -*- coding: utf-8 -*-
-"""Test i18n data."""
-from __future__ import unicode_literals
-
-msg = {
- 'de': {
- 'test-plural': u'Bot: Ändere %(num)d {{PLURAL:num|Seite|Seiten}}.',
- 'test-multiple-plurals':
- u'Bot: %(action)s %(line)s {{PLURAL:line|Zeile|Zeilen}} von '
- u'{{PLURAL:%(page)d|einer|mehreren}} {{PLURAL:page|Seite|Seiten}}.',
- },
- 'en': {
- 'test-localized': u'test-localized EN',
- 'test-semi-localized': u'test-semi-localized EN',
- 'test-non-localized': u'test-non-localized EN',
- 'test-plural': u'Bot: Changing %(num)s {{PLURAL:%(num)d|page|pages}}.',
- },
- 'nl': {
- 'test-localized': u'test-localized NL',
- 'test-semi-localized': u'test-semi-localized NL',
- 'test-plural': u'Bot: Pas {{PLURAL:num|1 pagina|%(num)d pagina\'s}} aan.',
- },
- 'fr': {
- 'test-plural': u'Robot: Changer %(descr)s {{PLURAL:num|une page|quelques pages}}.',
- },
- 'fy': {
- 'test-localized': u'test-localized FY'
- },
- 'ja': {
- 'test-no-english': u'test-no-english JA'
- }
-}
diff --git a/tests/i18n/test/de.json b/tests/i18n/test/de.json
new file mode 100644
index 0000000..5309641
--- /dev/null
+++ b/tests/i18n/test/de.json
@@ -0,0 +1,4 @@
+{
+ "test-multiple-plurals": "Bot: %(action)s %(line)s {{PLURAL:line|Zeile|Zeilen}} von {{PLURAL:%(page)d|einer|mehreren}} {{PLURAL:page|Seite|Seiten}}.",
+ "test-plural": "Bot: Ändere %(num)d {{PLURAL:num|Seite|Seiten}}."
+}
\ No newline at end of file
diff --git a/tests/i18n/test/en.json b/tests/i18n/test/en.json
new file mode 100644
index 0000000..1d53316
--- /dev/null
+++ b/tests/i18n/test/en.json
@@ -0,0 +1,6 @@
+{
+ "test-localized": "test-localized EN",
+ "test-non-localized": "test-non-localized EN",
+ "test-plural": "Bot: Changing %(num)s {{PLURAL:%(num)d|page|pages}}.",
+ "test-semi-localized": "test-semi-localized EN"
+}
\ No newline at end of file
diff --git a/tests/i18n/test/fr.json b/tests/i18n/test/fr.json
new file mode 100644
index 0000000..4066396
--- /dev/null
+++ b/tests/i18n/test/fr.json
@@ -0,0 +1,3 @@
+{
+ "test-plural": "Robot: Changer %(descr)s {{PLURAL:num|une page|quelques pages}}."
+}
\ No newline at end of file
diff --git a/tests/i18n/test/fy.json b/tests/i18n/test/fy.json
new file mode 100644
index 0000000..5e207fb
--- /dev/null
+++ b/tests/i18n/test/fy.json
@@ -0,0 +1,3 @@
+{
+ "test-localized": "test-localized FY"
+}
\ No newline at end of file
diff --git a/tests/i18n/test/ja.json b/tests/i18n/test/ja.json
new file mode 100644
index 0000000..d9a9084
--- /dev/null
+++ b/tests/i18n/test/ja.json
@@ -0,0 +1,3 @@
+{
+ "test-no-english": "test-no-english JA"
+}
\ No newline at end of file
diff --git a/tests/i18n/test/nl.json b/tests/i18n/test/nl.json
new file mode 100644
index 0000000..7f695c9
--- /dev/null
+++ b/tests/i18n/test/nl.json
@@ -0,0 +1,5 @@
+{
+ "test-localized": "test-localized NL",
+ "test-plural": "Bot: Pas {{PLURAL:num|1 pagina|%(num)d pagina's}} aan.",
+ "test-semi-localized": "test-semi-localized NL"
+}
\ No newline at end of file
--
To view, visit https://gerrit.wikimedia.org/r/151114
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I6e2c63db039854dd8d009d837b2e6b9d11ba217a
Gerrit-PatchSet: 69
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Nikerabbit <niklas.laxstrom(a)gmail.com>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: [IMPROV] docs: Mention that Python 3 is supported
......................................................................
[IMPROV] docs: Mention that Python 3 is supported
Change-Id: Ie8973a90485a1bd6f05de383e5dd8ea902f6e713
---
M docs/index.rst
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Ladsgroup: Looks good to me, approved
jenkins-bot: Verified
diff --git a/docs/index.rst b/docs/index.rst
index 692f264..bcf202d 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -12,7 +12,7 @@
Pywikibot is a full-stack framework for editing `MediaWiki <https://mediawiki.org>`_ wiki's. (more intro)
-Pywikibot supports Python 2.6.5 and 2.7; Python 3 support is under way.
+Pywikibot supports Python 2.6.5, 2.7 and 3.2+.
Pywikibot is licensed under the :ref:`MIT license <licenses-MIT>`; the documentation is licensed under the :ref:`CC-BY-SA-3.0 <licenses-CCBYSA>` license.
--
To view, visit https://gerrit.wikimedia.org/r/213229
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie8973a90485a1bd6f05de383e5dd8ea902f6e713
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>