jenkins-bot has submitted this change and it was merged.
Change subject: [FEAT] pagefromfile: Support separator with append
......................................................................
[FEAT] pagefromfile: Support separator with append
When using appendbottom or appendtop it is possible to add any text between the
old and new text.
Change-Id: I116cbb644135fa3ddc25c5a01f0f3415a865e681
---
M scripts/pagefromfile.py
1 file changed, 23 insertions(+), 13 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/pagefromfile.py b/scripts/pagefromfile.py
index c32f7d7..1eda597 100755
--- a/scripts/pagefromfile.py
+++ b/scripts/pagefromfile.py
@@ -1,6 +1,6 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
-"""
+r"""
Bot to upload pages from a file.
This bot takes its input from a file that contains a number of
@@ -46,6 +46,11 @@
-appendtop add the text to the top of it
-appendbottom add the text to the bottom of it
-force overwrite the existing page
+
+It's possible to define a separator after the 'append' modes which is added
+between the exisiting and new text. For example -appendtop:foo would add 'foo'
+between the parts. The \n (two separate characters) is replaced by the newline
+character.
"""
#
# (C) Andre Engels, 2004
@@ -138,16 +143,19 @@
nocontent.lower() in pagecontents):
pywikibot.output('Page has %s so it is skipped' % nocontent)
return
- if self.getOption('append') == 'top':
- pywikibot.output(u"Page %s already exists, appending on top!"
- % title)
- contents = contents + pagecontents
- comment = comment_top
- elif self.getOption('append') == 'bottom':
- pywikibot.output(u"Page %s already exists, appending on bottom!"
- % title)
- contents = pagecontents + contents
- comment = comment_bottom
+ if self.getOption('append'):
+ separator = self.getOption('append')[1]
+ if separator == r'\n':
+ separator = '\n'
+ if self.getOption('append')[0] == 'top':
+ above, below = contents, pagecontents
+ comment = comment_top
+ else:
+ above, below = pagecontents, contents
+ comment = comment_bottom
+ pywikibot.output('Page {0} already exists, appending on {1}!'.format(
+ title, self.getOption('append')[0]))
+ contents = above + separator + below
elif self.getOption('force'):
pywikibot.output(u"Page %s already exists, ***overwriting!"
% title)
@@ -278,8 +286,10 @@
filename = arg[6:]
elif arg == "-include":
include = True
- elif arg.startswith('-append') and arg[7:] in ('top', 'bottom'):
- options['append'] = arg[7:]
+ elif arg.startswith('-appendbottom'):
+ options['append'] = ('bottom', arg[len('-appendbottom:'):])
+ elif arg.startswith('-appendtop'):
+ options['append'] = ('top', arg[len('-appendtop:'):])
elif arg == "-force":
options['force'] = True
elif arg == "-safe":
--
To view, visit https://gerrit.wikimedia.org/r/213524
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I116cbb644135fa3ddc25c5a01f0f3415a865e681
Gerrit-PatchSet: 2
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: T109077 Performance decreased saving pages for sites with non-alphabetical code interwiki sort (interwiki_putfirst specified)
......................................................................
T109077 Performance decreased saving pages for sites with non-alphabetical code interwiki sort (interwiki_putfirst specified)
Change-Id: I97e8ecf44c3e8099113068c343f35af94bc02e84
---
M pywikibot/textlib.py
1 file changed, 2 insertions(+), 1 deletion(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/textlib.py b/pywikibot/textlib.py
index 77454fc..bd7adbd 100644
--- a/pywikibot/textlib.py
+++ b/pywikibot/textlib.py
@@ -962,8 +962,9 @@
if putfirst:
# In this case I might have to change the order
firstsites = []
+ validlanglinks = insite.validLanguageLinks()
for code in putfirst:
- if code in insite.validLanguageLinks():
+ if code in validlanglinks:
site = insite.getSite(code=code)
if site in sites:
del sites[sites.index(site)]
--
To view, visit https://gerrit.wikimedia.org/r/231577
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I97e8ecf44c3e8099113068c343f35af94bc02e84
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Malafaya <malafaya(a)clix.pt>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>
Build Update for wikimedia/pywikibot-core
-------------------------------------
Build: #2654
Status: Broken
Duration: 40 minutes and 50 seconds
Commit: a451aed (master)
Author: Fabian Neundorf
Message: [FIX] scripts: unicode_literals import
These two are hopefully the last scripts with wrong unicode_literals import
since its introduction in 1e54a7d6.
Change-Id: I5306cbf637cea833f791b2b63c2415e757ab7e7f
View the changeset: https://github.com/wikimedia/pywikibot-core/compare/4239bff69db9...a451aed6…
View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/75618511
--
You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications
jenkins-bot has submitted this change and it was merged.
Change subject: [IMPROV] replace: Ask for summary if necessary
......................................................................
[IMPROV] replace: Ask for summary if necessary
At the moment it always asks for a default summary unless one was defined via
the command line. But if only fixes are used and all replacements in these
fixes have a summary defined (either via the fix itself or separate for each
replacement) it shouldn't ask for this. This makes it easier to be used
autonomously.
As a byproduct it now knows which replacements from fixes have no summary
defined and lists them before asking for a default summary.
It also adds an option (-automaticsummary) which does prevent that the script
asks the operator what summary it uses and allows to use it without
user intervention but using the automatically generated summary.
Change-Id: Ie72ed81550631c3f53f7991fc688ae73a9b2515e
---
M scripts/replace.py
M tests/data/fixes.py
M tests/replacebot_tests.py
M tox.ini
4 files changed, 92 insertions(+), 20 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/replace.py b/scripts/replace.py
index 9dacfb8..ec8d114 100755
--- a/scripts/replace.py
+++ b/scripts/replace.py
@@ -53,7 +53,10 @@
-summary:XYZ Set the summary message text for the edit to XYZ, bypassing
the predefined message texts with original and replacements
- inserted.
+ inserted. Can't be used with -automaticsummary.
+
+-automaticsummary Uses an automatic summary for all replacements which don't
+ have a summary defined. Can't be used with -summary.
-sleep:123 If you use -fix you can check multiple regex at the same time
in every page. This can lead to a great waste of CPU because
@@ -790,6 +793,8 @@
add_cat = arg[8:]
elif arg.startswith('-summary:'):
edit_summary = arg[9:]
+ elif arg.startswith('-automaticsummary'):
+ edit_summary = True
elif arg.startswith('-allowoverlap'):
allowoverlap = True
elif arg.startswith('-manualinput'):
@@ -846,6 +851,7 @@
'Please enter another text that should be replaced,'
'\nor press Enter to start:')
+ # The summary stored here won't be actually used but is only an example
single_summary = None
for i in range(0, len(commandline_replacements), 2):
replacement = Replacement(commandline_replacements[i],
@@ -858,22 +864,11 @@
)
replacements.append(replacement)
- if not edit_summary:
- if single_summary:
- pywikibot.output(u'The summary message for the command line '
- 'replacements will be something like: %s'
- % single_summary)
- if fixes_set:
- pywikibot.output('If a summary is defined for the fix, this '
- 'default summary won\'t be applied.')
- edit_summary = pywikibot.input(
- 'Press Enter to use this automatic message, or enter a '
- 'description of the\nchanges your bot will make:')
-
# Perform one of the predefined actions.
- for fix in fixes_set:
+ missing_fixes_summaries = [] # which a fixes/replacements miss a summary
+ for fix_name in fixes_set:
try:
- fix = fixes.fixes[fix]
+ fix = fixes.fixes[fix_name]
except KeyError:
pywikibot.output(u'Available predefined fixes are: %s'
% ', '.join(fixes.fixes.keys()))
@@ -881,6 +876,10 @@
pywikibot.output('The user fixes file could not be found: '
'{0}'.format(fixes.filename))
return
+ if not fix['replacements']:
+ pywikibot.warning('No replacements defined for fix '
+ '"{0}"'.format(fix_name))
+ continue
if "msg" in fix:
if isinstance(fix['msg'], basestring):
set_summary = i18n.twtranslate(site, str(fix['msg']))
@@ -892,8 +891,14 @@
fix.get('exceptions'),
fix.get('nocase'),
set_summary)
- for replacement in fix['replacements']:
+ # Whether some replacements have a summary, if so only show which
+ # have none, otherwise just mention the complete fix
+ missing_fix_summaries = []
+ for index, replacement in enumerate(fix['replacements'], start=1):
summary = None if len(replacement) < 3 else replacement[2]
+ if not set_summary and not summary:
+ missing_fix_summaries.append(
+ '"{0}" (replacement #{1})'.format(fix_name, index))
if chars.contains_invisible(replacement[0]):
pywikibot.warning('The old string "{0}" contains formatting '
'characters like U+200E'.format(
@@ -908,9 +913,34 @@
fix_set=replacement_set,
edit_summary=summary,
))
+
if replacement_set:
replacements.extend(replacement_set)
+ if len(fix['replacements']) == len(missing_fix_summaries):
+ missing_fixes_summaries.append(
+ '"{0}" (all replacements)'.format(fix_name))
+ else:
+ missing_fixes_summaries += missing_fix_summaries
+
+ if ((not edit_summary or edit_summary is True) and
+ (missing_fixes_summaries or single_summary)):
+ if single_summary:
+ pywikibot.output(u'The summary message for the command line '
+ 'replacements will be something like: %s'
+ % single_summary)
+ if missing_fixes_summaries:
+ pywikibot.output('The summary will not be used when the fix has '
+ 'one defined but the following fix(es) do(es) not '
+ 'have a summary defined: '
+ '{0}'.format(', '.join(missing_fixes_summaries)))
+ if edit_summary is not True:
+ edit_summary = pywikibot.input(
+ 'Press Enter to use this automatic message, or enter a '
+ 'description of the\nchanges your bot will make:')
+ else:
+ edit_summary = ''
+
# Set the regular expression flags
flags = re.UNICODE
if caseInsensitive:
diff --git a/tests/data/fixes.py b/tests/data/fixes.py
index ef76906..e750b98 100644
--- a/tests/data/fixes.py
+++ b/tests/data/fixes.py
@@ -32,6 +32,21 @@
]
}
+fixes['all-repl-msg'] = {
+ 'regex': False,
+ 'replacements': [
+ ('1', '2', 'M1'),
+ ]
+}
+
+fixes['partial-repl-msg'] = {
+ 'regex': False,
+ 'replacements': [
+ ('1', '2', 'M1'),
+ ('3', '4'),
+ ]
+}
+
fixes['has-msg-multiple'] = {
'regex': False,
'msg': {
diff --git a/tests/replacebot_tests.py b/tests/replacebot_tests.py
index 610b811..cd854c8 100644
--- a/tests/replacebot_tests.py
+++ b/tests/replacebot_tests.py
@@ -97,11 +97,17 @@
self.assertEqual(replacement.old, str(offset * 2 + 1))
self.assertEqual(replacement.new, str(offset * 2 + 2))
- def _test_fix_replacement(self, replacement, length=1, offset=0):
+ def _test_fix_replacement(self, replacement, length=1, offset=0, msg=False):
"""Test a replacement from a fix."""
assert length > offset
self._test_replacement(replacement, replace.ReplacementListEntry,
offset)
+ if msg:
+ self.assertEqual(replacement.edit_summary,
+ 'M{0}'.format(offset + 1))
+ else:
+ self.assertIs(replacement.edit_summary,
+ replacement.fix_set.edit_summary)
self.assertIsInstance(replacement.fix_set, replace.ReplacementList)
self.assertIsInstance(replacement.fix_set, list)
self.assertIn(replacement, replacement.fix_set)
@@ -128,15 +134,22 @@
self.assertEqual(len(bot.replacements), 1)
self._test_replacement(bot.replacements[0])
+ def test_cmd_automatic(self):
+ """Test command line replacements with automatic summary."""
+ bot = self._get_bot(None, '1', '2', '-automaticsummary')
+ self.assertEqual(len(bot.replacements), 1)
+ self._test_replacement(bot.replacements[0])
+ self.assertEqual(self.inputs, [])
+
def test_only_fix_global_message(self):
"""Test fixes replacements only."""
- bot = self._get_bot(True, '-fix:has-msg')
+ bot = self._get_bot(None, '-fix:has-msg')
self.assertEqual(len(bot.replacements), 1)
self._test_fix_replacement(bot.replacements[0])
def test_only_fix_global_message_tw(self):
"""Test fixes replacements only."""
- bot = self._get_bot(True, '-fix:has-msg-tw')
+ bot = self._get_bot(None, '-fix:has-msg-tw')
self.assertEqual(len(bot.replacements), 1)
self._test_fix_replacement(bot.replacements[0])
@@ -146,9 +159,22 @@
self.assertEqual(len(bot.replacements), 1)
self._test_fix_replacement(bot.replacements[0])
+ def test_only_fix_all_replacement_summary(self):
+ """Test fixes replacements only."""
+ bot = self._get_bot(None, '-fix:all-repl-msg')
+ self.assertEqual(len(bot.replacements), 1)
+ self._test_fix_replacement(bot.replacements[0], msg=True)
+
+ def test_only_fix_partial_replacement_summary(self):
+ """Test fixes replacements only."""
+ bot = self._get_bot(True, '-fix:partial-repl-msg')
+ for offset, replacement in enumerate(bot.replacements):
+ self._test_fix_replacement(replacement, 2, offset, offset == 0)
+ self.assertEqual(len(bot.replacements), 2)
+
def test_only_fix_multiple(self):
"""Test fixes replacements only."""
- bot = self._get_bot(True, '-fix:has-msg-multiple')
+ bot = self._get_bot(None, '-fix:has-msg-multiple')
for offset, replacement in enumerate(bot.replacements):
self._test_fix_replacement(replacement, 3, offset)
self.assertEqual(len(bot.replacements), 3)
diff --git a/tox.ini b/tox.ini
index c85de9f..e65a2cc 100644
--- a/tox.ini
+++ b/tox.ini
@@ -142,6 +142,7 @@
tests/protectbot_tests.py \
tests/pwb/ \
tests/pwb_tests.py \
+ tests/replacebot_tests.py \
tests/script_tests.py \
tests/site_detect_tests.py \
tests/tests_tests.py \
--
To view, visit https://gerrit.wikimedia.org/r/221377
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie72ed81550631c3f53f7991fc688ae73a9b2515e
Gerrit-PatchSet: 7
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: Fall back to utf-8 console encoding
......................................................................
Fall back to utf-8 console encoding
Bug: T93474
Change-Id: Iec28824bd74377c33a08964307449cebc90279e4
---
M pywikibot/config2.py
1 file changed, 6 insertions(+), 5 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/config2.py b/pywikibot/config2.py
index 0ef584b..13c4578 100644
--- a/pywikibot/config2.py
+++ b/pywikibot/config2.py
@@ -1007,12 +1007,13 @@
"Please inform the maintainers if you depend on it." % _key,
_ConfigurationDeprecationWarning)
-# Fix up default console_encoding
+# If we cannot auto-detect the console encoding (e.g. when piping data)
+# assume utf-8. On Linux, this will typically be correct; on windows,
+# this can be an issue when piping through more. However, the behavior
+# when redirecting to a file utf-8 is more reasonable.
+
if console_encoding is None:
- if sys.platform == 'win32':
- console_encoding = 'cp850'
- else:
- console_encoding = 'iso-8859-1'
+ console_encoding = 'utf-8'
# Fix up transliteration_target
if transliteration_target == 'not set':
--
To view, visit https://gerrit.wikimedia.org/r/231568
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Iec28824bd74377c33a08964307449cebc90279e4
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: [FIX] scripts: unicode_literals import
......................................................................
[FIX] scripts: unicode_literals import
These two are hopefully the last scripts with wrong unicode_literals import
since its introduction in 1e54a7d6.
Change-Id: I5306cbf637cea833f791b2b63c2415e757ab7e7f
---
M scripts/cosmetic_changes.py
M scripts/flickrripper.py
2 files changed, 4 insertions(+), 1 deletion(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/cosmetic_changes.py b/scripts/cosmetic_changes.py
index de6afcf..cb2c318 100644
--- a/scripts/cosmetic_changes.py
+++ b/scripts/cosmetic_changes.py
@@ -31,6 +31,8 @@
#
# Distributed under the terms of the MIT license.
#
+from __future__ import unicode_literals
+
__version__ = '$Id$'
#
diff --git a/scripts/flickrripper.py b/scripts/flickrripper.py
index 439c6ba..b449d71 100755
--- a/scripts/flickrripper.py
+++ b/scripts/flickrripper.py
@@ -15,7 +15,6 @@
#**Add a nice hotcat lookalike for the adding of categories
#**Filter the categories
#*Upload the image
-from __future__ import unicode_literals
Todo:
*Check if the image is already uploaded (SHA hash)
@@ -30,6 +29,8 @@
#
# Distributed under the terms of the MIT license.
#
+from __future__ import unicode_literals
+
__version__ = '$Id$'
#
--
To view, visit https://gerrit.wikimedia.org/r/231562
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I5306cbf637cea833f791b2b63c2415e757ab7e7f
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>