Pywikibot-commits April 2018

pywikibot-commits@lists.wikimedia.org

1 participants
88 discussions

[Gerrit] pywikibot/core[master]: [fix] charset is a keyword argument of request
by jenkins-bot (Code Review) 23 Apr '18

23 Apr '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428417 ) Change subject: [fix] charset is a keyword argument of request ...................................................................... [fix] charset is a keyword argument of request Change-Id: I3dfc91da47bcded58509d153f0b3074c6d251ec6 --- M pywikibot/comms/http.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Dalba: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/comms/http.py b/pywikibot/comms/http.py index 0ac5187..4c096e0 100644 --- a/pywikibot/comms/http.py +++ b/pywikibot/comms/http.py @@ -285,7 +285,7 @@ @type site: L{pywikibot.site.BaseSite} @param uri: the URI to retrieve @type uri: str - @param charset: Either a valid charset (usable for str.decode()) or None + @keyword charset: Either a valid charset (usable for str.decode()) or None to automatically chose the charset from the returned header (defaults to latin-1) @type charset: CodecInfo, str, None -- To view, visit https://gerrit.wikimedia.org/r/428417 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3dfc91da47bcded58509d153f0b3074c6d251ec6 Gerrit-Change-Number: 428417 Gerrit-PatchSet: 1 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] pywikibot/core[master]: [cleanup] Throttle.lastwait was never used and can be deprecated
by jenkins-bot (Code Review) 23 Apr '18

23 Apr '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428423 ) Change subject: [cleanup] Throttle.lastwait was never used and can be deprecated ...................................................................... [cleanup] Throttle.lastwait was never used and can be deprecated Change-Id: I3b923b880260a056ac5130d097c9ac24f3c87728 --- M pywikibot/throttle.py 1 file changed, 8 insertions(+), 2 deletions(-) Approvals: Dalba: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/throttle.py b/pywikibot/throttle.py index 40022fb..ebad441 100644 --- a/pywikibot/throttle.py +++ b/pywikibot/throttle.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- """Mechanics to slow down wiki read and/or write rate.""" # -# (C) Pywikibot team, 2008-2017 +# (C) Pywikibot team, 2008-2018 # # Distributed under the terms of the MIT license. # @@ -13,6 +13,7 @@ import pywikibot from pywikibot import config +from pywikibot.tools import deprecated _logger = "wiki.throttle" @@ -64,7 +65,6 @@ # Free the process id after this many seconds: self.releasepid = 1200 - self.lastwait = 0.0 self.delay = 0 self.checktime = 0 self.multiplydelay = multiplydelay @@ -72,6 +72,12 @@ self.checkMultiplicity() self.setDelays() + @deprecated + @property + def lastwait(self): + """DEPRECATED property.""" + return 0.0 + def checkMultiplicity(self): """Count running processes for site and set process_multiplicity.""" global pid -- To view, visit https://gerrit.wikimedia.org/r/428423 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3b923b880260a056ac5130d097c9ac24f3c87728 Gerrit-Change-Number: 428423 Gerrit-PatchSet: 3 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] pywikibot/core[master]: [PEP 479] Change StopIteration handling inside generators
by jenkins-bot (Code Review) 23 Apr '18

23 Apr '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/426819 ) Change subject: [PEP 479] Change StopIteration handling inside generators ...................................................................... [PEP 479] Change StopIteration handling inside generators User.uploadedImages: just return instead of raising StopIteration interwiki.generateMore: StopIteration was raised only to leave the the outer for loop. Implement a more suitable way to escape from that loop: Use for loop for the pageGenerator which stops looping if the generator is empty. In that case the outer for loop could be finished. set self.pageGenerator to None then. The IOError is from old compat code which used their TextfileGenerator to retrieve pages from the text file. Now pickle is used and IOError is handled when loading from file. https://www.python.org/dev/peps/pep-0479/#why-not-fix-all-next-methods Change-Id: Ic342346698d01f8dd0050a2558e689edaebdfb6c --- M pywikibot/page.py M scripts/interwiki.py 2 files changed, 51 insertions(+), 53 deletions(-) Approvals: Dalba: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/page.py b/pywikibot/page.py index 97c6681..b5b0396 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -3589,9 +3589,8 @@ @type total: int """ if not self.isRegistered(): - raise StopIteration - for item in self.logevents( - logtype='upload', total=total): + return + for item in self.logevents(logtype='upload', total=total): yield (item.page(), unicode(item.timestamp()), item.comment(), diff --git a/scripts/interwiki.py b/scripts/interwiki.py index 8235e14..0333585 100755 --- a/scripts/interwiki.py +++ b/scripts/interwiki.py @@ -2063,60 +2063,59 @@ 'NOTE: Number of pages queued is {0}, trying to add {1} more.' .format(len(self.subjects), number)) for i in range(number): - try: - while True: + for page in self.pageGenerator: + if page in self.conf.skip: + pywikibot.output('Skipping: {0} is in the skip list' + .format(page)) + continue + if self.conf.skipauto: + dictName, year = page.autoFormat() + if dictName is not None: + pywikibot.output( + 'Skipping: {0} is an auto entry {1}({2})' + .format(page, dictName, year)) + continue + if self.conf.parenthesesonly: + # Only yield pages that have ( ) in titles + if '(' not in page.title(): + continue + if page.isTalkPage(): + pywikibot.output('Skipping: {} is a talk page' + .format(page)) + continue + if page.namespace() == 10: try: - page = next(self.pageGenerator) - except IOError: - pywikibot.output(u'IOError occurred; skipping') + tmpl, loc = moved_links[page.site.code] + del tmpl + except KeyError: + loc = None + if loc is not None and loc in page.title(): + pywikibot.output( + 'Skipping: {} is a templates subpage' + .format(page.title())) continue - if page in self.conf.skip: - pywikibot.output('Skipping: {0} is in the skip list' - .format(page)) - continue - if self.conf.skipauto: - dictName, year = page.autoFormat() - if dictName is not None: - pywikibot.output( - 'Skipping: {0} is an auto entry {1}({2})' - .format(page, dictName, year)) - continue - if self.conf.parenthesesonly: - # Only yield pages that have ( ) in titles - if "(" not in page.title(): - continue - if page.isTalkPage(): - pywikibot.output(u'Skipping: %s is a talk page' % page) - continue - if page.namespace() == 10: - try: - tmpl, loc = moved_links[page.site.code] - del tmpl - except KeyError: - loc = None - if loc is not None and loc in page.title(): - pywikibot.output( - 'Skipping: %s is a templates subpage' - % page.title()) - continue + break + else: # generator stopped + break + + if self.generateUntil: + until = self.generateUntil + page_namespace = ( + page.site.namespaces[int(page.namespace())]) + if page_namespace.case == 'first-letter': + until = first_upper(until) + if page.title(withNamespace=False) > until: break - if self.generateUntil: - until = self.generateUntil - page_namespace = ( - page.site.namespaces[int(page.namespace())]) - if page_namespace.case == 'first-letter': - until = first_upper(until) - if page.title(withNamespace=False) > until: - raise StopIteration - self.add(page, hints=self.conf.hints) - self.generated += 1 - if self.generateNumber: - if self.generated >= self.generateNumber: - raise StopIteration - except StopIteration: - self.pageGenerator = None - break + self.add(page, hints=self.conf.hints) + self.generated += 1 + if self.generateNumber: + if self.generated >= self.generateNumber: + break + else: + return + # for loop was exited by break statement + self.pageGenerator = None def firstSubject(self): """Return the first subject that is still being worked on.""" -- To view, visit https://gerrit.wikimedia.org/r/426819 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic342346698d01f8dd0050a2558e689edaebdfb6c Gerrit-Change-Number: 426819 Gerrit-PatchSet: 4 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zhuyifei1999 <zhuyifei1999(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] pywikibot/core[master]: page.Category.articles: Fix the equivalant of startFrom in core
by jenkins-bot (Code Review) 22 Apr '18

22 Apr '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428263 ) Change subject: page.Category.articles: Fix the equivalant of startFrom in core ...................................................................... page.Category.articles: Fix the equivalant of startFrom in core The way `startFrom` works in compat is equal to `startprefix` in core.[1] `startsort`, which is deprecated in MW 1.24, needs to be a binary string,[2] otherwise the user will get the wrong results because of the wrong start. This means that compat's `startFrom` has no equivalant in core if the MediaWiki version is less than 1.18. [1]: https://phabricator.wikimedia.org/diffusion/PWBO/browse/master/catlib.py;df… [2]: https://www.mediawiki.org/wiki/API:Categorymembers Change-Id: I10620e82d2d0792579889199ffa5e94a0cce0d07 --- M pywikibot/page.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Zhuyifei1999: Looks good to me, but someone else must approve Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/page.py b/pywikibot/page.py index 97c6681..93d5f40 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -2798,7 +2798,7 @@ if total == 0: return - @deprecated_args(startFrom='startsort', step=None) + @deprecated_args(startFrom='startprefix', step=None) def articles(self, recurse=False, total=None, content=False, namespaces=None, sortby=None, reverse=False, starttime=None, endtime=None, -- To view, visit https://gerrit.wikimedia.org/r/428263 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I10620e82d2d0792579889199ffa5e94a0cce0d07 Gerrit-Change-Number: 428263 Gerrit-PatchSet: 1 Gerrit-Owner: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zhuyifei1999 <zhuyifei1999(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] pywikibot/core[master]: CategorizedPageGenerator: Use startprefix parameter of category.articles
by jenkins-bot (Code Review) 22 Apr '18

22 Apr '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428148 ) Change subject: CategorizedPageGenerator: Use startprefix parameter of category.articles ...................................................................... CategorizedPageGenerator: Use startprefix parameter of category.articles The previous method of passing `start` to `startsort` was wrong. It does not work as expected because `startsort` needs to be a binary string.[1] For older versions of MediaWiki implement a workaround by iterating over all category.articles. This is inefficient, but no better way found. Remove `kwargs['sortby'] = 'sortkey'`, 'sortkey' is API's default. [1]: https://www.mediawiki.org/wiki/API:Categorymembers Bug: T74101 Bug: T143120 Change-Id: I7eece66e258659f84479fbdd1094a8df9ee69479 --- M pywikibot/pagegenerators.py 1 file changed, 12 insertions(+), 2 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py index e503671..9071e1a 100644 --- a/pywikibot/pagegenerators.py +++ b/pywikibot/pagegenerators.py @@ -56,6 +56,7 @@ ) from pywikibot.logentries import LogEntryFactory from pywikibot.proofreadpage import ProofreadPage +from pywikibot.tools import MediaWikiVersion if sys.version_info[0] > 2: basestring = (str, ) @@ -1386,8 +1387,17 @@ 'content': content, 'namespaces': namespaces, } if start: - kwargs['sortby'] = 'sortkey' - kwargs['startsort'] = start + if category.site.version() < MediaWikiVersion('1.18'): + kwargs.pop('total', None) + count = 0 + for article in category.articles(**kwargs): + if article.title() >= start: + count += 1 + yield article + if count == total: + return + return + kwargs['startprefix'] = start for a in category.articles(**kwargs): yield a -- To view, visit https://gerrit.wikimedia.org/r/428148 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7eece66e258659f84479fbdd1094a8df9ee69479 Gerrit-Change-Number: 428148 Gerrit-PatchSet: 6 Gerrit-Owner: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] pywikibot/core[master]: [IMPR] reduce code complexity of add_text
by jenkins-bot (Code Review) 22 Apr '18

22 Apr '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428127 ) Change subject: [IMPR] reduce code complexity of add_text ...................................................................... [IMPR] reduce code complexity of add_text - Reduce cyclomatic complexity of update_page from E33 to D22 and improve readability by splitting code in parts. Change-Id: I42e61ea9a0391aafe714b52e3f3418fc93c395d7 --- M scripts/add_text.py M tests/add_text_tests.py 2 files changed, 96 insertions(+), 76 deletions(-) Approvals: Framawiki: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/add_text.py b/scripts/add_text.py index c3e1d74..22c8069 100755 --- a/scripts/add_text.py +++ b/scripts/add_text.py @@ -57,7 +57,7 @@ # # (C) Filnik, 2007-2010 -# (C) Pywikibot team, 2007-2017 +# (C) Pywikibot team, 2007-2018 # # Distributed under the terms of the MIT license. # @@ -79,6 +79,57 @@ } +def get_text(page, old, create): + """Get the old text.""" + if old is None: + try: + text = page.get() + except pywikibot.NoPage: + if create: + pywikibot.output( + "{} doesn't exist, creating it!".format(page.title())) + text = '' + else: + pywikibot.output( + "{} doesn't exist, skip!".format(page.title())) + return None + except pywikibot.IsRedirectPage: + pywikibot.output('{} is a redirect, skip!'.format(page.title())) + return None + else: + text = old + return text + + +def put_text(page, new, summary, count, asynchronous=False): + """Save the new text.""" + page.text = new + try: + page.save(summary=summary, asynchronous=asynchronous, + minorEdit=page.namespace() != 3) + except pywikibot.EditConflict: + pywikibot.output('Edit conflict! skip!') + except pywikibot.ServerError: + if count <= config.max_retries: + pywikibot.output('Server Error! Wait..') + time.sleep(config.retry_wait) + return None + else: + raise pywikibot.ServerError( + 'Server Error! Maximum retries exceeded') + except pywikibot.SpamfilterError as e: + pywikibot.output( + 'Cannot change {} because of blacklist entry {}' + .format(page.title(), e.url)) + except pywikibot.LockedPage: + pywikibot.output('Skipping {} (locked page)'.format(page.title())) + except pywikibot.PageNotSaved as error: + pywikibot.output('Error putting page: {}'.format(error.args)) + else: + return True + return False + + def add_text(page, addText, summary=None, regexSkip=None, regexSkipUrl=None, always=False, up=False, putText=True, oldTextGiven=None, reorderEnabled=True, create=False): @@ -91,27 +142,13 @@ if not summary: summary = i18n.twtranslate(site, 'add_text-adding', {'adding': addText[:200]}) - - errorCount = 0 - if putText: pywikibot.output(u'Loading %s...' % page.title()) - if oldTextGiven is None: - try: - text = page.get() - except pywikibot.NoPage: - if create: - pywikibot.output(u"%s doesn't exist, creating it!" - % page.title()) - text = u'' - else: - pywikibot.output(u"%s doesn't exist, skip!" % page.title()) - return (False, False, always) - except pywikibot.IsRedirectPage: - pywikibot.output(u"%s is a redirect, skip!" % page.title()) - return (False, False, always) - else: - text = oldTextGiven + + text = get_text(page, oldTextGiven, create) + if text is None: + return (False, False, always) + # Understand if the bot has to skip the page or not # In this way you can use both -except and -excepturl if regexSkipUrl is not None: @@ -159,67 +196,43 @@ newtext += u"%s%s" % (config.line_separator, addText) else: newtext = addText + config.line_separator + text + if putText and text != newtext: pywikibot.output(color_format( '\n\n>>> {lightpurple}{0}{default} <<<', page.title())) pywikibot.showDiff(text, newtext) + # Let's put the changes. + error_count = 0 while True: # If someone load it as module, maybe it's not so useful to put the # text in the page - if putText: - if not always: - try: - choice = pywikibot.input_choice( - 'Do you want to accept these changes?', - [('Yes', 'y'), ('No', 'n'), ('All', 'a'), - ('open in Browser', 'b')], 'n') - except QuitKeyboardInterrupt: - sys.exit('User quit bot run.') - if choice == 'a': - always = True - elif choice == 'n': - return (False, False, always) - elif choice == 'b': - pywikibot.bot.open_webbrowser(page) - if always or choice == 'y': - try: - if always: - page.put(newtext, summary, - minorEdit=page.namespace() != 3) - else: - page.put_async(newtext, summary, - minorEdit=page.namespace() != 3) - except pywikibot.EditConflict: - pywikibot.output(u'Edit conflict! skip!') - return (False, False, always) - except pywikibot.ServerError: - errorCount += 1 - if errorCount < config.max_retries: - pywikibot.output(u'Server Error! Wait..') - time.sleep(config.retry_wait) - continue - else: - raise pywikibot.ServerError(u'Fifth Server Error!') - except pywikibot.SpamfilterError as e: - pywikibot.output( - u'Cannot change %s because of blacklist entry %s' - % (page.title(), e.url)) - return (False, False, always) - except pywikibot.LockedPage: - pywikibot.output(u'Skipping %s (locked page)' - % page.title()) - return (False, False, always) - except pywikibot.PageNotSaved as error: - pywikibot.output(u'Error putting page: %s' % error.args) - return (False, False, always) - else: - # Break only if the errors are one after the other... - errorCount = 0 - return (True, True, always) - else: + if not putText: return (text, newtext, always) + if not always: + try: + choice = pywikibot.input_choice( + 'Do you want to accept these changes?', + [('Yes', 'y'), ('No', 'n'), ('All', 'a'), + ('open in Browser', 'b')], 'n') + except QuitKeyboardInterrupt: + sys.exit('User quit bot run.') + + if choice == 'a': + always = True + elif choice == 'n': + return (False, False, always) + elif choice == 'b': + pywikibot.bot.open_webbrowser(page) + + if always or choice == 'y': + result = put_text(page, newtext, summary, error_count, + asynchronous=not always) + if result is not None: + return (result, result, always) + error_count += 1 + def main(*args): """ diff --git a/tests/add_text_tests.py b/tests/add_text_tests.py index d696809..c96fa93 100644 --- a/tests/add_text_tests.py +++ b/tests/add_text_tests.py @@ -9,7 +9,7 @@ import pywikibot -from scripts.add_text import add_text +from scripts.add_text import add_text, get_text from tests.aspects import unittest, TestCase @@ -23,11 +23,15 @@ dry = True + def setUp(self): + """Setup test.""" + super(TestAdding, self).setUp() + self.page = pywikibot.Page(self.site, 'foo') + def test_basic(self): """Test adding text.""" - page = pywikibot.Page(self.site, 'foo') (text, newtext, always) = add_text( - page, 'bar', putText=False, + self.page, 'bar', putText=False, oldTextGiven='foo\n{{linkfa}}') self.assertEqual( 'foo\n{{linkfa}}\nbar', @@ -35,14 +39,17 @@ def test_with_category(self): """Test adding text before categories.""" - page = pywikibot.Page(self.site, 'foo') (text, newtext, always) = add_text( - page, 'bar', putText=False, + self.page, 'bar', putText=False, oldTextGiven='foo\n[[Category:Foo]]') self.assertEqual( 'foo\nbar\n\n[[Category:Foo]]', newtext) + def test_get_text(self): + """Test get_text with given text.""" + self.assertEqual(get_text(self.page, 'foo', False), 'foo') + if __name__ == '__main__': # pragma: no cover unittest.main() -- To view, visit https://gerrit.wikimedia.org/r/428127 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I42e61ea9a0391aafe714b52e3f3418fc93c395d7 Gerrit-Change-Number: 428127 Gerrit-PatchSet: 2 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Framawiki <framawiki(a)tools.wmflabs.org> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] pywikibot/core[master]: [IMPR] Improvements for upload.py
by jenkins-bot (Code Review) 22 Apr '18

22 Apr '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428130 ) Change subject: [IMPR] Improvements for upload.py ...................................................................... [IMPR] Improvements for upload.py - improve argument parsing - split code in parts, improve readability and decrease code complexity from F42 to D22 Change-Id: I9616a6c4280c17673795c38ac288ceb3e85e6944 --- M scripts/upload.py 1 file changed, 64 insertions(+), 53 deletions(-) Approvals: Framawiki: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/upload.py b/scripts/upload.py index b98bb2a..2d40ba4 100755 --- a/scripts/upload.py +++ b/scripts/upload.py @@ -50,7 +50,7 @@ """ # # (C) Rob W.W. Hooft, Andre Engels 2003-2004 -# (C) Pywikibot team, 2003-2017 +# (C) Pywikibot team, 2003-2018 # # Distributed under the terms of the MIT license. # @@ -63,6 +63,31 @@ import pywikibot from pywikibot.bot import suggest_help from pywikibot.specialbots import UploadRobot + + +def get_chunk_size(match): + """Get chunk size.""" + if not match: + pywikibot.error('Chunk size parameter is not valid.') + chunk_size = 0 + elif match.group(1): # number was in there + base = float(match.group(1)) + if match.group(2): # suffix too + suffix = match.group(2).lower() + if suffix == 'k': + suffix = 1000 + elif suffix == 'm': + suffix = 1000000 + elif suffix == 'ki': + suffix = 1 << 10 + elif suffix == 'mi': + suffix = 1 << 20 + else: + suffix = 1 + chunk_size = math.trunc(base * suffix) + else: + chunk_size = 1 << 20 # default to 1 MiB + return chunk_size def main(*args): @@ -83,65 +108,48 @@ verifyDescription = True aborts = set() ignorewarn = set() - chunk_size = 0 - chunk_size_regex = r'^-chunked(?::(\d+(?:\.\d+)?)[ \t]*(k|ki|m|mi)?b?)?$' - chunk_size_regex = re.compile(chunk_size_regex, re.I) + chunk_size_regex = re.compile( + r'^-chunked(?::(\d+(?:\.\d+)?)[ \t]*(k|ki|m|mi)?b?)?$', re.I) recursive = False # process all global bot args # returns a list of non-global args, i.e. args for upload.py - for arg in pywikibot.handle_args(args): - if arg: - if arg == '-always': - keepFilename = True - always = True - verifyDescription = False - elif arg == '-recursive': - recursive = True - elif arg.startswith('-keep'): - keepFilename = True - elif arg.startswith('-filename:'): - useFilename = arg[10:] - elif arg.startswith('-summary'): - summary = arg[9:] - elif arg.startswith('-noverify'): - verifyDescription = False - elif arg.startswith('-abortonwarn'): - if len(arg) > len('-abortonwarn:') and aborts is not True: - aborts.add(arg[len('-abortonwarn:'):]) - else: - aborts = True - elif arg.startswith('-ignorewarn'): - if len(arg) > len('-ignorewarn:') and ignorewarn is not True: - ignorewarn.add(arg[len('-ignorewarn:'):]) - else: - ignorewarn = True - elif arg.startswith('-chunked'): - match = chunk_size_regex.match(arg) - if match: - if match.group(1): # number was in there - base = float(match.group(1)) - if match.group(2): # suffix too - suffix = match.group(2).lower() - if suffix == "k": - suffix = 1000 - elif suffix == "m": - suffix = 1000000 - elif suffix == "ki": - suffix = 1 << 10 - elif suffix == "mi": - suffix = 1 << 20 - else: - suffix = 1 - chunk_size = math.trunc(base * suffix) - else: - chunk_size = 1 << 20 # default to 1 MiB - else: - pywikibot.error('Chunk size parameter is not valid.') - elif url == u'': + local_args = pywikibot.handle_args(args) + for option in local_args: + arg, _, value = option.partition(':') + if arg == '-always': + keepFilename = True + always = True + verifyDescription = False + elif arg == '-recursive': + recursive = True + elif arg == '-keep': + keepFilename = True + elif arg == '-filename': + useFilename = value + elif arg == '-summary': + summary = value + elif arg == '-noverify': + verifyDescription = False + elif arg == '-abortonwarn': + if value and aborts is not True: + aborts.add(value) + else: + aborts = True + elif arg == '-ignorewarn': + if value and ignorewarn is not True: + ignorewarn.add(value) + else: + ignorewarn = True + elif arg == '-chunked': + match = chunk_size_regex.match(option) + chunk_size = get_chunk_size(match) + elif arg and not value: + if not url: url = arg else: description.append(arg) + description = u' '.join(description) while not ("://" in url or os.path.exists(url)): if not url: @@ -156,6 +164,7 @@ else: pywikibot.output(error) url = pywikibot.input(u'URL, file or directory where files are now:') + if always and ((aborts is not True and ignorewarn is not True) or not description or url is None): additional = '' @@ -171,6 +180,7 @@ additional += 'Unable to run in -always mode' suggest_help(missing_parameters=missing, additional_text=additional) return False + if os.path.isdir(url): file_list = [] for directory_info in os.walk(url): @@ -182,6 +192,7 @@ url = file_list else: url = [url] + bot = UploadRobot(url, description=description, useFilename=useFilename, keepFilename=keepFilename, verifyDescription=verifyDescription, -- To view, visit https://gerrit.wikimedia.org/r/428130 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9616a6c4280c17673795c38ac288ceb3e85e6944 Gerrit-Change-Number: 428130 Gerrit-PatchSet: 5 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Framawiki <framawiki(a)tools.wmflabs.org> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] pywikibot/core[master]: [IMPR] reduce code complexity of QueryGenerator.__iter__
by jenkins-bot (Code Review) 22 Apr '18

22 Apr '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428069 ) Change subject: [IMPR] reduce code complexity of QueryGenerator.__iter__ ...................................................................... [IMPR] reduce code complexity of QueryGenerator.__iter__ - Reduce cyclomatic complexity of submit from E37 to D23 and improve readability by splitting code in parts. Change-Id: Iaa1427ca3a0afc6bcc16e8ff379dabf5853fd36c --- M pywikibot/data/api.py 1 file changed, 73 insertions(+), 65 deletions(-) Approvals: Dalba: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/data/api.py b/pywikibot/data/api.py index 1fed1f9..b9c54e0 100644 --- a/pywikibot/data/api.py +++ b/pywikibot/data/api.py @@ -2732,6 +2732,74 @@ value = str(value) self.request[key] = value + def _handle_query_limit(self, prev_limit, new_limit, count, had_data): + """Handle query limit.""" + if self.query_limit is None: + return prev_limit, new_limit + + prev_limit = new_limit + if self.limit is None: + new_limit = self.query_limit + elif self.limit > 0: + if had_data: + # self.resultkey in data in last request.submit() + new_limit = min(self.query_limit, self.limit - count) + else: + # only "(query-)continue" returned. See Bug T74209. + # increase new_limit to advance faster until new + # useful data are found again. + new_limit = min(new_limit * 2, self.query_limit) + else: + new_limit = None + + if new_limit and 'rvprop' in self.request \ + and 'content' in self.request['rvprop']: + # queries that retrieve page content have lower limits + # Note: although API allows up to 500 pages for content + # queries, these sometimes result in server-side errors + # so use 250 as a safer limit + new_limit = min(new_limit, self.api_limit // 10, 250) + + if new_limit is not None: + self.request[self.prefix + 'limit'] = str(new_limit) + + if prev_limit != new_limit: + pywikibot.debug( + '{name}: query_limit: {query}, api_limit: {api}, ' + 'limit: {limit}, new_limit: {new}, count: {count}\n' + '{name}: {prefix}limit: {value}' + .format(name=self.__class__.__name__, + query=self.query_limit, + api=self.api_limit, + limit=self.limit, + new=new_limit, + count=count, + prefix=self.prefix, + value=self.request[self.prefix + 'limit']), + _logger) + return prev_limit, new_limit + + def _get_resultdata(self): + """Get resultdata and verify result.""" + resultdata = keys = self.data['query'][self.resultkey] + if isinstance(resultdata, dict): + keys = list(resultdata.keys()) + if 'results' in resultdata: + resultdata = resultdata['results'] + elif 'pageids' in self.data['query']: + # this ensures that page data will be iterated + # in the same order as received from server + resultdata = [resultdata[k] + for k in self.data['query']['pageids']] + else: + resultdata = [resultdata[k] + for k in sorted(resultdata.keys())] + pywikibot.debug('{name} received {keys}; limit={limit}' + .format(name=self.__class__.__name__, + keys=keys, limit=self.limit), + _logger) + return resultdata + def __iter__(self): """Submit request and iterate the response based on self.resultkey. @@ -2743,78 +2811,18 @@ count = 0 while True: - if self.query_limit is not None: - prev_limit = new_limit - if self.limit is None: - new_limit = self.query_limit - elif self.limit > 0: - if previous_result_had_data: - # self.resultkey in data in last request.submit() - new_limit = min(self.query_limit, self.limit - count) - else: - # only "(query-)continue" returned. See Bug T74209. - # increase new_limit to advance faster until new - # useful data are found again. - new_limit = min(new_limit * 2, self.query_limit) - else: - new_limit = None - - if new_limit and \ - "rvprop" in self.request \ - and "content" in self.request["rvprop"]: - # queries that retrieve page content have lower limits - # Note: although API allows up to 500 pages for content - # queries, these sometimes result in server-side errors - # so use 250 as a safer limit - new_limit = min(new_limit, self.api_limit // 10, 250) - if new_limit is not None: - self.request[self.prefix + "limit"] = str(new_limit) - if prev_limit != new_limit: - pywikibot.debug( - u"%s: query_limit: %s, api_limit: %s, " - u"limit: %s, new_limit: %s, count: %s" - % (self.__class__.__name__, - self.query_limit, self.api_limit, - self.limit, new_limit, count), - _logger) - pywikibot.debug( - u"%s: %s: %s" - % (self.__class__.__name__, - self.prefix + "limit", - self.request[self.prefix + "limit"]), - _logger) + prev_limit, new_limit = self._handle_query_limit( + prev_limit, new_limit, count, previous_result_had_data) if not hasattr(self, "data"): self.data = self.request.submit() if not self.data or not isinstance(self.data, dict): pywikibot.debug( - u"%s: stopped iteration because no dict retrieved from api." - % self.__class__.__name__, + '{}: stopped iteration because no dict retrieved from api.' + .format(self.__class__.__name__), _logger) return if 'query' in self.data and self.resultkey in self.data["query"]: - resultdata = self.data["query"][self.resultkey] - if isinstance(resultdata, dict): - pywikibot.debug(u"%s received %s; limit=%s" - % (self.__class__.__name__, - list(resultdata.keys()), - self.limit), - _logger) - if "results" in resultdata: - resultdata = resultdata["results"] - elif "pageids" in self.data["query"]: - # this ensures that page data will be iterated - # in the same order as received from server - resultdata = [resultdata[k] - for k in self.data["query"]["pageids"]] - else: - resultdata = [resultdata[k] - for k in sorted(resultdata.keys())] - else: - pywikibot.debug(u"%s received %s; limit=%s" - % (self.__class__.__name__, - resultdata, - self.limit), - _logger) + resultdata = self._get_resultdata() if "normalized" in self.data["query"]: self.normalized = { item['to']: item['from'] -- To view, visit https://gerrit.wikimedia.org/r/428069 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iaa1427ca3a0afc6bcc16e8ff379dabf5853fd36c Gerrit-Change-Number: 428069 Gerrit-PatchSet: 2 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] pywikibot/core[master]: [bugfix] Fix category title handling in makecat.py
by jenkins-bot (Code Review) 21 Apr '18

21 Apr '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428046 ) Change subject: [bugfix] Fix category title handling in makecat.py ...................................................................... [bugfix] Fix category title handling in makecat.py The makecat.py was broken due to double colon in processed category title. The script was not working at all and also the docs were missing to the script (T187009). Bug: T192706 Change-Id: I4d84e2d021c2ae85c11f45a898443f8c7b618312 --- M scripts/makecat.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/makecat.py b/scripts/makecat.py index 0f647ad..4305fdd 100755 --- a/scripts/makecat.py +++ b/scripts/makecat.py @@ -217,7 +217,7 @@ summary = i18n.twtranslate(mysite, 'makecat-create', {'cat': workingcatname}) workingcat = pywikibot.Category(mysite, - u'%s:%s' + '%s%s' % (mysite.namespaces.CATEGORY, workingcatname)) filename = pywikibot.config.datafilepath( -- To view, visit https://gerrit.wikimedia.org/r/428046 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4d84e2d021c2ae85c11f45a898443f8c7b618312 Gerrit-Change-Number: 428046 Gerrit-PatchSet: 2 Gerrit-Owner: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] pywikibot/core[master]: [L10N] Update languages_by_size
by jenkins-bot (Code Review) 21 Apr '18

21 Apr '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428049 ) Change subject: [L10N] Update languages_by_size ...................................................................... [L10N] Update languages_by_size including new created sites Change-Id: I9f780ed1fd4b41a363bbce48e4e1b5123052e061 --- M pywikibot/families/wikibooks_family.py M pywikibot/families/wikipedia_family.py M pywikibot/families/wikisource_family.py M pywikibot/families/wiktionary_family.py 4 files changed, 17 insertions(+), 17 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/families/wikibooks_family.py b/pywikibot/families/wikibooks_family.py index f71669d..e269438 100644 --- a/pywikibot/families/wikibooks_family.py +++ b/pywikibot/families/wikibooks_family.py @@ -37,10 +37,10 @@ self.languages_by_size = [ 'en', 'hu', 'de', 'fr', 'pt', 'ja', 'it', 'nl', 'es', 'pl', 'th', 'he', 'id', 'sq', 'fi', 'fa', 'zh', 'ca', 'ru', 'az', 'vi', 'da', - 'sv', 'gl', 'sr', 'ko', 'cs', 'hr', 'ba', 'no', 'tr', 'ar', 'ta', + 'sv', 'gl', 'ko', 'sr', 'cs', 'hr', 'ba', 'no', 'tr', 'ar', 'ta', 'sa', 'sk', 'uk', 'hi', 'is', 'ro', 'eo', 'si', 'mk', 'bn', 'bg', 'ka', 'ms', 'lt', 'tt', 'li', 'el', 'ur', 'sl', 'km', 'tl', 'kk', - 'et', 'ml', 'oc', 'be', 'ia', 'ne', 'eu', 'pa', 'hy', 'la', 'cv', + 'et', 'ml', 'oc', 'be', 'ia', 'eu', 'ne', 'pa', 'hy', 'la', 'cv', 'tg', 'fy', 'ku', 'bs', 'cy', 'te', 'af', 'mr', 'mg', 'ky', ] diff --git a/pywikibot/families/wikipedia_family.py b/pywikibot/families/wikipedia_family.py index de99534..0960980 100644 --- a/pywikibot/families/wikipedia_family.py +++ b/pywikibot/families/wikipedia_family.py @@ -37,30 +37,30 @@ 'hy', 'da', 'zh-min-nan', 'sk', 'he', 'min', 'kk', 'hr', 'lt', 'et', 'ce', 'sl', 'be', 'gl', 'el', 'ur', 'nn', 'az', 'simple', 'uz', 'la', 'hi', 'th', 'ka', 'vo', 'ta', 'cy', 'mk', 'tg', 'mg', - 'oc', 'lv', 'tl', 'ky', 'tt', 'bs', 'ast', 'sq', 'new', 'azb', + 'oc', 'lv', 'tl', 'ky', 'tt', 'bs', 'ast', 'sq', 'azb', 'new', 'te', 'zh-yue', 'br', 'pms', 'be-tarask', 'bn', 'ml', 'jv', 'ht', - 'lb', 'mr', 'sco', 'af', 'ga', 'pnb', 'is', 'ba', 'cv', 'sw', 'fy', + 'lb', 'sco', 'mr', 'af', 'ga', 'pnb', 'is', 'ba', 'cv', 'sw', 'fy', 'su', 'my', 'lmo', 'an', 'yo', 'ne', 'nds', 'pa', 'gu', 'io', 'scn', 'bar', 'bpy', 'als', 'ku', 'kn', 'ckb', 'ia', 'qu', 'arz', 'mn', 'bat-smg', 'si', 'gd', 'wa', 'nap', 'yi', 'am', 'bug', 'or', - 'cdo', 'map-bms', 'fo', 'hsb', 'mzn', 'mai', 'xmf', 'li', 'sah', + 'cdo', 'map-bms', 'hsb', 'fo', 'mzn', 'mai', 'xmf', 'li', 'sah', 'sa', 'vec', 'ilo', 'os', 'mrj', 'mhr', 'hif', 'eml', 'bh', 'sd', 'roa-tara', 'ps', 'diq', 'pam', 'hak', 'wuu', 'nso', - 'zh-classical', 'bcl', 'se', 'ace', 'mi', 'nah', 'szl', 'nds-nl', + 'zh-classical', 'bcl', 'se', 'ace', 'mi', 'szl', 'nah', 'nds-nl', 'frr', 'rue', 'gan', 'vls', 'km', 'bo', 'crh', 'sc', 'vep', 'glk', 'co', 'fiu-vro', 'tk', 'lrc', 'kv', 'csb', 'myv', 'gv', 'as', 'nv', 'so', 'zea', 'udm', 'ay', 'lez', 'stq', 'ie', 'lad', 'nrm', 'kw', - 'pcd', 'ug', 'mwl', 'gom', 'rm', 'gn', 'sn', 'koi', 'ab', 'lij', + 'ug', 'pcd', 'mwl', 'gom', 'rm', 'gn', 'sn', 'koi', 'ab', 'lij', 'mt', 'fur', 'dsb', 'dv', 'ang', 'ln', 'cbk-zam', 'frp', 'kab', 'ext', 'dty', 'ksh', 'lo', 'gag', 'olo', 'pag', 'pi', 'av', 'haw', 'bxr', 'pfl', 'xal', 'krc', 'pap', 'kaa', 'rw', 'pdc', 'bjn', 'ha', 'to', 'nov', 'kl', 'arc', 'jam', 'kbd', 'tyv', 'tpi', 'kbp', 'tet', - 'ki', 'ig', 'na', 'jbo', 'lbe', 'roa-rup', 'ty', 'mdf', 'za', 'kg', - 'bi', 'wo', 'lg', 'srn', 'tcy', 'zu', 'chr', 'ltg', 'sm', 'om', - 'xh', 'rmy', 'cu', 'tn', 'pih', 'rn', 'chy', 'tw', 'bm', 'tum', - 'ts', 'got', 'st', 'ak', 'atj', 'pnt', 'ss', 'ch', 'fj', 'ady', - 'iu', 'ny', 'ee', 'ks', 'ik', 'sg', 've', 'ff', 'dz', 'ti', 'cr', - 'din', 'gor', 'inh', 'lfn', + 'ki', 'lfn', 'ig', 'na', 'jbo', 'lbe', 'roa-rup', 'ty', 'mdf', + 'za', 'kg', 'gor', 'bi', 'wo', 'lg', 'srn', 'tcy', 'zu', 'chr', + 'ltg', 'sm', 'om', 'inh', 'xh', 'rmy', 'cu', 'tn', 'pih', 'rn', + 'chy', 'tw', 'bm', 'tum', 'ts', 'got', 'st', 'ak', 'atj', 'pnt', + 'ss', 'ch', 'fj', 'ady', 'iu', 'ny', 'ee', 'ks', 'ik', 'sg', 've', + 'ff', 'dz', 'ti', 'cr', 'din', ] # Sites we want to edit but not count as real languages diff --git a/pywikibot/families/wikisource_family.py b/pywikibot/families/wikisource_family.py index 623d872..0a40422 100644 --- a/pywikibot/families/wikisource_family.py +++ b/pywikibot/families/wikisource_family.py @@ -34,8 +34,8 @@ 'te', 'sa', 'el', 'ro', 'fi', 'uk', 'hy', 'ja', 'vi', 'az', 'th', 'ca', 'ta', 'kn', 'br', 'nl', 'hr', 'is', 'la', 'no', 'vec', 'eo', 'tr', 'et', 'be', 'mk', 'yi', 'da', 'id', 'bg', 'li', 'mr', 'as', - 'or', 'bs', 'sah', 'lt', 'gl', 'sk', 'cy', 'pa', 'zh-min-nan', - 'fo', 'eu', + 'or', 'bs', 'sah', 'lt', 'gl', 'sk', 'eu', 'cy', 'pa', + 'zh-min-nan', 'fo', ] super(Family, self).__init__() diff --git a/pywikibot/families/wiktionary_family.py b/pywikibot/families/wiktionary_family.py index da4cca2..2319c96 100644 --- a/pywikibot/families/wiktionary_family.py +++ b/pywikibot/families/wiktionary_family.py @@ -37,8 +37,8 @@ 'kn', 'pt', 'hy', 'vi', 'sr', 'ja', 'chr', 'hi', 'th', 'ro', 'no', 'id', 'ml', 'et', 'my', 'uz', 'li', 'or', 'te', 'cs', 'fa', 'eo', 'ar', 'jv', 'az', 'eu', 'gl', 'oc', 'da', 'br', 'lo', 'uk', 'hr', - 'fj', 'tg', 'bg', 'simple', 'ps', 'sk', 'cy', 'vo', 'wa', 'is', - 'la', 'zh-min-nan', 'af', 'ky', 'scn', 'ast', 'he', 'tl', 'sw', + 'fj', 'tg', 'bg', 'simple', 'ps', 'sk', 'cy', 'vo', 'wa', 'la', + 'is', 'zh-min-nan', 'ky', 'af', 'scn', 'ast', 'he', 'tl', 'sw', 'fy', 'nn', 'lv', 'bn', 'pa', 'co', 'pnb', 'mn', 'ka', 'nds', 'sl', 'sq', 'lb', 'bs', 'nah', 'sa', 'kk', 'tk', 'ur', 'km', 'sm', 'mk', 'hsb', 'be', 'ms', 'ga', 'an', 'wo', 'vec', 'ang', 'tt', 'sd', -- To view, visit https://gerrit.wikimedia.org/r/428049 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9f780ed1fd4b41a363bbce48e4e1b5123052e061 Gerrit-Change-Number: 428049 Gerrit-PatchSet: 1 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

1 0

← Newer
1
2
3
4
5
6
7
8
9
Older →

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits April 2018