Pywikibot-commits February 2015

pywikibot-commits@lists.wikimedia.org

2 participants
126 discussions

[Gerrit] Enable edit failure tests in travis builds - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Enable edit failure tests in travis builds ...................................................................... Enable edit failure tests in travis builds Adds new environment variable PYWIKIBOT2_TEST_WRITE_FAIL which when set to '1' will run edit failure tests, which are much safer than the other 'write' tests. Also improve documentation for travis builds. Change-Id: I85bf198c2a30b69a9e29d220d8696a92642272bc --- M .travis.yml M tests/README.rst M tests/aspects.py M tests/edit_failure_tests.py 4 files changed, 67 insertions(+), 21 deletions(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve XZise: Looks good to me, approved jenkins-bot: Verified diff --git a/.travis.yml b/.travis.yml index f0c970b..1233f57 100644 --- a/.travis.yml +++ b/.travis.yml @@ -32,15 +32,20 @@ - echo "maximum_GET_length = 5000" >> ~/.pywikibot/user-config.py - echo "console_encoding = 'utf8'" >> ~/.pywikibot/user-config.py - - if [[ "$GITHUB_USER" == "wikimedia" ]]; then PYWIKIBOT2_USERNAME="Pywikibot-test"; fi + - if [[ "$GITHUB_USER" == "wikimedia" ]]; then + PYWIKIBOT2_USERNAME="Pywikibot-test" ; + else + export PYWIKIBOT2_TEST_WRITE_FAIL=1 ; + fi - - if [[ -n "$USER_PASSWORD" && -n "$PYWIKIBOT2_USERNAME" ]]; then echo "usernames['$FAMILY']['$LANGUAGE'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py; fi - - if [[ -n "$USER_PASSWORD" && -n "$PYWIKIBOT2_USERNAME" ]]; then echo "usernames['wikipedia']['en'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py; fi - - if [[ -n "$USER_PASSWORD" && -n "$PYWIKIBOT2_USERNAME" ]]; then echo "usernames['wikipedia']['test'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py; fi - - if [[ -n "$USER_PASSWORD" && -n "$PYWIKIBOT2_USERNAME" ]]; then echo "usernames['wikidata']['test'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py; fi - - if [[ -n "$USER_PASSWORD" && -n "$PYWIKIBOT2_USERNAME" ]]; then echo "usernames['commons']['commons'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py; fi - - - if [[ -n "$USER_PASSWORD" && -n "$PYWIKIBOT2_USERNAME" ]]; then echo "('$PYWIKIBOT2_USERNAME', '$USER_PASSWORD')" > ~/.pywikibot/passwordfile; fi + - if [[ -n "$USER_PASSWORD" && -n "$PYWIKIBOT2_USERNAME" ]]; then + echo "usernames['$FAMILY']['$LANGUAGE'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py ; + echo "usernames['wikipedia']['en'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py ; + echo "usernames['wikipedia']['test'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py ; + echo "usernames['wikidata']['test'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py ; + echo "usernames['commons']['commons'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py ; + echo "('$PYWIKIBOT2_USERNAME', '$USER_PASSWORD')" > ~/.pywikibot/passwordfile ; + fi - cd externals/httplib2 - python setup.py install diff --git a/tests/README.rst b/tests/README.rst index 4258016..214c110 100644 --- a/tests/README.rst +++ b/tests/README.rst @@ -83,7 +83,7 @@ use the Wikimedia global (SUL) account 'Pywikibot-test', which has a password securely stored in .travis.yml . See section env:global:secure. -Anyone can run these tests on travis-ci.org using their own account, with +Anyone can run these tests on travis-ci.org using their own github account, with code changes that have not been merged into the main repository. To do this: 1. create a github and travis-ci account @@ -103,9 +103,35 @@ 2. Add a new variable named PYWIKIBOT2_USERNAME and a value of a valid Wikimedia SUL username 3. Add another variable named USER_PASSWORD, with the private password for - the Wikimedia SUL username used in step 2 + the Wikimedia SUL username used in step 2. Check that this + environment variable has "Display value in build logs" set to OFF, so + the password does not leak into the build logs. 4. The next build should run tests that require a logged in user +While passwords in travis-ci environment variables are not leaked in normal +operations, you are responsible for your own passwords. + +It is strongly recommended that an untrusted bot account is created for +travis tests, using a password that is not shared with trusted accounts. + +There are a set of 'edit failure' tests, which attempt to write to the wikis +and **should** fail. If there is a bug in pywikibot or MediaWiki, these +tests **may** actually perform a write operation. + +These 'edit failure' tests are disabled by default for the 'wikimedia' builds, +but are enabled by default on builds by any other github account. + +To disable 'edit failure' tests in travis, add PYWIKIBOT2_TEST_WRITE_FAIL=0 + +There are also several other 'write' tests which also attempt to perform +write operations successfully. These **will** write to the wikis, and they +should always only write to 'test' wikis. + +These 'write' tests are disabled in travis builds, and currently can not be +run on travis as they require interaction using a terminal. + +To enable 'write' tests in travis, add PYWIKIBOT2_TEST_WRITE=1 + Contributing tests ================== diff --git a/tests/aspects.py b/tests/aspects.py index be21cf8..8147a70 100644 --- a/tests/aspects.py +++ b/tests/aspects.py @@ -518,24 +518,37 @@ """ Set up the test class. - Prevent test classes to write to the site and also cache results. + Reject write test classes configured with non-test wikis, or caching. - Skip the test class if environment variable PYWIKIBOT2_TEST_WRITE - does not equal 1. + Prevent test classes from writing to the site by default. + + If class attribute 'write' is -1, the test class is skipped unless + environment variable PYWIKIBOT2_TEST_WRITE_FAIL is set to 1. + + Otherwise the test class is skipped unless environment variable + PYWIKIBOT2_TEST_WRITE is set to 1. """ - if os.environ.get('PYWIKIBOT2_TEST_WRITE', '0') != '1': + super(SiteWriteMixin, cls).setUpClass() + + site = cls.get_site() + assert('test' in (site.family.name, site.code)) + + if cls.write == -1: + env_var = 'PYWIKIBOT2_TEST_WRITE_FAIL' + else: + env_var = 'PYWIKIBOT2_TEST_WRITE' + + if os.environ.get(env_var, '0') != '1': raise unittest.SkipTest( '%r write tests disabled. ' - 'Set PYWIKIBOT2_TEST_WRITE=1 to enable.' - % cls.__name__) + 'Set %s=1 to enable.' + % (cls.__name__, env_var)) if issubclass(cls, ForceCacheMixin): raise Exception( '%s can not be a subclass of both ' 'SiteEditTestCase and ForceCacheMixin' % cls.__name__) - - super(SiteWriteMixin, cls).setUpClass() class RequireUserMixin(TestCaseBase): diff --git a/tests/edit_failure_tests.py b/tests/edit_failure_tests.py index 57d3ffd..fa3c1db 100644 --- a/tests/edit_failure_tests.py +++ b/tests/edit_failure_tests.py @@ -4,6 +4,8 @@ These tests should never write to the wiki, unless something has broken badly. + +These tests use special code 'write = -1' for edit failures. """ # # (C) Pywikibot team, 2014 @@ -27,7 +29,7 @@ """Test cases for edits which should fail to save.""" - write = True + write = -1 family = 'wikipedia' code = 'test' @@ -55,7 +57,7 @@ """Test cases for actions which should fail to save.""" - write = True + write = -1 family = 'wikipedia' code = 'test' @@ -87,7 +89,7 @@ family = 'wikidata' code = 'test' - write = True + write = -1 def test_itempage_save(self): """Test ItemPage save method inherited from superclass Page.""" -- To view, visit https://gerrit.wikimedia.org/r/190635 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I85bf198c2a30b69a9e29d220d8696a92642272bc Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 2 months

[Gerrit] Normalize usernames in password file - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Normalize usernames in password file ...................................................................... Normalize usernames in password file Capitalize the first letter of username in password file when used according to Wikipedia username policy. Warn if the format of tuple in password file is not correct. Bug: T74974 Change-Id: I2035bf03bad62f6f505b2a9129758c2cc4f6b5ce --- M pywikibot/exceptions.py M pywikibot/login.py 2 files changed, 27 insertions(+), 5 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/exceptions.py b/pywikibot/exceptions.py index 3bb6a7a..2a8105f 100644 --- a/pywikibot/exceptions.py +++ b/pywikibot/exceptions.py @@ -57,6 +57,7 @@ UserWarning: warnings targetted at users - config2._ConfigurationDeprecationWarning: user configuration file problems + - login._PasswordFileWarning: password file problems - ArgumentDeprecationWarning: command line argument problems - FamilyMaintenanceWarning: missing information in family definition """ diff --git a/pywikibot/login.py b/pywikibot/login.py index db7be8f..816378f 100644 --- a/pywikibot/login.py +++ b/pywikibot/login.py @@ -3,18 +3,27 @@ """Library to log the bot in to a wiki account.""" # # (C) Rob W.W. Hooft, 2003 -# (C) Pywikibot team, 2003-2012 +# (C) Pywikibot team, 2003-2015 # # Distributed under the terms of the MIT license. # __version__ = '$Id$' # import codecs +from warnings import warn import pywikibot from pywikibot import config from pywikibot.tools import deprecated_args from pywikibot.exceptions import NoUsername + + +class _PasswordFileWarning(UserWarning): + + """The format of password file is incorrect.""" + + pass + _logger = "wiki.login" @@ -169,18 +178,30 @@ for line in password_f: if not line.strip(): continue - entry = eval(line) + try: + entry = eval(line) + except SyntaxError: + entry = None + if type(entry) is not tuple: + warn('Invalid tuple', _PasswordFileWarning) + continue + if not 2 <= len(entry) <= 4: + warn('The length of tuple should be 2 to 4 (%s given)' + % len(entry), _PasswordFileWarning) + continue + username = entry[-2] + username = username[0].upper() + username[1:] if len(entry) == 4: # for userinfo included code and family if entry[0] == self.site.code and \ entry[1] == self.site.family.name and \ - entry[2] == self.username: + username == self.username: self.password = entry[3] elif len(entry) == 3: # for userinfo included family if entry[0] == self.site.family.name and \ - entry[1] == self.username: + username == self.username: self.password = entry[2] elif len(entry) == 2: # for default userinfo - if entry[0] == self.username: + if username == self.username: self.password = entry[1] password_f.close() -- To view, visit https://gerrit.wikimedia.org/r/190638 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I2035bf03bad62f6f505b2a9129758c2cc4f6b5ce Gerrit-PatchSet: 4 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: VcamX <vcamx3(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: VcamX <vcamx3(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

9 years, 2 months

[Gerrit] Fix anarchopedia family file - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Fix anarchopedia family file ...................................................................... Fix anarchopedia family file Anarchopedia wishes to use two letter codes in interwiki links, and three letter codes should be replaced when found. To acheive this, three letter codes need to map to two letter codes, and the language subdomains need to use those two letter codes. - Add Esperanto project which already existed in obsolete mapping - Map all ISO 639-2 Norwegian language codes to 'no' - Map Bosnian language and Serbian language to 'hr', but mark them as non-compliant with ISO 639-2->ISO 639-1 mappings Bug: T72444 Change-Id: I75fef66dbce80f9e7e8ee7ae2761bf491603ef6b --- M pywikibot/families/anarchopedia_family.py 1 file changed, 22 insertions(+), 17 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/families/anarchopedia_family.py b/pywikibot/families/anarchopedia_family.py index 789ae2c..08ba6f3 100644 --- a/pywikibot/families/anarchopedia_family.py +++ b/pywikibot/families/anarchopedia_family.py @@ -18,10 +18,10 @@ self.name = 'anarchopedia' self.languages_by_size = [ - 'ar', 'en', 'ger', 'de', 'nl', 'el', 'it', 'fa', 'fi', 'fr', 'he', - 'es', 'hy', 'id', 'meta', 'ja', 'ko', 'lv', 'lit', 'no', 'hrv', - 'pl', 'pt', 'ro', 'ru', 'sr', 'sq', 'da', 'sv', 'tr', 'zh', 'gre', - 'chi', + 'ar', 'en', 'de', 'nl', 'el', 'it', 'fa', 'fi', 'fr', 'he', + 'es', 'hy', 'id', 'meta', 'ja', 'ko', 'lv', 'lt', 'no', 'hr', + 'pl', 'pt', 'ro', 'ru', 'sr', 'sq', 'da', 'sv', 'tr', 'zh', + 'eo', ] for l in self.languages_by_size: self.langs[l] = '%s.anarchopedia.org' % l @@ -29,9 +29,9 @@ self.nocapitalize = list(self.langs.keys()) self.obsolete = { + # ISO 639-2 -> ISO 639-1 mappings 'ara': 'ar', - 'bos': 'bs', - 'zho': 'zh', + 'chi': 'zh', 'dan': 'da', 'deu': 'de', 'dut': 'nl', @@ -39,33 +39,38 @@ 'eng': 'en', 'epo': 'eo', 'fas': 'fa', - 'fra': 'fr', 'fin': 'fi', + 'fra': 'fr', + 'ger': 'de', + 'gre': 'el', 'heb': 'he', + 'hye': 'hy', 'ind': 'id', 'ita': 'it', 'jpn': 'ja', - 'lit': 'lt', + 'kor': 'ko', 'lav': 'lv', + 'lit': 'lt', + 'nno': 'no', + 'nob': 'no', 'nor': 'no', - 'nsh': 'sh', 'pol': 'pl', 'por': 'pt', 'rum': 'ro', 'rus': 'ru', 'spa': 'es', 'srp': 'sr', - 'srp': 'hr', - 'swe': 'sv', - 'kor': 'ko', 'sqi': 'sq', - 'hye': 'hy', + 'swe': 'sv', 'tur': 'tr', + 'zho': 'zh', - 'ell': 'gre', - 'srp': 'hrv', - 'nno': None, - 'nob': None, + # ISO 639-1 -> ISO 639-1 mappings + 'bs': 'hr', + + # Non-compliant mappings + 'bos': 'hr', + 'nsh': 'hr', } @deprecated('APISite.version()') -- To view, visit https://gerrit.wikimedia.org/r/190429 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I75fef66dbce80f9e7e8ee7ae2761bf491603ef6b Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 2 months

Broken: wikimedia/pywikibot-core#1986 (master - 6551067)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #1986 Status: Broken Duration: 14 minutes and 9 seconds Commit: 6551067 (master) Author: jenkins-bot Message: Merge "[FIX] Upload: Fixing uploads" View the changeset: https://github.com/wikimedia/pywikibot-core/compare/464bd248384f...65510677… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/50706978 -- You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications

9 years, 2 months

[Gerrit] [FIX] Upload: Fixing uploads - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: [FIX] Upload: Fixing uploads ...................................................................... [FIX] Upload: Fixing uploads When there is a page name problem, it reports it and it might then exit the loop. But if there is no problem it doesn't continue but asks the user and the loop continues. It also calls upload_file internally and thus preserving forwarding the URL, as upload_image was assuming all URLs are a string. Change-Id: I0020e214db1cf0e276d9267c94d49859802b0a6e --- M scripts/upload.py 1 file changed, 6 insertions(+), 4 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/upload.py b/scripts/upload.py index b02068b..ba258b3 100755 --- a/scripts/upload.py +++ b/scripts/upload.py @@ -6,7 +6,7 @@ Arguments: -keep Keep the filename as is - -filename Target filename + -filename Target filename without the namespace prefix -noverify Do not ask for verification of the upload description if one is given -abortonwarn: Abort upload on the specified warning type. If no warning type @@ -229,8 +229,8 @@ first_check = True while True: if not first_check: - filename = pywikibot.input(u'Enter a better name, ' - 'or press enter to skip:') + filename = pywikibot.input('Enter a better name, or press ' + 'enter to skip the file:') if not filename: return None first_check = False @@ -270,6 +270,8 @@ "repository and cannot be overwritten." % filename) continue + else: + break except pywikibot.NoPage: break @@ -372,7 +374,7 @@ if answer: self.ignoreWarning = True self.keepFilename = True - return self.upload_image(debug) + return self.upload_file(file_url, debug) else: pywikibot.output(u"Upload aborted.") return -- To view, visit https://gerrit.wikimedia.org/r/190560 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I0020e214db1cf0e276d9267c94d49859802b0a6e Gerrit-PatchSet: 4 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot <>

9 years, 2 months

Fixed: wikimedia/pywikibot-core#1985 (master - 464bd24)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #1985 Status: Fixed Duration: 47 minutes and 39 seconds Commit: 464bd24 (master) Author: John Vandenberg Message: Move import of 'gui' to top of file Using 'import foo.bar.x' in the middle of function defines 'foo' in that context, and if the import fails 'foo' is no longer bound. Bug: T89485 Change-Id: Ic43af9d08dd5c4e006ebfb12f03dd90ae7ec5423 View the changeset: https://github.com/wikimedia/pywikibot-core/compare/5e895f8e41fe...464bd248… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/50696783 -- You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications

9 years, 2 months

[Gerrit] Move import of 'gui' to top of file - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Move import of 'gui' to top of file ...................................................................... Move import of 'gui' to top of file Using 'import foo.bar.x' in the middle of function defines 'foo' in that context, and if the import fails 'foo' is no longer bound. Bug: T89485 Change-Id: Ic43af9d08dd5c4e006ebfb12f03dd90ae7ec5423 --- M pywikibot/editor.py 1 file changed, 8 insertions(+), 4 deletions(-) Approvals: Merlijn van Deen: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/editor.py b/pywikibot/editor.py index 32afdc6..7d1687d 100644 --- a/pywikibot/editor.py +++ b/pywikibot/editor.py @@ -17,8 +17,14 @@ import tempfile import pywikibot + from pywikibot import config from pywikibot.tools import deprecated + +try: + from pywikibot.userinterfaces import gui # noqa +except ImportError as e: + gui = e class TextEditor(object): @@ -107,14 +113,12 @@ finally: os.unlink(tempFilename) - try: - import pywikibot.userinterfaces.gui # noqa - except ImportError as e: + if isinstance(gui, ImportError): raise pywikibot.Error( 'Could not load GUI modules: %s\nNo editor available.\n' 'Set your favourite editor in user-config.py "editor", ' 'or install python packages tkinter and idlelib, which ' 'are typically part of Python but may be packaged separately ' - 'on your platform.\n' % e) + 'on your platform.\n' % gui) return pywikibot.ui.editText(text, jumpIndex=jumpIndex, highlight=highlight) -- To view, visit https://gerrit.wikimedia.org/r/190561 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ic43af9d08dd5c4e006ebfb12f03dd90ae7ec5423 Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot <>

9 years, 2 months

[Gerrit] page.py: Add string methods to class Revision - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: page.py: Add string methods to class Revision ...................................................................... page.py: Add string methods to class Revision Added: - Revision.__unicode__() - Revision.__repr__() Change-Id: Ie0ad1aa6ca30468b288502d12c280e70c20a5420 --- M pywikibot/page.py 1 file changed, 11 insertions(+), 1 deletion(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/page.py b/pywikibot/page.py index 3d357b8..ecbbe0d 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -4144,7 +4144,7 @@ } -class Revision(object): +class Revision(pywikibot.UnicodeMixin): """A structure holding information about a single revision of a Page.""" @@ -4213,6 +4213,16 @@ """ return getattr(self, key) + def __unicode__(self): + """Return string representation.""" + _content = u', '.join( + u'{0}: {1}'.format(k, v) for k, v in self.__dict__.items()) + return u'{{{0}}}'.format(_content) + + def __repr__(self): + """Return a more complete string representation.""" + return self.__dict__.__repr__() + class FileInfo(pywikibot.UnicodeMixin): -- To view, visit https://gerrit.wikimedia.org/r/190349 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ie0ad1aa6ca30468b288502d12c280e70c20a5420 Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 2 months

Failed: wikimedia/pywikibot-core#1983 (master - fecc123)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #1983 Status: Failed Duration: 46 minutes and 33 seconds Commit: fecc123 (master) Author: jenkins-bot Message: Merge "[FEAT] Category: Get newest pages" View the changeset: https://github.com/wikimedia/pywikibot-core/compare/a80c15d3a58b...fecc1235… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/50568908 -- You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications

9 years, 2 months

[Gerrit] [FEAT] Category: Get newest pages - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: [FEAT] Category: Get newest pages ...................................................................... [FEAT] Category: Get newest pages This returns the pages which have been added to a category ordered by the creation date from newest to oldest. To do that it needs to cache all pages and check on each iteration if any previously cached page was created after the current page is added. If that is the case the cached pages are newer than the current page and any other page which will be checked as the generator progresses as they are ordered by addition to the category and only pages are returned which were added to the category before the current page was added. Change-Id: I9bb3f74bbe2e3319ed2dbcdefab414a3204c2c32 --- M pywikibot/page.py M tests/category_tests.py 2 files changed, 90 insertions(+), 0 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/page.py b/pywikibot/page.py index c63af01..61e1d1b 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -2439,6 +2439,74 @@ """ return self.site.categoryinfo(self) + def newest_pages(self, total=None): + """ + Return pages in a category ordered by the creation date. + + If two or more pages are created at the same time, the pages are + returned in the order they were added to the category. The most recently + added page is returned first. + + It only allows to return the pages ordered from newest to oldest, as it + is impossible to determine the oldest page in a category without + checking all pages. But it is possible to check the category in order + with the newly added first and it yields all pages which were created + after the currently checked page was added (and thus there is no page + created after any of the cached but added before the currently checked). + + @param total: The total number of pages queried. + @type total: int + @return: A page generator of all pages in a category ordered by the + creation date. From newest to oldest. Note: It currently only + returns Page instances and not a subclass of it if possible. This + might change so don't expect to only get Page instances. + @rtype: generator + """ + def check_cache(latest): + """Return the cached pages in order and not more than total.""" + cached = [] + for timestamp in sorted((ts for ts in cache if ts > latest), + reverse=True): + # The complete list can be removed, it'll either yield all of + # them, or only a portion but will skip the rest anyway + cached += cache.pop(timestamp)[:None if total is None else + total - len(cached)] + if total and len(cached) >= total: + break # already got enough + assert(total is None or len(cached) <= total) + return cached + + # all pages which have been checked but where created before the + # current page was added, at some point they will be created after + # the current page was added. It saves all pages via the creation + # timestamp. Be prepared for multiple pages. + cache = defaultdict(list) + # TODO: Make site.categorymembers is usable as it returns pages + # There is no total defined, as it's not known how many pages need to be + # checked before the total amount of new pages was found. In worst case + # all pages of a category need to be checked. + for member in pywikibot.data.api.QueryGenerator( + site=self.site, list='categorymembers', cmsort='timestamp', + cmdir='older', cmprop='timestamp|title', + cmtitle=self.title()): + # TODO: Upcast to suitable class + page = pywikibot.Page(self.site, member['title']) + assert(page.namespace() == member['ns']) + cached = check_cache(pywikibot.Timestamp.fromISOformat( + member['timestamp'])) + for cached_page in cached: + yield cached_page + if total is not None: + total -= len(cached) + if total <= 0: + break + cache[page.oldest_revision.timestamp] += [page] + else: + # clear cache + assert(total is None or total > 0) + for cached_page in check_cache(pywikibot.Timestamp.min): + yield cached_page + # ### DEPRECATED METHODS #### @deprecated("list(Category.subcategories(...))") def subcategoriesList(self, recurse=False): diff --git a/tests/category_tests.py b/tests/category_tests.py index 8923884..d9d543c 100644 --- a/tests/category_tests.py +++ b/tests/category_tests.py @@ -209,6 +209,28 @@ self.assertEqual(cat.aslink(sortKey='Foo'), '[[Category:Wikipedia categories|Foo]]') +class CategoryNewestPages(TestCase): + + """Test newest_pages feature on French Wikinews.""" + + family = 'wikinews' + code = 'fr' + + cached = True + + def test_newest_pages(self): + """Test that the pages are getting older.""" + cat = pywikibot.Category(self.get_site(), u'Catégorie:Yukon Quest 2015') + last = pywikibot.Timestamp.max + count = 0 + for page in cat.newest_pages(): + creation_stamp = page.oldest_revision.timestamp + self.assertLessEqual(creation_stamp, last) + last = creation_stamp + count += 1 + self.assertEqual(count, cat.categoryinfo['size']) + + if __name__ == '__main__': try: unittest.main() -- To view, visit https://gerrit.wikimedia.org/r/190137 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I9bb3f74bbe2e3319ed2dbcdefab414a3204c2c32 Gerrit-PatchSet: 5 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 2 months

← Newer
1
...
4
5
6
7
8
9
10
...
13
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits February 2015