jenkins-bot has submitted this change and it was merged.
Change subject: Enable edit failure tests in travis builds
......................................................................
Enable edit failure tests in travis builds
Adds new environment variable PYWIKIBOT2_TEST_WRITE_FAIL
which when set to '1' will run edit failure tests, which are
much safer than the other 'write' tests.
Also improve documentation for travis builds.
Change-Id: I85bf198c2a30b69a9e29d220d8696a92642272bc
---
M .travis.yml
M tests/README.rst
M tests/aspects.py
M tests/edit_failure_tests.py
4 files changed, 67 insertions(+), 21 deletions(-)
Approvals:
John Vandenberg: Looks good to me, but someone else must approve
XZise: Looks good to me, approved
jenkins-bot: Verified
diff --git a/.travis.yml b/.travis.yml
index f0c970b..1233f57 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -32,15 +32,20 @@
- echo "maximum_GET_length = 5000" >> ~/.pywikibot/user-config.py
- echo "console_encoding = 'utf8'" >> ~/.pywikibot/user-config.py
- - if [[ "$GITHUB_USER" == "wikimedia" ]]; then PYWIKIBOT2_USERNAME="Pywikibot-test"; fi
+ - if [[ "$GITHUB_USER" == "wikimedia" ]]; then
+ PYWIKIBOT2_USERNAME="Pywikibot-test" ;
+ else
+ export PYWIKIBOT2_TEST_WRITE_FAIL=1 ;
+ fi
- - if [[ -n "$USER_PASSWORD" && -n "$PYWIKIBOT2_USERNAME" ]]; then echo "usernames['$FAMILY']['$LANGUAGE'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py; fi
- - if [[ -n "$USER_PASSWORD" && -n "$PYWIKIBOT2_USERNAME" ]]; then echo "usernames['wikipedia']['en'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py; fi
- - if [[ -n "$USER_PASSWORD" && -n "$PYWIKIBOT2_USERNAME" ]]; then echo "usernames['wikipedia']['test'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py; fi
- - if [[ -n "$USER_PASSWORD" && -n "$PYWIKIBOT2_USERNAME" ]]; then echo "usernames['wikidata']['test'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py; fi
- - if [[ -n "$USER_PASSWORD" && -n "$PYWIKIBOT2_USERNAME" ]]; then echo "usernames['commons']['commons'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py; fi
-
- - if [[ -n "$USER_PASSWORD" && -n "$PYWIKIBOT2_USERNAME" ]]; then echo "('$PYWIKIBOT2_USERNAME', '$USER_PASSWORD')" > ~/.pywikibot/passwordfile; fi
+ - if [[ -n "$USER_PASSWORD" && -n "$PYWIKIBOT2_USERNAME" ]]; then
+ echo "usernames['$FAMILY']['$LANGUAGE'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py ;
+ echo "usernames['wikipedia']['en'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py ;
+ echo "usernames['wikipedia']['test'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py ;
+ echo "usernames['wikidata']['test'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py ;
+ echo "usernames['commons']['commons'] = '$PYWIKIBOT2_USERNAME'" >> ~/.pywikibot/user-config.py ;
+ echo "('$PYWIKIBOT2_USERNAME', '$USER_PASSWORD')" > ~/.pywikibot/passwordfile ;
+ fi
- cd externals/httplib2
- python setup.py install
diff --git a/tests/README.rst b/tests/README.rst
index 4258016..214c110 100644
--- a/tests/README.rst
+++ b/tests/README.rst
@@ -83,7 +83,7 @@
use the Wikimedia global (SUL) account 'Pywikibot-test', which has a password
securely stored in .travis.yml . See section env:global:secure.
-Anyone can run these tests on travis-ci.org using their own account, with
+Anyone can run these tests on travis-ci.org using their own github account, with
code changes that have not been merged into the main repository. To do this:
1. create a github and travis-ci account
@@ -103,9 +103,35 @@
2. Add a new variable named PYWIKIBOT2_USERNAME and a value of a valid
Wikimedia SUL username
3. Add another variable named USER_PASSWORD, with the private password for
- the Wikimedia SUL username used in step 2
+ the Wikimedia SUL username used in step 2. Check that this
+ environment variable has "Display value in build logs" set to OFF, so
+ the password does not leak into the build logs.
4. The next build should run tests that require a logged in user
+While passwords in travis-ci environment variables are not leaked in normal
+operations, you are responsible for your own passwords.
+
+It is strongly recommended that an untrusted bot account is created for
+travis tests, using a password that is not shared with trusted accounts.
+
+There are a set of 'edit failure' tests, which attempt to write to the wikis
+and **should** fail. If there is a bug in pywikibot or MediaWiki, these
+tests **may** actually perform a write operation.
+
+These 'edit failure' tests are disabled by default for the 'wikimedia' builds,
+but are enabled by default on builds by any other github account.
+
+To disable 'edit failure' tests in travis, add PYWIKIBOT2_TEST_WRITE_FAIL=0
+
+There are also several other 'write' tests which also attempt to perform
+write operations successfully. These **will** write to the wikis, and they
+should always only write to 'test' wikis.
+
+These 'write' tests are disabled in travis builds, and currently can not be
+run on travis as they require interaction using a terminal.
+
+To enable 'write' tests in travis, add PYWIKIBOT2_TEST_WRITE=1
+
Contributing tests
==================
diff --git a/tests/aspects.py b/tests/aspects.py
index be21cf8..8147a70 100644
--- a/tests/aspects.py
+++ b/tests/aspects.py
@@ -518,24 +518,37 @@
"""
Set up the test class.
- Prevent test classes to write to the site and also cache results.
+ Reject write test classes configured with non-test wikis, or caching.
- Skip the test class if environment variable PYWIKIBOT2_TEST_WRITE
- does not equal 1.
+ Prevent test classes from writing to the site by default.
+
+ If class attribute 'write' is -1, the test class is skipped unless
+ environment variable PYWIKIBOT2_TEST_WRITE_FAIL is set to 1.
+
+ Otherwise the test class is skipped unless environment variable
+ PYWIKIBOT2_TEST_WRITE is set to 1.
"""
- if os.environ.get('PYWIKIBOT2_TEST_WRITE', '0') != '1':
+ super(SiteWriteMixin, cls).setUpClass()
+
+ site = cls.get_site()
+ assert('test' in (site.family.name, site.code))
+
+ if cls.write == -1:
+ env_var = 'PYWIKIBOT2_TEST_WRITE_FAIL'
+ else:
+ env_var = 'PYWIKIBOT2_TEST_WRITE'
+
+ if os.environ.get(env_var, '0') != '1':
raise unittest.SkipTest(
'%r write tests disabled. '
- 'Set PYWIKIBOT2_TEST_WRITE=1 to enable.'
- % cls.__name__)
+ 'Set %s=1 to enable.'
+ % (cls.__name__, env_var))
if issubclass(cls, ForceCacheMixin):
raise Exception(
'%s can not be a subclass of both '
'SiteEditTestCase and ForceCacheMixin'
% cls.__name__)
-
- super(SiteWriteMixin, cls).setUpClass()
class RequireUserMixin(TestCaseBase):
diff --git a/tests/edit_failure_tests.py b/tests/edit_failure_tests.py
index 57d3ffd..fa3c1db 100644
--- a/tests/edit_failure_tests.py
+++ b/tests/edit_failure_tests.py
@@ -4,6 +4,8 @@
These tests should never write to the wiki,
unless something has broken badly.
+
+These tests use special code 'write = -1' for edit failures.
"""
#
# (C) Pywikibot team, 2014
@@ -27,7 +29,7 @@
"""Test cases for edits which should fail to save."""
- write = True
+ write = -1
family = 'wikipedia'
code = 'test'
@@ -55,7 +57,7 @@
"""Test cases for actions which should fail to save."""
- write = True
+ write = -1
family = 'wikipedia'
code = 'test'
@@ -87,7 +89,7 @@
family = 'wikidata'
code = 'test'
- write = True
+ write = -1
def test_itempage_save(self):
"""Test ItemPage save method inherited from superclass Page."""
--
To view, visit https://gerrit.wikimedia.org/r/190635
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I85bf198c2a30b69a9e29d220d8696a92642272bc
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: Normalize usernames in password file
......................................................................
Normalize usernames in password file
Capitalize the first letter of username in password file when used according to
Wikipedia username policy. Warn if the format of tuple in password file is not
correct.
Bug: T74974
Change-Id: I2035bf03bad62f6f505b2a9129758c2cc4f6b5ce
---
M pywikibot/exceptions.py
M pywikibot/login.py
2 files changed, 27 insertions(+), 5 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/exceptions.py b/pywikibot/exceptions.py
index 3bb6a7a..2a8105f 100644
--- a/pywikibot/exceptions.py
+++ b/pywikibot/exceptions.py
@@ -57,6 +57,7 @@
UserWarning: warnings targetted at users
- config2._ConfigurationDeprecationWarning: user configuration file problems
+ - login._PasswordFileWarning: password file problems
- ArgumentDeprecationWarning: command line argument problems
- FamilyMaintenanceWarning: missing information in family definition
"""
diff --git a/pywikibot/login.py b/pywikibot/login.py
index db7be8f..816378f 100644
--- a/pywikibot/login.py
+++ b/pywikibot/login.py
@@ -3,18 +3,27 @@
"""Library to log the bot in to a wiki account."""
#
# (C) Rob W.W. Hooft, 2003
-# (C) Pywikibot team, 2003-2012
+# (C) Pywikibot team, 2003-2015
#
# Distributed under the terms of the MIT license.
#
__version__ = '$Id$'
#
import codecs
+from warnings import warn
import pywikibot
from pywikibot import config
from pywikibot.tools import deprecated_args
from pywikibot.exceptions import NoUsername
+
+
+class _PasswordFileWarning(UserWarning):
+
+ """The format of password file is incorrect."""
+
+ pass
+
_logger = "wiki.login"
@@ -169,18 +178,30 @@
for line in password_f:
if not line.strip():
continue
- entry = eval(line)
+ try:
+ entry = eval(line)
+ except SyntaxError:
+ entry = None
+ if type(entry) is not tuple:
+ warn('Invalid tuple', _PasswordFileWarning)
+ continue
+ if not 2 <= len(entry) <= 4:
+ warn('The length of tuple should be 2 to 4 (%s given)'
+ % len(entry), _PasswordFileWarning)
+ continue
+ username = entry[-2]
+ username = username[0].upper() + username[1:]
if len(entry) == 4: # for userinfo included code and family
if entry[0] == self.site.code and \
entry[1] == self.site.family.name and \
- entry[2] == self.username:
+ username == self.username:
self.password = entry[3]
elif len(entry) == 3: # for userinfo included family
if entry[0] == self.site.family.name and \
- entry[1] == self.username:
+ username == self.username:
self.password = entry[2]
elif len(entry) == 2: # for default userinfo
- if entry[0] == self.username:
+ if username == self.username:
self.password = entry[1]
password_f.close()
--
To view, visit https://gerrit.wikimedia.org/r/190638
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I2035bf03bad62f6f505b2a9129758c2cc4f6b5ce
Gerrit-PatchSet: 4
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: VcamX <vcamx3(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: VcamX <vcamx3(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: [FIX] Upload: Fixing uploads
......................................................................
[FIX] Upload: Fixing uploads
When there is a page name problem, it reports it and it might then exit
the loop. But if there is no problem it doesn't continue but asks the
user and the loop continues.
It also calls upload_file internally and thus preserving forwarding the
URL, as upload_image was assuming all URLs are a string.
Change-Id: I0020e214db1cf0e276d9267c94d49859802b0a6e
---
M scripts/upload.py
1 file changed, 6 insertions(+), 4 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/upload.py b/scripts/upload.py
index b02068b..ba258b3 100755
--- a/scripts/upload.py
+++ b/scripts/upload.py
@@ -6,7 +6,7 @@
Arguments:
-keep Keep the filename as is
- -filename Target filename
+ -filename Target filename without the namespace prefix
-noverify Do not ask for verification of the upload description if one
is given
-abortonwarn: Abort upload on the specified warning type. If no warning type
@@ -229,8 +229,8 @@
first_check = True
while True:
if not first_check:
- filename = pywikibot.input(u'Enter a better name, '
- 'or press enter to skip:')
+ filename = pywikibot.input('Enter a better name, or press '
+ 'enter to skip the file:')
if not filename:
return None
first_check = False
@@ -270,6 +270,8 @@
"repository and cannot be overwritten."
% filename)
continue
+ else:
+ break
except pywikibot.NoPage:
break
@@ -372,7 +374,7 @@
if answer:
self.ignoreWarning = True
self.keepFilename = True
- return self.upload_image(debug)
+ return self.upload_file(file_url, debug)
else:
pywikibot.output(u"Upload aborted.")
return
--
To view, visit https://gerrit.wikimedia.org/r/190560
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I0020e214db1cf0e276d9267c94d49859802b0a6e
Gerrit-PatchSet: 4
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: jenkins-bot <>
Build Update for wikimedia/pywikibot-core
-------------------------------------
Build: #1985
Status: Fixed
Duration: 47 minutes and 39 seconds
Commit: 464bd24 (master)
Author: John Vandenberg
Message: Move import of 'gui' to top of file
Using 'import foo.bar.x' in the middle of function defines 'foo'
in that context, and if the import fails 'foo' is no longer bound.
Bug: T89485
Change-Id: Ic43af9d08dd5c4e006ebfb12f03dd90ae7ec5423
View the changeset: https://github.com/wikimedia/pywikibot-core/compare/5e895f8e41fe...464bd248…
View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/50696783
--
You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications
jenkins-bot has submitted this change and it was merged.
Change subject: Move import of 'gui' to top of file
......................................................................
Move import of 'gui' to top of file
Using 'import foo.bar.x' in the middle of function defines 'foo'
in that context, and if the import fails 'foo' is no longer bound.
Bug: T89485
Change-Id: Ic43af9d08dd5c4e006ebfb12f03dd90ae7ec5423
---
M pywikibot/editor.py
1 file changed, 8 insertions(+), 4 deletions(-)
Approvals:
Merlijn van Deen: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/editor.py b/pywikibot/editor.py
index 32afdc6..7d1687d 100644
--- a/pywikibot/editor.py
+++ b/pywikibot/editor.py
@@ -17,8 +17,14 @@
import tempfile
import pywikibot
+
from pywikibot import config
from pywikibot.tools import deprecated
+
+try:
+ from pywikibot.userinterfaces import gui # noqa
+except ImportError as e:
+ gui = e
class TextEditor(object):
@@ -107,14 +113,12 @@
finally:
os.unlink(tempFilename)
- try:
- import pywikibot.userinterfaces.gui # noqa
- except ImportError as e:
+ if isinstance(gui, ImportError):
raise pywikibot.Error(
'Could not load GUI modules: %s\nNo editor available.\n'
'Set your favourite editor in user-config.py "editor", '
'or install python packages tkinter and idlelib, which '
'are typically part of Python but may be packaged separately '
- 'on your platform.\n' % e)
+ 'on your platform.\n' % gui)
return pywikibot.ui.editText(text, jumpIndex=jumpIndex, highlight=highlight)
--
To view, visit https://gerrit.wikimedia.org/r/190561
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic43af9d08dd5c4e006ebfb12f03dd90ae7ec5423
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: [FEAT] Category: Get newest pages
......................................................................
[FEAT] Category: Get newest pages
This returns the pages which have been added to a category ordered by
the creation date from newest to oldest. To do that it needs to cache
all pages and check on each iteration if any previously cached page was
created after the current page is added. If that is the case the cached
pages are newer than the current page and any other page which will be
checked as the generator progresses as they are ordered by addition to
the category and only pages are returned which were added to the
category before the current page was added.
Change-Id: I9bb3f74bbe2e3319ed2dbcdefab414a3204c2c32
---
M pywikibot/page.py
M tests/category_tests.py
2 files changed, 90 insertions(+), 0 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/page.py b/pywikibot/page.py
index c63af01..61e1d1b 100644
--- a/pywikibot/page.py
+++ b/pywikibot/page.py
@@ -2439,6 +2439,74 @@
"""
return self.site.categoryinfo(self)
+ def newest_pages(self, total=None):
+ """
+ Return pages in a category ordered by the creation date.
+
+ If two or more pages are created at the same time, the pages are
+ returned in the order they were added to the category. The most recently
+ added page is returned first.
+
+ It only allows to return the pages ordered from newest to oldest, as it
+ is impossible to determine the oldest page in a category without
+ checking all pages. But it is possible to check the category in order
+ with the newly added first and it yields all pages which were created
+ after the currently checked page was added (and thus there is no page
+ created after any of the cached but added before the currently checked).
+
+ @param total: The total number of pages queried.
+ @type total: int
+ @return: A page generator of all pages in a category ordered by the
+ creation date. From newest to oldest. Note: It currently only
+ returns Page instances and not a subclass of it if possible. This
+ might change so don't expect to only get Page instances.
+ @rtype: generator
+ """
+ def check_cache(latest):
+ """Return the cached pages in order and not more than total."""
+ cached = []
+ for timestamp in sorted((ts for ts in cache if ts > latest),
+ reverse=True):
+ # The complete list can be removed, it'll either yield all of
+ # them, or only a portion but will skip the rest anyway
+ cached += cache.pop(timestamp)[:None if total is None else
+ total - len(cached)]
+ if total and len(cached) >= total:
+ break # already got enough
+ assert(total is None or len(cached) <= total)
+ return cached
+
+ # all pages which have been checked but where created before the
+ # current page was added, at some point they will be created after
+ # the current page was added. It saves all pages via the creation
+ # timestamp. Be prepared for multiple pages.
+ cache = defaultdict(list)
+ # TODO: Make site.categorymembers is usable as it returns pages
+ # There is no total defined, as it's not known how many pages need to be
+ # checked before the total amount of new pages was found. In worst case
+ # all pages of a category need to be checked.
+ for member in pywikibot.data.api.QueryGenerator(
+ site=self.site, list='categorymembers', cmsort='timestamp',
+ cmdir='older', cmprop='timestamp|title',
+ cmtitle=self.title()):
+ # TODO: Upcast to suitable class
+ page = pywikibot.Page(self.site, member['title'])
+ assert(page.namespace() == member['ns'])
+ cached = check_cache(pywikibot.Timestamp.fromISOformat(
+ member['timestamp']))
+ for cached_page in cached:
+ yield cached_page
+ if total is not None:
+ total -= len(cached)
+ if total <= 0:
+ break
+ cache[page.oldest_revision.timestamp] += [page]
+ else:
+ # clear cache
+ assert(total is None or total > 0)
+ for cached_page in check_cache(pywikibot.Timestamp.min):
+ yield cached_page
+
# ### DEPRECATED METHODS ####
@deprecated("list(Category.subcategories(...))")
def subcategoriesList(self, recurse=False):
diff --git a/tests/category_tests.py b/tests/category_tests.py
index 8923884..d9d543c 100644
--- a/tests/category_tests.py
+++ b/tests/category_tests.py
@@ -209,6 +209,28 @@
self.assertEqual(cat.aslink(sortKey='Foo'), '[[Category:Wikipedia categories|Foo]]')
+class CategoryNewestPages(TestCase):
+
+ """Test newest_pages feature on French Wikinews."""
+
+ family = 'wikinews'
+ code = 'fr'
+
+ cached = True
+
+ def test_newest_pages(self):
+ """Test that the pages are getting older."""
+ cat = pywikibot.Category(self.get_site(), u'Catégorie:Yukon Quest 2015')
+ last = pywikibot.Timestamp.max
+ count = 0
+ for page in cat.newest_pages():
+ creation_stamp = page.oldest_revision.timestamp
+ self.assertLessEqual(creation_stamp, last)
+ last = creation_stamp
+ count += 1
+ self.assertEqual(count, cat.categoryinfo['size'])
+
+
if __name__ == '__main__':
try:
unittest.main()
--
To view, visit https://gerrit.wikimedia.org/r/190137
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I9bb3f74bbe2e3319ed2dbcdefab414a3204c2c32
Gerrit-PatchSet: 5
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>