jenkins-bot has submitted this change and it was merged.
Change subject: [FIX] intersect_generators must use daemon threads
......................................................................
[FIX] intersect_generators must use daemon threads
Set threads in intersect_generators() as daemon threads, so that they
are killed automatically on program exit.
With the quit mechanism implemented via QuitKeyboardInterrupt,
KeyboardInterrupt is not caught any longer in intersect_generators()
and scripts based on Bot class hang when using pagegenerators with
parameter -intersect.
Bug: T129186
Change-Id: I61b2770afce4fd0c86ca0f0a7996432375e8b18a
---
M pywikibot/tools/__init__.py
1 file changed, 1 insertion(+), 0 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/tools/__init__.py b/pywikibot/tools/__init__.py
index 94a6dcd..c2c5bba 100644
--- a/pywikibot/tools/__init__.py
+++ b/pywikibot/tools/__init__.py
@@ -695,6 +695,7 @@
for source in genlist:
threaded_gen = ThreadedGenerator(name=repr(source), target=source)
+ threaded_gen.daemon = True
thrlist.append(threaded_gen)
while True:
--
To view, visit https://gerrit.wikimedia.org/r/275718
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I61b2770afce4fd0c86ca0f0a7996432375e8b18a
Gerrit-PatchSet: 3
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: [bugfix] Enable script_tests for featured.py
......................................................................
[bugfix] Enable script_tests for featured.py
featured -simulate doesn't response "'Use -help for further information."
but there are 4 lines:
Link FA does not exist
<empty line>
NOTE: featured articles are not implemented at wikipedia.de
0 pages written.
We cannot use first or 3rd line because of the localized answer for
template and site. We can expect the last line as response.
The behavior of the response might be changed because the data is hold
by wikibase for wm sites.
Bug: T128955
Change-Id: Ia9ac862d2d07db47b1edc87c89aa4136c524891c
---
M tests/script_tests.py
1 file changed, 1 insertion(+), 0 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/tests/script_tests.py b/tests/script_tests.py
index cc4348c..c0bb625 100644
--- a/tests/script_tests.py
+++ b/tests/script_tests.py
@@ -146,6 +146,7 @@
no_args_expected_results = {
# TODO: until done here, remember to set editor = None in user_config.py
'editarticle': 'Nothing changed',
+ 'featured': '0 pages written.',
'freebasemappingupload': 'Cannot find ',
'harvest_template': 'ERROR: Please specify',
'imageuncat': 'WARNING: This script is primarily written for Wikimedia Commons',
--
To view, visit https://gerrit.wikimedia.org/r/275391
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia9ac862d2d07db47b1edc87c89aa4136c524891c
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: imageuncat: Deprecate -yesterday for -logevents
......................................................................
imageuncat: Deprecate -yesterday for -logevents
The cli argument -logevents uses LogeventsPageGenerator to find
pages based on logevent. This can be used for imageuncat instead
of the -yesterday cli arg. But it doesn't filter based on
timestamp.
This commit adds the start and end timestamps to -logevents and
the same args to LogeventsPageGenerator. Hence, this now deprecates
the older -yesterday cli arg in imageuncat.
Bug: T67192
Change-Id: I5cd14609a43bd7b7343f234c0832cc471ae0250b
---
M pywikibot/pagegenerators.py
M scripts/imageuncat.py
M tests/pagegenerators_tests.py
3 files changed, 98 insertions(+), 11 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py
index 2f80a3b..2668ddd 100644
--- a/pywikibot/pagegenerators.py
+++ b/pywikibot/pagegenerators.py
@@ -103,7 +103,11 @@
across all namespaces.
-logevents Work on articles that were on a specified Special:Log.
- The value may be a comma separated list of three values:
+ The value may be a comma separated list of these values:
+
+ logevent,username,start,end
+
+ or for backward compatibility:
logevent,username,total
@@ -123,6 +127,12 @@
-logevents:delete,,20 gives 20 pages from deletion log
-logevents:protect,Usr gives pages from protect by user Usr
-logevents:patrol,Usr,20 gives 20 patroled pages by user Usr
+ -logevents:upload,,20121231,20100101 gives upload pages
+ in the 2010s, 2011s, and 2012s
+ -logevents:review,,20121231 gives review pages since the
+ beginning till the 31 Dec 2012
+ -logevents:review,Usr,20121231 gives review pages by user
+ Usr since the beginning till the 31 Dec 2012
In some cases it must be written as -logevents:"patrol,Usr,20"
@@ -528,7 +538,7 @@
recurse=recurse,
content=content)
- def _parse_log_events(self, logtype, user=None, total=None):
+ def _parse_log_events(self, logtype, user=None, start=None, end=None):
"""
Parse the -logevent argument information.
@@ -537,25 +547,47 @@
@param user: A username associated to the log events. Ignored if
empty string or None.
@type user: str
- @param total: The total amount of pages returned.
- @type total: str (castable to int) or int (positive)
+ @param start: Timestamp to start listing from. For backward
+ compatibility, this can also be the total amount of pages
+ that should be returned. It is taken as 'total' if the value does
+ not have 8 digits.
+ @type start: str convertable to Timestamp in the format YYYYMMDD. If
+ the length is not 8: for backward compatibility to use this as
+ 'total', it can also be a str (castable to int) or int (positive).
+ @param end: Timestamp to end listing at
+ @type end: str convertable to Timestamp in the format YYYYMMDD
@return: The generator or None if invalid 'total' value.
@rtype: LogeventsPageGenerator
"""
# TODO: Check if logtype is one of the allowed log types
# TODO: -*log used 500 as default total, also use with -logevents?
- if total is not None:
+
+ # 'start or None', because start might be an empty string
+ start = start or None
+ if isinstance(start, basestring) and len(start) == 8:
+ total = None
+ start = pywikibot.Timestamp.strptime(start, '%Y%m%d')
+ elif start is not None:
try:
- total = int(total)
+ total = int(start)
if total <= 0:
raise ValueError
except ValueError:
pywikibot.error(u'Total number of log ({0}) events must be a '
'positive int.'.format(total))
return None
+ start = None
+
+ if end is not None:
+ if start is None:
+ pywikibot.error('End cannot be given if start is not given.')
+ return None
+ end = pywikibot.Timestamp.strptime(end, '%Y%m%d')
+
# 'user or None', because user might be an empty string when
# 'foo,,bar' was used.
- return LogeventsPageGenerator(logtype, user or None, total=total)
+ return LogeventsPageGenerator(logtype, user or None, total=total,
+ start=start, end=end)
def handleArg(self, arg):
"""Parse one argument at a time.
@@ -919,8 +951,8 @@
@deprecated_args(number="total", mode="logtype", repeat=None)
-def LogeventsPageGenerator(logtype=None, user=None, site=None,
- namespace=0, total=None):
+def LogeventsPageGenerator(logtype=None, user=None, site=None, namespace=0,
+ total=None, start=None, end=None, reverse=False):
"""
Generate Pages for specified modes of logevents.
@@ -934,11 +966,18 @@
@type namespace: int
@param total: Maximum number of pages to retrieve in total
@type total: int
+ @param start: Timestamp to start listing from
+ @type start: pywikibot.Timestamp
+ @param end: Timestamp to end listing at
+ @type end: pywikibot.Timestamp
+ @param reverse: if True, start with oldest changes (default: newest)
+ @type reverse: bool
"""
if site is None:
site = pywikibot.Site()
- for entry in site.logevents(total=total, logtype=logtype,
- user=user, namespace=namespace):
+ for entry in site.logevents(total=total, logtype=logtype, user=user,
+ namespace=namespace, start=start, end=end,
+ reverse=False):
try:
yield entry.page()
except KeyError as e:
diff --git a/scripts/imageuncat.py b/scripts/imageuncat.py
index 7efd746..b63f4f6 100755
--- a/scripts/imageuncat.py
+++ b/scripts/imageuncat.py
@@ -30,7 +30,9 @@
from datetime import timedelta
import pywikibot
+from pywikibot.exceptions import ArgumentDeprecationWarning
from pywikibot import pagegenerators
+from pywikibot.tools import issue_deprecation_warning
docuReplacements = {
'¶ms;': pagegenerators.parameterHelp,
@@ -1371,6 +1373,10 @@
for arg in local_args:
if arg.startswith('-yesterday'):
generator = uploadedYesterday(site)
+ issue_deprecation_warning(
+ 'The usage of "-yesterday"',
+ '-logevents:"upload,,YYYYMMDD,YYYYMMDD"',
+ 2, ArgumentDeprecationWarning)
elif arg.startswith('-recentchanges'):
generator = recentChanges(site=site, delay=120)
else:
diff --git a/tests/pagegenerators_tests.py b/tests/pagegenerators_tests.py
index 91519c2..6c70194 100755
--- a/tests/pagegenerators_tests.py
+++ b/tests/pagegenerators_tests.py
@@ -892,6 +892,15 @@
"""Test GeneratorFactory with pagegenerators.LogeventsPageGenerator."""
+ @classmethod
+ def setUpClass(cls):
+ """Setup test class."""
+ super(TestLogeventsFactoryGenerator, cls).setUpClass()
+ site = pywikibot.Site()
+ newuser_logevents = list(site.logevents(logtype='newusers', total=1))
+ if len(newuser_logevents) == 0:
+ raise unittest.SkipTest('No newuser logs found to test with.')
+
user = True
@unittest.expectedFailure
@@ -954,6 +963,39 @@
self.assertLessEqual(len(pages), 10)
self.assertTrue(all(isinstance(item, pywikibot.Page) for item in pages))
+ def test_logevents_with_start_timestamp(self):
+ """Test -logevents which uses timestamp for start."""
+ gf = pagegenerators.GeneratorFactory(site=self.site)
+ # We limit the results to 1 as running this on large websites like
+ # Wikipedia will give an insane number of results as it asks for all
+ # logevents since beginning till now.
+ self.assertTrue(gf.handleArg('-limit:1'))
+ self.assertTrue(gf.handleArg('-logevents:newusers,,21000101'))
+ gen = gf.getCombinedGenerator()
+ self.assertIsNotNone(gen)
+ pages = set(gen)
+ self.assertGreater(len(pages), 0)
+ self.assertTrue(all(isinstance(item, pywikibot.Page) for item in pages))
+
+ def test_logevents_with_start_and_end_timestamp(self):
+ """Test -logevents which uses timestamps for start and end."""
+ gf = pagegenerators.GeneratorFactory(site=self.site)
+ self.assertTrue(gf.handleArg('-logevents:newusers,,21000101,20990101'))
+ gen = gf.getCombinedGenerator()
+ self.assertIsNotNone(gen)
+ pages = set(gen)
+ self.assertEqual(len(pages), 0)
+
+ def test_logevents_with_total(self):
+ """Test -logevents which uses total."""
+ gf = pagegenerators.GeneratorFactory(site=self.site)
+ self.assertTrue(gf.handleArg('-logevents:newusers,,1'))
+ gen = gf.getCombinedGenerator()
+ self.assertIsNotNone(gen)
+ pages = set(gen)
+ self.assertEqual(len(pages), 1)
+ self.assertTrue(all(isinstance(item, pywikibot.Page) for item in pages))
+
class PageGeneratorIntersectTestCase(GeneratorIntersectTestCase,
RecentChangesTestCase):
--
To view, visit https://gerrit.wikimedia.org/r/275204
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I5cd14609a43bd7b7343f234c0832cc471ae0250b
Gerrit-PatchSet: 9
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: AbdealiJK <abdealikothari(a)gmail.com>
Gerrit-Reviewer: AbdealiJK <abdealikothari(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: Fix -cat pagegenerator
......................................................................
Fix -cat pagegenerator
-cat without an argument was broken in 343b875
This broken unit test test_pagegen_i18n_input
Also properly deprecated the old 'arg' argument for
public GeneratorFactory.getCategory and .getCategoryGen
which was also a breaking change in 343b875.
Bug: T129001
Change-Id: Ifca4c65737e1f67f14576f540bdafcd0b82daf4c
---
M pywikibot/pagegenerators.py
1 file changed, 4 insertions(+), 1 deletion(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py
index 2f80a3b..7c9854e 100644
--- a/pywikibot/pagegenerators.py
+++ b/pywikibot/pagegenerators.py
@@ -475,6 +475,7 @@
return dupfiltergen
+ @deprecated_args(arg='category')
def getCategory(self, category):
"""
Return Category and start as defined by category.
@@ -483,7 +484,7 @@
@type category: str
@rtype: tuple
"""
- if category.startswith('-'): # old arg parameter
+ if category and category.startswith('-'):
categoryname = category.partition(':')[2]
issue_deprecation_warning(
'The usage of "{0}" as actual parameter of '
@@ -491,6 +492,7 @@
categoryname, 2, ArgumentDeprecationWarning)
else:
categoryname = category
+
if not categoryname:
categoryname = i18n.input(
'pywikibot-enter-category-name',
@@ -512,6 +514,7 @@
defaultNamespace=14))
return cat, startfrom
+ @deprecated_args(arg='category')
def getCategoryGen(self, category, recurse=False, content=False,
gen_func=None):
"""
--
To view, visit https://gerrit.wikimedia.org/r/275272
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ifca4c65737e1f67f14576f540bdafcd0b82daf4c
Gerrit-PatchSet: 2
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: listpages.py: Fix help docstring
......................................................................
listpages.py: Fix help docstring
Replace -dir with -save in docstring.
Change-Id: Ic17d6255a532c3959ac0924314fe2f01f4d53ea6
(cherry picked from commit 08af9606decaf405a034f4f5a9275c7eb56a4b32)
---
M scripts/listpages.py
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Mpaa: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/listpages.py b/scripts/listpages.py
index c89be5a..30d71ca 100755
--- a/scripts/listpages.py
+++ b/scripts/listpages.py
@@ -43,7 +43,7 @@
-get Page content is printed.
-save Save Page content to a file named as page.title(as_filename=True).
- Directory can be set with -dir:dir_name
+ Directory can be set with -save:dir_name
If no dir is specified, current direcory will be used.
-encode File encoding can be specified with '-encode:name' (name must be a
--
To view, visit https://gerrit.wikimedia.org/r/275267
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic17d6255a532c3959ac0924314fe2f01f4d53ea6
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: 2.0
Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>