Build Update for wikimedia/pywikibot-core
-------------------------------------
Build: #1930
Status: Fixed
Duration: 42 minutes and 51 seconds
Commit: 04b730b (master)
Author: Geoffrey Mon
Message: Use unprotect i18n messages
When -unprotect flag is set or when the default value
for protection is 'all', use the unprotect i18n messages
when applicable for edit summary.
Fix generator_type extraction by removing colon from name
Bug: T63312
Change-Id: I2cda1fdd472070e4fd65392df9db2d0eebea41ba
View the changeset: https://github.com/wikimedia/pywikibot-core/compare/4aab14109aa8...04b730b1…
View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/47486541
--
You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications
jenkins-bot has submitted this change and it was merged.
Change subject: Use unprotect i18n messages
......................................................................
Use unprotect i18n messages
When -unprotect flag is set or when the default value
for protection is 'all', use the unprotect i18n messages
when applicable for edit summary.
Fix generator_type extraction by removing colon from name
Bug: T63312
Change-Id: I2cda1fdd472070e4fd65392df9db2d0eebea41ba
---
M scripts/protect.py
1 file changed, 17 insertions(+), 10 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/protect.py b/scripts/protect.py
index 18b6a6c..2bfaade 100644
--- a/scripts/protect.py
+++ b/scripts/protect.py
@@ -161,6 +161,7 @@
genFactory = pagegenerators.GeneratorFactory()
site = pywikibot.Site()
+ generator_type = None
protection_levels = set(site.protection_levels())
protection_types = site.protection_types()
if '' in protection_levels:
@@ -182,7 +183,7 @@
default_level = 'all'
elif arg.startswith('-default'):
if len(arg) == len('-default'):
- default_level = None
+ default_level = 'sysop'
else:
default_level = arg[len('-default:'):]
else:
@@ -198,19 +199,25 @@
if not is_p_type:
if not genFactory.handleArg(arg):
raise ValueError('Unknown parameter "{0}"'.format(arg))
- found = arg.find(':') + 1
+ found = arg.find(':')
if found:
- message_properties.update({'cat': arg[found:],
- 'page': arg[found:]})
+ message_properties.update({'cat': arg[found + 1:],
+ 'page': arg[found + 1:]})
if 'summary' not in options:
generator_type = arg[1:found] if found > 0 else arg[1:]
- if generator_type in default_summaries:
- message_type = default_summaries[generator_type]
- if message_type == 'simple' or message_properties:
- options['summary'] = i18n.twtranslate(
- site, 'protect-{0}'.format(message_type),
- message_properties)
+
+ if generator_type in default_summaries:
+ message_type = default_summaries[generator_type]
+ if message_type == 'simple' or message_properties:
+ if default_level == 'all':
+ options['summary'] = i18n.twtranslate(
+ site, 'unprotect-{0}'.format(message_type),
+ message_properties)
+ else:
+ options['summary'] = i18n.twtranslate(
+ site, 'protect-{0}'.format(message_type),
+ message_properties)
generator = genFactory.getCombinedGenerator()
# We are just protecting pages, so we have no need of using a preloading
--
To view, visit https://gerrit.wikimedia.org/r/185784
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I2cda1fdd472070e4fd65392df9db2d0eebea41ba
Gerrit-PatchSet: 9
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Sn1per <geofbot(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>
Build Update for wikimedia/pywikibot-core
-------------------------------------
Build: #1929
Status: Still Failing
Duration: 41 minutes and 48 seconds
Commit: 4aab141 (master)
Author: Geoffrey Mon
Message: Properly determine the valid protection types
self.protections.items() instead of self.protections on protect.py L105
Use Bot.treat() so we don't have to loop thru in run()
Enable automatic_quit because the exception is caught by Bot.run()
Use 'all' instead of 'none' and '' for unprotection level (its supported
by older versions of mediawiki)
Bug: T87124
Change-Id: I49615f7fddd548ee4bf5ad953317ed3c9dce883d
View the changeset: https://github.com/wikimedia/pywikibot-core/compare/6b796778ec9a...4aab1410…
View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/47482550
--
You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications
jenkins-bot has submitted this change and it was merged.
Change subject: Properly determine the valid protection types
......................................................................
Properly determine the valid protection types
self.protections.items() instead of self.protections on protect.py L105
Use Bot.treat() so we don't have to loop thru in run()
Enable automatic_quit because the exception is caught by Bot.run()
Use 'all' instead of 'none' and '' for unprotection level (its supported
by older versions of mediawiki)
Bug: T87124
Change-Id: I49615f7fddd548ee4bf5ad953317ed3c9dce883d
---
M scripts/i18n
M scripts/protect.py
2 files changed, 27 insertions(+), 29 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/i18n b/scripts/i18n
index 3aa58d7..4892b06 160000
--- a/scripts/i18n
+++ b/scripts/i18n
-Subproject commit 3aa58d74d10ff45803a84e6b78d84161972e37e7
+Subproject commit 4892b065142a239bb014ce6b754467df7a36196f
diff --git a/scripts/protect.py b/scripts/protect.py
index c8503bd..18b6a6c 100644
--- a/scripts/protect.py
+++ b/scripts/protect.py
@@ -15,18 +15,18 @@
the page selector. If no summary is supplied or couldn't
determine one from the selector it'll ask for one.
--unprotect Acts like "default:none"
+-unprotect Acts like "default:all"
-default: Sets the default protection level (default 'sysop'). If no
level is defined it doesn't change unspecified levels.
-[type]:[level] Set [type] protection level to [level]
-Usual values for [level] are: sysop, autoconfirmed, none; further levels may be
+Usual values for [level] are: sysop, autoconfirmed, all; further levels may be
provided by some wikis.
For all protection types (edit, move, etc.) it chooses the default protection
-level. This is "sysop" or "none" if -unprotect was selected. If multiple
+level. This is "sysop" or "all" if -unprotect was selected. If multiple
-unprotect or -default are used, only the last occurrence is applied.
Usage: python protect.py <OPTIONS>
@@ -82,29 +82,27 @@
self.generator = generator
self.protections = protections
- def run(self):
- """Start the bot's action.
+ def treat(self, page):
+ """Run the bot's action on each page.
- Loop through everything in the page generator and apply the
- protections.
+ Bot.run() loops through everything in the page generator and applies
+ the protections using this function.
"""
- for page in self.generator:
- self.current_page = page
- if not self.getOption('always'):
- choice = pywikibot.input_choice(
- u'Do you want to change the protection level of %s?'
- % page.title(asLink=True, forceInterwiki=True),
- [('yes', 'y'), ('No', 'n'), ('all', 'a')],
- 'n', automatic_quit=False)
- if choice == 'n':
- continue
- elif choice == 'a':
- self.options['always'] = True
- applicable = page.applicable_protections()
- protections = dict(
- [prot for prot in self.protections if prot[0] in applicable])
- page.protect(reason=self.getOption('summary'),
- protections=protections)
+ self.current_page = page
+ if not self.getOption('always'):
+ choice = pywikibot.input_choice(
+ u'Do you want to change the protection level of %s?'
+ % page.title(asLink=True, forceInterwiki=True),
+ [('yes', 'y'), ('No', 'n'), ('all', 'a')], 'n')
+ if choice == 'n':
+ return
+ elif choice == 'a':
+ self.options['always'] = True
+ applicable = page.applicable_protections()
+ protections = dict(
+ prot for prot in self.protections.items() if prot[0] in applicable)
+ page.protect(reason=self.getOption('summary'),
+ protections=protections)
def check_protection_level(operation, level, levels, default=None):
@@ -166,8 +164,7 @@
protection_levels = set(site.protection_levels())
protection_types = site.protection_types()
if '' in protection_levels:
- protection_levels.remove('')
- protection_levels.add('none')
+ protection_levels.add('all')
for arg in local_args:
if arg == '-always':
options['always'] = True
@@ -182,7 +179,7 @@
'Please use -imagelinks instead.\03{default}\n')
local_args.append('-imagelinks' + arg[7:])
elif arg.startswith('-unprotect'):
- default_level = 'none'
+ default_level = 'all'
elif arg.startswith('-default'):
if len(arg) == len('-default'):
default_level = None
@@ -230,8 +227,9 @@
for p_type, level in protections.items():
level = check_protection_level(p_type, level, protection_levels,
default_level)
- if level == 'none':
- level = ''
+ # '' is equivalent to 'all'
+ if level == 'none' or level == '':
+ level = 'all'
combined_protections[p_type] = level
if not options.get('summary'):
options['summary'] = pywikibot.input(
--
To view, visit https://gerrit.wikimedia.org/r/185661
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I49615f7fddd548ee4bf5ad953317ed3c9dce883d
Gerrit-PatchSet: 15
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Sn1per <geofbot(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: Unicodesnowman <admin(a)glados.cc>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>
Build Update for wikimedia/pywikibot-core
-------------------------------------
Build: #1928
Status: Broken
Duration: 44 minutes and 41 seconds
Commit: 6b79677 (master)
Author: Geoffrey Mon
Message: Updated pywikibot/core
Project: pywikibot/i18n 3aa58d74d10ff45803a84e6b78d84161972e37e7
Add unprotection messages
Add messages for edit summaries when bot
is unprotecting pages
Bug: T63312
Change-Id: I8255e075b40b40208b443633d99cf596e624c6be
View the changeset: https://github.com/wikimedia/pywikibot-core/compare/78c082c297cb...6b796778…
View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/47478934
--
You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications
jenkins-bot has submitted this change and it was merged.
Change subject: Add ItemClaimFilterPageGenerator
......................................................................
Add ItemClaimFilterPageGenerator
The generator filters ItemPages which does or does not contain
a specified claim.
Can be used via the -onlyif and -onlyifnot command line option.
Bug: T69568
Bug: T57005
Bug: T76547
Change-Id: I850f1063016fd0c8845c9509634f85b1830724ef
---
M pywikibot/page.py
M pywikibot/pagegenerators.py
M tests/pagegenerators_tests.py
3 files changed, 254 insertions(+), 0 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/page.py b/pywikibot/page.py
index 45e41c3..a10291f 100644
--- a/pywikibot/page.py
+++ b/pywikibot/page.py
@@ -3931,6 +3931,72 @@
else:
self.qualifiers[qualifier.getID()] = [qualifier]
+ def target_equals(self, value):
+ """
+ Check whether the Claim's target is equal to specified value.
+
+ The function checks for:
+ - ItemPage ID equality
+ - WbTime year equality
+ - Coordinate equality, regarding precision
+ - direct equality
+
+ @param value: the value to compare with
+ @return: true if the Claim's target is equal to the value provided,
+ false otherwise
+ @rtype: bool
+ """
+ if (isinstance(self.target, pywikibot.ItemPage) and
+ isinstance(value, str) and
+ self.target.id == value):
+ return True
+
+ if (isinstance(self.target, pywikibot.WbTime) and
+ not isinstance(value, pywikibot.WbTime) and
+ self.target.year == int(value)):
+ return True
+
+ if (isinstance(self.target, pywikibot.Coordinate) and
+ isinstance(value, str)):
+ coord_args = [float(x) for x in value.split(',')]
+ if len(coord_args) >= 3:
+ precision = coord_args[2]
+ else:
+ precision = 0.0001 # Default value (~10 m at equator)
+ try:
+ if self.target.precision is not None:
+ precision = max(precision, self.target.precision)
+ except TypeError:
+ pass
+
+ if (abs(self.target.lat - coord_args[0]) <= precision and
+ abs(self.target.lon - coord_args[1]) <= precision):
+ return True
+
+ if self.target == value:
+ return True
+
+ return False
+
+ def has_qualifier(self, qualifier_id, target):
+ """
+ Check whether Claim contains specified qualifier.
+
+ @param qualifier_id: id of the qualifier
+ @type qualifier_id: str
+ @param target: qualifier target to check presence of
+ @return: true if the qualifier was found, false otherwise
+ @rtype: bool
+ """
+ if self.isQualifier or self.isReference:
+ raise ValueError(u'Qualifiers and references cannot have '
+ u'qualifiers.')
+
+ for qualifier in self.qualifiers.get(qualifier_id, []):
+ if qualifier.target_equals(target):
+ return True
+ return False
+
def _formatValue(self):
"""
Format the target into the proper JSON value that Wikibase wants.
diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py
index aeb2d15..6f01605 100644
--- a/pywikibot/pagegenerators.py
+++ b/pywikibot/pagegenerators.py
@@ -235,6 +235,24 @@
Case insensitive regular expressions will be used and
dot matches any character, including a newline.
+-onlyif A claim the page needs to contain, otherwise the item won't
+ be returned.
+ The format is property=value,qualifier=value. Multiple (or
+ none) qualifiers can be passed, separated by commas.
+ Examples: P1=Q2 (property P1 must contain value Q2),
+ P3=Q4,P5=Q6,P6=Q7 (property P3 with value Q4 and
+ qualifiers: P5 with value Q6 and P6 with value Q7).
+ Value can be page ID, coordinate in format:
+ latitude,longitude[,precision] (all values are in decimal
+ degrees), year, or plain string.
+ The argument can be provided multiple times and the item
+ page will be returned only if all of the claims are present.
+ Argument can be also given as "-onlyif:expression".
+
+-onlyifnot A claim the page must not contain, otherwise the item won't
+ be returned.
+ For usage and examples, see -onlyif above.
+
-intersect Work on the intersection of all the provided generators.
"""
@@ -270,6 +288,7 @@
self.step = None
self.limit = None
self.articlefilter_list = []
+ self.claimfilter_list = []
self.intersect = False
self._site = site
@@ -353,6 +372,13 @@
else:
gensList = CombinedPageGenerator(self.gens)
dupfiltergen = DuplicateFilterPageGenerator(gensList)
+
+ if self.claimfilter_list:
+ dupfiltergen = PreloadingItemGenerator(dupfiltergen)
+ for claim in self.claimfilter_list:
+ dupfiltergen = ItemClaimFilterPageGenerator(dupfiltergen,
+ claim[0], claim[1],
+ claim[2], claim[3])
if self.articlefilter_list:
return RegexBodyFilterPageGenerator(
@@ -664,6 +690,20 @@
u'Which pattern do you want to grep?'))
else:
self.articlefilter_list.append(arg[6:])
+ return True
+ elif arg.startswith('-onlyif') or arg.startswith('-onlyifnot'):
+ ifnot = arg.startswith('-onlyifnot')
+ if (len(arg) == 7 and not ifnot) or (len(arg) == 10 and ifnot):
+ claim = pywikibot.input(u'Which claim do you want to filter?')
+ else:
+ claim = arg[11 if ifnot else 8:]
+
+ p = re.compile(r'(?<!\\),') # Match "," only if there no "\" before
+ temp = [] # Array to store split argument
+ for arg in p.split(claim):
+ temp.append(arg.replace('\,', ',').split('='))
+ self.claimfilter_list.append((temp[0][0], temp[0][1],
+ dict(temp[1:]), ifnot))
return True
elif arg.startswith('-yahoo'):
gen = YahooSearchPageGenerator(arg[7:], site=self.site)
@@ -1184,6 +1224,56 @@
yield page
+class ItemClaimFilter(object):
+
+ """Item claim filter."""
+
+ @classmethod
+ def __filter_match(cls, page, prop, claim, qualifiers=None):
+ """
+ Return true if the page contains the claim given.
+
+ @param page: the page to check
+ @return: true if page contains the claim, false otherwise
+ @rtype: bool
+ """
+ if not isinstance(page, pywikibot.ItemPage):
+ pywikibot.output(u'%s is not an ItemPage. Skipping.' % page)
+ return False
+ for page_claim in page.get()['claims'][prop]:
+ if page_claim.target_equals(claim):
+ if not qualifiers:
+ return True
+
+ for prop, val in qualifiers.items():
+ if not page_claim.has_qualifier(prop, val):
+ return False
+ return True
+
+ @classmethod
+ def filter(cls, generator, prop, claim, qualifiers=None, negate=False):
+ """
+ Yield all ItemPages which does contain certain claim in a property.
+
+ @param prop: property id to check
+ @type prop: str
+ @param claim: value of the property to check. Can be exact value (for
+ instance, ItemPage instance) or ItemPage ID string (e.g. 'Q37470').
+ @param qualifiers: dict of qualifiers that must be present, or None if
+ qualifiers are irrelevant
+ @type qualifiers: dict or None
+ @param negate: true if pages that does *not* contain specified claim
+ should be yielded, false otherwise
+ @type negate: bool
+ """
+ for page in generator:
+ if cls.__filter_match(page, prop, claim, qualifiers) and not negate:
+ yield page
+
+# name the generator methods
+ItemClaimFilterPageGenerator = ItemClaimFilter.filter
+
+
class RegexFilter(object):
"""Regex filter."""
diff --git a/tests/pagegenerators_tests.py b/tests/pagegenerators_tests.py
index 1bbecab..ae7e2d8 100755
--- a/tests/pagegenerators_tests.py
+++ b/tests/pagegenerators_tests.py
@@ -437,6 +437,73 @@
self.assertEqual(gf.namespaces, set([1, 6]))
+class TestItemClaimFilterPageGenerator(WikidataTestCase):
+
+ """Test item claim filter page generator generator."""
+
+ def _simple_claim_test(self, prop, claim, qualifiers, valid):
+ """
+ Test given claim on sample (India) page.
+
+ @param prop: the property to check
+ @type prop: str
+ @param claim: the claim the property should contain
+ @param qualifiers: qualifiers to check or None
+ @type qualifiers: dict or None
+ @param valid: true if the page should be yielded by the generator,
+ false otherwise
+ @type valid: bool
+ """
+ item = pywikibot.ItemPage(self.get_repo(), 'Q668')
+ gen = pagegenerators.ItemClaimFilterPageGenerator([item], prop,
+ claim, qualifiers)
+ pages = set(gen)
+ self.assertEqual(len(pages), 1 if valid else 0)
+
+ def _get_council_page(self):
+ """Return United Nations Security Council Wikidata page."""
+ site = self.get_site()
+ return pywikibot.Page(site, 'Q37470')
+
+ def test_valid_qualifiers(self):
+ """Test ItemClaimFilterPageGenerator on sample page using valid qualifiers."""
+ qualifiers = {
+ 'P580': pywikibot.WbTime(1950, 1, 1, precision=9,
+ site=self.get_site()),
+ 'P582': '1951',
+ }
+ self._simple_claim_test('P463', self._get_council_page(), qualifiers,
+ True)
+
+ def test_invalid_qualifiers(self):
+ """Test ItemClaimFilterPageGenerator on sample page using invalid qualifiers."""
+ qualifiers = {
+ 'P580': 1950,
+ 'P582': pywikibot.WbTime(1960, 1, 1, precision=9,
+ site=self.site),
+ }
+ self._simple_claim_test('P463', self._get_council_page(), qualifiers,
+ False)
+
+ def test_nonexisting_qualifiers(self):
+ """Test ItemClaimFilterPageGenerator on sample page using qualifiers the page doesn't have."""
+ qualifiers = {
+ 'P370': pywikibot.WbTime(1950, 1, 1, precision=9,
+ site=self.get_site()),
+ 'P232': pywikibot.WbTime(1960, 1, 1, precision=9,
+ site=self.get_site()),
+ }
+ self._simple_claim_test('P463', self._get_council_page(), qualifiers,
+ False)
+
+ def test_no_qualifiers(self):
+ """Test ItemClaimFilterPageGenerator on sample page without qualifiers."""
+ self._simple_claim_test('P474', '+91', None, True)
+ self._simple_claim_test('P463', 'Q37470', None, True)
+ self._simple_claim_test('P625', '21,78', None, True)
+ self._simple_claim_test('P625', '21,78.05,0.01', None, False)
+
+
class TestFactoryGenerator(DefaultSiteTestCase):
"""Test pagegenerators.GeneratorFactory."""
@@ -560,6 +627,37 @@
self.assertPagesInNamespaces(gen, set([1, 3]))
+class TestFactoryGeneratorWikibase(WikidataTestCase):
+
+ """Test pagegenerators.GeneratorFactory on Wikibase site."""
+
+ def test_onlyif(self):
+ """Test -onlyif without qualifiers."""
+ gf = pagegenerators.GeneratorFactory(site=self.site)
+ gf.handleArg('-page:Q15745378')
+ gf.handleArg('-onlyif:P357=International Journal of Minerals\, '
+ 'Metallurgy\, and Materials')
+ gen = gf.getCombinedGenerator()
+ self.assertEqual(len(set(gen)), 1)
+
+ def test_onlyifnot(self):
+ """Test -onlyifnot without qualifiers."""
+ gf = pagegenerators.GeneratorFactory(site=self.site)
+ gf.handleArg('-page:Q15745378')
+ gf.handleArg('-onlyifnot:P357=International Journal of Minerals\, '
+ 'Metallurgy\, and Materials')
+ gen = gf.getCombinedGenerator()
+ self.assertEqual(len(set(gen)), 0)
+
+ def test_onlyif_qualifiers(self):
+ """Test -onlyif with qualifiers."""
+ gf = pagegenerators.GeneratorFactory(site=self.site)
+ gf.handleArg('-page:Q668')
+ gf.handleArg('-onlyif:P47=Q837,P805=Q3088768')
+ gen = gf.getCombinedGenerator()
+ self.assertEqual(len(set(gen)), 1)
+
+
class TestLogeventsFactoryGenerator(DefaultSiteTestCase):
"""Test GeneratorFactory with pagegenerators.LogeventsPageGenerator."""
--
To view, visit https://gerrit.wikimedia.org/r/179158
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I850f1063016fd0c8845c9509634f85b1830724ef
Gerrit-PatchSet: 16
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: M4tx <m4tx(a)m4tx.pl>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: M4tx <m4tx(a)m4tx.pl>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: Multichill <maarten(a)mdammers.nl>
Gerrit-Reviewer: Ricordisamoa <ricordisamoa(a)openmailbox.org>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>