jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/609295 )
Change subject: Use set in Page.templatesWithParams
......................................................................
Use set in Page.templatesWithParams
Set membership lookups are faster.
Change-Id: I6cf46fdfe6c8494857f38acc468fc4b90c03b56a
---
M pywikibot/page/__init__.py
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/page/__init__.py b/pywikibot/page/__init__.py
index a4605a0..31468ae 100644
--- a/pywikibot/page/__init__.py
+++ b/pywikibot/page/__init__.py
@@ -2429,7 +2429,7 @@
"""
# WARNING: may not return all templates used in particularly
# intricate cases such as template substitution
- titles = [t.title() for t in self.templates()]
+ titles = {t.title() for t in self.templates()}
templates = self.raw_extracted_templates
# backwards-compatibility: convert the dict returned as the second
# element into a list in the format used by old scripts
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/609295
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I6cf46fdfe6c8494857f38acc468fc4b90c03b56a
Gerrit-Change-Number: 609295
Gerrit-PatchSet: 2
Gerrit-Owner: Matěj Suchánek <matejsuchanek97(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/608658 )
Change subject: [IMPR] newitem.py: ignore pages with {{delete}} templates
......................................................................
[IMPR] newitem.py: ignore pages with {{delete}} templates
- DELETION_TEMPLATES hold all WD items for deletion templates
- introduce _skipping_templates attribute which holds all deletion templates
including redirects for each site. This enable newitem.py to be a
MultipleSites bot.
- derive NewItemRobot from NoRedirectPageBot which skips redirects
- move all skipping conditions to skip_page method
- retrieve all skipping templates in get_skipping_templates method and
store the result as a set in _skipping_templates dict for each site
- skip_templates method checks for each template of a page if it is in
a skipping template and return it's title. skip_page method uses this
title for printing a message
- split initializer and move setting up pageAgeBefore and lastEditBefore
attributes to setup method
Bug: T256676
Change-Id: If7c36813606a89e11f17773611ec69fe080d62ca
---
M scripts/newitem.py
1 file changed, 92 insertions(+), 31 deletions(-)
Approvals:
Matěj Suchánek: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/newitem.py b/scripts/newitem.py
index 1e85ee0..1c7acf0 100755
--- a/scripts/newitem.py
+++ b/scripts/newitem.py
@@ -31,12 +31,16 @@
from textwrap import fill
import pywikibot
-from pywikibot import pagegenerators, WikidataBot
+from pywikibot import pagegenerators
+from pywikibot.bot import NoRedirectPageBot, WikidataBot
from pywikibot.exceptions import (LockedPage, NoCreateError, NoPage,
PageNotSaved)
-class NewItemRobot(WikidataBot):
+DELETION_TEMPLATES = ('Q4847311', 'Q6687153', 'Q21528265')
+
+
+class NewItemRobot(WikidataBot, NoRedirectPageBot):
"""A bot to create new items."""
@@ -56,17 +60,23 @@
self.generator = generator
self.pageAge = self.getOption('pageage')
self.lastEdit = self.getOption('lastedit')
+ self._skipping_templates = {}
+
+ def setup(self):
+ """Setup ages."""
+ super(NewItemRobot, self).setup()
+
self.pageAgeBefore = self.repo.server_time() - timedelta(
days=self.pageAge)
self.lastEditBefore = self.repo.server_time() - timedelta(
days=self.lastEdit)
pywikibot.output('Page age is set to {0} days so only pages created'
- '\nbefore {1} will be considered.'
+ '\nbefore {1} will be considered.\n'
.format(self.pageAge, self.pageAgeBefore.isoformat()))
pywikibot.output(
'Last edit is set to {0} days so only pages last edited'
- '\nbefore {1} will be considered.'.format(
- self.lastEdit, self.lastEditBefore.isoformat()))
+ '\nbefore {1} will be considered.\n'
+ .format(self.lastEdit, self.lastEditBefore.isoformat()))
@staticmethod
def _touch_page(page):
@@ -87,6 +97,83 @@
if exc is None and self.getOption('touch'):
self._touch_page(page)
+ def get_skipping_templates(self, site):
+ """Get templates which leads the page to be skipped.
+
+ If the script is used for multiple sites, hold the skipping templates
+ as attribute.
+ """
+ if site in self._skipping_templates:
+ return self._skipping_templates[site]
+
+ skipping_templates = set()
+ pywikibot.output('Retrieving skipping templates for site {}...'
+ .format(site))
+ for item in DELETION_TEMPLATES:
+ template = site.page_from_repository(item)
+
+ if template is None:
+ continue
+
+ skipping_templates.add(template)
+ # also add redirect templates
+ skipping_templates.update(
+ template.getReferences(follow_redirects=False,
+ with_template_inclusion=False,
+ filter_redirects=True,
+ namespaces=site.namespaces.TEMPLATE))
+ self._skipping_templates[site] = skipping_templates
+ return skipping_templates
+
+ def skip_templates(self, page):
+ """Check whether the page is to be skipped due to skipping template.
+
+ @param page: treated page
+ @type page: pywikibot.Page
+ @return: the template which leads to skip
+ @rtype: str
+ """
+ skipping_templates = self.get_skipping_templates(page.site)
+ for template, _ in page.templatesWithParams():
+ if template in skipping_templates:
+ return template.title(with_ns=False)
+ return ''
+
+ def skip_page(self, page):
+ """Skip pages which are unwanted to treat."""
+ if page.editTime() > self.lastEditBefore:
+ pywikibot.output(
+ 'Last edit on {page} was on {page.latest_revision.timestamp}.'
+ '\nToo recent. Skipping.'.format(page=page))
+ return True
+
+ if page.oldest_revision.timestamp > self.pageAgeBefore:
+ pywikibot.output(
+ 'Page creation of {page} on {page.oldest_revision.timestamp} '
+ 'is too recent. Skipping.'.format(page=page))
+ return True
+
+ if page.isCategoryRedirect():
+ pywikibot.output('{} is a category redirect. Skipping.'
+ .format(page))
+ return True
+
+ if page.langlinks():
+ # FIXME: Implement this
+ pywikibot.output(
+ 'Found language links (interwiki links) for {}.\n'
+ "Haven't implemented that yet so skipping."
+ .format(page))
+ return True
+
+ template = self.skip_templates(page)
+ if template:
+ pywikibot.output('%s contains {{%s}}. Skipping.'
+ % (page, template))
+ return True
+
+ return super(NewItemRobot, self).skip_page(page)
+
def treat_page_and_item(self, page, item):
"""Treat page/item."""
if item and item.exists():
@@ -96,32 +183,6 @@
self._touch_page(page)
return
- if page.isRedirectPage():
- pywikibot.output('{0} is a redirect page. Skipping.'.format(page))
- return
- if page.editTime() > self.lastEditBefore:
- pywikibot.output(
- 'Last edit on {0} was on {1}.\nToo recent. Skipping.'
- .format(page, page.editTime().isoformat()))
- return
-
- if page.oldest_revision.timestamp > self.pageAgeBefore:
- pywikibot.output(
- 'Page creation of {0} on {1} is too recent. Skipping.'
- .format(page, page.editTime().isoformat()))
- return
- if page.isCategoryRedirect():
- pywikibot.output('{0} is a category redirect. Skipping.'
- .format(page))
- return
-
- if page.langlinks():
- # FIXME: Implement this
- pywikibot.output(
- 'Found language links (interwiki links).\n'
- "Haven't implemented that yet so skipping.")
- return
-
self.create_item_for_page(
page, callback=lambda _, exc: self._callback(page, exc))
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/608658
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: If7c36813606a89e11f17773611ec69fe080d62ca
Gerrit-Change-Number: 608658
Gerrit-PatchSet: 4
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com>
Gerrit-Reviewer: JAn Dudík <jan.dudik(a)gmail.com>
Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/608947 )
Change subject: [IMPR] Wait for _putthread is done in BaseBot.exit()
......................................................................
[IMPR] Wait for _putthread is done in BaseBot.exit()
Re-implement waiting for for _putthread is done in BaseBot.exit()
but use stopme() function instead of _flush.
The reason for the revert of 87cfda61cd98 was that async put was blocked:
With _flush a tuple containing None was added to the page_put_queue and
the async_manager function leaved the loop and no longer processed any
queue items which were put later by async_request. This means all
following wheren't pocessed anymore i.e. no page was written to site.
in result the page_put_queue increases up to its max_queue_size and blocks
after that including KeyboardInterrupt wasn't funtional.
This problem only occures if the bot itself runs in a loop.
This partly reverts commit 4291f36321606 but re-implements most of the
87cfda61cd98 patch except that _flush was replaced by stopme,
Change-Id: Ic26928eafdf19844c950f40993073a4e6b9dd687
---
M pywikibot/bot.py
1 file changed, 15 insertions(+), 7 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/bot.py b/pywikibot/bot.py
index 9a5b42e..2d22767 100644
--- a/pywikibot/bot.py
+++ b/pywikibot/bot.py
@@ -1290,28 +1290,36 @@
May be overridden by subclasses.
"""
self.teardown()
+ if hasattr(self, '_start_ts'):
+ read_delta = pywikibot.Timestamp.now() - self._start_ts
+ read_seconds = int(read_delta.total_seconds())
+
+ # wait until pending threads finished but don't close the queue
+ pywikibot.stopme()
+
pywikibot.output('\n{} pages read'
'\n{} pages written'
'\n{} pages skipped'
.format(self._treat_counter,
self._save_counter,
self._skip_counter))
+
if hasattr(self, '_start_ts'):
- delta = (pywikibot.Timestamp.now() - self._start_ts)
- seconds = int(delta.total_seconds())
- if delta.days:
+ write_delta = pywikibot.Timestamp.now() - self._start_ts
+ write_seconds = int(write_delta.total_seconds())
+ if write_delta.days:
pywikibot.output(
'Execution time: {d.days} days, {d.seconds} seconds'
- .format(d=delta))
+ .format(d=write_delta))
else:
pywikibot.output('Execution time: {} seconds'
- .format(delta.seconds))
+ .format(write_delta.seconds))
if self._treat_counter:
pywikibot.output('Read operation time: {:.1f} seconds'
- .format(seconds / self._treat_counter))
+ .format(read_seconds / self._treat_counter))
if self._save_counter:
pywikibot.output('Write operation time: {:.1f} seconds'
- .format(seconds / self._save_counter))
+ .format(write_seconds / self._save_counter))
# exc_info contains exception from self.run() while terminating
exc_info = sys.exc_info()
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/608947
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: Ic26928eafdf19844c950f40993073a4e6b9dd687
Gerrit-Change-Number: 608947
Gerrit-PatchSet: 1
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/608420 )
Change subject: [doc] minor documentation updates
......................................................................
[doc] minor documentation updates
- Update CONTENT.rst: README-conversion.txt is only a hint to
Pywikibot manual and no longer a conversion guide.
- Update README-conversion.rst: take into account that
version 4 might be published
Change-Id: I0c791455dd615a9e8684ca242711d27631a1bfc7
---
M CONTENT.rst
M README-conversion.rst
2 files changed, 2 insertions(+), 3 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/CONTENT.rst b/CONTENT.rst
index a5aa3bb..8ba2abb 100644
--- a/CONTENT.rst
+++ b/CONTENT.rst
@@ -26,8 +26,7 @@
+---------------------------+------------------------------------------------------+
| README.rst | Short info string used by Pywikibot Nightlies |
+---------------------------+------------------------------------------------------+
- | README-conversion.txt | Guide to converting bot scripts from version 1 |
- | | of the Pywikibot framework to current version |
+ | README-conversion.txt | Compat to Core branch conversion hints |
+---------------------------+------------------------------------------------------+
| requirements.txt | General PIP requirements file |
+---------------------------+------------------------------------------------------+
diff --git a/README-conversion.rst b/README-conversion.rst
index 25c04a0..726686e 100644
--- a/README-conversion.rst
+++ b/README-conversion.rst
@@ -2,7 +2,7 @@
=========================
There is a guide to converting bot scripts from version 1 of the
-Pywikibot framework called trunk or compat to the new version 3,
+Pywikibot framework called trunk or compat to the new version 3+,
so called rewrite or core. Refer `compat-to-core conversion Manual
<https://www.mediawiki.org/wiki/Manual:Pywikibot/compat-to-core_conversion>`_
for further instructions.
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/608420
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I0c791455dd615a9e8684ca242711d27631a1bfc7
Gerrit-Change-Number: 608420
Gerrit-PatchSet: 1
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz>
Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/609280 )
Change subject: [IMPR] Show additional hints with ValueErrors in _WbDataPage._validate
......................................................................
[IMPR] Show additional hints with ValueErrors in _WbDataPage._validate
Bug: T257013
Change-Id: Ic62af7f29d20f235201e1cbaecaf0ce5090dc612
---
M pywikibot/__init__.py
1 file changed, 4 insertions(+), 2 deletions(-)
Approvals:
JJMC89: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/__init__.py b/pywikibot/__init__.py
index 9dde2ad..1e08813 100644
--- a/pywikibot/__init__.py
+++ b/pywikibot/__init__.py
@@ -982,11 +982,13 @@
@type site: str
"""
if not isinstance(page, Page):
- raise ValueError('Page must be a pywikibot.Page object.')
+ raise ValueError(
+ 'Page {} must be a pywikibot.Page object not a {}.'
+ .format(page, type(page)))
# validate page exists
if not page.exists():
- raise ValueError('Page must exist.')
+ raise ValueError('Page {} must exist.'.format(page))
# validate page is on the right site, and that site supports the type
if not data_site:
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/609280
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: Ic62af7f29d20f235201e1cbaecaf0ce5090dc612
Gerrit-Change-Number: 609280
Gerrit-PatchSet: 2
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: ChongDae <cdpark(a)gmail.com>
Gerrit-Reviewer: JJMC89 <JJMC89.Wikimedia(a)gmail.com>
Gerrit-Reviewer: Lokal Profil <andre.costa(a)wikimedia.se>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/608690 )
Change subject: [tests] Remove failing test_ASCII_compatible test
......................................................................
[tests] Remove failing test_ASCII_compatible test
- remove failing page_tests.TestPageRepr.test_ASCII_compatible
because Python 2 has been dropped and this issue will not be
solved. No need to test this known bug anymore
Bug: T130919
Bug: T66958
Change-Id: I94e168d07b3e1fb05e2d2a09d6a72949ffe22939
---
M tests/page_tests.py
1 file changed, 0 insertions(+), 8 deletions(-)
Approvals:
DannyS712: Looks good to me, but someone else must approve
Matěj Suchánek: Looks good to me, approved
jenkins-bot: Verified
diff --git a/tests/page_tests.py b/tests/page_tests.py
index 09eb93c..8a3ccbb 100644
--- a/tests/page_tests.py
+++ b/tests/page_tests.py
@@ -753,14 +753,6 @@
self.assertEqual('%r' % self.page, "Page('Ō')")
self.assertEqual('{0!r}'.format(self.page), "Page('Ō')")
- @unittest.skipIf(not PY2, 'Python 2 specific test')
- @unittest.expectedFailure
- def test_ASCII_compatible(self):
- """Test that repr returns ASCII compatible bytes in Python 2."""
- page = pywikibot.Page(self.site, 'ä')
- # Bug T95809, the repr in Python 2 should be decodable as ASCII
- repr(page).decode('ascii')
-
class TestPageReprASCII(TestPageBaseUnicode):
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/608690
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I94e168d07b3e1fb05e2d2a09d6a72949ffe22939
Gerrit-Change-Number: 608690
Gerrit-PatchSet: 1
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: DannyS712 <DannyS712.enwiki(a)gmail.com>
Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz>
Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged