jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/791498 )
Change subject: [bugfix] fix suggest_help usage
......................................................................
[bugfix] fix suggest_help usage
Change-Id: I7d2f4ef3204de8c42197471cee6f13daa13e63d6
Signed-off-by: Xqt <info(a)gno.de>
---
M scripts/basic.py
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/basic.py b/scripts/basic.py
index 27fb367..30d5433 100755
--- a/scripts/basic.py
+++ b/scripts/basic.py
@@ -164,7 +164,7 @@
gen = gen_factory.getCombinedGenerator(preload=True)
# check if further help is needed
- if not pywikibot.bot.suggest_help(missing_generator=gen):
+ if not pywikibot.bot.suggest_help(missing_generator=not gen):
# pass generator and private options to the bot
bot = BasicBot(generator=gen, **options)
bot.run() # guess what it does
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/791498
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I7d2f4ef3204de8c42197471cee6f13daa13e63d6
Gerrit-Change-Number: 791498
Gerrit-PatchSet: 2
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
Xqt has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/791563 )
Change subject: [doc] Additional documentation for basic script class usage
......................................................................
[doc] Additional documentation for basic script class usage
Change-Id: I23f02d6dc4e2c12f70a9678cf583266d9e885d7d
---
M docs/library_usage.rst
M scripts/basic.py
2 files changed, 32 insertions(+), 18 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/docs/library_usage.rst b/docs/library_usage.rst
index b1c0846..6aae756 100644
--- a/docs/library_usage.rst
+++ b/docs/library_usage.rst
@@ -47,29 +47,43 @@
**Explanations:**
-:1-3: Import necessary framework code parts
-:5: The bot is derived from ExistingPageBot. All pages from generator
- which does not exists are skipped.
+:1-3: Import necessary framework code parts:
+ :mod:`pywikibot`, :mod:`pywikibot.pagegenerators`,
+ :class:`pywikibot.bot.ExistingPageBot`.
+:5: The bot is derived from
+ :class:`ExistingPageBot<pywikibot.bot.ExistingPageBot>`.
+ All pages from generator which does not exists are skipped.
:7: Every Bot has an *always* option which autoconfirms any changes if
set to True. To expand all available options of a bot and set the
- default values of them, use `update_options` attribute or update
- `available_options` like it is shown in BasicBot below.
-:12: All changes for each page are made in this method.
-:14: `currentpage` is the current :py:obj:`pywikibot.Page` object from
- generator.
+ default values of them, use
+ :attr:`update_options<pywikibot.bot.BaseBot.update_options>`
+ attribute or update
+ :attr:`available_options<pywikibot.bot.BaseBot.available_options>`
+ like it is shown in :ref:`BasicBot<Basic script>` below.
+:12: All changes for each page are made in this
+ :meth:`treat_page<pywikibot.bot.CurrentPageBot.treat_page>` method.
+:14: :attr:`current_page<pywikibot.bot.BaseBot.current_page>` is the current
+ :class:`pywikibot.Page` object from
+ :attr:`generator<pywikibot.bot.BaseBot.generator>`.
:15: All bot options which are passed to the bot class when
- instantiating it are accessable via opt attribute. `opt.always`,
- `opt.text` and `opt.summary` are all available options for this
+ instantiating it are accessable via
+ :class:`opt attribute<pywikibot.bot.OptionHandler>`. ``opt.always``,
+ ``opt.text`` and ``opt.summary`` are all available options for this
bot class.
-:16: Save the changes to the live wiki.
+:16: Save the changes to the live wiki with
+ :meth:`put_current<pywikibot.bot.CurrentPageBot.put_current>`.
:18: Parse command line options inside this function.
:20: A dict which holds all options for the bot.
:21: :py:obj:`pywikibot.pagegenerators.GeneratorFactory` supports
generators and filter options.
-:23: Pywikibot provides global options like site specification or
- a simulate switch to prevent live wiki changes.
-:24: Generators and filter options of :py:obj:`pywikibot.pagegenerators`
- are parsed here.
+:23: Pywikibot provides :ref:`global options<Global Options>` like site
+ specification or a *simulate* switch to prevent live wiki changes.
+ These options are parsed by :func:`pywikibot.handle_args`.
+:24: :ref:`Generators<Generator Options>` and
+ :ref:`filter options<Filter Options>` of
+ :mod:`pywikibot.pagegenerators` are parsed here using
+ :meth:`GeneratorFactory.handle_args()
+ <pywikibot.pagegenerators.GeneratorFactory.handle_args>`.
:25: Local options which are are available for the current bot are
parsed in this loop.
:29: Create the bot passing keyword only parameters and run it.
diff --git a/scripts/basic.py b/scripts/basic.py
index 1bcd7ba..27fb367 100755
--- a/scripts/basic.py
+++ b/scripts/basic.py
@@ -162,12 +162,12 @@
# The preloading option is responsible for downloading multiple
# pages from the wiki simultaneously.
gen = gen_factory.getCombinedGenerator(preload=True)
- if gen:
+
+ # check if further help is needed
+ if not pywikibot.bot.suggest_help(missing_generator=gen):
# pass generator and private options to the bot
bot = BasicBot(generator=gen, **options)
bot.run() # guess what it does
- else:
- pywikibot.bot.suggest_help(missing_generator=True)
if __name__ == '__main__':
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/791563
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I23f02d6dc4e2c12f70a9678cf583266d9e885d7d
Gerrit-Change-Number: 791563
Gerrit-PatchSet: 2
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/511334 )
Change subject: [IMPR] Weblinkchecker: throttle connections to the same host
......................................................................
[IMPR] Weblinkchecker: throttle connections to the same host
Bug: T152350
Change-Id: I894582d115013f5bf09e42bff6023c25bee6f02b
---
M scripts/weblinkchecker.py
1 file changed, 27 insertions(+), 2 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/weblinkchecker.py b/scripts/weblinkchecker.py
index 4c49ee9..7730707 100755
--- a/scripts/weblinkchecker.py
+++ b/scripts/weblinkchecker.py
@@ -114,6 +114,7 @@
import re
import threading
import time
+import urllib.parse as urlparse
from contextlib import suppress
from functools import partial
from http import HTTPStatus
@@ -122,6 +123,7 @@
import pywikibot
from pywikibot import comms, config, i18n, pagegenerators, textlib
+from pywikibot.backports import Dict, removeprefix
from pywikibot.bot import ExistingPageBot, SingleSiteBot, suggest_help
from pywikibot.exceptions import (
IsRedirectPageError,
@@ -289,6 +291,10 @@
After checking the page, it will die.
"""
+ #: Collecting start time of a thread for any host
+ hosts = {} # type: Dict[str, float]
+ lock = threading.Lock()
+
def __init__(self, page, url, history, http_ignores, day) -> None:
"""Initializer."""
self.page = page
@@ -307,12 +313,28 @@
self._use_fake_user_agent = config.fake_user_agent_default.get(
'weblinkchecker', False)
self.day = day
+ super().__init__()
- name = '{} - {}'.format(page.title(), url.encode('utf-8', 'replace'))
- super().__init__(name=name)
+ @classmethod
+ def get_delay(cls, name: str) -> float:
+ """Determine delay from class attribute.
+
+ Store the last call for a given hostname with an offset of
+ 6 seconds to ensure there are no more than 10 calls per minute
+ for the same host. Calculate the delay to start the run.
+
+ :param name: The key for the hosts class attribute
+ :return: The calulated delay to start the run
+ """
+ now = time.monotonic()
+ with cls.lock:
+ timestamp = cls.hosts.get(name, now)
+ cls.hosts[name] = max(now, timestamp) + 6
+ return max(0, timestamp - now)
def run(self):
"""Run the bot."""
+ time.sleep(self.get_delay(self.name))
try:
header = self.header
r = comms.http.fetch(
@@ -599,6 +621,9 @@
self.http_ignores, self.day)
# thread dies when program terminates
thread.daemon = True
+ # use hostname as thread.name
+ thread.name = removeprefix(
+ urlparse.urlparse(url).hostname, 'www.')
self.threads.append(thread)
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/511334
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I894582d115013f5bf09e42bff6023c25bee6f02b
Gerrit-Change-Number: 511334
Gerrit-PatchSet: 6
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com>
Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com>
Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz>
Gerrit-Reviewer: Huji <huji.huji(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: Zhuyifei1999 <zhuyifei1999(a)gmail.com>
Gerrit-Reviewer: jenkins-bot
Gerrit-CC: Rubin <rubin.happy(a)gmail.com>
Gerrit-MessageType: merged
Xqt has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/791483 )
Change subject: [doc] fix spelling in cfd script
......................................................................
[doc] fix spelling in cfd script
Change-Id: I1a44a8034e083e895d21732ec41396fa7924d77d
---
M docs/scripts/outdated.rst
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Xqt: Verified; Looks good to me, approved
diff --git a/docs/scripts/outdated.rst b/docs/scripts/outdated.rst
index eb82e4d..7961562 100644
--- a/docs/scripts/outdated.rst
+++ b/docs/scripts/outdated.rst
@@ -32,7 +32,7 @@
cfd script
----------
-**This script processes the Categories for discussion working pag**
+**This script processes the Categories for discussion working page**
It parses out the actions that need to be taken as a result of CFD
discussions (as posted to the working page by an administrator) and
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/791483
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I1a44a8034e083e895d21732ec41396fa7924d77d
Gerrit-Change-Number: 791483
Gerrit-PatchSet: 1
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-MessageType: merged