jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/971606 )
Change subject: [IMPR] use process_cpu_count to select the default number of worker
......................................................................
[IMPR] use process_cpu_count to select the default number of worker
os.cpu_count() returns the total number of CPUs that the current
machinevhas. On Unix, sched_setaffinity() can reduce the number of CPU
"usable" by a process. With Python 3.13 os.process_cpu_count() can be
used to get the number of logical CPUs usable by the calling thread of
the current process. This patch uses os.process_cpu_count() like in
Python 3.13 to select the default number of worker for
concurrent.futures.
Change-Id: Id66f9bb32cec80206434f39277db7b4c62867ec3
---
M pywikibot/scripts/preload_sites.py
1 file changed, 28 insertions(+), 5 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/scripts/preload_sites.py b/pywikibot/scripts/preload_sites.py
index afda243..4d4799e 100755
--- a/pywikibot/scripts/preload_sites.py
+++ b/pywikibot/scripts/preload_sites.py
@@ -22,15 +22,19 @@
#
# Distributed under the terms of the MIT license.
#
-import os
from concurrent.futures import ThreadPoolExecutor, wait
from datetime import datetime
from typing import Optional, Union
import pywikibot
-from pywikibot.backports import List, Set, removeprefix
+from pywikibot.backports import Dict, List, Set, removeprefix
from pywikibot.family import Family
+try: # Python 3.13
+ from os import process_cpu_count # type: ignore[attr-defined]
+except ImportError:
+ from os import cpu_count as process_cpu_count
+
#: supported families by this script
families_list = [
@@ -44,7 +48,9 @@
'wiktionary',
]
-exceptions = {
+# Ignore sites from preloading
+# example: {'wikiversity': ['beta'], }
+exceptions: Dict[str, List[str]] = {
}
@@ -80,8 +86,8 @@
"""
start = datetime.now()
if worker is None:
- # Python 3.8 default
- worker = min(32, (os.cpu_count() or 1) + 4)
+ # Python 3.13 default
+ worker = min(32, (process_cpu_count() or 1) + 4)
# to allow adding futures in preload_family the workers must be one
# more than families are handled
worker = max(len(families) * 2, worker)
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/971606
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: Id66f9bb32cec80206434f39277db7b4c62867ec3
Gerrit-Change-Number: 971606
Gerrit-PatchSet: 4
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/971568 )
Change subject: [IMPR] Filter does not have to be subclassed
......................................................................
[IMPR] Filter does not have to be subclassed
Any instance can be passed which has a filter method with the same
semantics:
https://docs.python.org/3/library/logging.html?highlight=logging%20filter#l…
This also solves an deepsource PY-A6006 issue.
Change-Id: Ic2d3c345725ed33809c484be709f0c3d8b282149
---
M pywikibot/userinterfaces/terminal_interface_base.py
1 file changed, 18 insertions(+), 3 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/userinterfaces/terminal_interface_base.py b/pywikibot/userinterfaces/terminal_interface_base.py
index 9ba7545..b58e9c0 100644
--- a/pywikibot/userinterfaces/terminal_interface_base.py
+++ b/pywikibot/userinterfaces/terminal_interface_base.py
@@ -608,12 +608,12 @@
self.UI.output(msg, targetStream=self.stream)
-class MaxLevelFilter(logging.Filter):
+class MaxLevelFilter():
"""Filter that only passes records at or below a specific level.
- (setting handler level only passes records at or *above* a specified level,
- so this provides the opposite functionality)
+ .. note:: setting handler level only passes records at or *above* a
+ specified level, so this provides the opposite functionality.
"""
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/971568
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: Ic2d3c345725ed33809c484be709f0c3d8b282149
Gerrit-Change-Number: 971568
Gerrit-PatchSet: 1
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/970820 )
Change subject: [FEAT] Add a filter to check if a page is redirect
......................................................................
[FEAT] Add a filter to check if a page is redirect
Added a -redirect filter to assert if page is a redirect. Likewise,
the filter can be negated by using -redirect:false.
This is a missing feature present in compat, which implemented -redirectonly,
but this is more useful because it can be negated, and the name is different
to make it clearer.
Bug: T261549
Change-Id: If1a2c9bb420d9b5af69371537c7ca724096d8a23
---
M pywikibot/pagegenerators/__init__.py
M pywikibot/pagegenerators/_factory.py
2 files changed, 39 insertions(+), 1 deletion(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/pagegenerators/__init__.py b/pywikibot/pagegenerators/__init__.py
index d3fe693..a18aeb6 100644
--- a/pywikibot/pagegenerators/__init__.py
+++ b/pywikibot/pagegenerators/__init__.py
@@ -500,6 +500,9 @@
Valid values are in range 0-4.
Multiple values can be comma-separated.
+-redirect Filter pages based on whether they are redirects. To return
+ only pages that are not redirects, use -redirect:false
+
-subpage -subpage:n filters pages to only those that have depth n
i.e. a depth of 0 filters out all pages that are subpages,
and a depth of 1 filters out all pages that are subpages of
diff --git a/pywikibot/pagegenerators/_factory.py b/pywikibot/pagegenerators/_factory.py
index 4a6f5c4..4a19361 100644
--- a/pywikibot/pagegenerators/_factory.py
+++ b/pywikibot/pagegenerators/_factory.py
@@ -33,6 +33,7 @@
ItemClaimFilterPageGenerator,
NamespaceFilterPageGenerator,
QualityFilterPageGenerator,
+ RedirectFilterPageGenerator,
RegexBodyFilterPageGenerator,
RegexFilterPageGenerator,
SubpageFilterGenerator,
@@ -54,6 +55,7 @@
WikibaseSearchItemPageGenerator,
WikidataSPARQLPageGenerator,
)
+from pywikibot.tools import strtobool
from pywikibot.tools.collections import DequeGenerator
from pywikibot.tools.itertools import (
filter_unique,
@@ -113,6 +115,7 @@
self.catfilter_list: List['pywikibot.Category'] = []
self.intersect = False
self.subpage_max_depth: Optional[int] = None
+ self.redirectfilter: Optional[bool] = None
self._site = site
self._positional_arg_name = positional_arg_name
self._sparql: Optional[str] = None
@@ -241,7 +244,8 @@
self.claimfilter_list,
self.catfilter_list,
self.qualityfilter_list,
- self.subpage_max_depth is not None)):
+ self.subpage_max_depth is not None,
+ self.redirectfilter is not None)):
pywikibot.warning('filter(s) specified but no generators.')
return None
@@ -264,6 +268,12 @@
dupfiltergen = SubpageFilterGenerator(
dupfiltergen, self.subpage_max_depth)
+ if self.redirectfilter is not None:
+ # Generator expects second parameter true to exclude redirects, but
+ # our logic is true to assert it is a redirect, false when it isn't
+ dupfiltergen = RedirectFilterPageGenerator(
+ dupfiltergen, not self.redirectfilter)
+
if self.claimfilter_list:
for claim in self.claimfilter_list:
dupfiltergen = ItemClaimFilterPageGenerator(dupfiltergen,
@@ -902,6 +912,14 @@
f'Invalid -logevents parameter "{params[0]}"')
return self._parse_log_events(*params)
+ def _handle_redirect(self, value: str) -> HANDLER_RETURN_TYPE:
+ """Handle `-redirect` argument."""
+ if not value:
+ # True by default
+ value = 'true'
+ self.redirectfilter = strtobool(value)
+ return True
+
def handle_args(self, args: Iterable[str]) -> List[str]:
"""Handle command line arguments and return the rest as a list.
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/970820
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: If1a2c9bb420d9b5af69371537c7ca724096d8a23
Gerrit-Change-Number: 970820
Gerrit-PatchSet: 2
Gerrit-Owner: Martineznovo <martineznovo(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/967304 )
Change subject: remove check for purge right from APISite.purgepages
......................................................................
remove check for purge right from APISite.purgepages
The right has not been checked by MediaWiki since 2016.
Bug: T349348
Change-Id: I798f3d0e847acc2fc8be28322f0cc02894dc0d43
---
M pywikibot/site/_apisite.py
1 file changed, 12 insertions(+), 1 deletion(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/site/_apisite.py b/pywikibot/site/_apisite.py
index ba3e8ca..ba185c0 100644
--- a/pywikibot/site/_apisite.py
+++ b/pywikibot/site/_apisite.py
@@ -2736,7 +2736,6 @@
unwatch_s = 'unwatched' if unwatch else 'watched'
return all(unwatch_s in r for r in results['watch'])
- @need_right('purge')
def purgepages(
self,
pages: List['pywikibot.page.BasePage'],
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/967304
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I798f3d0e847acc2fc8be28322f0cc02894dc0d43
Gerrit-Change-Number: 967304
Gerrit-PatchSet: 2
Gerrit-Owner: JJMC89 <JJMC89.Wikimedia(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged