Xqt has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/789085 )
Change subject: [doc] update ROADMAP.rst
......................................................................
[doc] update ROADMAP.rst
Change-Id: I16c002047347ba0f315067fc95665db3076d8f52
---
M ROADMAP.rst
1 file changed, 6 insertions(+), 0 deletions(-)
Approvals:
jenkins-bot: Verified
Xqt: Looks good to me, approved
diff --git a/ROADMAP.rst b/ROADMAP.rst
index dfe85ae..422b6af 100644
--- a/ROADMAP.rst
+++ b/ROADMAP.rst
@@ -1,6 +1,12 @@
Current release 7.3.0
^^^^^^^^^^^^^^^^^^^^^
+* Preserve more workers than families are handled for preload_sites.py
+* Close logging handlers before deleting them (:phab:`T91375`, :phab:`T286127`)
+* Clear _sites cache if called with pwb wrapper (:phab:`T225594`)
+* Enable short creation of a site if family name is equal to site code
+* Use `exc_info=True` with pywikibot.exception() by default (:phab:`T306762`)
+* Make IndexPage more robust when getting links in Page ns (:phab:`T307280`)
* Do not print log header twice in log files (:phab:`T264235`)
* Do not delegate logging output to the root logger (:phab:`T281643`)
* Add `get_charset_from_content_type` to extract the charset from the content-type response header
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/789085
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I16c002047347ba0f315067fc95665db3076d8f52
Gerrit-Change-Number: 789085
Gerrit-PatchSet: 1
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/788701 )
Change subject: [bugfix] Preserve more workers than families are handled
......................................................................
[bugfix] Preserve more workers than families are handled
If worker is less than families are handled the script blocks because
no future can be added in preload_family function because all workers
are busy already.
Change-Id: Id6a6819271dcf51f6bb13327dd65e62303f6b668
---
M scripts/maintenance/preload_sites.py
1 file changed, 20 insertions(+), 5 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/maintenance/preload_sites.py b/scripts/maintenance/preload_sites.py
index 7e5b051..3c0afcd 100755
--- a/scripts/maintenance/preload_sites.py
+++ b/scripts/maintenance/preload_sites.py
@@ -16,18 +16,21 @@
"""
#
-# (C) Pywikibot team, 2022
+# (C) Pywikibot team, 2021-2022
#
# Distributed under the terms of the MIT license.
#
+import os
from concurrent.futures import ThreadPoolExecutor, wait
from datetime import datetime
+from typing import Optional, Union
import pywikibot
+from pywikibot.backports import List, Set
from pywikibot.family import Family
-# supported families by this script
+#: supported families by this script
families_list = [
'wikibooks',
'wikinews',
@@ -43,7 +46,7 @@
}
-def preload_family(family, executor):
+def preload_family(family: str, executor: ThreadPoolExecutor) -> None:
"""Preload all sites of a single family file."""
msg = 'Preloading sites of {} family{}'
pywikibot.output(msg.format(family, '...'))
@@ -64,9 +67,21 @@
pywikibot.output(msg.format(family, ' completed.'))
-def preload_families(families, worker):
- """Preload all sites of all given family files."""
+def preload_families(families: Union[List[str], Set[str]],
+ worker: Optional[int]) -> None:
+ """Preload all sites of all given family files.
+
+ .. versionchanged:: 7.3
+ Default of worker is calculated like for Python 3.8 but preserves
+ at least one worker more than families_list elements to ensure a
+ worker can be added in :func:`preload_family`.
+ """
start = datetime.now()
+ if worker is None:
+ # Python 3.8 default
+ worker = min(32, (os.cpu_count() or 1) + 4)
+ # allow to add futures in preload_family
+ worker = max(len(families) + 1, worker)
with ThreadPoolExecutor(worker) as executor:
futures = {executor.submit(preload_family, family, executor)
for family in families}
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/788701
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: Id6a6819271dcf51f6bb13327dd65e62303f6b668
Gerrit-Change-Number: 788701
Gerrit-PatchSet: 6
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
Xqt has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/788710 )
Change subject: [doc] Note regex usage of referencesSections
......................................................................
[doc] Note regex usage of referencesSections
Bug: T179653
Change-Id: I12e04eded02f4f67b68b12036a9ee726e69a1a82
---
M scripts/noreferences.py
1 file changed, 3 insertions(+), 3 deletions(-)
Approvals:
jenkins-bot: Verified
Xqt: Looks good to me, approved
diff --git a/scripts/noreferences.py b/scripts/noreferences.py
index fa0a37e..35aff3d 100755
--- a/scripts/noreferences.py
+++ b/scripts/noreferences.py
@@ -233,9 +233,9 @@
}
# Titles of sections where a reference tag would fit into.
-# The first title should be the preferred one: It's the one that
-# will be used when a new section has to be created.
-# Except for the first, others are tested as regexes.
+# The first title should be the preferred one: It's the one that will be
+# used when a new section has to be created. Section titles can be regex
+# patterns except of the first.
referencesSections = {
'wikipedia': {
'ar': [ # not sure about which ones are preferred.
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/788710
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I12e04eded02f4f67b68b12036a9ee726e69a1a82
Gerrit-Change-Number: 788710
Gerrit-PatchSet: 1
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged