jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/908987 )
Change subject: [8.2] Prepare next release 8.2
......................................................................
[8.2] Prepare next release 8.2
Change-Id: I30f62e54da08a44f4b8b2a2f30cda675d6b08b78
---
M .appveyor.yml
M ROADMAP.rst
M scripts/CHANGELOG.rst
M tests/utils.py
M pywikibot/__metadata__.py
M HISTORY.rst
M pywikibot/site/_apisite.py
M pywikibot/data/api/_requests.py
M tests/pagegenerators_tests.py
9 files changed, 43 insertions(+), 28 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/.appveyor.yml b/.appveyor.yml
index ca689a2..8130c22 100644
--- a/.appveyor.yml
+++ b/.appveyor.yml
@@ -1,7 +1,7 @@
image: Visual Studio 2022
clone_depth: 50
skip_tags: true
-version: 8.1.{build}
+version: 8.2.{build}
environment:
PYWIKIBOT_DIR: "%appdata%\\Pywikibot"
diff --git a/HISTORY.rst b/HISTORY.rst
index 560affb..d68ff25 100644
--- a/HISTORY.rst
+++ b/HISTORY.rst
@@ -1,6 +1,29 @@
Release history
===============
+8.1.0
+-----
+*16 April 2023*
+
+* :mod:`generate_family_file<pywikibot.scripts.generate_family_file>` script was improved (:phab:`T334775`)
+* A ``quiet`` parameter was added to :meth:`APISite.preloadpages()
+ <pywikibot.site._generators.GeneratorsMixin.preloadpages>` which is True by default
+* Fix getting HTTPStatus enum in site_detect check_response (:phab:`T334728`)
+* Do not show a logging in message if password is entered (:phab:`T178061`)
+* Enable preleading ``Bot:`` prefix with twtranslate messages (:phab:`T161459`)
+* Disable command.log if -nolog option is given (:phab:`T334381`)
+* Guess the last needed token key if the token is not found (:phab:`T334288`)
+* Show parameters with APIError (:phab:`T333957`)
+* Raise :exc:`exceptions.NoSiteLinkError` instead of :exc:`exceptions.NoPageError` when sitelink
+ is missing in :meth:`ItemPage.getSitelink()<pywikibot.ItemPage.getSitelink>` (:phab:`T332341`)
+* :exc:`exceptions.ClientError` was added
+* Raise :exc:`exceptions.NoPageError` when deleting a missing Page (:phab:`T332924`)
+* ``text`` parameter of :class:`proofreadpage.PagesTagParser` has a default value
+* L10N updates
+* Ignore talk pages with :meth:`APISite.watched_pages()<pywikibot.site._generators.GeneratorsMixin.watched_pages>` (:phab:`T330806`)
+* Load page info when creating a page if not updated previously (:phab:`T330980`)
+* Improve flush exception logging
+
8.0.4
-----
diff --git a/ROADMAP.rst b/ROADMAP.rst
index c2b94e8..1adb1ab 100644
--- a/ROADMAP.rst
+++ b/ROADMAP.rst
@@ -1,24 +1,7 @@
Current release
---------------
-* :mod:`generate_family_file<pywikibot.scripts.generate_family_file>` script was improved (:phab:`T334775`)
-* A ``quiet`` parameter was added to :meth:`APISite.preloadpages()
- <pywikibot.site._generators.GeneratorsMixin.preloadpages>` which is True by default
-* Fix getting HTTPStatus enum in site_detect check_response (:phab:`T334728`)
-* Do not show a logging in message if password is entered (:phab:`T178061`)
-* Enable preleading ``Bot:`` prefix with twtranslate messages (:phab:`T161459`)
-* Disable command.log if -nolog option is given (:phab:`T334381`)
-* Guess the last needed token key if the token is not found (:phab:`T334288`)
-* Show parameters with APIError (:phab:`T333957`)
-* Raise :exc:`exceptions.NoSiteLinkError` instead of :exc:`exceptions.NoPageError` when sitelink
- is missing in :meth:`ItemPage.getSitelink()<pywikibot.ItemPage.getSitelink>` (:phab:`T332341`)
-* :exc:`exceptions.ClientError` was added
-* Raise :exc:`exceptions.NoPageError` when deleting a missing Page (:phab:`T332924`)
-* ``text`` parameter of :class:`proofreadpage.PagesTagParser` has a default value
-* L10N updates
-* Ignore talk pages with :meth:`APISite.watched_pages()<pywikibot.site._generators.GeneratorsMixin.watched_pages>` (:phab:`T330806`)
-* Load page info when creating a page if not updated previously (:phab:`T330980`)
-* Improve flush exception logging
+* *(no changes yet)*
Deprecations
diff --git a/pywikibot/__metadata__.py b/pywikibot/__metadata__.py
index 8e50130..a816904 100644
--- a/pywikibot/__metadata__.py
+++ b/pywikibot/__metadata__.py
@@ -11,7 +11,7 @@
__name__ = 'pywikibot'
-__version__ = '8.1.0'
+__version__ = '8.2.0.dev0'
__description__ = 'Python MediaWiki Bot Framework'
__maintainer__ = 'The Pywikibot team'
__maintainer_email__ = 'pywikibot(a)lists.wikimedia.org'
diff --git a/pywikibot/data/api/_requests.py b/pywikibot/data/api/_requests.py
index 0377577..892c5a2 100644
--- a/pywikibot/data/api/_requests.py
+++ b/pywikibot/data/api/_requests.py
@@ -940,7 +940,7 @@
def submit(self) -> dict:
"""Submit a query and parse the response.
- .. versionchanged:: 8.1
+ .. versionchanged:: 8.0.4
in addition to *readapidenied* also try to login when API
response is *notloggedin*.
diff --git a/pywikibot/site/_apisite.py b/pywikibot/site/_apisite.py
index 673d619..a9decae 100644
--- a/pywikibot/site/_apisite.py
+++ b/pywikibot/site/_apisite.py
@@ -332,9 +332,9 @@
) -> None:
"""Log the user in if not already logged in.
- .. versionchanged:: 8.0
- lazy load cookies when logging in. This was dropped in 8.1
- .. versionchanged:: 8.1
+ .. versionchanged:: 8.0.0
+ lazy load cookies when logging in. This was dropped in 8.0.4
+ .. versionchanged:: 8.0.4
the *cookie_only* parameter was added and cookies are loaded
whenever the site is initialized.
diff --git a/scripts/CHANGELOG.rst b/scripts/CHANGELOG.rst
index c737b9e..4478ca1 100644
--- a/scripts/CHANGELOG.rst
+++ b/scripts/CHANGELOG.rst
@@ -22,7 +22,7 @@
clean_sandbox
~~~~~~~~~~~~~
-* L10N es-wikis
+* L10N for es-wikis
8.0.1
-----
diff --git a/tests/pagegenerators_tests.py b/tests/pagegenerators_tests.py
index 73cb357..24470be 100755
--- a/tests/pagegenerators_tests.py
+++ b/tests/pagegenerators_tests.py
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
"""Test pagegenerators module."""
#
-# (C) Pywikibot team, 2009-2022
+# (C) Pywikibot team, 2009-2023
#
# Distributed under the terms of the MIT license.
import calendar
diff --git a/tests/utils.py b/tests/utils.py
index 38c4105..5a49e16 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -1,6 +1,6 @@
"""Test utilities."""
#
-# (C) Pywikibot team, 2013-2022
+# (C) Pywikibot team, 2013-2023
#
# Distributed under the terms of the MIT license.
#
@@ -403,7 +403,7 @@
def login(self, *args, cookie_only=False, **kwargs):
"""Overwrite login which is called when a site is initialized.
- .. versionadded: 8.1
+ .. versionadded: 8.0.4
"""
if cookie_only:
return
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/908987
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I30f62e54da08a44f4b8b2a2f30cda675d6b08b78
Gerrit-Change-Number: 908987
Gerrit-PatchSet: 1
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/908966 )
Change subject: [bugfix] Remove sites from FamilyFileGenerator.langs on error
......................................................................
[bugfix] Remove sites from FamilyFileGenerator.langs on error
- If loading a site fails, remove it from langs attribute of
FamilyFileGenerator
- [s]trict: add the ability to process sites with the same domain only
- with [e]dit sites can be given delimited by space, comma or both
- collect self.prefixes for tests after FamilyFileGenerator.getapis()
was processed
- use [s]trict option for Wikimedia site for within tests
- remove old implementation removing obsolete sites in tests
Bug: T334775
Bug: T334714
Change-Id: I5b049ead55e19f2a4c2ba522f34412e541737669
---
M tests/generate_family_file_tests.py
M pywikibot/scripts/generate_family_file.py
2 files changed, 79 insertions(+), 22 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/scripts/generate_family_file.py b/pywikibot/scripts/generate_family_file.py
index d5c00cc..8b0607c 100755
--- a/pywikibot/scripts/generate_family_file.py
+++ b/pywikibot/scripts/generate_family_file.py
@@ -12,7 +12,7 @@
<url>: an url from where the family settings are loaded
<name>: the family name without "_family.py" tail.
- <dointerwiki>: predefined answer (y|n) to add multiple site codes
+ <dointerwiki>: predefined answer (y|s|n) to add multiple site codes
<verify>: disable certificate validaton `(y|n)
Example::
@@ -25,14 +25,18 @@
.. versionchanged:: 7.0
moved to pywikibot.scripts folder; create family files in families
folder of your base directory instead of pywikibot/families.
+.. versionchanged:: 8.1
+ [s]trict can be given for <dointerwiki> parameter to ensure that
+ sites are from the given domain.
"""
#
-# (C) Pywikibot team, 2010-2022
+# (C) Pywikibot team, 2010-2023
#
# Distributed under the terms of the MIT license
#
import codecs
import os
+import re
import string
import sys
from contextlib import suppress
@@ -63,8 +67,10 @@
:param url: an url from where the family settings are loaded
:param name: the family name without "_family.py" tail.
:param dointerwiki: Predefined answer to add multiple site
- codes. Pass `Y` or `y` for yes `N` or `n` for no and
- `E` or `e` if you want to edit the collection of sites.
+ codes. Pass `Y` or `y` for yes, `S` or `s` for strict which
+ only includes site of the same domain (usually for Wikimedia
+ sites), `N` or `n` for no and `E` or `e` if you want to edit
+ the collection of sites.
:param verify: If a certificate verification failes, you may
pass `Y` or `y` to disable certificate validaton `N` or `n`
to keep it enabled.
@@ -149,7 +155,14 @@
self.writefile(verify)
def getlangs(self, w) -> None:
- """Determine site code of a family."""
+ """Determine site code of a family.
+
+ .. versionchanged:: 8.1
+ with [e]dit the interwiki list can be given delimited by
+ space or comma or both. With [s]trict only sites with the
+ same domain are collected. A [h]elp answer was added to show
+ more information about possible answers.
+ """
print('Determining other sites...', end='')
try:
self.langs = w.langs
@@ -169,21 +182,39 @@
code_len = len(self.langs)
if code_len > 1:
if self.dointerwiki is None:
- makeiw = input(
- '\nThere are {} sites available.'
- '\nDo you want to generate interwiki links? '
- 'This might take a long time. ([y]es/[N]o/[e]dit)'
- .format(code_len)).lower()
+ while True:
+ makeiw = input(
+ '\n'
+ f'There are {code_len} sites available.'
+ ' Do you want to generate interwiki links?\n'
+ 'This might take a long time. '
+ '([y]es, [s]trict, [N]o, [e]dit), [h]elp) ').lower()
+ if makeiw in ('y', 's', 'n', 'e', ''):
+ break
+ print(
+ '\n'
+ '[y]es: create interwiki links for all sites\n'
+ '[s]trict: yes, but for sites with same domain only\n'
+ '[N]o: no, use the current site only (default)\n'
+ '[e]dit: get a list delimited with space or comma\n'
+ '[h]elp: this help message'
+ )
else:
makeiw = self.dointerwiki
- if makeiw == 'n':
+ if makeiw in ('n', ''):
self.langs = [wiki for wiki in self.langs
if wiki['url'] == w.iwpath]
+ elif makeiw == 's':
+ domain = '.'.join(urlparse(w.server).hostname.split('.')[1:])
+ self.langs = [wiki for wiki in self.langs
+ if domain in wiki['url']]
+
elif makeiw == 'e':
for wiki in self.langs:
print(wiki['prefix'], wiki['url'])
- do_langs = input('Which sites do you want: ')
+ do_langs = re.split(' *,| +',
+ input('Which sites do you want: '))
self.langs = [wiki for wiki in self.langs
if wiki['prefix'] in do_langs
or wiki['url'] == w.iwpath]
@@ -196,7 +227,8 @@
def getapis(self) -> None:
"""Load other site pages."""
- print('Loading wikis... ')
+ print(f'Loading {len(self.langs)} wikis... ')
+ remove = []
for lang in self.langs:
key = lang['prefix']
print(f' * {key}... ', end='')
@@ -206,17 +238,21 @@
print('downloaded')
except Exception as e: # pragma: no cover
print(e)
+ remove.append(lang)
else:
print('in cache')
+ for lang in remove:
+ self.langs.remove(lang)
+
def writefile(self, verify) -> None:
"""Write the family file."""
fn = os.path.join(self.base_dir, 'families',
f'{self.name}_family.py')
print(f'Writing {fn}... ')
- if os.path.exists(fn) and input('{} already exists. Overwrite? (y/n)'
- .format(fn)).lower() == 'n':
+ if os.path.exists(fn) and input(
+ f'{fn} already exists. Overwrite? (y/n) ').lower() == 'n':
print('Terminating.')
sys.exit(1)
diff --git a/tests/generate_family_file_tests.py b/tests/generate_family_file_tests.py
index f219ac6..e97f884 100755
--- a/tests/generate_family_file_tests.py
+++ b/tests/generate_family_file_tests.py
@@ -11,6 +11,7 @@
from urllib.parse import urlparse
from pywikibot import Site
+from pywikibot.family import WikimediaFamily
from pywikibot.scripts import generate_family_file
from tests.aspects import DefaultSiteTestCase
from tests.utils import skipping
@@ -29,15 +30,15 @@
self.langs.append(wiki)
break
- self.prefixes = [item['prefix'] for item in self.langs]
super().getapis()
+ self.prefixes = [item['prefix'] for item in self.langs]
self.langs = save
def writefile(self, verify):
"""Pass writing."""
-class TestGenerateFamilyFiles(DefaultSiteTestCase):
+class TestGenerateFamilyFile(DefaultSiteTestCase):
"""Test generate_family_file functionality."""
@@ -55,15 +56,17 @@
def setUp(self):
"""Set up tests."""
super().setUp()
+ answer = 's' if isinstance(self.site.family, WikimediaFamily) else 'y'
self.generator_instance = FamilyTestGenerator(
- url=self.site.base_url(''), name=self.familyname, dointerwiki='y')
+ url=self.site.base_url(''), name=self.familyname,
+ dointerwiki=answer)
def test_initial_attributes(self):
"""Test initial FamilyFileGenerator attributes."""
self.assertEqual(self.generator_instance.base_url,
self.site.base_url(''))
self.assertEqual(self.generator_instance.name, self.familyname)
- self.assertEqual(self.generator_instance.dointerwiki, 'y')
+ self.assertIn(self.generator_instance.dointerwiki, ['s', 'y'])
self.assertIsInstance(self.generator_instance.wikis, dict)
self.assertIsInstance(self.generator_instance.langs, list)
@@ -80,9 +83,7 @@
with self.subTest(test='Test element counts'):
if self.site.lang not in gen.prefixes:
gen.prefixes.append(self.site.lang)
- obsolete = self.site.family.interwiki_removals
- self.assertCountEqual(set(gen.prefixes) - obsolete,
- set(gen.wikis) - obsolete)
+ self.assertCountEqual(gen.prefixes, gen.wikis)
# test creating Site from url
# only test Sites for downloaded wikis (T241413)
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/908966
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I5b049ead55e19f2a4c2ba522f34412e541737669
Gerrit-Change-Number: 908966
Gerrit-PatchSet: 3
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/908943 )
Change subject: [tests] Only run a sample of cache tests
......................................................................
[tests] Only run a sample of cache tests
Processing all cached files need moe than 5 minutes on CI.
To decrease test time only run a sample of tests.
Bug: T334777
Change-Id: I453d1168744e4b46a44486698c3c387d1ebef794
---
M tests/cache_tests.py
M scripts/maintenance/cache.py
2 files changed, 24 insertions(+), 4 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/maintenance/cache.py b/scripts/maintenance/cache.py
index 1918d6a..400ff41 100755
--- a/scripts/maintenance/cache.py
+++ b/scripts/maintenance/cache.py
@@ -64,7 +64,7 @@
uniquedesc(entry)
"""
#
-# (C) Pywikibot team, 2014-2022
+# (C) Pywikibot team, 2014-2023
#
# Distributed under the terms of the MIT license.
#
@@ -74,6 +74,7 @@
import pickle
import sys
from pathlib import Path
+from random import sample
from typing import Optional
import pywikibot
@@ -219,7 +220,8 @@
def process_entries(cache_path, func, use_accesstime: Optional[bool] = None,
- output_func=None, action_func=None):
+ output_func=None, action_func=None, *,
+ tests: Optional[int] = None):
"""Check the contents of the cache.
This program tries to use file access times to determine whether
@@ -230,6 +232,7 @@
:param use_accesstime: Whether access times should be used. `None`
for detect, `False` for don't use and `True` for always use.
+ :param tests: Only process a test sample of files
"""
if not cache_path:
cache_path = os.path.join(pywikibot.config.base_dir,
@@ -245,6 +248,9 @@
else:
filenames = [cache_path]
+ if tests:
+ filenames = sample(filenames, min(len(filenames), tests))
+
for filepath in filenames:
filename = os.path.basename(filepath)
cache_dir = os.path.dirname(filepath)
@@ -260,7 +266,7 @@
continue
# Skip foreign python specific directory
- _, _, version = cache_path.partition('-')
+ *_, version = cache_path.partition('-')
if version and version[-1] != str(PYTHON_VERSION[0]):
pywikibot.error(
"Skipping {} directory, can't read content with python {}"
diff --git a/tests/cache_tests.py b/tests/cache_tests.py
index 761ec32..5b57083 100755
--- a/tests/cache_tests.py
+++ b/tests/cache_tests.py
@@ -34,7 +34,8 @@
def test_cache(self):
"""Test the apicache by doing _check_cache_entry over each entry."""
- cache.process_entries(join_cache_path(), self._check_cache_entry)
+ cache.process_entries(join_cache_path(), self._check_cache_entry,
+ tests=25)
if __name__ == '__main__': # pragma: no cover
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/908943
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I453d1168744e4b46a44486698c3c387d1ebef794
Gerrit-Change-Number: 908943
Gerrit-PatchSet: 2
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/908886 )
Change subject: [tests] Fix obsolete type in TestGenerateFamilyFile
......................................................................
[tests] Fix obsolete type in TestGenerateFamilyFile
interwiki_removals is either a frozen_set or a list
Bug: T334714
Change-Id: I573954fd3a92d22269708b6f58cee1bd58616d66
---
M tests/generate_family_file_tests.py
1 file changed, 17 insertions(+), 3 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/tests/generate_family_file_tests.py b/tests/generate_family_file_tests.py
index f219ac6..ea9e0b8 100755
--- a/tests/generate_family_file_tests.py
+++ b/tests/generate_family_file_tests.py
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
"""Test generate_family_file script."""
#
-# (C) Pywikibot team, 2018-2022
+# (C) Pywikibot team, 2018-2023
#
# Distributed under the terms of the MIT license.
#
@@ -37,7 +37,7 @@
"""Pass writing."""
-class TestGenerateFamilyFiles(DefaultSiteTestCase):
+class TestGenerateFamilyFile(DefaultSiteTestCase):
"""Test generate_family_file functionality."""
@@ -80,7 +80,9 @@
with self.subTest(test='Test element counts'):
if self.site.lang not in gen.prefixes:
gen.prefixes.append(self.site.lang)
- obsolete = self.site.family.interwiki_removals
+
+ # interwiki_removals is either a frozen_set or a list
+ obsolete = set(self.site.family.interwiki_removals)
self.assertCountEqual(set(gen.prefixes) - obsolete,
set(gen.wikis) - obsolete)
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/908886
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I573954fd3a92d22269708b6f58cee1bd58616d66
Gerrit-Change-Number: 908886
Gerrit-PatchSet: 1
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged