Pywikibot-commits April 2021

pywikibot-commits@lists.wikimedia.org

1 participants
138 discussions

[Gerrit] ...core[master]: [parser] Deprecate textlib usage without MediaWiki markup parser

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/680294 ) Change subject: [parser] Deprecate textlib usage without MediaWiki markup parser ...................................................................... [parser] Deprecate textlib usage without MediaWiki markup parser mwparserfromhell or wikitextparser will be mandatory in near future - check whether mwparserfromhell or wikitextparser is installed and show a FutureWarning in extract_templates_and_params if no package is available - catch an ImportError in pwb wrapper like for other dependency checks. The ImportError will be raised when parser is mandatory. - deprecate extract_templates_and_params_regex - update tests and remove TestGenericTemplateParams because dispatcher will be removed. Don't test the outdated regex parser anymore. - add wikitextparser to setup.py extra_deps - update tox.ini - update some documentation hints Patch detached from 908abeb94b2ee Bug: T106763 Change-Id: I967698f79d958f233becb6e7a60336361f57c0f5 --- M README.rst M pwb.py M pywikibot/textlib.py M requirements.txt M setup.py M tests/__init__.py M tests/textlib_tests.py M tox.ini 8 files changed, 111 insertions(+), 137 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/README.rst b/README.rst index 6b1a91f..8872f7d 100644 --- a/README.rst +++ b/README.rst @@ -44,6 +44,7 @@ :: + pip install requests git clone https://gerrit.wikimedia.org/r/pywikibot/core.git cd core git submodule update --init @@ -56,6 +57,18 @@ pip install -U setuptools pip install pywikibot +In addition a MediaWiki markup parser is required. Please install one of them: + +:: + + pip install mwparserfromhell + +or + +:: + + pip install wikitextparser + Our `installation guide <https://www.mediawiki.org/wiki/Manual:Pywikibot/Installation>`_ has more details for advanced usage. diff --git a/pwb.py b/pwb.py index 80bbc7e..95abb09 100755 --- a/pwb.py +++ b/pwb.py @@ -14,7 +14,7 @@ python pwb.py -lang:de bot_tests -v """ -# (C) Pywikibot team, 2012-2020 +# (C) Pywikibot team, 2012-2021 # # Distributed under the terms of the MIT license. # @@ -182,9 +182,8 @@ try: if not check_modules(): raise RuntimeError('') # no further output needed -except RuntimeError as e: - print(e) - sys.exit() +except RuntimeError as e: # setup.py may also raise RuntimeError + sys.exit(e) from pathlib import Path # noqa: E402 @@ -213,6 +212,8 @@ # we need to re-start the entire process. Ask the user to do so. print('Now, you have to re-execute the command to start your script.') sys.exit(1) +except ImportError as e: # raised in textlib + sys.exit(e) def find_alternates(filename, script_paths): diff --git a/pywikibot/textlib.py b/pywikibot/textlib.py index 2ee62d1..83f8872 100644 --- a/pywikibot/textlib.py +++ b/pywikibot/textlib.py @@ -1570,8 +1570,8 @@ # -------------------------------- def extract_templates_and_params(text: str, - remove_disabled_parts: Optional[bool] = None, - strip: Optional[bool] = None) -> ETPType: + remove_disabled_parts: bool = False, + strip: bool = False) -> ETPType: """Return a list of templates found in text. Return value is a list of tuples. There is one tuple for each use of a @@ -1582,16 +1582,19 @@ parameters, and if this results multiple parameters with the same name only the last value provided will be returned. - This uses the package L{mwparserfromhell} (mwpfh) if it is installed. - Otherwise it falls back on a regex based implementation. + This uses the package L{mwparserfromhell} or L{wikitextparser} as + MediaWiki markup parser. Otherwise it falls back on a regex based + implementation but it becomes mandatory that one of them is + installed. There are minor differences between the two implementations. - The two implementations return nested templates in a different order. - i.e. for {{a|b={{c}}}}, mwpfh returns [a, c], whereas regex returns [c, a]. + The two implementations return nested templates in a different + order, i.e. for {{a|b={{c}}}}, parsers returns [a, c], whereas regex + returns [c, a]. - mwpfh preserves whitespace in parameter names and values. regex excludes - anything between  before parsing the text. + The parser packages preserves whitespace in parameter names and + values. If there are multiple numbered parameters in the wikitext for the same position, MediaWiki will only use the last parameter value. @@ -1599,27 +1602,32 @@ To replicate that behaviour, enable both remove_disabled_parts and strip. @param text: The wikitext from which templates are extracted - @param remove_disabled_parts: Remove disabled wikitext such as comments - and pre. If None (default), this is enabled when neither - mwparserfromhell not wikitextparser package is available and - disabled otherwise. - @param strip: if enabled, strip arguments and values of templates. - If None (default), this is enabled when neither mwparserfromhell - nor wikitextparser package is available and disabled otherwise. + @param remove_disabled_parts: If enabled, remove disabled wikitext + such as comments and pre. + @param strip: If enabled, strip arguments and values of templates. @return: list of template name and params """ use_regex = isinstance(wikitextparser, ImportError) - if remove_disabled_parts is None: - remove_disabled_parts = use_regex if remove_disabled_parts: text = removeDisabledParts(text) - if strip is None: - strip = use_regex - if use_regex: - return extract_templates_and_params_regex(text, False, strip) + issue_deprecation_warning(""" +Pywikibot needs a MediaWiki markup parser. +Please install the requested module with either + + pip install "mwparserfromhell>=0.5.0" + +or + + pip install "wikitextparser>=0.47.0" + +Using pywikibot without MediaWiki markup parser""", + warning_class=FutureWarning, + since='20210416') + + return _extract_templates_and_params_regex(text, False, strip) return _extract_templates_and_params_parser(text, strip) @@ -1683,7 +1691,7 @@ future_warning=True) def extract_templates_and_params_mwpfh(text: str, strip: bool = False) -> ETPType: - """Extract templates with params using mwparserfromhell.""" + """DEPRECATED. Extract templates with params using mwparserfromhell.""" global wikitextparser saved_parser = wikitextparser import mwparserfromhell as wikitextparser @@ -1692,11 +1700,12 @@ return result +@deprecated('extract_templates_and_params', since='20210331', + future_warning=True) def extract_templates_and_params_regex(text: str, remove_disabled_parts: bool = True, strip: bool = True) -> ETPType: - """ - Extract templates with params using a regex with additional processing. + """DEPRECATED. Extract templates with params using a regex. This function should not be called directly. @@ -1708,6 +1717,14 @@ @param strip: if enabled, strip arguments and values of templates @return: list of template name and params """ + return _extract_templates_and_params_regex(text, remove_disabled_parts, + strip) + + +def _extract_templates_and_params_regex(text: str, + remove_disabled_parts: bool = True, + strip: bool = True) -> ETPType: + """DEPRECATED. Extract templates with params using a regex.""" # remove commented-out stuff etc. if remove_disabled_parts: thistxt = removeDisabledParts(text) diff --git a/requirements.txt b/requirements.txt index 8b64aec..2a6cbc4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6,7 +6,6 @@ # It is organised so that simple requirements # are processed first, and more difficult packages # are last. -# All dependencies other than requests are optional. # # It is good practise to install packages using the system # package manager if it has a packaged version. If you are @@ -18,12 +17,18 @@ # or # $ awk -F '[#>=]' '{print $1}' requirements.txt | xargs apt-cache search -# mandatory +# mandatory dependencies, others are optional requests>=2.20.1, < 2.26.0; python_version < '3.6' requests>=2.20.1 ; python_version >= '3.6' setuptools>=20.2, !=50.0.0, <50.2.0 ; python_version < '3.6' setuptools>=20.2 ; python_version >= '3.6' +# MediaWiki markup parser +# mwparserfromhell is default, wikitextparser can be used instead +# mwparserfromhell is still required for commons_information.py and patrol.py +# wikitextparser>=0.47.0 +mwparserfromhell>=0.5.0 + # OAuth support # mwoauth 0.2.4 is needed because it supports getting identity information # about the user @@ -42,9 +47,6 @@ google >= 1.7 sseclient >= 0.0.18,!=0.0.23,!=0.0.24 -# textlib.py, commons_information and patrol.py -mwparserfromhell>=0.5.0 - # The mysql generator in pagegenerators depends on PyMySQL PyMySQL >= 0.6.7, < 1.0.0 ; python_version < '3.6' PyMySQL >= 1.0.0 ; python_version >= '3.6' diff --git a/setup.py b/setup.py index 6b34da3..2ae5ca7 100644 --- a/setup.py +++ b/setup.py @@ -62,6 +62,7 @@ 'Graphviz': ['pydot>=1.2'], 'Google': ['google>=1.7'], 'mwparserfromhell': ['mwparserfromhell>=0.5.0'], + 'wikitextparser': ['wikitextparser>=0.47.0'], 'Tkinter': [ # vulnerability found in Pillow<8.1.1 'Pillow>=8.1.1;python_version>="3.6"', ], @@ -110,7 +111,7 @@ 'setuptools>=20.2, !=50.0.0, <50.2.0 ; python_version < "3.6"', 'setuptools>=20.2 ; python_version >= "3.6"', ] - +# in addition either mwparserfromhell or wikitextparser is required # ------- setup tests_require ------- # test_deps = ['mock'] diff --git a/tests/__init__.py b/tests/__init__.py index 04f61c3..e0ac4b6 100644 --- a/tests/__init__.py +++ b/tests/__init__.py @@ -24,7 +24,8 @@ # Verify that the unit tests have a base working environment: # - requests is mandatory # however if unavailable this will fail on use; see pywikibot/tools.py -# - mwparserfromhell is optional, so is only imported in textlib_tests +# - mwparserfromhell or wikitextparser should be used but the dependency +# is checked by textlib already import requests # noqa: F401 import pywikibot.data.api diff --git a/tests/textlib_tests.py b/tests/textlib_tests.py index dadc2a3..aed174f 100644 --- a/tests/textlib_tests.py +++ b/tests/textlib_tests.py @@ -23,8 +23,10 @@ from pywikibot import UnknownSite from tests.aspects import ( - require_modules, TestCase, DefaultDrySiteTestCase, - PatchingTestCase, SiteAttributeTestCase, + DefaultDrySiteTestCase, + require_modules, + SiteAttributeTestCase, + TestCase, ) from tests import mock @@ -302,7 +304,7 @@ 'Invalid category title extracted: nasty{{{!}}') -WARNING_MSG = (r'.*extract_templates_and_params_mwpfh .*' +WARNING_MSG = (r'.*extract_templates_and_params_.*' r'is deprecated for .*; use extract_templates_and_params') @@ -502,7 +504,7 @@ @require_modules('mwparserfromhell') def test_extract_templates_params_parser_stripped(self): """Test using mwparserfromhell with stripping.""" - func = functools.partial(textlib._extract_templates_and_params_parser, + func = functools.partial(textlib.extract_templates_and_params, strip=True) self._common_results(func) @@ -535,37 +537,39 @@ """Test using many complex regexes.""" func = functools.partial(textlib.extract_templates_and_params_regex, remove_disabled_parts=False, strip=False) - self._common_results(func) - self._order_differs(func) - self._unstripped(func) - - self.assertEqual(func('{{a|b={} }}'), []) # FIXME: {} is normal text + with suppress_warnings(WARNING_MSG, category=FutureWarning): + self._common_results(func) + self._order_differs(func) + self._unstripped(func) + # FIXME: {} is normal text + self.assertEqual(func('{{a|b={} }}'), []) def test_extract_templates_params_regex_stripped(self): """Test using many complex regexes with stripping.""" func = textlib.extract_templates_and_params_regex + with suppress_warnings(WARNING_MSG, category=FutureWarning): + self._common_results(func) + self._order_differs(func) + self._stripped(func) - self._common_results(func) - self._order_differs(func) - self._stripped(func) + self.assertEqual(func('{{a|b=}}'), + [('a', OrderedDict((('b', ''), )))]) - self.assertEqual(func('{{a|b=}}'), - [('a', OrderedDict((('b', ''), )))]) + # Identical to mwpfh + self.assertCountEqual(func('{{a|{{c|{{d}}}}}}'), + [('c', OrderedDict((('1', '{{d}}'), ))), + ('a', OrderedDict([('1', '{{c|{{d}}}}')])), + ('d', OrderedDict()) + ]) - # Identical to mwpfh - self.assertCountEqual(func('{{a|{{c|{{d}}}}}}'), - [('c', OrderedDict((('1', '{{d}}'), ))), - ('a', OrderedDict([('1', '{{c|{{d}}}}')])), - ('d', OrderedDict()) - ]) + # However fails to correctly handle three levels of balanced + # brackets with empty parameters + self.assertCountEqual(func('{{a|{{c|{{d|}}}}}}'), + [('c', OrderedDict((('1', '{{d|}}}'), ))), + ('d', OrderedDict([('1', '}')])) + ]) - # However fails to correctly handle three levels of balanced brackets - # with empty parameters - self.assertCountEqual(func('{{a|{{c|{{d|}}}}}}'), - [('c', OrderedDict((('1', '{{d|}}}'), ))), - ('d', OrderedDict([('1', '}')])) - ]) - + @require_modules('mwparserfromhell') def test_extract_templates_params(self): """Test that the normal entry point works.""" func = functools.partial(textlib.extract_templates_and_params, @@ -699,79 +703,6 @@ self.assertTrue(m.group(0).endswith('foo {{bar}}')) -class TestGenericTemplateParams(PatchingTestCase): - - """Test whether the generic function forwards the call correctly.""" - - net = False - - @PatchingTestCase.patched(textlib, '_extract_templates_and_params_parser') - def extract_mwpfh(self, text, *args, **kwargs): - """Patched call to extract_templates_and_params_mwpfh.""" - self._text = text - self._args = args - self._mwpfh = True - - @PatchingTestCase.patched(textlib, 'extract_templates_and_params_regex') - def extract_regex(self, text, *args, **kwargs): - """Patched call to extract_templates_and_params_regex.""" - self._text = text - self._args = args - self._mwpfh = False - - def test_removing_disabled_parts_regex(self): - """Test removing disabled parts when using the regex variant.""" - self.patch(textlib, 'wikitextparser', ImportError()) - textlib.extract_templates_and_params('{{a}}', True) - self.assertEqual(self._text, '{{a}}') - self.assertFalse(self._mwpfh) - textlib.extract_templates_and_params('{{a}}', False) - self.assertEqual(self._text, '{{a}}') - self.assertFalse(self._mwpfh) - textlib.extract_templates_and_params('{{a}}') - self.assertEqual(self._text, '{{a}}') - self.assertFalse(self._mwpfh) - - @require_modules('mwparserfromhell') - def test_removing_disabled_parts_mwpfh(self): - """Test removing disabled parts when using the mwpfh variant.""" - textlib.extract_templates_and_params('{{a}}', True) - self.assertEqual(self._text, '{{a}}') - self.assertTrue(self._mwpfh) - textlib.extract_templates_and_params('{{a}}', False) - self.assertEqual(self._text, '{{a}}') - self.assertTrue(self._mwpfh) - textlib.extract_templates_and_params('{{a}}') - self.assertEqual(self._text, '{{a}}') - self.assertTrue(self._mwpfh) - - def test_strip_regex(self): - """Test stripping values when using the regex variant.""" - self.patch(textlib, 'wikitextparser', ImportError()) - textlib.extract_templates_and_params('{{a| foo }}', False, True) - self.assertEqual(self._args, (False, True)) - self.assertFalse(self._mwpfh) - textlib.extract_templates_and_params('{{a| foo }}', False, False) - self.assertEqual(self._args, (False, False)) - self.assertFalse(self._mwpfh) - textlib.extract_templates_and_params('{{a| foo }}', False) - self.assertEqual(self._args, (False, True)) - self.assertFalse(self._mwpfh) - - @require_modules('mwparserfromhell') - def test_strip_mwpfh(self): - """Test stripping values when using the mwpfh variant.""" - textlib.extract_templates_and_params('{{a| foo }}', None, True) - self.assertEqual(self._args, (True, )) - self.assertTrue(self._mwpfh) - textlib.extract_templates_and_params('{{a| foo }}', None, False) - self.assertEqual(self._args, (False, )) - self.assertTrue(self._mwpfh) - textlib.extract_templates_and_params('{{a| foo }}') - self.assertEqual(self._args, (False, )) - self.assertTrue(self._mwpfh) - - class TestDisabledParts(DefaultDrySiteTestCase): """Test the removeDisabledParts function in textlib.""" diff --git a/tox.ini b/tox.ini index d68e21d..97ebeaf 100644 --- a/tox.ini +++ b/tox.ini @@ -40,8 +40,15 @@ fasttest: mock fasttest: .[scripts] - fasttest-py35: mwparserfromhell - fasttest-py35: beautifulsoup4 + fasttest-py35: .[html] + fasttest-py35: .[mwparserfromhell] + fasttest-py37: .[wikitextparser] + + deeptest: .[html] + deeptest: .[mwparserfromhell] + deeptest: .[scripts] + deeptest: .[wikitextparser] + [testenv:commit-message] basepython = python3 @@ -56,6 +63,7 @@ nosetests --with-doctest pywikibot {[params]doctest_skip} deps = nose + .[mwparserfromhell] [testenv:venv] commands = {posargs} -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/680294 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: I967698f79d958f233becb6e7a60336361f57c0f5 Gerrit-Change-Number: 680294 Gerrit-PatchSet: 3 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot Gerrit-MessageType: merged

3 years

[Gerrit] ...core[master]: [IMPR] Remove distutils

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/679612 ) Change subject: [IMPR] Remove distutils ...................................................................... [IMPR] Remove distutils Python 3.10 deprecated distutils and it will be removed in 3.12. They advise that we use packaging instead... https://docs.python.org/3.10/whatsnew/3.10.html#distutils-deprecated We mostly used distutils for version parsing, for which packaging provides a simple replacement... https://stackoverflow.com/a/11887885 https://packaging.pypa.io/en/latest/version.html#packaging.version.parse Setuptools contains a copy of packaging under 'pkg_resources.extern.packaging' and pkg_resources.parse_version() is effectively an alias of packaging.version.parse(). Since we already have a depency on Setuptools this uses its copy. As for get_python_lib() sysconfig can provide the same... >>> import distutils.sysconfig >>> distutils.sysconfig.get_python_lib(standard_lib=True) '/usr/lib/python3.8' >>> import sysconfig >>> sysconfig.get_paths()['stdlib'] '/usr/lib/python3.8' Bug: T274696 Change-Id: I6245895d4249d32cfae34043eed489cc94ddea2e --- M pywikibot/comms/eventstreams.py M pywikibot/data/mysql.py M pywikibot/tools/__init__.py M pywikibot/version.py M tests/tools_tests.py 5 files changed, 76 insertions(+), 24 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/comms/eventstreams.py b/pywikibot/comms/eventstreams.py index 662598a..6964089 100644 --- a/pywikibot/comms/eventstreams.py +++ b/pywikibot/comms/eventstreams.py @@ -12,11 +12,11 @@ # # Distributed under the terms of the MIT license. # -from distutils.version import LooseVersion -from functools import partial import json import socket +from functools import partial +from pkg_resources import parse_version from typing import Optional from requests import __version__ as requests_version @@ -31,7 +31,7 @@ from pywikibot import config, debug, Timestamp, Site, warning from pywikibot.tools import deprecated_args -if LooseVersion(requests_version) < LooseVersion('2.20.1'): +if parse_version(requests_version) < parse_version('2.20.1'): raise ImportError( 'requests >= 2.20.1 is required for EventStreams;\n' "install it with 'pip install \"requests>=2.20.1\"'\n") diff --git a/pywikibot/data/mysql.py b/pywikibot/data/mysql.py index 9ab934b..f3caa95 100644 --- a/pywikibot/data/mysql.py +++ b/pywikibot/data/mysql.py @@ -4,7 +4,8 @@ # # Distributed under the terms of the MIT license. # -from distutils.version import LooseVersion +import pkg_resources + from typing import Optional import pywikibot @@ -62,7 +63,10 @@ charset='utf8', defer_connect=query == 'test', # for tests **credentials) - if LooseVersion(pymysql.__version__) < LooseVersion('1.0.0'): + + pymysql_version = pkg_resources.parse_version(pymysql.__version__) + + if pymysql_version < pkg_resources.parse_version('1.0.0'): from contextlib import closing connection = closing(connection) diff --git a/pywikibot/tools/__init__.py b/pywikibot/tools/__init__.py index f891c09..77a5494 100644 --- a/pywikibot/tools/__init__.py +++ b/pywikibot/tools/__init__.py @@ -10,6 +10,7 @@ import inspect import itertools import os +import pkg_resources import queue import re import stat @@ -23,7 +24,6 @@ from collections import defaultdict from contextlib import suppress from datetime import datetime -from distutils.version import LooseVersion, Version from functools import wraps from importlib import import_module from inspect import getfullargspec @@ -78,24 +78,25 @@ def has_module(module, version=None): - """Check whether a module can be imported.""" + """Check if a module can be imported.""" try: m = import_module(module) except ImportError: - pass + return False else: - if version is None: - return True - try: - module_version = LooseVersion(m.__version__) - except AttributeError: - pass - else: - if module_version >= LooseVersion(version): - return True - warn('Module version {} is lower than requested version {}' - .format(module_version, version), ImportWarning) - return False + if version: + if not hasattr(m, '__version__'): + return False + + required_version = pkg_resources.parse_version(version) + module_version = pkg_resources.parse_version(m.__version__) + + if module_version < required_version: + warn('Module version {} is lower than requested version {}' + .format(module_version, required_version), ImportWarning) + return False + + return True def empty_iterator(): @@ -501,7 +502,7 @@ return first_upper(username) -class MediaWikiVersion(Version): +class MediaWikiVersion: """ Version object to allow comparing 'wmf' versions with normal ones. @@ -526,6 +527,15 @@ MEDIAWIKI_VERSION = re.compile( r'(\d+(?:\.\d+)+)(-?wmf\.?(\d+)|alpha|beta(\d+)|-?rc\.?(\d+)|.*)?$') + def __init__(self, version_str): + """ + Initializer. + + @param version_str: version to parse + @type version: str + """ + self.parse(version_str) + @classmethod def from_generator(cls, generator): """Create instance using the generator string.""" @@ -583,6 +593,21 @@ return -1 return 0 + def __eq__(self, other): + return self._cmp(other) == 0 + + def __lt__(self, other): + return self._cmp(other) < 0 + + def __le__(self, other): + return self._cmp(other) <= 0 + + def __gt__(self, other): + return self._cmp(other) > 0 + + def __ge__(self, other): + return self._cmp(other) >= 0 + class ThreadedGenerator(threading.Thread): diff --git a/pywikibot/version.py b/pywikibot/version.py index 20242f5..a71772b 100644 --- a/pywikibot/version.py +++ b/pywikibot/version.py @@ -11,11 +11,11 @@ import socket import subprocess import sys +import sysconfig import time import xml.dom.minidom from contextlib import closing, suppress -from distutils.sysconfig import get_python_lib from importlib import import_module from io import BytesIO from typing import Optional @@ -415,7 +415,7 @@ if not modules: modules = sys.modules.keys() - std_lib_dir = pathlib.Path(get_python_lib(standard_lib=True)) + std_lib_dir = pathlib.Path(sysconfig.get_paths()['stdlib']) root_packages = {key.split('.')[0] for key in modules} diff --git a/tests/tools_tests.py b/tests/tools_tests.py index ea0c7a3..cda8191 100644 --- a/tests/tools_tests.py +++ b/tests/tools_tests.py @@ -16,7 +16,7 @@ from importlib import import_module from pywikibot import tools -from pywikibot.tools import classproperty +from pywikibot.tools import classproperty, has_module, suppress_warnings from tests import join_xml_data_path, mock from tests.aspects import require_modules, TestCase @@ -742,6 +742,29 @@ self.assertEqual(result, 'HelloWorld') +class TestHasModule(TestCase): + + """Unit test class for has_module.""" + + net = False + + def test_when_present(self): + """Test when the module is available.""" + self.assertTrue(has_module('setuptools')) + self.assertTrue(has_module('setuptools', '1.0')) + + def test_when_missing(self): + """Test when the module is unavailable.""" + self.assertFalse(has_module('no-such-module')) + + @suppress_warnings( + r'^Module version .* is lower than requested version 99999$', + ImportWarning) + def test_when_insufficient_version(self): + """Test when the module is older than what we need.""" + self.assertFalse(has_module('setuptools', '99999')) + + if __name__ == '__main__': # pragma: no cover with suppress(SystemExit): unittest.main() -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/679612 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: I6245895d4249d32cfae34043eed489cc94ddea2e Gerrit-Change-Number: 679612 Gerrit-PatchSet: 3 Gerrit-Owner: Damian <atagar1(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot Gerrit-MessageType: merged

3 years

[Gerrit] ...core[master]: [cleanup] Deprecate DotReadableDict

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/680030 ) Change subject: [cleanup] Deprecate DotReadableDict ...................................................................... [cleanup] Deprecate DotReadableDict DotReadableDict only defines itself as "Parent class of Revision() and FileInfo()" rather than providing anything useful. Revision no longer uses it, and DotReadableDict is a pretty trivial class so merging what it has into FileInfo. Change-Id: I96eb5476e12c2afe8cceeb97419ffef5426ad3ab --- M pywikibot/page/__init__.py M pywikibot/tools/__init__.py 2 files changed, 14 insertions(+), 2 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/page/__init__.py b/pywikibot/page/__init__.py index a5aa643..2a6ba98 100644 --- a/pywikibot/page/__init__.py +++ b/pywikibot/page/__init__.py @@ -58,7 +58,6 @@ deprecated, deprecate_arg, deprecated_args, - DotReadableDict, first_upper, redirect_func, remove_last_args, @@ -4873,7 +4872,7 @@ } -class FileInfo(DotReadableDict): +class FileInfo: """ A structure holding imageinfo of latest rev. of FilePage. @@ -4895,6 +4894,14 @@ self.__dict__.update(file_revision) self.timestamp = pywikibot.Timestamp.fromISOformat(self.timestamp) + def __getitem__(self, key): + """Give access to class values by key.""" + return getattr(self, key) + + def __repr__(self): + """Return a more complete string representation.""" + return repr(self.__dict__) + def __eq__(self, other): """Test if two File_info objects are equal.""" return self.__dict__ == other.__dict__ diff --git a/pywikibot/tools/__init__.py b/pywikibot/tools/__init__.py index f891c09..ac1ce9b 100644 --- a/pywikibot/tools/__init__.py +++ b/pywikibot/tools/__init__.py @@ -1900,3 +1900,8 @@ option_msg += '\n' + ' ' * indent option_msg += option_line return '{} ({}):'.format(message, option_msg) + + +wrapper = ModuleDeprecationWrapper(__name__) +wrapper._add_deprecated_attr('DotReadableDict', replacement_name='', + since='20210416', future_warning=True) -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/680030 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: I96eb5476e12c2afe8cceeb97419ffef5426ad3ab Gerrit-Change-Number: 680030 Gerrit-PatchSet: 3 Gerrit-Owner: Damian <atagar1(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot Gerrit-MessageType: merged

3 years

[Gerrit] ...core[master]: Revert "[parser] Make mwparserfromhell or wikitextparser mandatory"

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/680016 ) Change subject: Revert "[parser] Make mwparserfromhell or wikitextparser mandatory" ...................................................................... Revert "[parser] Make mwparserfromhell or wikitextparser mandatory" This reverts commit 908abeb94b2eecf6f2210cff2a9280a19db85787. Reason for revert: CI tests are failing; postponed to a later release until deprecation time has expired Change-Id: I10ca06e5ca9944aa76c71f6be9269efae4f8e142 --- M README.rst M pwb.py M pywikibot/textlib.py M requirements.txt M setup.py M tests/__init__.py M tests/textlib_tests.py M tox.ini 8 files changed, 165 insertions(+), 100 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/README.rst b/README.rst index 8872f7d..6b1a91f 100644 --- a/README.rst +++ b/README.rst @@ -44,7 +44,6 @@ :: - pip install requests git clone https://gerrit.wikimedia.org/r/pywikibot/core.git cd core git submodule update --init @@ -57,18 +56,6 @@ pip install -U setuptools pip install pywikibot -In addition a MediaWiki markup parser is required. Please install one of them: - -:: - - pip install mwparserfromhell - -or - -:: - - pip install wikitextparser - Our `installation guide <https://www.mediawiki.org/wiki/Manual:Pywikibot/Installation>`_ has more details for advanced usage. diff --git a/pwb.py b/pwb.py index 95abb09..80bbc7e 100755 --- a/pwb.py +++ b/pwb.py @@ -14,7 +14,7 @@ python pwb.py -lang:de bot_tests -v """ -# (C) Pywikibot team, 2012-2021 +# (C) Pywikibot team, 2012-2020 # # Distributed under the terms of the MIT license. # @@ -182,8 +182,9 @@ try: if not check_modules(): raise RuntimeError('') # no further output needed -except RuntimeError as e: # setup.py may also raise RuntimeError - sys.exit(e) +except RuntimeError as e: + print(e) + sys.exit() from pathlib import Path # noqa: E402 @@ -212,8 +213,6 @@ # we need to re-start the entire process. Ask the user to do so. print('Now, you have to re-execute the command to start your script.') sys.exit(1) -except ImportError as e: # raised in textlib - sys.exit(e) def find_alternates(filename, script_paths): diff --git a/pywikibot/textlib.py b/pywikibot/textlib.py index 8807069..2ee62d1 100644 --- a/pywikibot/textlib.py +++ b/pywikibot/textlib.py @@ -35,18 +35,8 @@ except ImportError: try: import mwparserfromhell as wikitextparser - except ImportError: - # print required because pywikibot is not imported completely - raise ImportError(""" -Pywikibot is missing a MediaWiki markup parser which is necessary. -Please update the required module with either - - pip install "mwparserfromhell>=0.5.0" - -or - - pip install "wikitextparser>=0.47.0" -""") from None + except ImportError as e: + wikitextparser = e ETPType = List[Tuple[str, OrderedDictType[str, str]]] @@ -1580,8 +1570,8 @@ # -------------------------------- def extract_templates_and_params(text: str, - remove_disabled_parts: bool = False, - strip: bool = False) -> ETPType: + remove_disabled_parts: Optional[bool] = None, + strip: Optional[bool] = None) -> ETPType: """Return a list of templates found in text. Return value is a list of tuples. There is one tuple for each use of a @@ -1592,14 +1582,16 @@ parameters, and if this results multiple parameters with the same name only the last value provided will be returned. - This uses the package L{mwparserfromhell} or L{wikitextparser} as - MediaWiki markup parser. It is mandatory that one of them is - installed. + This uses the package L{mwparserfromhell} (mwpfh) if it is installed. + Otherwise it falls back on a regex based implementation. There are minor differences between the two implementations. - The parser packages preserves whitespace in parameter names and - values. + The two implementations return nested templates in a different order. + i.e. for {{a|b={{c}}}}, mwpfh returns [a, c], whereas regex returns [c, a]. + + mwpfh preserves whitespace in parameter names and values. regex excludes + anything between  before parsing the text. If there are multiple numbered parameters in the wikitext for the same position, MediaWiki will only use the last parameter value. @@ -1607,9 +1599,43 @@ To replicate that behaviour, enable both remove_disabled_parts and strip. @param text: The wikitext from which templates are extracted - @param remove_disabled_parts: If enabled, remove disabled wikitext - such as comments and pre. - @param strip: If enabled, strip arguments and values of templates. + @param remove_disabled_parts: Remove disabled wikitext such as comments + and pre. If None (default), this is enabled when neither + mwparserfromhell not wikitextparser package is available and + disabled otherwise. + @param strip: if enabled, strip arguments and values of templates. + If None (default), this is enabled when neither mwparserfromhell + nor wikitextparser package is available and disabled otherwise. + @return: list of template name and params + """ + use_regex = isinstance(wikitextparser, ImportError) + + if remove_disabled_parts is None: + remove_disabled_parts = use_regex + if remove_disabled_parts: + text = removeDisabledParts(text) + + if strip is None: + strip = use_regex + + if use_regex: + return extract_templates_and_params_regex(text, False, strip) + return _extract_templates_and_params_parser(text, strip) + + +def _extract_templates_and_params_parser(text: str, + strip: bool = False) -> ETPType: + """ + Extract templates with params using mwparserfromhell. + + This function should not be called directly. + + Use extract_templates_and_params, which will select this parser + implementation if the mwparserfromhell or wikitextparser package is + installed. + + @param text: The wikitext from which templates are extracted + @param strip: if enabled, strip arguments and values of templates @return: list of template name and params """ def explicit(param): @@ -1619,9 +1645,6 @@ attr = not param.positional return attr - if remove_disabled_parts: - text = removeDisabledParts(text) - parser_name = wikitextparser.__name__ pywikibot.log('Using {!r} wikitext parser'.format(parser_name)) @@ -1660,21 +1683,20 @@ future_warning=True) def extract_templates_and_params_mwpfh(text: str, strip: bool = False) -> ETPType: - """DEPRECATED. Extract templates with params using mwparserfromhell.""" + """Extract templates with params using mwparserfromhell.""" global wikitextparser saved_parser = wikitextparser import mwparserfromhell as wikitextparser - result = extract_templates_and_params(text, strip=strip) + result = _extract_templates_and_params_parser(text, strip) wikitextparser = saved_parser return result -@deprecated('extract_templates_and_params', since='20210331', - future_warning=True) def extract_templates_and_params_regex(text: str, remove_disabled_parts: bool = True, strip: bool = True) -> ETPType: - """DEPRECATED. Extract templates with params using a regex. + """ + Extract templates with params using a regex with additional processing. This function should not be called directly. diff --git a/requirements.txt b/requirements.txt index 2a6cbc4..8b64aec 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6,6 +6,7 @@ # It is organised so that simple requirements # are processed first, and more difficult packages # are last. +# All dependencies other than requests are optional. # # It is good practise to install packages using the system # package manager if it has a packaged version. If you are @@ -17,18 +18,12 @@ # or # $ awk -F '[#>=]' '{print $1}' requirements.txt | xargs apt-cache search -# mandatory dependencies, others are optional +# mandatory requests>=2.20.1, < 2.26.0; python_version < '3.6' requests>=2.20.1 ; python_version >= '3.6' setuptools>=20.2, !=50.0.0, <50.2.0 ; python_version < '3.6' setuptools>=20.2 ; python_version >= '3.6' -# MediaWiki markup parser -# mwparserfromhell is default, wikitextparser can be used instead -# mwparserfromhell is still required for commons_information.py and patrol.py -# wikitextparser>=0.47.0 -mwparserfromhell>=0.5.0 - # OAuth support # mwoauth 0.2.4 is needed because it supports getting identity information # about the user @@ -47,6 +42,9 @@ google >= 1.7 sseclient >= 0.0.18,!=0.0.23,!=0.0.24 +# textlib.py, commons_information and patrol.py +mwparserfromhell>=0.5.0 + # The mysql generator in pagegenerators depends on PyMySQL PyMySQL >= 0.6.7, < 1.0.0 ; python_version < '3.6' PyMySQL >= 1.0.0 ; python_version >= '3.6' diff --git a/setup.py b/setup.py index 2ae5ca7..6b34da3 100644 --- a/setup.py +++ b/setup.py @@ -62,7 +62,6 @@ 'Graphviz': ['pydot>=1.2'], 'Google': ['google>=1.7'], 'mwparserfromhell': ['mwparserfromhell>=0.5.0'], - 'wikitextparser': ['wikitextparser>=0.47.0'], 'Tkinter': [ # vulnerability found in Pillow<8.1.1 'Pillow>=8.1.1;python_version>="3.6"', ], @@ -111,7 +110,7 @@ 'setuptools>=20.2, !=50.0.0, <50.2.0 ; python_version < "3.6"', 'setuptools>=20.2 ; python_version >= "3.6"', ] -# in addition either mwparserfromhell or wikitextparser is required + # ------- setup tests_require ------- # test_deps = ['mock'] diff --git a/tests/__init__.py b/tests/__init__.py index 596f1ce..04f61c3 100644 --- a/tests/__init__.py +++ b/tests/__init__.py @@ -24,8 +24,7 @@ # Verify that the unit tests have a base working environment: # - requests is mandatory # however if unavailable this will fail on use; see pywikibot/tools.py -# - mwparserfromhell or wikitextparser is mandatory but the dependency -# is checked by textlib already +# - mwparserfromhell is optional, so is only imported in textlib_tests import requests # noqa: F401 import pywikibot.data.api diff --git a/tests/textlib_tests.py b/tests/textlib_tests.py index aed174f..dadc2a3 100644 --- a/tests/textlib_tests.py +++ b/tests/textlib_tests.py @@ -23,10 +23,8 @@ from pywikibot import UnknownSite from tests.aspects import ( - DefaultDrySiteTestCase, - require_modules, - SiteAttributeTestCase, - TestCase, + require_modules, TestCase, DefaultDrySiteTestCase, + PatchingTestCase, SiteAttributeTestCase, ) from tests import mock @@ -304,7 +302,7 @@ 'Invalid category title extracted: nasty{{{!}}') -WARNING_MSG = (r'.*extract_templates_and_params_.*' +WARNING_MSG = (r'.*extract_templates_and_params_mwpfh .*' r'is deprecated for .*; use extract_templates_and_params') @@ -504,7 +502,7 @@ @require_modules('mwparserfromhell') def test_extract_templates_params_parser_stripped(self): """Test using mwparserfromhell with stripping.""" - func = functools.partial(textlib.extract_templates_and_params, + func = functools.partial(textlib._extract_templates_and_params_parser, strip=True) self._common_results(func) @@ -537,39 +535,37 @@ """Test using many complex regexes.""" func = functools.partial(textlib.extract_templates_and_params_regex, remove_disabled_parts=False, strip=False) - with suppress_warnings(WARNING_MSG, category=FutureWarning): - self._common_results(func) - self._order_differs(func) - self._unstripped(func) - # FIXME: {} is normal text - self.assertEqual(func('{{a|b={} }}'), []) + self._common_results(func) + self._order_differs(func) + self._unstripped(func) + + self.assertEqual(func('{{a|b={} }}'), []) # FIXME: {} is normal text def test_extract_templates_params_regex_stripped(self): """Test using many complex regexes with stripping.""" func = textlib.extract_templates_and_params_regex - with suppress_warnings(WARNING_MSG, category=FutureWarning): - self._common_results(func) - self._order_differs(func) - self._stripped(func) - self.assertEqual(func('{{a|b=}}'), - [('a', OrderedDict((('b', ''), )))]) + self._common_results(func) + self._order_differs(func) + self._stripped(func) - # Identical to mwpfh - self.assertCountEqual(func('{{a|{{c|{{d}}}}}}'), - [('c', OrderedDict((('1', '{{d}}'), ))), - ('a', OrderedDict([('1', '{{c|{{d}}}}')])), - ('d', OrderedDict()) - ]) + self.assertEqual(func('{{a|b=}}'), + [('a', OrderedDict((('b', ''), )))]) - # However fails to correctly handle three levels of balanced - # brackets with empty parameters - self.assertCountEqual(func('{{a|{{c|{{d|}}}}}}'), - [('c', OrderedDict((('1', '{{d|}}}'), ))), - ('d', OrderedDict([('1', '}')])) - ]) + # Identical to mwpfh + self.assertCountEqual(func('{{a|{{c|{{d}}}}}}'), + [('c', OrderedDict((('1', '{{d}}'), ))), + ('a', OrderedDict([('1', '{{c|{{d}}}}')])), + ('d', OrderedDict()) + ]) - @require_modules('mwparserfromhell') + # However fails to correctly handle three levels of balanced brackets + # with empty parameters + self.assertCountEqual(func('{{a|{{c|{{d|}}}}}}'), + [('c', OrderedDict((('1', '{{d|}}}'), ))), + ('d', OrderedDict([('1', '}')])) + ]) + def test_extract_templates_params(self): """Test that the normal entry point works.""" func = functools.partial(textlib.extract_templates_and_params, @@ -703,6 +699,79 @@ self.assertTrue(m.group(0).endswith('foo {{bar}}')) +class TestGenericTemplateParams(PatchingTestCase): + + """Test whether the generic function forwards the call correctly.""" + + net = False + + @PatchingTestCase.patched(textlib, '_extract_templates_and_params_parser') + def extract_mwpfh(self, text, *args, **kwargs): + """Patched call to extract_templates_and_params_mwpfh.""" + self._text = text + self._args = args + self._mwpfh = True + + @PatchingTestCase.patched(textlib, 'extract_templates_and_params_regex') + def extract_regex(self, text, *args, **kwargs): + """Patched call to extract_templates_and_params_regex.""" + self._text = text + self._args = args + self._mwpfh = False + + def test_removing_disabled_parts_regex(self): + """Test removing disabled parts when using the regex variant.""" + self.patch(textlib, 'wikitextparser', ImportError()) + textlib.extract_templates_and_params('{{a}}', True) + self.assertEqual(self._text, '{{a}}') + self.assertFalse(self._mwpfh) + textlib.extract_templates_and_params('{{a}}', False) + self.assertEqual(self._text, '{{a}}') + self.assertFalse(self._mwpfh) + textlib.extract_templates_and_params('{{a}}') + self.assertEqual(self._text, '{{a}}') + self.assertFalse(self._mwpfh) + + @require_modules('mwparserfromhell') + def test_removing_disabled_parts_mwpfh(self): + """Test removing disabled parts when using the mwpfh variant.""" + textlib.extract_templates_and_params('{{a}}', True) + self.assertEqual(self._text, '{{a}}') + self.assertTrue(self._mwpfh) + textlib.extract_templates_and_params('{{a}}', False) + self.assertEqual(self._text, '{{a}}') + self.assertTrue(self._mwpfh) + textlib.extract_templates_and_params('{{a}}') + self.assertEqual(self._text, '{{a}}') + self.assertTrue(self._mwpfh) + + def test_strip_regex(self): + """Test stripping values when using the regex variant.""" + self.patch(textlib, 'wikitextparser', ImportError()) + textlib.extract_templates_and_params('{{a| foo }}', False, True) + self.assertEqual(self._args, (False, True)) + self.assertFalse(self._mwpfh) + textlib.extract_templates_and_params('{{a| foo }}', False, False) + self.assertEqual(self._args, (False, False)) + self.assertFalse(self._mwpfh) + textlib.extract_templates_and_params('{{a| foo }}', False) + self.assertEqual(self._args, (False, True)) + self.assertFalse(self._mwpfh) + + @require_modules('mwparserfromhell') + def test_strip_mwpfh(self): + """Test stripping values when using the mwpfh variant.""" + textlib.extract_templates_and_params('{{a| foo }}', None, True) + self.assertEqual(self._args, (True, )) + self.assertTrue(self._mwpfh) + textlib.extract_templates_and_params('{{a| foo }}', None, False) + self.assertEqual(self._args, (False, )) + self.assertTrue(self._mwpfh) + textlib.extract_templates_and_params('{{a| foo }}') + self.assertEqual(self._args, (False, )) + self.assertTrue(self._mwpfh) + + class TestDisabledParts(DefaultDrySiteTestCase): """Test the removeDisabledParts function in textlib.""" diff --git a/tox.ini b/tox.ini index cb76128..d68e21d 100644 --- a/tox.ini +++ b/tox.ini @@ -38,17 +38,10 @@ fasttest: pytest-attrib>=0.1.3 fasttest: pytest-subtests >= 0.3.2 fasttest: mock - fasttest: .[mwparserfromhell] fasttest: .[scripts] - fasttest-py35: .[html] - fasttest-py37: .[wikitextparser] - - deeptest: .[html] - deeptest: .[mwparserfromhell] - deeptest: .[scripts] - deeptest: .[wikitextparser] - + fasttest-py35: mwparserfromhell + fasttest-py35: beautifulsoup4 [testenv:commit-message] basepython = python3 @@ -63,7 +56,6 @@ nosetests --with-doctest pywikibot {[params]doctest_skip} deps = nose - .[mwparserfromhell] [testenv:venv] commands = {posargs} -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/680016 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: I10ca06e5ca9944aa76c71f6be9269efae4f8e142 Gerrit-Change-Number: 680016 Gerrit-PatchSet: 2 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com> Gerrit-Reviewer: DannyS712 <dannys712.wiki(a)gmail.com> Gerrit-Reviewer: JJMC89 <JJMC89.Wikimedia(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot Gerrit-CC: Dalba <dalba.wiki(a)gmail.com> Gerrit-CC: Dvorapa <dvorapa(a)seznam.cz> Gerrit-MessageType: merged

3 years

[Gerrit] ...i18n[master]: Localisation updates from https://translatewiki.net.

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/i18n/+/679759 ) Change subject: Localisation updates from https://translatewiki.net. ...................................................................... Localisation updates from https://translatewiki.net. Change-Id: I82d0dc1526a5fd3541efe1ee52332fe4d06374cb --- M pywikibot/sv.json 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: L10n-bot: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/sv.json b/pywikibot/sv.json index 8046e87..fdc740e 100644 --- a/pywikibot/sv.json +++ b/pywikibot/sv.json @@ -1,6 +1,7 @@ { "@metadata": { "authors": [ + "Anhn", "Cybjit", "Jopparn", "Lokal Profil", @@ -19,5 +20,5 @@ "pywikibot-fixes-fckeditor": "Bot: Åtgärdar Rich-editor-html", "pywikibot-fixes-html": "Bot: Konverterar/korrigerar HTML", "pywikibot-fixes-syntax": "Bot: Fixar wikisyntax", - "pywikibot-touch": "Pywikibot-nolleredigering" + "pywikibot-touch": "Pywikibot noll-redigering" } -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/i18n/+/679759 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/i18n Gerrit-Branch: master Gerrit-Change-Id: I82d0dc1526a5fd3541efe1ee52332fe4d06374cb Gerrit-Change-Number: 679759 Gerrit-PatchSet: 1 Gerrit-Owner: L10n-bot <l10n-bot(a)translatewiki.net> Gerrit-Reviewer: L10n-bot <l10n-bot(a)translatewiki.net> Gerrit-Reviewer: jenkins-bot Gerrit-MessageType: merged

3 years

[Gerrit] ...core[master]: [IMPR] use http.HTTPStatus description, phrase and value

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/678395 ) Change subject: [IMPR] use http.HTTPStatus description, phrase and value ...................................................................... [IMPR] use http.HTTPStatus description, phrase and value Change-Id: Iee4c6f20bbf3f62f4175016076768e9a176b1b50 --- M pywikibot/comms/http.py M pywikibot/page/__init__.py M pywikibot/pagegenerators.py M pywikibot/proofreadpage.py M pywikibot/site_detect.py M pywikibot/specialbots/_upload.py M scripts/download_dump.py M scripts/reflinks.py M scripts/weblinkchecker.py M tests/aspects.py M tests/http_tests.py M tests/site_detect_tests.py M tests/site_tests.py 13 files changed, 79 insertions(+), 49 deletions(-) Approvals: DannyS712: Looks good to me, but someone else must approve Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/comms/http.py b/pywikibot/comms/http.py index 5807013..65cddd1 100644 --- a/pywikibot/comms/http.py +++ b/pywikibot/comms/http.py @@ -21,7 +21,7 @@ import sys from contextlib import suppress -from http import cookiejar +from http import cookiejar, HTTPStatus from string import Formatter from typing import Optional, Union from urllib.parse import quote, urlparse @@ -295,11 +295,11 @@ error('An error occurred for uri ' + response.request.url) raise response from None - if response.status_code == 504: + if response.status_code == HTTPStatus.GATEWAY_TIMEOUT: raise Server504Error('Server {} timed out' .format(urlparse(response.url).netloc)) - if response.status_code == 414: + if response.status_code == HTTPStatus.REQUEST_URI_TOO_LONG: raise Server414Error('Too long GET request') # TODO: shall it raise? this might break some code, TBC @@ -307,7 +307,7 @@ # HTTP status 207 is also a success status for Webdav FINDPROP, # used by the version module. - if response.status_code not in (200, 207): + if response.status_code not in (HTTPStatus.OK, HTTPStatus.MULTI_STATUS): warning('Http response status {}'.format(response.status_code)) diff --git a/pywikibot/page/__init__.py b/pywikibot/page/__init__.py index 17a2814..a5aa643 100644 --- a/pywikibot/page/__init__.py +++ b/pywikibot/page/__init__.py @@ -23,6 +23,7 @@ from collections import Counter, defaultdict, OrderedDict from contextlib import suppress from html.entities import name2codepoint +from http import HTTPStatus from itertools import chain from typing import Any, Optional, Union from urllib.parse import quote_from_bytes, unquote_to_bytes @@ -2458,7 +2459,7 @@ revision = self.latest_file_info req = http.fetch(revision.url, stream=True) - if req.status_code == 200: + if req.status_code == HTTPStatus.OK: try: with open(filename, 'wb') as f: for chunk in req.iter_content(chunk_size): diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py index fbc585f..ce0398d 100644 --- a/pywikibot/pagegenerators.py +++ b/pywikibot/pagegenerators.py @@ -29,6 +29,7 @@ from collections import namedtuple from datetime import timedelta from functools import partial +from http import HTTPStatus from itertools import zip_longest from requests.exceptions import ReadTimeout from typing import Optional, Union @@ -2967,11 +2968,13 @@ try: req = http.fetch(url, params=self.opts) except ReadTimeout: - raise ServerError( - 'received ReadTimeout from {}'.format(url)) - if 500 <= req.status_code < 600: + raise ServerError('received ReadTimeout from {}'.format(url)) + + server_err = HTTPStatus.INTERNAL_SERVER_ERROR + if server_err <= req.status_code < server_err + 100: raise ServerError( 'received {} status from {}'.format(req.status_code, req.url)) + j = json.loads(req.text) raw_pages = j['*'][0]['a']['*'] yield from raw_pages diff --git a/pywikibot/proofreadpage.py b/pywikibot/proofreadpage.py index 92ec48e..e8cb902 100644 --- a/pywikibot/proofreadpage.py +++ b/pywikibot/proofreadpage.py @@ -28,9 +28,11 @@ import time from functools import partial -from requests.exceptions import ReadTimeout +from http import HTTPStatus from typing import Optional +from requests.exceptions import ReadTimeout + try: from bs4 import BeautifulSoup except ImportError as e: @@ -618,7 +620,7 @@ else: return True, ReadTimeout - if 400 <= response.status_code < 600: + if HTTPStatus.BAD_REQUEST <= response.status_code < 600: return True, 'Http response status {}'.format(response.status_code) data = json.loads(response.text) diff --git a/pywikibot/site_detect.py b/pywikibot/site_detect.py index b509fb0..dcc6117 100644 --- a/pywikibot/site_detect.py +++ b/pywikibot/site_detect.py @@ -8,8 +8,10 @@ from contextlib import suppress from html.parser import HTMLParser +from http import HTTPStatus from typing import Optional from urllib.parse import urljoin, urlparse + from requests.exceptions import RequestException import pywikibot @@ -273,11 +275,9 @@ def check_response(response): """Raise ServerError if the response indicates a server error.""" - if response.status_code == 503: - raise ServerError('Service Unavailable') - if response.status_code == 502: - raise ServerError('Bad Gateway') - if response.status_code == 500: - raise ServerError('Internal Server Error') - if response.status_code == 200 and SERVER_DB_ERROR_MSG in response.text: + if response.status_code >= HTTPStatus.INTERNAL_SERVER_ERROR: + raise ServerError(HTTPStatus(response.status_code).phrase) + + if response.status_code == HTTPStatus.OK \ + and SERVER_DB_ERROR_MSG in response.text: raise ServerError('Server cannot access the database') diff --git a/pywikibot/specialbots/_upload.py b/pywikibot/specialbots/_upload.py index 5f76d8f..8b1f49a 100644 --- a/pywikibot/specialbots/_upload.py +++ b/pywikibot/specialbots/_upload.py @@ -13,6 +13,7 @@ import tempfile from contextlib import suppress +from http import HTTPStatus from pathlib import Path from typing import Optional, Union from urllib.parse import urlparse @@ -158,7 +159,8 @@ # exit criteria if size is not available # error on last iteration is OK, we're requesting # {'Range': 'bytes=file_len-'} - if response.status_code == 416 and path.stat().st_size: + err = HTTPStatus.REQUESTED_RANGE_NOT_SATISFIABLE + if response.status_code == err and path.stat().st_size: break raise FatalServerError(str(e)) from e diff --git a/scripts/download_dump.py b/scripts/download_dump.py index 4f38605..9ba08e7 100755 --- a/scripts/download_dump.py +++ b/scripts/download_dump.py @@ -20,6 +20,7 @@ import binascii import os.path +from http import HTTPStatus from os import remove, replace, symlink, urandom import pywikibot @@ -108,13 +109,16 @@ pywikibot.output('Downloading file from ' + url) response = fetch(url, stream=True) - if response.status_code != 200: - if response.status_code == 404: + if response.status_code != HTTPStatus.OK: + if response.status_code == HTTPStatus.NOT_FOUND: pywikibot.output( 'File with name {filename!r}, from dumpdate ' '{dumpdate!r}, and wiki {wikiname!r} ({url}) ' "isn't available in the Wikimedia Dumps" .format(url=url, **self.opt)) + else: + pywikibot.output( + HTTPStatus(response.status_code).description) return with open(file_current_storepath, 'wb') as result_file: diff --git a/scripts/reflinks.py b/scripts/reflinks.py index 559da2e..9c816f4 100755 --- a/scripts/reflinks.py +++ b/scripts/reflinks.py @@ -49,11 +49,10 @@ from contextlib import suppress from functools import partial +from http import HTTPStatus from textwrap import shorten from urllib.error import URLError -from requests import codes - import pywikibot from pywikibot import comms, i18n, pagegenerators, textlib @@ -584,16 +583,15 @@ 'Redirect to root : {0} ', ref.link)) continue - if r.status_code != codes.ok: + if r.status_code != HTTPStatus.OK: pywikibot.stdout('HTTP error ({}) for {} on {}' .format(r.status_code, ref.url, page.title(as_link=True))) # 410 Gone, indicates that the resource has been # purposely removed - if r.status_code == 410 \ - or (r.status_code == 404 - and '\t{}\t'.format( - ref.url) in self.dead_links): + if r.status_code == HTTPStatus.GONE \ + or (r.status_code == HTTPStatus.NOT_FOUND + and '\t{}\t'.format(ref.url) in self.dead_links): repl = ref.refDead() new_text = new_text.replace(match.group(), repl) continue diff --git a/scripts/weblinkchecker.py b/scripts/weblinkchecker.py index 192fc9f..b2b42cb 100755 --- a/scripts/weblinkchecker.py +++ b/scripts/weblinkchecker.py @@ -117,6 +117,7 @@ from contextlib import suppress from functools import partial +from http import HTTPStatus import requests @@ -321,9 +322,8 @@ .format(self.url, self.page.title())) raise - if r.status_code != requests.codes.ok \ - or r.status_code in self.HTTPignore: - message = str(r.status_code) + if r.status_code != HTTPStatus.OK or r.status_code in self.HTTPignore: + message = HTTPStatus(r.status_code).phrase pywikibot.output('*{} links to {} - {}.' .format(self.page.title(as_link=True), self.url, message)) diff --git a/tests/aspects.py b/tests/aspects.py index 18caee2..87da3bb 100644 --- a/tests/aspects.py +++ b/tests/aspects.py @@ -20,6 +20,7 @@ from contextlib import contextmanager, suppress from collections.abc import Sized +from http import HTTPStatus from unittest.util import safe_repr import pywikibot @@ -451,8 +452,15 @@ r = http.fetch(hostname, method='HEAD', default_error_handling=False) - if r.status_code not in {200, 301, 302, 303, 307, 308}: - raise ServerError('HTTP status: {}'.format(r.status_code)) + if r.status_code not in {HTTPStatus.OK, + HTTPStatus.MOVED_PERMANENTLY, + HTTPStatus.FOUND, + HTTPStatus.SEE_OTHER, + HTTPStatus.TEMPORARY_REDIRECT, + HTTPStatus.PERMANENT_REDIRECT}: + raise ServerError( + 'HTTP status: {} - {}'.format( + r.status_code, HTTPStatus(r.status_code).phrase)) except Exception as e: pywikibot.error('{}: accessing {} caused exception:' .format(cls.__name__, hostname)) diff --git a/tests/http_tests.py b/tests/http_tests.py index aa2fa67..5656c5d 100644 --- a/tests/http_tests.py +++ b/tests/http_tests.py @@ -9,6 +9,7 @@ import warnings from contextlib import suppress +from http import HTTPStatus import requests @@ -42,7 +43,7 @@ """Test http.fetch using http://www.wikipedia.org/.""" r = http.fetch('http://www.wikipedia.org/') self.assertIsInstance(r, requests.Response) - self.assertEqual(r.status_code, 200) + self.assertEqual(r.status_code, HTTPStatus.OK.value) self.assertIn('<html lang="en"', r.text) self.assertIsInstance(r.text, str) self.assertIsInstance(r.content, bytes) @@ -167,12 +168,12 @@ """Test follow 301 redirects correctly.""" # The following will redirect from ' ' -> '_', and maybe to https:// r = http.fetch('http://en.wikipedia.org/wiki/Main%20Page') - self.assertEqual(r.status_code, 200) + self.assertEqual(r.status_code, HTTPStatus.OK.value) self.assertIsNotNone(r.history) self.assertIn('//en.wikipedia.org/wiki/Main_Page', r.url) r = http.fetch('http://en.wikia.com') - self.assertEqual(r.status_code, 200) + self.assertEqual(r.status_code, HTTPStatus.OK.value) self.assertEqual(r.url, 'https://community.fandom.com/wiki/Community_Central') @@ -555,10 +556,13 @@ def test_no_params(self): """Test fetch method with no parameters.""" r = http.fetch(self.url, params={}) - if r.status_code == 503: # T203637 - self.skipTest( - '503: Service currently not available for ' + self.url) - self.assertEqual(r.status_code, 200) + + fail_status = HTTPStatus.SERVICE_UNAVAILABLE + if r.status_code == fail_status: # T203637 + self.skipTest('{status.value}: {status.description} for {url}' + .format(status=fail_status, url=self.url)) + + self.assertEqual(r.status_code, HTTPStatus.OK) content = json.loads(r.text) self.assertDictEqual(content['args'], {}) @@ -571,10 +575,13 @@ should be the same as what we get out. """ r = http.fetch(self.url, params={'fish&chips': 'delicious'}) - if r.status_code == 503: # T203637 - self.skipTest( - '503: Service currently not available for ' + self.url) - self.assertEqual(r.status_code, 200) + + fail_status = HTTPStatus.SERVICE_UNAVAILABLE + if r.status_code == fail_status: # T203637 + self.skipTest('{status.value}: {status.description} for {url}' + .format(status=fail_status, url=self.url)) + + self.assertEqual(r.status_code, HTTPStatus.OK) content = json.loads(r.text) self.assertDictEqual(content['args'], {'fish&chips': 'delicious'}) @@ -587,10 +594,13 @@ should be the same as what we get out. """ r = http.fetch(self.url, params={'fish%26chips': 'delicious'}) - if r.status_code == 503: # T203637 - self.skipTest( - '503: Service currently not available for ' + self.url) - self.assertEqual(r.status_code, 200) + + fail_status = HTTPStatus.SERVICE_UNAVAILABLE + if r.status_code == fail_status: # T203637 + self.skipTest('{status.value}: {status.description} for {url}' + .format(status=fail_status, url=self.url)) + + self.assertEqual(r.status_code, HTTPStatus.OK) content = json.loads(r.text) self.assertDictEqual(content['args'], {'fish%26chips': 'delicious'}) diff --git a/tests/site_detect_tests.py b/tests/site_detect_tests.py index 813e890..807d105 100644 --- a/tests/site_detect_tests.py +++ b/tests/site_detect_tests.py @@ -7,6 +7,7 @@ import unittest from contextlib import suppress +from http import HTTPStatus from urllib.parse import urlparse from requests.exceptions import ConnectionError, Timeout @@ -290,7 +291,7 @@ return type(str('Response'), (object,), - {'status_code': 200, + {'status_code': HTTPStatus.OK.value, 'text': self._responses[parsed_url.path], 'url': url}) diff --git a/tests/site_tests.py b/tests/site_tests.py index 677d1a4..877445d 100644 --- a/tests/site_tests.py +++ b/tests/site_tests.py @@ -12,6 +12,7 @@ from collections.abc import Iterable, Mapping from contextlib import suppress +from http import HTTPStatus import pywikibot @@ -3287,7 +3288,7 @@ self.assertEqual(site.hostname(), 'mh.wikipedia.org') r = http.fetch('http://mh.wikipedia.org/w/api.php', default_error_handling=False) - self.assertEqual(r.status_code, 200) + self.assertEqual(r.status_code, HTTPStatus.OK.value) self.assertEqual(site.siteinfo['lang'], 'mh') def test_removed_site(self): -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/678395 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: Iee4c6f20bbf3f62f4175016076768e9a176b1b50 Gerrit-Change-Number: 678395 Gerrit-PatchSet: 4 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com> Gerrit-Reviewer: DannyS712 <dannys712.wiki(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot Gerrit-CC: Zhuyifei1999 <zhuyifei1999(a)gmail.com> Gerrit-MessageType: merged

3 years

[Gerrit] ...core[master]: [cleanup] Show a FutureWarnig with not qualified string passed to che...

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/677477 ) Change subject: [cleanup] Show a FutureWarnig with not qualified string passed to checkstr ...................................................................... [cleanup] Show a FutureWarnig with not qualified string passed to checkstr Change-Id: I44d95ed4495bffe46f7e4753337f0b55ad3badec --- M scripts/archivebot.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: JJMC89: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/archivebot.py b/scripts/archivebot.py index 8c947d6..e103d3d 100755 --- a/scripts/archivebot.py +++ b/scripts/archivebot.py @@ -238,7 +238,7 @@ key = 's' duration = string issue_deprecation_warning('Time period without qualifier', - string + key, 1, UserWarning, + string + key, 1, FutureWarning, since='20161009') else: key = string[-1] -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/677477 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: I44d95ed4495bffe46f7e4753337f0b55ad3badec Gerrit-Change-Number: 677477 Gerrit-PatchSet: 1 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com> Gerrit-Reviewer: JJMC89 <JJMC89.Wikimedia(a)gmail.com> Gerrit-Reviewer: jenkins-bot Gerrit-MessageType: merged

3 years

[Gerrit] ...core[master]: Use Special:MyLanguage for the code of conduct link

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/678699 ) Change subject: Use Special:MyLanguage for the code of conduct link ...................................................................... Use Special:MyLanguage for the code of conduct link LibUp adds it automatically for php repos, but it should be added here too Change-Id: I2dccfc13206bb826debb941c560e43c41770eec5 --- M CODE_OF_CONDUCT.rst 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Zhuyifei1999: Looks good to me, approved jenkins-bot: Verified diff --git a/CODE_OF_CONDUCT.rst b/CODE_OF_CONDUCT.rst index 3615c22..da720ee 100644 --- a/CODE_OF_CONDUCT.rst +++ b/CODE_OF_CONDUCT.rst @@ -2,5 +2,5 @@ --------------- The development of this software is covered by a -`Code of Conduct <https://www.mediawiki.org/wiki/Code_of_Conduct>`_. +`Code of Conduct <https://www.mediawiki.org/wiki/Special:MyLanguage/Code_of_Conduct>`_. -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/678699 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: I2dccfc13206bb826debb941c560e43c41770eec5 Gerrit-Change-Number: 678699 Gerrit-PatchSet: 2 Gerrit-Owner: DannyS712 <dannys712.wiki(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zhuyifei1999 <zhuyifei1999(a)gmail.com> Gerrit-Reviewer: jenkins-bot Gerrit-MessageType: merged

3 years

[Gerrit] ...core[master]: [doc] Update documentation

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/678705 ) Change subject: [doc] Update documentation ...................................................................... [doc] Update documentation * Update file extensions for files ** Doc not updated when reStructuredText was used Change-Id: I4fc5c3b6bf3dbc02662c89e29eca0a85b4be87b6 --- M CONTENT.rst 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/CONTENT.rst b/CONTENT.rst index e96efb5..f9a1b5c 100644 --- a/CONTENT.rst +++ b/CONTENT.rst @@ -4,11 +4,11 @@ +---------------------------------------------------------------------------------------+ | README and config files: | +===========================+===========================================================+ - | CODE_OF_CONDUCT.md | Code of conduct reference | + | CODE_OF_CONDUCT.rst | Code of conduct reference | +---------------------------+-----------------------------------------------------------+ | CONTENT.rst | This Content description file | +---------------------------+-----------------------------------------------------------+ - | CREDITS | List of major contributors to this module | + | CREDITS.rst | List of major contributors to this module | +---------------------------+-----------------------------------------------------------+ | dev-requirements.txt | PIP requirements file for development dependencies | +---------------------------+-----------------------------------------------------------+ -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/678705 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: I4fc5c3b6bf3dbc02662c89e29eca0a85b4be87b6 Gerrit-Change-Number: 678705 Gerrit-PatchSet: 3 Gerrit-Owner: Meno25 <meno25mail(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot Gerrit-MessageType: merged

3 years

[Gerrit] ...core[master]: [parser] Make mwparserfromhell or wikitextparser mandatory

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/675170 ) Change subject: [parser] Make mwparserfromhell or wikitextparser mandatory ...................................................................... [parser] Make mwparserfromhell or wikitextparser mandatory - check whether mwparserfromhell or wikitextparser is installed and raise ImportError if no package is available - catch the ImportError in pwb wrapper like for other dependency checks - deprecate extract_templates_and_params_regex - update tests and remove TestGenericTemplateParams because dispatcher was removed - add wikitextparser to setup.py extra_deps - update tox.ini - update some documentation hints Bug: T106763 Change-Id: Ie35c6399616d3fbe660b3387f16146d939fd51b7 --- M README.rst M pwb.py M pywikibot/textlib.py M requirements.txt M setup.py M tests/__init__.py M tests/textlib_tests.py M tox.ini 8 files changed, 100 insertions(+), 165 deletions(-) Approvals: JJMC89: Looks good to me, approved jenkins-bot: Verified diff --git a/README.rst b/README.rst index 6b1a91f..8872f7d 100644 --- a/README.rst +++ b/README.rst @@ -44,6 +44,7 @@ :: + pip install requests git clone https://gerrit.wikimedia.org/r/pywikibot/core.git cd core git submodule update --init @@ -56,6 +57,18 @@ pip install -U setuptools pip install pywikibot +In addition a MediaWiki markup parser is required. Please install one of them: + +:: + + pip install mwparserfromhell + +or + +:: + + pip install wikitextparser + Our `installation guide <https://www.mediawiki.org/wiki/Manual:Pywikibot/Installation>`_ has more details for advanced usage. diff --git a/pwb.py b/pwb.py index 80bbc7e..95abb09 100755 --- a/pwb.py +++ b/pwb.py @@ -14,7 +14,7 @@ python pwb.py -lang:de bot_tests -v """ -# (C) Pywikibot team, 2012-2020 +# (C) Pywikibot team, 2012-2021 # # Distributed under the terms of the MIT license. # @@ -182,9 +182,8 @@ try: if not check_modules(): raise RuntimeError('') # no further output needed -except RuntimeError as e: - print(e) - sys.exit() +except RuntimeError as e: # setup.py may also raise RuntimeError + sys.exit(e) from pathlib import Path # noqa: E402 @@ -213,6 +212,8 @@ # we need to re-start the entire process. Ask the user to do so. print('Now, you have to re-execute the command to start your script.') sys.exit(1) +except ImportError as e: # raised in textlib + sys.exit(e) def find_alternates(filename, script_paths): diff --git a/pywikibot/textlib.py b/pywikibot/textlib.py index 2ee62d1..8807069 100644 --- a/pywikibot/textlib.py +++ b/pywikibot/textlib.py @@ -35,8 +35,18 @@ except ImportError: try: import mwparserfromhell as wikitextparser - except ImportError as e: - wikitextparser = e + except ImportError: + # print required because pywikibot is not imported completely + raise ImportError(""" +Pywikibot is missing a MediaWiki markup parser which is necessary. +Please update the required module with either + + pip install "mwparserfromhell>=0.5.0" + +or + + pip install "wikitextparser>=0.47.0" +""") from None ETPType = List[Tuple[str, OrderedDictType[str, str]]] @@ -1570,8 +1580,8 @@ # -------------------------------- def extract_templates_and_params(text: str, - remove_disabled_parts: Optional[bool] = None, - strip: Optional[bool] = None) -> ETPType: + remove_disabled_parts: bool = False, + strip: bool = False) -> ETPType: """Return a list of templates found in text. Return value is a list of tuples. There is one tuple for each use of a @@ -1582,16 +1592,14 @@ parameters, and if this results multiple parameters with the same name only the last value provided will be returned. - This uses the package L{mwparserfromhell} (mwpfh) if it is installed. - Otherwise it falls back on a regex based implementation. + This uses the package L{mwparserfromhell} or L{wikitextparser} as + MediaWiki markup parser. It is mandatory that one of them is + installed. There are minor differences between the two implementations. - The two implementations return nested templates in a different order. - i.e. for {{a|b={{c}}}}, mwpfh returns [a, c], whereas regex returns [c, a]. - - mwpfh preserves whitespace in parameter names and values. regex excludes - anything between  before parsing the text. + The parser packages preserves whitespace in parameter names and + values. If there are multiple numbered parameters in the wikitext for the same position, MediaWiki will only use the last parameter value. @@ -1599,43 +1607,9 @@ To replicate that behaviour, enable both remove_disabled_parts and strip. @param text: The wikitext from which templates are extracted - @param remove_disabled_parts: Remove disabled wikitext such as comments - and pre. If None (default), this is enabled when neither - mwparserfromhell not wikitextparser package is available and - disabled otherwise. - @param strip: if enabled, strip arguments and values of templates. - If None (default), this is enabled when neither mwparserfromhell - nor wikitextparser package is available and disabled otherwise. - @return: list of template name and params - """ - use_regex = isinstance(wikitextparser, ImportError) - - if remove_disabled_parts is None: - remove_disabled_parts = use_regex - if remove_disabled_parts: - text = removeDisabledParts(text) - - if strip is None: - strip = use_regex - - if use_regex: - return extract_templates_and_params_regex(text, False, strip) - return _extract_templates_and_params_parser(text, strip) - - -def _extract_templates_and_params_parser(text: str, - strip: bool = False) -> ETPType: - """ - Extract templates with params using mwparserfromhell. - - This function should not be called directly. - - Use extract_templates_and_params, which will select this parser - implementation if the mwparserfromhell or wikitextparser package is - installed. - - @param text: The wikitext from which templates are extracted - @param strip: if enabled, strip arguments and values of templates + @param remove_disabled_parts: If enabled, remove disabled wikitext + such as comments and pre. + @param strip: If enabled, strip arguments and values of templates. @return: list of template name and params """ def explicit(param): @@ -1645,6 +1619,9 @@ attr = not param.positional return attr + if remove_disabled_parts: + text = removeDisabledParts(text) + parser_name = wikitextparser.__name__ pywikibot.log('Using {!r} wikitext parser'.format(parser_name)) @@ -1683,20 +1660,21 @@ future_warning=True) def extract_templates_and_params_mwpfh(text: str, strip: bool = False) -> ETPType: - """Extract templates with params using mwparserfromhell.""" + """DEPRECATED. Extract templates with params using mwparserfromhell.""" global wikitextparser saved_parser = wikitextparser import mwparserfromhell as wikitextparser - result = _extract_templates_and_params_parser(text, strip) + result = extract_templates_and_params(text, strip=strip) wikitextparser = saved_parser return result +@deprecated('extract_templates_and_params', since='20210331', + future_warning=True) def extract_templates_and_params_regex(text: str, remove_disabled_parts: bool = True, strip: bool = True) -> ETPType: - """ - Extract templates with params using a regex with additional processing. + """DEPRECATED. Extract templates with params using a regex. This function should not be called directly. diff --git a/requirements.txt b/requirements.txt index 8b64aec..2a6cbc4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6,7 +6,6 @@ # It is organised so that simple requirements # are processed first, and more difficult packages # are last. -# All dependencies other than requests are optional. # # It is good practise to install packages using the system # package manager if it has a packaged version. If you are @@ -18,12 +17,18 @@ # or # $ awk -F '[#>=]' '{print $1}' requirements.txt | xargs apt-cache search -# mandatory +# mandatory dependencies, others are optional requests>=2.20.1, < 2.26.0; python_version < '3.6' requests>=2.20.1 ; python_version >= '3.6' setuptools>=20.2, !=50.0.0, <50.2.0 ; python_version < '3.6' setuptools>=20.2 ; python_version >= '3.6' +# MediaWiki markup parser +# mwparserfromhell is default, wikitextparser can be used instead +# mwparserfromhell is still required for commons_information.py and patrol.py +# wikitextparser>=0.47.0 +mwparserfromhell>=0.5.0 + # OAuth support # mwoauth 0.2.4 is needed because it supports getting identity information # about the user @@ -42,9 +47,6 @@ google >= 1.7 sseclient >= 0.0.18,!=0.0.23,!=0.0.24 -# textlib.py, commons_information and patrol.py -mwparserfromhell>=0.5.0 - # The mysql generator in pagegenerators depends on PyMySQL PyMySQL >= 0.6.7, < 1.0.0 ; python_version < '3.6' PyMySQL >= 1.0.0 ; python_version >= '3.6' diff --git a/setup.py b/setup.py index 6b34da3..2ae5ca7 100644 --- a/setup.py +++ b/setup.py @@ -62,6 +62,7 @@ 'Graphviz': ['pydot>=1.2'], 'Google': ['google>=1.7'], 'mwparserfromhell': ['mwparserfromhell>=0.5.0'], + 'wikitextparser': ['wikitextparser>=0.47.0'], 'Tkinter': [ # vulnerability found in Pillow<8.1.1 'Pillow>=8.1.1;python_version>="3.6"', ], @@ -110,7 +111,7 @@ 'setuptools>=20.2, !=50.0.0, <50.2.0 ; python_version < "3.6"', 'setuptools>=20.2 ; python_version >= "3.6"', ] - +# in addition either mwparserfromhell or wikitextparser is required # ------- setup tests_require ------- # test_deps = ['mock'] diff --git a/tests/__init__.py b/tests/__init__.py index 04f61c3..596f1ce 100644 --- a/tests/__init__.py +++ b/tests/__init__.py @@ -24,7 +24,8 @@ # Verify that the unit tests have a base working environment: # - requests is mandatory # however if unavailable this will fail on use; see pywikibot/tools.py -# - mwparserfromhell is optional, so is only imported in textlib_tests +# - mwparserfromhell or wikitextparser is mandatory but the dependency +# is checked by textlib already import requests # noqa: F401 import pywikibot.data.api diff --git a/tests/textlib_tests.py b/tests/textlib_tests.py index dadc2a3..aed174f 100644 --- a/tests/textlib_tests.py +++ b/tests/textlib_tests.py @@ -23,8 +23,10 @@ from pywikibot import UnknownSite from tests.aspects import ( - require_modules, TestCase, DefaultDrySiteTestCase, - PatchingTestCase, SiteAttributeTestCase, + DefaultDrySiteTestCase, + require_modules, + SiteAttributeTestCase, + TestCase, ) from tests import mock @@ -302,7 +304,7 @@ 'Invalid category title extracted: nasty{{{!}}') -WARNING_MSG = (r'.*extract_templates_and_params_mwpfh .*' +WARNING_MSG = (r'.*extract_templates_and_params_.*' r'is deprecated for .*; use extract_templates_and_params') @@ -502,7 +504,7 @@ @require_modules('mwparserfromhell') def test_extract_templates_params_parser_stripped(self): """Test using mwparserfromhell with stripping.""" - func = functools.partial(textlib._extract_templates_and_params_parser, + func = functools.partial(textlib.extract_templates_and_params, strip=True) self._common_results(func) @@ -535,37 +537,39 @@ """Test using many complex regexes.""" func = functools.partial(textlib.extract_templates_and_params_regex, remove_disabled_parts=False, strip=False) - self._common_results(func) - self._order_differs(func) - self._unstripped(func) - - self.assertEqual(func('{{a|b={} }}'), []) # FIXME: {} is normal text + with suppress_warnings(WARNING_MSG, category=FutureWarning): + self._common_results(func) + self._order_differs(func) + self._unstripped(func) + # FIXME: {} is normal text + self.assertEqual(func('{{a|b={} }}'), []) def test_extract_templates_params_regex_stripped(self): """Test using many complex regexes with stripping.""" func = textlib.extract_templates_and_params_regex + with suppress_warnings(WARNING_MSG, category=FutureWarning): + self._common_results(func) + self._order_differs(func) + self._stripped(func) - self._common_results(func) - self._order_differs(func) - self._stripped(func) + self.assertEqual(func('{{a|b=}}'), + [('a', OrderedDict((('b', ''), )))]) - self.assertEqual(func('{{a|b=}}'), - [('a', OrderedDict((('b', ''), )))]) + # Identical to mwpfh + self.assertCountEqual(func('{{a|{{c|{{d}}}}}}'), + [('c', OrderedDict((('1', '{{d}}'), ))), + ('a', OrderedDict([('1', '{{c|{{d}}}}')])), + ('d', OrderedDict()) + ]) - # Identical to mwpfh - self.assertCountEqual(func('{{a|{{c|{{d}}}}}}'), - [('c', OrderedDict((('1', '{{d}}'), ))), - ('a', OrderedDict([('1', '{{c|{{d}}}}')])), - ('d', OrderedDict()) - ]) + # However fails to correctly handle three levels of balanced + # brackets with empty parameters + self.assertCountEqual(func('{{a|{{c|{{d|}}}}}}'), + [('c', OrderedDict((('1', '{{d|}}}'), ))), + ('d', OrderedDict([('1', '}')])) + ]) - # However fails to correctly handle three levels of balanced brackets - # with empty parameters - self.assertCountEqual(func('{{a|{{c|{{d|}}}}}}'), - [('c', OrderedDict((('1', '{{d|}}}'), ))), - ('d', OrderedDict([('1', '}')])) - ]) - + @require_modules('mwparserfromhell') def test_extract_templates_params(self): """Test that the normal entry point works.""" func = functools.partial(textlib.extract_templates_and_params, @@ -699,79 +703,6 @@ self.assertTrue(m.group(0).endswith('foo {{bar}}')) -class TestGenericTemplateParams(PatchingTestCase): - - """Test whether the generic function forwards the call correctly.""" - - net = False - - @PatchingTestCase.patched(textlib, '_extract_templates_and_params_parser') - def extract_mwpfh(self, text, *args, **kwargs): - """Patched call to extract_templates_and_params_mwpfh.""" - self._text = text - self._args = args - self._mwpfh = True - - @PatchingTestCase.patched(textlib, 'extract_templates_and_params_regex') - def extract_regex(self, text, *args, **kwargs): - """Patched call to extract_templates_and_params_regex.""" - self._text = text - self._args = args - self._mwpfh = False - - def test_removing_disabled_parts_regex(self): - """Test removing disabled parts when using the regex variant.""" - self.patch(textlib, 'wikitextparser', ImportError()) - textlib.extract_templates_and_params('{{a}}', True) - self.assertEqual(self._text, '{{a}}') - self.assertFalse(self._mwpfh) - textlib.extract_templates_and_params('{{a}}', False) - self.assertEqual(self._text, '{{a}}') - self.assertFalse(self._mwpfh) - textlib.extract_templates_and_params('{{a}}') - self.assertEqual(self._text, '{{a}}') - self.assertFalse(self._mwpfh) - - @require_modules('mwparserfromhell') - def test_removing_disabled_parts_mwpfh(self): - """Test removing disabled parts when using the mwpfh variant.""" - textlib.extract_templates_and_params('{{a}}', True) - self.assertEqual(self._text, '{{a}}') - self.assertTrue(self._mwpfh) - textlib.extract_templates_and_params('{{a}}', False) - self.assertEqual(self._text, '{{a}}') - self.assertTrue(self._mwpfh) - textlib.extract_templates_and_params('{{a}}') - self.assertEqual(self._text, '{{a}}') - self.assertTrue(self._mwpfh) - - def test_strip_regex(self): - """Test stripping values when using the regex variant.""" - self.patch(textlib, 'wikitextparser', ImportError()) - textlib.extract_templates_and_params('{{a| foo }}', False, True) - self.assertEqual(self._args, (False, True)) - self.assertFalse(self._mwpfh) - textlib.extract_templates_and_params('{{a| foo }}', False, False) - self.assertEqual(self._args, (False, False)) - self.assertFalse(self._mwpfh) - textlib.extract_templates_and_params('{{a| foo }}', False) - self.assertEqual(self._args, (False, True)) - self.assertFalse(self._mwpfh) - - @require_modules('mwparserfromhell') - def test_strip_mwpfh(self): - """Test stripping values when using the mwpfh variant.""" - textlib.extract_templates_and_params('{{a| foo }}', None, True) - self.assertEqual(self._args, (True, )) - self.assertTrue(self._mwpfh) - textlib.extract_templates_and_params('{{a| foo }}', None, False) - self.assertEqual(self._args, (False, )) - self.assertTrue(self._mwpfh) - textlib.extract_templates_and_params('{{a| foo }}') - self.assertEqual(self._args, (False, )) - self.assertTrue(self._mwpfh) - - class TestDisabledParts(DefaultDrySiteTestCase): """Test the removeDisabledParts function in textlib.""" diff --git a/tox.ini b/tox.ini index d68e21d..cb76128 100644 --- a/tox.ini +++ b/tox.ini @@ -38,10 +38,17 @@ fasttest: pytest-attrib>=0.1.3 fasttest: pytest-subtests >= 0.3.2 fasttest: mock + fasttest: .[mwparserfromhell] fasttest: .[scripts] - fasttest-py35: mwparserfromhell - fasttest-py35: beautifulsoup4 + fasttest-py35: .[html] + fasttest-py37: .[wikitextparser] + + deeptest: .[html] + deeptest: .[mwparserfromhell] + deeptest: .[scripts] + deeptest: .[wikitextparser] + [testenv:commit-message] basepython = python3 @@ -56,6 +63,7 @@ nosetests --with-doctest pywikibot {[params]doctest_skip} deps = nose + .[mwparserfromhell] [testenv:venv] commands = {posargs} -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/675170 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: Ie35c6399616d3fbe660b3387f16146d939fd51b7 Gerrit-Change-Number: 675170 Gerrit-PatchSet: 13 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com> Gerrit-Reviewer: DannyS712 <dannys712.wiki(a)gmail.com> Gerrit-Reviewer: JJMC89 <JJMC89.Wikimedia(a)gmail.com> Gerrit-Reviewer: jenkins-bot Gerrit-CC: Dalba <dalba.wiki(a)gmail.com> Gerrit-CC: Dvorapa <dvorapa(a)seznam.cz> Gerrit-MessageType: merged

3 years

← Newer
1
2
3
4
5
6
7
8
9
...
14
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits April 2021