Pywikibot-commits

pywikibot-commits@lists.wikimedia.org

1 participants
13430 discussions

[Gerrit] ...core[master]: [doc] Update ROADMAP.rst, CHANGELOG.rst and some code documentations
by jenkins-bot (Code Review) 22 Jun '24

22 Jun '24

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1048832?usp=email ) Change subject: [doc] Update ROADMAP.rst, CHANGELOG.rst and some code documentations ...................................................................... [doc] Update ROADMAP.rst, CHANGELOG.rst and some code documentations Change-Id: If46121fb857cc49cc8521947f6fadecccdcb5ee3 --- M ROADMAP.rst M docs/api_ref/pywikibot.data.rst M pywikibot/data/superset.py M pywikibot/pagegenerators/_generators.py M scripts/CHANGELOG.rst M tests/superset_tests.py 6 files changed, 64 insertions(+), 62 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/ROADMAP.rst b/ROADMAP.rst index b244348..b956e38 100644 --- a/ROADMAP.rst +++ b/ROADMAP.rst @@ -1,6 +1,12 @@ Current Release Changes ======================= +* Add support for ``btmwiki`` to Pywikibot (:phab:`T368069`) +* Include image repository extensions in :attr:`site.APISite.file_extensions + <pywikibot.site._apisite.APISite.file_extensions>` +* Ignore :exc:`ValueError` durig upcast of :class:`FilePage<pywikibot.page.FilePage>` due to invalid file extension + (:phab:`T367777`) +* Add :func:`pagegenerators.SupersetPageGenerator` pagegenerator (:phab:`T367684`) * No longer wait in :meth:`data.api.Request._http_request` for ``ImportError`` and ``NameError`` * Replace ``requests.utils.urlparse`` with ``urllib.parse.urlparse`` in :func:`comms.http.get_authentication` (:phab:`T367649`) diff --git a/docs/api_ref/pywikibot.data.rst b/docs/api_ref/pywikibot.data.rst index cd4f8c3..0caf1b9 100644 --- a/docs/api_ref/pywikibot.data.rst +++ b/docs/api_ref/pywikibot.data.rst @@ -29,6 +29,12 @@ .. automodule:: data.sparql :synopsis: SPARQL Query interface +:mod:`data.superset` --- Superset requests +========================================== + +.. automodule:: data.superset + :synopsis: Superset Query interface + :mod:`data.wikistats` --- WikiStats requests ============================================ diff --git a/pywikibot/data/superset.py b/pywikibot/data/superset.py index 85aebc0..2da5a10 100644 --- a/pywikibot/data/superset.py +++ b/pywikibot/data/superset.py @@ -1,10 +1,16 @@ -"""Superset Query interface.""" +"""Superset Query interface. + +.. versionadded:: 9.2 +""" # # (C) Pywikibot team, 2024 # # Distributed under the terms of the MIT license. # +from __future__ import annotations + from textwrap import fill +from typing import TYPE_CHECKING, Any import pywikibot from pywikibot.comms import http @@ -12,6 +18,10 @@ from pywikibot.exceptions import NoUsernameError, ServerError +if TYPE_CHECKING: + from pywikibot.site import BaseSite + + class SupersetQuery(WaitingMixin): """Superset Query class. @@ -19,27 +29,21 @@ service. """ - def __init__(self, schema_name=None, - site=None, database_id=None): - """ - Create superset endpoint with initial defaults. + def __init__(self, + schema_name: str | None = None, + site: BaseSite | None = None, + database_id: int | None = None) -> None: + """Create superset endpoint with initial defaults. Either site OR schema_name is required. Site and schema_name are mutually exclusive. Database id will be retrieved automatically if needed. :param site: The mediawiki site to be queried - :type site: pywikibot.Site, optional - - :param schema_name: superset database schema name. - Example value "enwiki_p" - :type schema_name: str, optional - + :param schema_name: superset database schema name. Example value + "enwiki_p" :param database_id: superset database id. - :type database_id: int, optional - :raises TypeError: if site and schema_name are both defined' - """ if site and schema_name: msg = 'Only one of schema_name and site parameters can be defined' @@ -59,20 +63,16 @@ self.last_response = None self.superset_url = 'https://superset.wmcloud.org' - def login(self): - """ - Login to superset. + def login(self) -> bool: + """Login to superset. - Function logins first to meta.wikimedia.org - and then OAUTH login to superset.wmcloud.org. - Working login expects that the user has manually - permitted the username to login to the superset. + Function logins first to meta.wikimedia.org and then OAUTH login + to superset.wmcloud.org. Working login expects that the user has + manually permitted the username to login to the superset. :raises NoUsernameError: if not not logged in. :raises ServerError: For other errors - :return: True if user has been logged to superset - :rtype bool """ # superset uses meta for OAUTH authentication loginsite = pywikibot.Site('meta') @@ -108,16 +108,14 @@ return self.connected - def get_csrf_token(self): - """ - Get superset CSRF token. + def get_csrf_token(self) -> str: + """Get superset CSRF token. - Method retrieves a CSRF token from the Superset service. - If the instance is not connected, it attempts to log in first. + Method retrieves a CSRF token from the Superset service. If the + instance is not connected, it attempts to log in first. :raises ServerError: For any http errors :return CSRF token string - :rtype str """ if not self.connected: self.login() @@ -132,19 +130,14 @@ status_code = self.last_response.status_code raise ServerError(f'CSRF token error: {status_code}') - def get_database_id_by_schema_name(self, schema_name): - """ - Get superset database_id using superset schema name. + def get_database_id_by_schema_name(self, schema_name: str) -> int: + """Get superset database_id using superset schema name. - :param schema_name: superset database schema name. - Example value "enwiki_p" - :type schema_name: str - + :param schema_name: superset database schema name. Example value + "enwiki_p" :raises KeyError: If the database ID could found. :raises ServerError: For any other http errors - :return: database id - :rtype: int """ if not self.connected: self.login() @@ -169,27 +162,18 @@ raise KeyError(f'Schema "{schema_name}" not found in {url}.') def merge_query_arguments(self, - database_id=None, - schema_name=None, - site=None): - """ - Determine and validate the database_id and schema_name. + database_id: int | None = None, + schema_name: str | None = None, + site: BaseSite = None) -> tuple(int, str): + """Determine and validate the database_id and schema_name. :param database_id: The superset database ID. - :type database_id: int, optional - :param schema_name: The superset schema name. - :type schema_name: str, optional - :param site: The target site - :type site: pywikibot.Site, optional - :raises TypeError: if site and schema_name are both defined' :raises TypeError: If determined database_id is not an integer. :raises TypeError: If neither site nor schema_name is determined. - :return A tuple containing database_id and schema_name. - :rtype: tuple """ if site and schema_name: msg = 'Only one of schema_name and site parameters can be defined' @@ -222,21 +206,17 @@ return database_id, schema_name - def query(self, sql, database_id=None, schema_name=None, site=None): - """ - Execute SQL queries on Superset. + def query(self, sql: str, + database_id: int | None = None, + schema_name: str | None = None, + site: BaseSite = None) -> list[Any]: + """Execute SQL queries on Superset. :param sql: The SQL query to execute. - :type sql: str :param database_id: The database ID. - :type database_id: int, optional :param schema_name: The schema name. - :type schema_name: str, optional - :raises RuntimeError: If the query execution fails. - :return: The data returned from the query execution. - :rtype: list """ if not self.connected: self.login() diff --git a/pywikibot/pagegenerators/_generators.py b/pywikibot/pagegenerators/_generators.py index 9c6c6db..b335fc8 100644 --- a/pywikibot/pagegenerators/_generators.py +++ b/pywikibot/pagegenerators/_generators.py @@ -1,6 +1,6 @@ """Page filter generators provided by the pagegenerators module.""" # -# (C) Pywikibot team, 2008-2023 +# (C) Pywikibot team, 2008-2024 # # Distributed under the terms of the MIT license. # @@ -981,6 +981,8 @@ FROM page LIMIT 10 + .. versionadded:: 9.2 + :param query: the SQL query string. :param site: Site for generator results. :param schema_name: target superset schema name diff --git a/scripts/CHANGELOG.rst b/scripts/CHANGELOG.rst index 45d2b18..31ae292 100644 --- a/scripts/CHANGELOG.rst +++ b/scripts/CHANGELOG.rst @@ -4,6 +4,11 @@ 9.2.0 ----- +addwikis +^^^^^^^^ + +* This maintenance script was added to add wikis to the Family.codes set + commons_information ^^^^^^^^^^^^^^^^^^^ diff --git a/tests/superset_tests.py b/tests/superset_tests.py index 171835f..744b430 100755 --- a/tests/superset_tests.py +++ b/tests/superset_tests.py @@ -1,5 +1,8 @@ #!/usr/bin/env python3 -"""Tests for superset module.""" +"""Tests for superset module. + +.. versionadded:: 9.2 +""" # # (C) Pywikibot team, 2024 # -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1048832?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: If46121fb857cc49cc8521947f6fadecccdcb5ee3 Gerrit-Change-Number: 1048832 Gerrit-PatchSet: 3 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: D3r1ck01 <dalangi-ctr(a)wikimedia.org> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] ...core[master]: [maintenance] Add new addwikis script to add wikis to a family file
by jenkins-bot (Code Review) 21 Jun '24

21 Jun '24

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1048367?usp=email ) Change subject: [maintenance] Add new addwikis script to add wikis to a family file ...................................................................... [maintenance] Add new addwikis script to add wikis to a family file Change-Id: I8ce9dc7c374a281286c8b2e1890f570413e6a6e1 --- M docs/scripts/maintenance.rst M docs/scripts_ref/scripts.maintenance.rst M scripts/README.rst A scripts/maintenance/addwikis.py 4 files changed, 120 insertions(+), 0 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/docs/scripts/maintenance.rst b/docs/scripts/maintenance.rst index 2eefdcd..da7df5d 100644 --- a/docs/scripts/maintenance.rst +++ b/docs/scripts/maintenance.rst @@ -2,6 +2,13 @@ Utility scripts *************** +addwikis script +=============== + +.. automodule:: scripts.maintenance.addwikis + :no-members: + :noindex: + cache script ============ diff --git a/docs/scripts_ref/scripts.maintenance.rst b/docs/scripts_ref/scripts.maintenance.rst index af4359d..b24603d 100644 --- a/docs/scripts_ref/scripts.maintenance.rst +++ b/docs/scripts_ref/scripts.maintenance.rst @@ -9,6 +9,11 @@ Maintenance submodules ====================== +scripts.maintenance.addwikis +---------------------------- + +.. automodule:: scripts.maintenance.addwikis + scripts.maintenance.cache ------------------------- diff --git a/scripts/README.rst b/scripts/README.rst index 162933a..3815f29 100644 --- a/scripts/README.rst +++ b/scripts/README.rst @@ -166,6 +166,8 @@ +------------------------+---------------------------------------------------------+ | maintenance | Framework helper scripts | +========================+=========================================================+ +| addwikis.py | Script to add wikis to a family file. | ++------------------------+---------------------------------------------------------+ | cache.py | Script for showing and deleting API cache. | +------------------------+---------------------------------------------------------+ | colors.py | Utility to show pywikibot colors. | diff --git a/scripts/maintenance/addwikis.py b/scripts/maintenance/addwikis.py new file mode 100644 index 0000000..0fa7b0d --- /dev/null +++ b/scripts/maintenance/addwikis.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +"""Script that adds new wikis to the codes set in Wikimedia family files. + +Usage: + + python pwb.py addwiki [-family:<family>] {<wiki>} + + +.. versionadded:: 9.2 +""" +# +# (C) Pywikibot team, 2024 +# +# Distributed under the terms of the MIT license. +# +from __future__ import annotations + +import re +import sys +from pathlib import Path + +import pywikibot +from pywikibot.family import Family + + +# supported families by this script +families_list = [ + 'wikibooks', + 'wikinews', + 'wikipedia', + 'wikiquote', + 'wikisource', + 'wikiversity', + 'wikivoyage', + 'wiktionary', +] + + +def update_family(family, wikis): + """Update codes set in family file.""" + joined_wikis = "', '".join(wikis) + pywikibot.info(f"Adding '{joined_wikis}' to {family} family...\n") + + original = Family.load(family).codes + new_codes = set() + for wiki in list(wikis): + if wiki in original: + pywikibot.warning(f'{wiki!r} is alread in Family.codes; ignoring.') + else: + new_codes.add(wiki) + + if not new_codes: + pywikibot.info('No wikis to add.') + return + + # combine new codes set + new = sorted(original | new_codes) + pywikibot.info("The lists don't match, the new list is:\n") + text = ' codes = {\n' + line = ' ' * 7 + for code in new: + if len(line) + len(code) >= 76: + text += line + '\n' + line = ' ' * 7 + line += f" '{code}'," + text += line + '\n' + text += ' }' + pywikibot.info(text) + + # update codes + filepath = Path(f'pywikibot/families/{family}_family.py') + family_text = filepath.read_text(encoding='utf8') + family_text = re.sub(r'(?ms)^ {4}codes = \{.+?\}', + text, family_text, count=1) + filepath.write_text(family_text, encoding='utf8') + + +def main(*args: str) -> None: + """Script entry point to handle args.""" + if not args: + args = sys.argv[1:] + sys.argv = [sys.argv[0]] + + family = 'wikipedia' + wikis = [] + for arg in args: + if arg.startswith('-family'): + family = arg.split(':')[1] + elif arg == 'help': + pywikibot.show_help() + return + else: + wikis.append(arg) + + if not wikis: + pywikibot.bot.suggest_help( + additional_text='No wiki is specified to be added.') + elif family not in families_list: + pywikibot.bot.suggest_help( + additional_text=f'Script cannot be used for {family} family.') + else: + update_family(family, wikis) + + +if __name__ == '__main__': + main() -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1048367?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: I8ce9dc7c374a281286c8b2e1890f570413e6a6e1 Gerrit-Change-Number: 1048367 Gerrit-PatchSet: 5 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: D3r1ck01 <dalangi-ctr(a)wikimedia.org> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] ...core[master]: [tests] skip superset_tests on ConnectionError
by jenkins-bot (Code Review) 21 Jun '24

21 Jun '24

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1048283?usp=email ) Change subject: [tests] skip superset_tests on ConnectionError ...................................................................... [tests] skip superset_tests on ConnectionError Also - reorder imports - login is already done because of login class attribute Bug: T367939 Change-Id: If22af2fdafc0a4c006f7e0b0c9b10fc454e031cf --- M tests/superset_tests.py 1 file changed, 7 insertions(+), 6 deletions(-) Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/tests/superset_tests.py b/tests/superset_tests.py index 772fa05..171835f 100755 --- a/tests/superset_tests.py +++ b/tests/superset_tests.py @@ -9,11 +9,12 @@ import unittest from contextlib import suppress + import pywikibot from pywikibot.data.superset import SupersetQuery from pywikibot.exceptions import NoUsernameError -from tests.aspects import TestCase from pywikibot.pagegenerators import SupersetPageGenerator +from tests.aspects import TestCase class TestSupersetWithoutAuth(TestCase): @@ -52,17 +53,17 @@ """Superset login and queries.""" sql = 'SELECT page_id, page_title FROM page LIMIT 2;' site = self.get_site() - site.login() - self.assertTrue(site.logged_in()) # Test login and initial site parameters + superset = SupersetQuery(site=site) try: - superset = SupersetQuery(site=site) superset.login() except NoUsernameError: - pywikibot.warning('Oauth permission is missing. SKIPPING TESTS.') - return + self.skipTest('Oauth permission is missing.') + except ConnectionError as e: + self.skipTest(e) + self.assertTrue(site.logged_in()) self.assertTrue(superset.connected) rows = superset.query(sql) self.assertLength(rows, 2) -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1048283?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: If22af2fdafc0a4c006f7e0b0c9b10fc454e031cf Gerrit-Change-Number: 1048283 Gerrit-PatchSet: 2 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] ...core[master]: [L10N] Add support for btmwiki to Pywikibot
by jenkins-bot (Code Review) 21 Jun '24

21 Jun '24

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1048373?usp=email ) Change subject: [L10N] Add support for btmwiki to Pywikibot ...................................................................... [L10N] Add support for btmwiki to Pywikibot Bug: T368069 Change-Id: I813d9e7a85c6e69024a5baab22ab92e4ee9042ff --- M pywikibot/families/wikipedia_family.py 1 file changed, 19 insertions(+), 19 deletions(-) Approvals: Zabe: Looks good to me, but someone else must approve jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/pywikibot/families/wikipedia_family.py b/pywikibot/families/wikipedia_family.py index 531a61a..2fb443e 100644 --- a/pywikibot/families/wikipedia_family.py +++ b/pywikibot/families/wikipedia_family.py @@ -32,9 +32,9 @@ 'anp', 'ar', 'arc', 'ary', 'arz', 'as', 'ast', 'atj', 'av', 'avk', 'awa', 'ay', 'az', 'azb', 'ba', 'ban', 'bar', 'bat-smg', 'bbc', 'bcl', 'be', 'be-tarask', 'bew', 'bg', 'bh', 'bi', 'bjn', 'blk', 'bm', 'bn', - 'bo', 'bpy', 'br', 'bs', 'bug', 'bxr', 'ca', 'cbk-zam', 'cdo', 'ce', - 'ceb', 'ch', 'chr', 'chy', 'ckb', 'co', 'cr', 'crh', 'cs', 'csb', 'cu', - 'cv', 'cy', 'da', 'dag', 'de', 'dga', 'din', 'diq', 'dsb', 'dtp', + 'bo', 'bpy', 'br', 'bs', 'btm', 'bug', 'bxr', 'ca', 'cbk-zam', 'cdo', + 'ce', 'ceb', 'ch', 'chr', 'chy', 'ckb', 'co', 'cr', 'crh', 'cs', 'csb', + 'cu', 'cv', 'cy', 'da', 'dag', 'de', 'dga', 'din', 'diq', 'dsb', 'dtp', 'dty', 'dv', 'dz', 'ee', 'el', 'eml', 'en', 'eo', 'es', 'et', 'eu', 'ext', 'fa', 'fat', 'ff', 'fi', 'fiu-vro', 'fj', 'fo', 'fon', 'fr', 'frp', 'frr', 'fur', 'fy', 'ga', 'gag', 'gan', 'gcr', 'gd', 'gl', @@ -44,22 +44,22 @@ 'is', 'it', 'iu', 'ja', 'jam', 'jbo', 'jv', 'ka', 'kaa', 'kab', 'kbd', 'kbp', 'kcg', 'kg', 'ki', 'kk', 'kl', 'km', 'kn', 'ko', 'koi', 'krc', 'ks', 'ksh', 'ku', 'kus', 'kv', 'kw', 'ky', 'la', 'lad', 'lb', 'lbe', - 'lez', 'lfn', 'lg', 'li', 'lij', 'lld', 'lmo', 'ln', 'lo', 'lt', - 'ltg', 'lv', 'mad', 'mai', 'map-bms', 'mdf', 'mg', 'mhr', 'mi', 'min', - 'mk', 'ml', 'mn', 'mni', 'mnw', 'mr', 'mrj', 'ms', 'mt', 'mwl', 'my', - 'myv', 'mzn', 'nah', 'nap', 'nds', 'nds-nl', 'ne', 'new', 'nia', 'nl', - 'nn', 'no', 'nov', 'nqo', 'nrm', 'nso', 'nv', 'ny', 'oc', 'olo', 'om', - 'or', 'os', 'pa', 'pag', 'pam', 'pap', 'pcd', 'pcm', 'pdc', 'pfl', - 'pi', 'pih', 'pl', 'pms', 'pnb', 'pnt', 'ps', 'pt', 'pwn', 'qu', 'rm', - 'rmy', 'rn', 'ro', 'roa-rup', 'roa-tara', 'ru', 'rue', 'rw', 'sa', - 'sah', 'sat', 'sc', 'scn', 'sco', 'sd', 'se', 'sg', 'sh', 'shi', - 'shn', 'si', 'simple', 'sk', 'skr', 'sl', 'sm', 'smn', 'sn', 'so', - 'sq', 'sr', 'srn', 'ss', 'st', 'stq', 'su', 'sv', 'sw', 'szl', 'szy', - 'ta', 'tay', 'tcy', 'te', 'tet', 'tg', 'th', 'ti', 'tk', 'tl', 'tly', - 'tn', 'to', 'tpi', 'tr', 'trv', 'ts', 'tt', 'tum', 'tw', 'ty', 'tyv', - 'udm', 'ug', 'uk', 'ur', 'uz', 've', 'vec', 'vep', 'vi', 'vls', 'vo', - 'wa', 'war', 'wo', 'wuu', 'xal', 'xh', 'xmf', 'yi', 'yo', 'za', 'zea', - 'zgh', 'zh', 'zh-classical', 'zh-min-nan', 'zh-yue', 'zu', + 'lez', 'lfn', 'lg', 'li', 'lij', 'lld', 'lmo', 'ln', 'lo', 'lt', 'ltg', + 'lv', 'mad', 'mai', 'map-bms', 'mdf', 'mg', 'mhr', 'mi', 'min', 'mk', + 'ml', 'mn', 'mni', 'mnw', 'mr', 'mrj', 'ms', 'mt', 'mwl', 'my', 'myv', + 'mzn', 'nah', 'nap', 'nds', 'nds-nl', 'ne', 'new', 'nia', 'nl', 'nn', + 'no', 'nov', 'nqo', 'nrm', 'nso', 'nv', 'ny', 'oc', 'olo', 'om', 'or', + 'os', 'pa', 'pag', 'pam', 'pap', 'pcd', 'pcm', 'pdc', 'pfl', 'pi', + 'pih', 'pl', 'pms', 'pnb', 'pnt', 'ps', 'pt', 'pwn', 'qu', 'rm', 'rmy', + 'rn', 'ro', 'roa-rup', 'roa-tara', 'ru', 'rue', 'rw', 'sa', 'sah', + 'sat', 'sc', 'scn', 'sco', 'sd', 'se', 'sg', 'sh', 'shi', 'shn', 'si', + 'simple', 'sk', 'skr', 'sl', 'sm', 'smn', 'sn', 'so', 'sq', 'sr', + 'srn', 'ss', 'st', 'stq', 'su', 'sv', 'sw', 'szl', 'szy', 'ta', 'tay', + 'tcy', 'te', 'tet', 'tg', 'th', 'ti', 'tk', 'tl', 'tly', 'tn', 'to', + 'tpi', 'tr', 'trv', 'ts', 'tt', 'tum', 'tw', 'ty', 'tyv', 'udm', 'ug', + 'uk', 'ur', 'uz', 've', 'vec', 'vep', 'vi', 'vls', 'vo', 'wa', 'war', + 'wo', 'wuu', 'xal', 'xh', 'xmf', 'yi', 'yo', 'za', 'zea', 'zgh', 'zh', + 'zh-classical', 'zh-min-nan', 'zh-yue', 'zu', } # Sites we want to edit but not count as real languages -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1048373?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: I813d9e7a85c6e69024a5baab22ab92e4ee9042ff Gerrit-Change-Number: 1048373 Gerrit-PatchSet: 4 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zabe <zabe(a)avorwerk.net> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] ...i18n[master]: Localisation updates from https://translatewiki.net.
by jenkins-bot (Code Review) 20 Jun '24

20 Jun '24

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/i18n/+/1047970?usp=email ) Change subject: Localisation updates from https://translatewiki.net. ...................................................................... Localisation updates from https://translatewiki.net. Change-Id: I12f82818c1399a00b9672fa869a2799ef7b1125f --- M checkimages/fa.json M imagetransfer/fa.json M misspelling/fa.json M weblinkchecker/fa.json 4 files changed, 4 insertions(+), 0 deletions(-) Approvals: L10n-bot: Looks good to me, approved jenkins-bot: Verified diff --git a/checkimages/fa.json b/checkimages/fa.json index 6572dc8..92d2b2d 100644 --- a/checkimages/fa.json +++ b/checkimages/fa.json @@ -2,6 +2,7 @@ "@metadata": { "authors": [ "Amir Sarabadani", + "Ebrahim", "Ebraminio", "FarsiNevis", "Huji", diff --git a/imagetransfer/fa.json b/imagetransfer/fa.json index 350dc24..100c884 100644 --- a/imagetransfer/fa.json +++ b/imagetransfer/fa.json @@ -2,6 +2,7 @@ "@metadata": { "authors": [ "Amir Sarabadani", + "Ebrahim", "Ebraminio", "Reza1615" ] diff --git a/misspelling/fa.json b/misspelling/fa.json index 87b48cf..4f2fce7 100644 --- a/misspelling/fa.json +++ b/misspelling/fa.json @@ -1,6 +1,7 @@ { "@metadata": { "authors": [ + "Ebrahim", "Ebraminio" ] }, diff --git a/weblinkchecker/fa.json b/weblinkchecker/fa.json index b5e654d..75a059a 100644 --- a/weblinkchecker/fa.json +++ b/weblinkchecker/fa.json @@ -1,6 +1,7 @@ { "@metadata": { "authors": [ + "Ebrahim", "Ebraminio", "Reza1615" ] -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/i18n/+/1047970?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: pywikibot/i18n Gerrit-Branch: master Gerrit-Change-Id: I12f82818c1399a00b9672fa869a2799ef7b1125f Gerrit-Change-Number: 1047970 Gerrit-PatchSet: 1 Gerrit-Owner: L10n-bot <l10n-bot(a)translatewiki.net> Gerrit-Reviewer: L10n-bot <l10n-bot(a)translatewiki.net> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] ...core[master]: include image repository extensions in APISite.file_extensions
by jenkins-bot (Code Review) 19 Jun '24

19 Jun '24

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1047202?usp=email ) Change subject: include image repository extensions in APISite.file_extensions ...................................................................... include image repository extensions in APISite.file_extensions the site's image repository may have different file extensions configured which should also be considered valid for the site Change-Id: Ibf449cbf41f5e464ebef0bf4e6927e0dac14c49c --- M pywikibot/site/_apisite.py 1 file changed, 7 insertions(+), 3 deletions(-) Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/pywikibot/site/_apisite.py b/pywikibot/site/_apisite.py index 9a2037c..e1af5e0 100644 --- a/pywikibot/site/_apisite.py +++ b/pywikibot/site/_apisite.py @@ -492,8 +492,13 @@ """File extensions enabled on the wiki. .. versionadded:: 8.4 + .. versionchanged:: 9.2 + also include extensions from the image repository """ - return sorted(e['ext'] for e in self.siteinfo.get('fileextensions')) + ext = self.siteinfo.get('fileextensions') + if self.has_image_repository: + ext.extend(self.image_repository().siteinfo.get('fileextensions')) + return sorted({e['ext'] for e in ext}) @property def maxlimit(self) -> int: @@ -1273,8 +1278,7 @@ @property def has_image_repository(self) -> bool: """Return True if site has a shared image repository like Commons.""" - code, fam = self.shared_image_repository() - return bool(code or fam) + return self.image_repository() is not None @property def has_data_repository(self) -> bool: -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1047202?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: Ibf449cbf41f5e464ebef0bf4e6927e0dac14c49c Gerrit-Change-Number: 1047202 Gerrit-PatchSet: 2 Gerrit-Owner: JJMC89 <JJMC89.Wikimedia(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] ...core[master]: [IMPR] ignore ValueError durig upcast of FilePage due to invalid file...
by jenkins-bot (Code Review) 18 Jun '24

18 Jun '24

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1047079?usp=email ) Change subject: [IMPR] ignore ValueError durig upcast of FilePage due to invalid file extension ...................................................................... [IMPR] ignore ValueError durig upcast of FilePage due to invalid file extension Also remove retry loop in TestFilePageDataItem.test_data_item_when_file_exist_but_without_item of file_tests which doesn't make any sense because the generator was empty after the exception. Now let the test fail if an exception might occur but skip the test if no related FilePage was found. Bug: T367777 Change-Id: Ife4b1a7e648f770df98d6141902a28e0d7cb62f5 --- M pywikibot/pagegenerators/_generators.py M tests/file_tests.py 2 files changed, 20 insertions(+), 14 deletions(-) Approvals: Zache-tool: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/pagegenerators/_generators.py b/pywikibot/pagegenerators/_generators.py index af59e3d..2e5fcee 100644 --- a/pywikibot/pagegenerators/_generators.py +++ b/pywikibot/pagegenerators/_generators.py @@ -155,8 +155,7 @@ namespaces: NamespaceArgType = (0, ), total: int | None = None ) -> Generator[pywikibot.page.Page, None, None]: - """ - Iterate Page objects for all new titles in a single namespace. + """Iterate Page objects for all new titles in a single namespace. :param site: Site for generator results. :param namespace: namespace to retrieve pages from @@ -185,8 +184,14 @@ .. versionchanged:: 8.2 The YieldType depends on namespace. It can be - :class:`pywikibot.Page`, :class:`pywikibot.User`, - :class:`pywikibot.FilePage` or :class:`pywikibot.Category`. + :class:`pywikibot.Page<pywikibot.page.Page>`, + :class:`pywikibot.User<pywikibot.page.User>`, + :class:`pywikibot.FilePage<pywikibot.page.FilePage>` or + :class:`pywikibot.Category<pywikibot.page.Category>`. + .. versionchanged:: 9.2 + Ignore :class:`pywikibot.FilePage<pywikibot.page.FilePage>` if it + raises a :exc:`ValueError` during upcast e.g. due to an invaild + file extension. :param site: Site for generator results. """ @@ -206,7 +211,12 @@ pageclass = pywikibot.Category else: pageclass = pywikibot.Page - yield pageclass(site, rc['title']) + try: + yield pageclass(site, rc['title']) + except ValueError: + if pageclass == pywikibot.FilePage: + pywikibot.exception() + raise if site is None: site = pywikibot.Site() diff --git a/tests/file_tests.py b/tests/file_tests.py index f55d73c..6650bcc 100755 --- a/tests/file_tests.py +++ b/tests/file_tests.py @@ -415,16 +415,12 @@ ) # Seek to first page without mediainfo. - # Retry loop is for excepting incorrect files - for retry in range(5): - try: - for page in gen: - slots = page.latest_revision.slots - if 'mediainfo' not in slots: - break + for page in gen: + slots = page.latest_revision.slots + if 'mediainfo' not in slots: break - except ValueError: - pass + else: + self.skipTest('No page found without mediainfo') item = page.data_item() self.assertIsInstance(item, pywikibot.MediaInfo) -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1047079?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: Ife4b1a7e648f770df98d6141902a28e0d7cb62f5 Gerrit-Change-Number: 1047079 Gerrit-PatchSet: 3 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: JJMC89 <JJMC89.Wikimedia(a)gmail.com> Gerrit-Reviewer: Zache-tool <kimmo.virtanen(a)gmail.com> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] ...core[master]: Add superset pagegenerator
by jenkins-bot (Code Review) 18 Jun '24

18 Jun '24

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1046603?usp=email ) Change subject: Add superset pagegenerator ...................................................................... Add superset pagegenerator Add support for SQL queries through https://superset.wmcloud.org and use results as pagegenerator. Bug: T367684 Change-Id: I3d3b3e4cc09497e3806f9552f5e0bc81700cdcbf --- A pywikibot/data/superset.py M pywikibot/pagegenerators/__init__.py M pywikibot/pagegenerators/_factory.py M pywikibot/pagegenerators/_generators.py A tests/superset_tests.py 5 files changed, 570 insertions(+), 0 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/data/superset.py b/pywikibot/data/superset.py new file mode 100644 index 0000000..85aebc0 --- /dev/null +++ b/pywikibot/data/superset.py @@ -0,0 +1,275 @@ +"""Superset Query interface.""" +# +# (C) Pywikibot team, 2024 +# +# Distributed under the terms of the MIT license. +# +from textwrap import fill + +import pywikibot +from pywikibot.comms import http +from pywikibot.data import WaitingMixin +from pywikibot.exceptions import NoUsernameError, ServerError + + +class SupersetQuery(WaitingMixin): + """Superset Query class. + + This class allows to run SQL queries against wikimedia superset + service. + """ + + def __init__(self, schema_name=None, + site=None, database_id=None): + """ + Create superset endpoint with initial defaults. + + Either site OR schema_name is required. Site and schema_name are + mutually exclusive. Database id will be retrieved automatically + if needed. + + :param site: The mediawiki site to be queried + :type site: pywikibot.Site, optional + + :param schema_name: superset database schema name. + Example value "enwiki_p" + :type schema_name: str, optional + + :param database_id: superset database id. + :type database_id: int, optional + + :raises TypeError: if site and schema_name are both defined' + + """ + if site and schema_name: + msg = 'Only one of schema_name and site parameters can be defined' + raise TypeError(msg) + + # Validate database_id + if database_id: + if not isinstance(database_id, int): + msg = f'database_id should be integer, but got "{database_id}"' + raise TypeError(msg) + + self.site = site + self.schema_name = schema_name + self.database_id = database_id + + self.connected = False + self.last_response = None + self.superset_url = 'https://superset.wmcloud.org' + + def login(self): + """ + Login to superset. + + Function logins first to meta.wikimedia.org + and then OAUTH login to superset.wmcloud.org. + Working login expects that the user has manually + permitted the username to login to the superset. + + :raises NoUsernameError: if not not logged in. + :raises ServerError: For other errors + + :return: True if user has been logged to superset + :rtype bool + """ + # superset uses meta for OAUTH authentication + loginsite = pywikibot.Site('meta') + if not loginsite.logged_in(): + loginsite.login() + if not loginsite.logged_in(): + msg = 'User is not logged in on meta.wikimedia.org' + raise NoUsernameError(msg) + + # Superset oauth login + url = f'{self.superset_url}/login/mediawiki?next=' + self.last_response = http.fetch(url) + + # Test if uset has been succesfully logged in + url = f'{self.superset_url}/api/v1/me/' + self.last_response = http.fetch(url) + + # Handle error cases + if self.last_response.status_code == 200: + self.connected = True + elif self.last_response.status_code == 401: + self.connected = False + raise NoUsernameError(fill( + 'User not logged in. You need to log in to ' + 'meta.wikimedia.org and give OAUTH permission. ' + 'Open https://superset.wmcloud.org/login/ ' + 'with browser to login and give permission.' + )) + else: + self.connected = False + status_code = self.last_response.status_code + raise ServerError(f'Unknown error: {status_code}') + + return self.connected + + def get_csrf_token(self): + """ + Get superset CSRF token. + + Method retrieves a CSRF token from the Superset service. + If the instance is not connected, it attempts to log in first. + + :raises ServerError: For any http errors + :return CSRF token string + :rtype str + """ + if not self.connected: + self.login() + + # Load CSRF token + url = f'{self.superset_url}/api/v1/security/csrf_token/' + self.last_response = http.fetch(url) + + if self.last_response.status_code == 200: + return self.last_response.json()['result'] + else: + status_code = self.last_response.status_code + raise ServerError(f'CSRF token error: {status_code}') + + def get_database_id_by_schema_name(self, schema_name): + """ + Get superset database_id using superset schema name. + + :param schema_name: superset database schema name. + Example value "enwiki_p" + :type schema_name: str + + :raises KeyError: If the database ID could found. + :raises ServerError: For any other http errors + + :return: database id + :rtype: int + """ + if not self.connected: + self.login() + + for database_id in range(1, 20): + url = self.superset_url + url += f'/api/v1/database/{database_id}/schemas/?q=(force:!f)' + self.last_response = http.fetch(url) + + if self.last_response.status_code == 200: + schemas = self.last_response.json()['result'] + if schema_name in schemas: + return database_id + + elif self.last_response.status_code == 404: + break + else: + status_code = self.last_response.status_code + raise ServerError(f'Unknown error: {status_code}') + + url = self.superset_url + raise KeyError(f'Schema "{schema_name}" not found in {url}.') + + def merge_query_arguments(self, + database_id=None, + schema_name=None, + site=None): + """ + Determine and validate the database_id and schema_name. + + :param database_id: The superset database ID. + :type database_id: int, optional + + :param schema_name: The superset schema name. + :type schema_name: str, optional + + :param site: The target site + :type site: pywikibot.Site, optional + + :raises TypeError: if site and schema_name are both defined' + :raises TypeError: If determined database_id is not an integer. + :raises TypeError: If neither site nor schema_name is determined. + + :return A tuple containing database_id and schema_name. + :rtype: tuple + """ + if site and schema_name: + msg = 'Only one of schema_name and site parameters can be defined' + raise TypeError(msg) + + # Determine schema_name + if not schema_name: + if site: + schema_name = f'{site.dbName()}_p' + elif self.schema_name: + schema_name = self.schema_name + elif self.site: + schema_name = f'{self.site.dbName()}_p' + + # Determine database_id + if not database_id: + if self.database_id: + database_id = int(self.database_id) + else: + database_id = self.get_database_id_by_schema_name(schema_name) + + # Validate database_id + if not isinstance(database_id, int): + msg = f'database_id should be integer, but got "{database_id}"' + raise TypeError(msg) + + # Ensure either site or schema_name is provided + if not (self.site or schema_name): + raise TypeError('Either site or schema_name must be provided') + + return database_id, schema_name + + def query(self, sql, database_id=None, schema_name=None, site=None): + """ + Execute SQL queries on Superset. + + :param sql: The SQL query to execute. + :type sql: str + :param database_id: The database ID. + :type database_id: int, optional + :param schema_name: The schema name. + :type schema_name: str, optional + + :raises RuntimeError: If the query execution fails. + + :return: The data returned from the query execution. + :rtype: list + """ + if not self.connected: + self.login() + + token = self.get_csrf_token() + + headers = { + 'X-CSRFToken': token, + 'Content-Type': 'application/json', + 'referer': 'https://superset.wmcloud.org/sqllab/' + } + + database_id, schema_name = self.merge_query_arguments(database_id, + schema_name, + site) + + sql_query_payload = { + 'database_id': database_id, + 'schema': schema_name, + 'sql': sql, + 'json': True, + 'runAsync': False, + } + + url = f'{self.superset_url}/api/v1/sqllab/execute/' + try: + self.last_response = http.fetch(uri=url, + json=sql_query_payload, + method='POST', + headers=headers) + self.last_response.raise_for_status() + json = self.last_response.json() + return json['data'] + + except Exception as e: + raise RuntimeError(f'Failed to execute query: {e}') diff --git a/pywikibot/pagegenerators/__init__.py b/pywikibot/pagegenerators/__init__.py index b9f02d7..64c8089 100644 --- a/pywikibot/pagegenerators/__init__.py +++ b/pywikibot/pagegenerators/__init__.py @@ -68,6 +68,7 @@ SearchPageGenerator, ShortPagesPageGenerator, SubCategoriesPageGenerator, + SupersetPageGenerator, TextIOPageGenerator, UnCategorizedCategoryGenerator, UnCategorizedImageGenerator, @@ -139,6 +140,7 @@ 'SearchPageGenerator', 'ShortPagesPageGenerator', 'SubCategoriesPageGenerator', + 'SupersetPageGenerator', 'TextIOPageGenerator', 'UnCategorizedCategoryGenerator', 'UnCategorizedImageGenerator', @@ -355,6 +357,12 @@ and treats the resulting pages. See :manpage:`MySQL` for more details. +-supersetquery Takes a SQL query string like + "SELECT page_namespace, page_title FROM page + WHERE page_namespace = 0" and run it in + https://superset.wmcloud.org/ and treats + the resulting pages. + -sparql Takes a SPARQL SELECT query string including ?item and works on the resulting pages. diff --git a/pywikibot/pagegenerators/_factory.py b/pywikibot/pagegenerators/_factory.py index adb10cd..0e2c135 100644 --- a/pywikibot/pagegenerators/_factory.py +++ b/pywikibot/pagegenerators/_factory.py @@ -46,6 +46,7 @@ PrefixingPageGenerator, RecentChangesPageGenerator, SubCategoriesPageGenerator, + SupersetPageGenerator, TextIOPageGenerator, UserContributionsGenerator, WikibaseSearchItemPageGenerator, @@ -901,6 +902,12 @@ value = pywikibot.input('Mysql query string:') return MySQLPageGenerator(value, site=self.site) + def _handle_supersetquery(self, value: str) -> HANDLER_GEN_TYPE: + """Handle `-supersetquery` argument.""" + if not value: + value = pywikibot.input('Superset SQL query string:') + return SupersetPageGenerator(value, site=self.site) + def _handle_intersect(self, value: str) -> Literal[True]: """Handle `-intersect` argument.""" self.intersect = True diff --git a/pywikibot/pagegenerators/_generators.py b/pywikibot/pagegenerators/_generators.py index af59e3d..bc9ed2c 100644 --- a/pywikibot/pagegenerators/_generators.py +++ b/pywikibot/pagegenerators/_generators.py @@ -922,6 +922,125 @@ yield page +def SupersetPageGenerator(query: str, + site: BaseSite | None = None, + schema_name: str | None = None, + database_id: int | None = None + ) -> Iterator[pywikibot.page.Page]: + """Generate pages that result from the given SPARQL query. + + Pages are generated using site in following order: + + 1. site retrieved using page_wikidb column in SQL result + 2. site as parameter + 3. site retrieved using schema_name + + SQL columns used are + + - page_id + - page_namespace + page_title + - page_wikidb + + Example SQL queries + + .. code-block:: sql + + SELECT + gil_wiki AS page_wikidb, + gil_page AS page_id + FROM globalimagelinks + GROUP BY gil_wiki + LIMIT 10 + + OR + + .. code-block:: sql + + SELECT + page_id + FROM page + LIMIT 10 + + OR + + .. code-block:: sql + + SELECT + page_namespace, + page_title + FROM page + LIMIT 10 + + :param query: the SQL query string. + :param site: Site for generator results. + :param schema_name: target superset schema name + :param database_id: target superset database id + """ + from pywikibot.data.superset import SupersetQuery + + # Do not pass site to superset if schema_name is defined. + # The user may use schema_name to point to different + # wikimedia db on purpose and use site for + # generating result pages. + + superset_site = None if schema_name else site + + superset = SupersetQuery(site=superset_site, + schema_name=schema_name, + database_id=database_id) + + try: + rows = superset.query(query) + except Exception as e: + pywikibot.error(f'Error executing query: {query}\n{e}') + return + + sites = {} + + # If there is no site then retrieve it using schema_name + if not site: + if not schema_name: + raise TypeError('Schema name or site must be provided.') + + wikidb = re.sub('_p$', '', schema_name) + site = pywikibot.site.APISite.fromDBName(wikidb) + + for row in rows: + # If page_wikidb column in SQL result then use it to retrieve site + if 'page_wikidb' in row: + # remove "_p" suffix + wikidb = re.sub('_p$', '', row['page_wikidb']) + + # Caching sites + if wikidb not in sites: + try: + sites[wikidb] = pywikibot.site.APISite.fromDBName(wikidb) + except ValueError: + msg = f'Cannot parse a site from {wikidb} for {row}.' + pywikibot.warning(msg) + continue + site = sites[wikidb] + + # Generate page objects + + # Create page object from page_id + if 'page_id' in row: + page_ids = [row['page_id']] + pages = site.load_pages_from_pageids(page_ids) + for page in pages: + yield page + + # Create page object from page_namespace + page_title + elif 'page_title' in rows[0] and 'page_namespace' in rows[0]: + page_namespace = int(row['page_namespace']) + page_title = row['page_title'] + page = pywikibot.Page(site, page_title, ns=page_namespace) + yield page + + else: + raise ValueError('The SQL result is in wrong format.') + + class XMLDumpPageGenerator(abc.Iterator): # type: ignore[type-arg] """Xml iterator that yields Page objects. diff --git a/tests/superset_tests.py b/tests/superset_tests.py new file mode 100755 index 0000000..772fa05 --- /dev/null +++ b/tests/superset_tests.py @@ -0,0 +1,161 @@ +#!/usr/bin/env python3 +"""Tests for superset module.""" +# +# (C) Pywikibot team, 2024 +# +# Distributed under the terms of the MIT license. +# +from __future__ import annotations + +import unittest +from contextlib import suppress +import pywikibot +from pywikibot.data.superset import SupersetQuery +from pywikibot.exceptions import NoUsernameError +from tests.aspects import TestCase +from pywikibot.pagegenerators import SupersetPageGenerator + + +class TestSupersetWithoutAuth(TestCase): + """Test Superset without auth.""" + + family = 'meta' + code = 'meta' + + def test_init(self): + """Test init validation functions.""" + # Test initial database_id parameter in wrong format + site = self.get_site() + + superset = SupersetQuery(schema_name='fiwiki_p') + self.assertIsInstance(superset, SupersetQuery) + + msg = 'Only one of schema_name and site parameters can be defined' + with self.assertRaisesRegex(TypeError, msg): + superset = SupersetQuery(schema_name='enwiki_p', + site=site) + + msg = 'database_id should be integer' + with self.assertRaisesRegex(TypeError, msg): + superset = SupersetQuery(schema_name='enwiki_p', + database_id='foo') + + +class TestSupersetWithAuth(TestCase): + """Test Superset with auth.""" + + login = True + family = 'meta' + code = 'meta' + + def test_login_and_oauth_permisson(self): + """Superset login and queries.""" + sql = 'SELECT page_id, page_title FROM page LIMIT 2;' + site = self.get_site() + site.login() + self.assertTrue(site.logged_in()) + + # Test login and initial site parameters + try: + superset = SupersetQuery(site=site) + superset.login() + except NoUsernameError: + pywikibot.warning('Oauth permission is missing. SKIPPING TESTS.') + return + + self.assertTrue(superset.connected) + rows = superset.query(sql) + self.assertLength(rows, 2) + + # Test initial schema_name parameter + superset = SupersetQuery(schema_name='fiwiki_p') + rows = superset.query(sql) + self.assertLength(rows, 2) + + # Test initial schema_name and database_id parameters + superset = SupersetQuery(schema_name='enwiki_p', database_id=1) + rows = superset.query(sql) + self.assertLength(rows, 2) + + # Test get_database_id_by_schema_name() + database_id = superset.get_database_id_by_schema_name('fiwiki_p') + self.assertEqual(database_id, 2) + + # Test incorrect initial schema_name parameter + superset = SupersetQuery(schema_name='foowiki_p') + msg = 'Schema "foowiki_p" not found in https://superset.wmcloud.org.' + with self.assertRaisesRegex(KeyError, msg): + rows = superset.query(sql) + + # Test incorrect initial database_id parameter + # superset.wmcloud.org fails with 500 server error + # so this is expected to be changed when server side + # is updated + + superset = SupersetQuery(schema_name='enwiki_p', database_id=2) + with self.assertRaises(RuntimeError): + rows = superset.query(sql) + + # Test overriding database_id in query + rows = superset.query(sql, database_id=1) + self.assertLength(rows, 2) + + # Test overriding schema_name in query + rows = superset.query(sql, schema_name='fiwiki_p') + self.assertLength(rows, 2) + + # Test overriding schema using site + testsite = pywikibot.Site('fi', 'wikipedia') + rows = superset.query(sql, site=testsite) + self.assertLength(rows, 2) + + # Test that overriding both schema_name and site fails + msg = 'Only one of schema_name and site parameters can be defined' + with self.assertRaisesRegex(TypeError, msg): + rows = superset.query(sql, schema_name='fiwiki_p', site=site) + + def test_superset_generators(self): + """Superset generator.""" + site = self.get_site() + query = 'SELECT page_id FROM page LIMIT 2' + gen = SupersetPageGenerator(query, site=site) + for page in gen: + t = str(page) + self.assertTrue(t) + + query = 'SELECT page_title, page_namespace FROM page LIMIT 2' + gen = SupersetPageGenerator(query, site=site) + for page in gen: + t = str(page) + self.assertTrue(t) + + query = 'SELECT page_namespace, page_title FROM page LIMIT 2' + gen = SupersetPageGenerator(query, schema_name='fiwiki_p') + for page in gen: + t = str(page) + self.assertTrue(t) + + query = ('SELECT * FROM (' + 'SELECT gil_wiki AS page_wikidb, gil_page AS page_id ' + 'FROM globalimagelinks GROUP BY gil_wiki ' + ') AS t LIMIT 2') + gen = SupersetPageGenerator(query, schema_name='commonswiki_p') + for page in gen: + t = str(page) + self.assertTrue(t) + + query = ('SELECT * FROM ( ' + 'SELECT gil_wiki AS page_wikidb, ' + 'gil_page_namespace_id AS page_namespace, ' + 'gil_page_title AS page_title ' + 'FROM globalimagelinks GROUP BY gil_wiki ' + ') AS t LIMIT 2') + gen = SupersetPageGenerator(query, schema_name='commonswiki_p') + for page in gen: + t = str(page) + self.assertTrue(t) + + +if __name__ == '__main__': + with suppress(SystemExit): + unittest.main() -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1046603?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: I3d3b3e4cc09497e3806f9552f5e0bc81700cdcbf Gerrit-Change-Number: 1046603 Gerrit-PatchSet: 5 Gerrit-Owner: Zache-tool <kimmo.virtanen(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] ...core[master]: Revert "[9.2] Publish Pywikibot 9.2"
by Xqt (Code Review) 17 Jun '24

17 Jun '24

Xqt has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1046711?usp=email ) Change subject: Revert "[9.2] Publish Pywikibot 9.2" ...................................................................... Revert "[9.2] Publish Pywikibot 9.2" This reverts commit 7564bfde26e9237033886d60ab16bf316208903a. Reason for revert: stable failed Change-Id: I078335ed186f44d0837cc10edab3540b1d63d479 --- M pywikibot/__metadata__.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Xqt: Verified; Looks good to me, approved diff --git a/pywikibot/__metadata__.py b/pywikibot/__metadata__.py index be801c8..6e5a9d0 100644 --- a/pywikibot/__metadata__.py +++ b/pywikibot/__metadata__.py @@ -12,6 +12,6 @@ from time import strftime -__version__ = '9.2.0' +__version__ = '9.2.0.dev3' __url__ = 'https://www.mediawiki.org/wiki/Manual:Pywikibot' __copyright__ = f'2003-{strftime("%Y")}, Pywikibot team' -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1046711?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: I078335ed186f44d0837cc10edab3540b1d63d479 Gerrit-Change-Number: 1046711 Gerrit-PatchSet: 1 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] ...core[master]: [9.2] Publish Pywikibot 9.2
by Xqt (Code Review) 17 Jun '24

17 Jun '24

Xqt has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1046694?usp=email ) Change subject: [9.2] Publish Pywikibot 9.2 ...................................................................... [9.2] Publish Pywikibot 9.2 Change-Id: If81d19966b886cd6913d6c74ad1bf9f0bb24ba71 --- M pywikibot/__metadata__.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/__metadata__.py b/pywikibot/__metadata__.py index 6e5a9d0..be801c8 100644 --- a/pywikibot/__metadata__.py +++ b/pywikibot/__metadata__.py @@ -12,6 +12,6 @@ from time import strftime -__version__ = '9.2.0.dev3' +__version__ = '9.2.0' __url__ = 'https://www.mediawiki.org/wiki/Manual:Pywikibot' __copyright__ = f'2003-{strftime("%Y")}, Pywikibot team' -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1046694?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: If81d19966b886cd6913d6c74ad1bf9f0bb24ba71 Gerrit-Change-Number: 1046694 Gerrit-PatchSet: 1 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot

1 0

← Newer
1
2
3
4
5
6
7
8
9
...
1343
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits