Pywikibot-commits September 2015

pywikibot-commits@lists.wikimedia.org

2 participants
243 discussions

[Gerrit] [FIX] wikimediachapter: Use dk prefix instead - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: [FIX] wikimediachapter: Use dk prefix instead ...................................................................... [FIX] wikimediachapter: Use dk prefix instead With 28cd83f7 the site constructor assumed that `dk:wikimediachapter` assumed that it's obsolete and replaced by `da:wikimediachapter` which is not the case for `wikimediachapter`. This is removing all code aliases as they are designed for Wikipedia. Change-Id: I2841df3d4e5b5df42096151a49ad1705015763a0 --- M pywikibot/families/wikimediachapter_family.py 1 file changed, 2 insertions(+), 0 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/families/wikimediachapter_family.py b/pywikibot/families/wikimediachapter_family.py index 9f3bbdf..891fc5b 100644 --- a/pywikibot/families/wikimediachapter_family.py +++ b/pywikibot/families/wikimediachapter_family.py @@ -11,6 +11,8 @@ """Family for Wikimedia chapters hosted on wikimedia.org.""" + code_aliases = {} + def __init__(self): """Constructor.""" super(Family, self).__init__() -- To view, visit https://gerrit.wikimedia.org/r/235942 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I2841df3d4e5b5df42096151a49ad1705015763a0 Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

8 years, 8 months

[Gerrit] Update WikimediaFamily cross_projects - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Update WikimediaFamily cross_projects ...................................................................... Update WikimediaFamily cross_projects Move cross_projects to be a class attribute. Break down the list of cross_projects into different types, adding missing project families. Update wikimediachapter family to be a WikimediaFamily subclass, and remove an obsolete 'continue' in the Family tests related to the wikimediachapter family. Bug: T103254 Change-Id: I29e720088502757b0461489a9476e40d434ba0d5 --- M pywikibot/families/wikimediachapter_family.py M pywikibot/family.py M tests/family_tests.py 3 files changed, 49 insertions(+), 16 deletions(-) Approvals: XZise: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/families/wikimediachapter_family.py b/pywikibot/families/wikimediachapter_family.py index b240954..dccf1ce 100644 --- a/pywikibot/families/wikimediachapter_family.py +++ b/pywikibot/families/wikimediachapter_family.py @@ -7,13 +7,13 @@ from pywikibot import family -class Family(family.Family): +class Family(family.WikimediaFamily): """Family for Wikimedia chapters hosted on wikimedia.org.""" def __init__(self): """Constructor.""" - family.Family.__init__(self) + super(Family, self).__init__() self.name = 'wikimediachapter' self.countries = [ diff --git a/pywikibot/family.py b/pywikibot/family.py index 6c2e071..e809240 100644 --- a/pywikibot/family.py +++ b/pywikibot/family.py @@ -691,7 +691,8 @@ } # A list of projects that share cross-project sessions. - self.cross_projects = [] + if not hasattr(self, 'cross_projects'): + self.cross_projects = [] # A list with the name for cross-project cookies. # default for wikimedia centralAuth extensions. @@ -1411,6 +1412,51 @@ """Class for all wikimedia families.""" + multi_language_content_families = [ + 'wikipedia', 'test', + 'wiktionary', 'wikisource', + 'wikibooks', 'wikinews', 'wikiquote', + 'wikiversity', 'wikivoyage', + ] + + wikimedia_org_content_families = [ + 'commons', 'incubator', 'species', + ] + + wikimedia_org_meta_families = [ + 'meta', 'outreach', 'strategy', + 'wikimediachapter', + ] + + wikimedia_org_other_families = [ + 'wikitech', + ] + + other_content_families = [ + 'wikidata' + 'mediawiki', + ] + + content_families = set( + multi_language_content_families + + wikimedia_org_content_families + + other_content_families + ) + + wikimedia_org_families = set( + wikimedia_org_content_families + + wikimedia_org_meta_families + + wikimedia_org_other_families + ) + + # CentralAuth cross avaliable projects. + cross_projects = set( + multi_language_content_families + + wikimedia_org_content_families + + wikimedia_org_meta_families + + other_content_families + ) + # Code mappings which are only an alias, and there is no 'old' wiki. # For all except 'nl_nds', subdomains do exist as a redirect, but that # should not be relied upon. @@ -1447,16 +1493,6 @@ # Romanian was to be the replacement. 'mo': 'ro', } - - def __init__(self): - super(WikimediaFamily, self).__init__() - - # CentralAuth cross avaliable projects. - self.cross_projects = [ - 'commons', 'incubator', 'mediawiki', 'meta', 'species', 'test', - 'wikibooks', 'wikidata', 'wikinews', 'wikipedia', 'wikiquote', - 'wikisource', 'wikiversity', 'wiktionary', - ] @property def interwiki_removals(self): diff --git a/tests/family_tests.py b/tests/family_tests.py index 6fe8823..7a08da7 100644 --- a/tests/family_tests.py +++ b/tests/family_tests.py @@ -33,9 +33,6 @@ f = Family.load(name) self.assertIsInstance(f.langs, dict) self.assertNotEqual(f.langs, {}) - # There is one inconsistency - if f.name == 'wikimediachapter' and name == 'wikimedia': - continue self.assertEqual(f.name, name) def test_family_load_invalid(self): -- To view, visit https://gerrit.wikimedia.org/r/219613 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I29e720088502757b0461489a9476e40d434ba0d5 Gerrit-PatchSet: 6 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Nemo bis <federicoleva(a)tiscali.it> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

8 years, 8 months

[Gerrit] Fix HTML regex version detection - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Fix HTML regex version detection ...................................................................... Fix HTML regex version detection 2f28bd3e68 included a bug in the HTML regex version detection. Bug: T74846 Change-Id: I1d5ff1e64ab83962e0ed7b970123561283a5f04b --- M pywikibot/site_detect.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: XZise: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/site_detect.py b/pywikibot/site_detect.py index 7dd2587..c4d07ce 100644 --- a/pywikibot/site_detect.py +++ b/pywikibot/site_detect.py @@ -114,7 +114,7 @@ % self.fromurl) try: self.version = MediaWikiVersion( - self.REwgVersion.search(data).groups(0)) + self.REwgVersion.search(data).group(1)) except AttributeError: pass -- To view, visit https://gerrit.wikimedia.org/r/235660 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I1d5ff1e64ab83962e0ed7b970123561283a5f04b Gerrit-PatchSet: 3 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

8 years, 8 months

[Gerrit] Disable multi-site on betawp and betaws - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Disable multi-site on betawp and betaws ...................................................................... Disable multi-site on betawp and betaws Multi-site family generation does not work correctly on the beta cluster, as the beta site matrix points to the production cluster, causing the family file to include production wikis, and Family.from_url fails. Change-Id: I9c9e43eeb8732ec653797e9ba5f9e231019546dd --- M .travis.yml 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve XZise: Looks good to me, approved jenkins-bot: Verified diff --git a/.travis.yml b/.travis.yml index e74a0c3..e18b15a 100644 --- a/.travis.yml +++ b/.travis.yml @@ -52,10 +52,10 @@ - if [[ "$FAMILY" == 'oraintest' ]]; then python pwb.py generate_family_file 'https://test.orain.org/' oraintest ; fi - python pwb.py generate_family_file 'https://wiki.musicbrainz.org/' musicbrainz - if [[ $FAMILY == 'wpbeta' ]]; then - python -m generate_family_file 'http://'$LANGUAGE'.wikipedia.beta.wmflabs.org/' 'wpbeta' 'y' ; + python -m generate_family_file 'http://'$LANGUAGE'.wikipedia.beta.wmflabs.org/' 'wpbeta' 'n' ; fi - if [[ $FAMILY == 'wsbeta' ]]; then - python -m generate_family_file 'http://'$LANGUAGE'.wikisource.beta.wmflabs.org/' 'wsbeta' 'y' ; + python -m generate_family_file 'http://'$LANGUAGE'.wikisource.beta.wmflabs.org/' 'wsbeta' 'n' ; fi - python -W error::UserWarning -m generate_user_files -dir:~/.pywikibot/ -family:$FAMILY -lang:$LANGUAGE -v -user:"$PYWIKIBOT2_USERNAME" -- To view, visit https://gerrit.wikimedia.org/r/235724 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I9c9e43eeb8732ec653797e9ba5f9e231019546dd Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

8 years, 8 months

Failed: wikimedia/pywikibot-core#2765 (master - 3e2444a)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #2765 Status: Failed Duration: 41 minutes and 55 seconds Commit: 3e2444a (master) Author: John Vandenberg Message: Multi site family detection regression 53141af26 migrated the site detection logic from generate_family_file.py to pywikibot.site_detect.MWSite. (Merlijn van Deen was Author: for attribution only) This introduced an error as MWSite.langs failed, preventing multi site family from being detected, and thus always creating a single site family. Bug: T74846 Change-Id: I157c2b48cf49dcc7b94eca439f6f3d3e45bd969e View the changeset: https://github.com/wikimedia/pywikibot-core/compare/ad5477edce4a...3e2444a3… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/78529615 -- You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications

8 years, 8 months

[Gerrit] Multi site family detection regression - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Multi site family detection regression ...................................................................... Multi site family detection regression 53141af26 migrated the site detection logic from generate_family_file.py to pywikibot.site_detect.MWSite. (Merlijn van Deen was Author: for attribution only) This introduced an error as MWSite.langs failed, preventing multi site family from being detected, and thus always creating a single site family. Bug: T74846 Change-Id: I157c2b48cf49dcc7b94eca439f6f3d3e45bd969e --- M pywikibot/site_detect.py 1 file changed, 2 insertions(+), 3 deletions(-) Approvals: XZise: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/site_detect.py b/pywikibot/site_detect.py index 7dd2587..23e3c22 100644 --- a/pywikibot/site_detect.py +++ b/pywikibot/site_detect.py @@ -102,9 +102,8 @@ if 'error' in iw: raise RuntimeError('%s - %s' % (iw['error']['code'], iw['error']['info'])) - self.langs = [wiki for wiki in iw['query']['interwikimap'] - if u'language' in wiki] - return self.langs + return [wiki for wiki in iw['query']['interwikimap'] + if u'language' in wiki] def _parse_pre_117(self, data): """Parse HTML.""" -- To view, visit https://gerrit.wikimedia.org/r/235662 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I157c2b48cf49dcc7b94eca439f6f3d3e45bd969e Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

8 years, 8 months

Errored: wikimedia/pywikibot-core#2764 (master - ad5477e)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #2764 Status: Errored Duration: 17 minutes and 30 seconds Commit: ad5477e (master) Author: jenkins-bot Message: Merge "Improve HTML parser to detect all IWM MW sites" View the changeset: https://github.com/wikimedia/pywikibot-core/compare/3663c12c8d01...ad5477ed… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/78496120 -- You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications

8 years, 8 months

[Gerrit] Improve HTML parser to detect all IWM MW sites - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Improve HTML parser to detect all IWM MW sites ...................................................................... Improve HTML parser to detect all IWM MW sites The HTML parser used only EditURI to determine the api.php endpoint. Add HTML parser support for * OpenSearch's opensearch_desc.php, introduced in MW 1.8, and * Resource Loader's load.php, introduced in MW 1.17. Also raise an exception for unsupported versions, lower than MW 1.14. And fix flake8 issues in site_detect module. Bug: T111007 Change-Id: Iebadd7f782ab1b471d3de71fa1c5a52c0d6f1018 --- M pywikibot/site_detect.py M tests/site_detect_tests.py M tox.ini 3 files changed, 244 insertions(+), 56 deletions(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve XZise: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/site_detect.py b/pywikibot/site_detect.py index 13bfe27..7dd2587 100644 --- a/pywikibot/site_detect.py +++ b/pywikibot/site_detect.py @@ -13,17 +13,21 @@ import json import re -from distutils.version import LooseVersion as V +import pywikibot from pywikibot.comms.http import fetch -from pywikibot.tools import PY2, PYTHON_VERSION +from pywikibot.exceptions import ServerError +from pywikibot.tools import MediaWikiVersion, PY2, PYTHON_VERSION if not PY2: from html.parser import HTMLParser - from urllib.parse import urljoin + from urllib.parse import urljoin, urlparse else: - from HTMLParser import HTMLParser - from urlparse import urljoin + try: + from future.backports.html.parser import HTMLParser + except ImportError: + from HTMLParser import HTMLParser + from urlparse import urljoin, urlparse class MWSite(object): @@ -38,25 +42,59 @@ REwgVersion = re.compile(r'wgVersion ?= ?"([^"]*)"') def __init__(self, fromurl): - self.fromurl = fromurl + """ + Constructor. + + @raises ServerError: a server error occurred while loading the site + @raises Timeout: a timeout occurred while loading the site + @raises RuntimeError: Version not found or version less than 1.14 + """ if fromurl.endswith("$1"): fromurl = fromurl[:-2] - data = fetch(fromurl).content + r = fetch(fromurl) + if r.status == 503: + raise ServerError('Service Unavailable') - wp = WikiHTMLPageParser() + if fromurl != r.data.url: + pywikibot.log('{0} redirected to {1}'.format(fromurl, r.data.url)) + fromurl = r.data.url + + self.fromurl = fromurl + + data = r.content + + wp = WikiHTMLPageParser(fromurl) wp.feed(data) - try: - self.version = wp.generator.replace("MediaWiki ", "") - except Exception: - self.version = "0.0" - if V(self.version) < V("1.17.0"): + self.version = wp.version + self.server = wp.server + self.scriptpath = wp.scriptpath + self.articlepath = None + + try: self._parse_pre_117(data) - else: - self._parse_post_117(wp, fromurl) + except Exception as e: + pywikibot.log('MW pre-1.17 detection failed: {0!r}'.format(e)) + + if self.api: + try: + self._parse_post_117() + except Exception as e: + pywikibot.log('MW 1.17+ detection failed: {0!r}'.format(e)) + + if not self.version: + self._fetch_old_version() + + if not self.api: + raise RuntimeError('Unsupported url: {0}'.format(self.fromurl)) + + if (not self.version or + self.version < MediaWikiVersion('1.14')): + raise RuntimeError('Unsupported version: {0}'.format(self.version)) @property def langs(self): + """Build interwikimap.""" response = fetch( self.api + "?action=query&meta=siteinfo&siprop=interwikimap&sifilteriw=local&format=json") @@ -69,52 +107,78 @@ return self.langs def _parse_pre_117(self, data): + """Parse HTML.""" if not self.REwgEnableApi.search(data): - print("*** WARNING: Api does not seem to be enabled on %s" - % self.fromurl) + pywikibot.log( + 'wgEnableApi is not enabled in HTML of %s' + % self.fromurl) try: - self.version = self.REwgVersion.search(data).groups()[0] + self.version = MediaWikiVersion( + self.REwgVersion.search(data).groups(0)) except AttributeError: - self.version = None + pass self.server = self.REwgServer.search(data).groups()[0] self.scriptpath = self.REwgScriptPath.search(data).groups()[0] self.articlepath = self.REwgArticlePath.search(data).groups()[0] self.lang = self.REwgContentLanguage.search(data).groups()[0] + def _fetch_old_version(self): + """Extract the version from API help with ?version enabled.""" if self.version is None: - # try to get version using api try: - d = json.loads(fetch(self.api + '?version&format=json').content) + d = fetch(self.api + '?version&format=json').content + try: + d = json.loads(d) + except ValueError: + # Fallback for old versions which didnt wrap help in json + d = {'error': {'*': d}} + self.version = list(filter( lambda x: x.startswith("MediaWiki"), [l.strip() for l in d['error']['*'].split("\n")]))[0].split()[1] except Exception: pass + else: + self.version = MediaWikiVersion(self.version) - def _parse_post_117(self, wp, fromurl): - apipath = wp.edituri.split("?")[0] - fullurl = urljoin(fromurl, apipath) - response = fetch(fullurl + '?action=query&meta=siteinfo&format=json') + def _parse_post_117(self): + """Parse 1.17+ siteinfo data.""" + response = fetch(self.api + '?action=query&meta=siteinfo&format=json') info = json.loads(response.content)['query']['general'] - self.server = urljoin(fromurl, info['server']) + self.version = MediaWikiVersion.from_generator(info['generator']) + if self.version < MediaWikiVersion('1.17'): + return + + self.server = urljoin(self.fromurl, info['server']) for item in ['scriptpath', 'articlepath', 'lang']: setattr(self, item, info[item]) - def __cmp__(self, other): + def __eq__(self, other): + """Return True if equal to other.""" return (self.server + self.scriptpath == other.server + other.scriptpath) def __hash__(self): + """Get hashable representation.""" return hash(self.server + self.scriptpath) @property def api(self): + """ + Get api URL. + + @rtype: str or None + """ + if self.server is None or self.scriptpath is None: + return + return self.server + self.scriptpath + "/api.php" @property def iwpath(self): + """Get article path URL.""" return self.server + self.articlepath @@ -122,18 +186,85 @@ """Wiki HTML page parser.""" - def __init__(self): + def __init__(self, url): + """Constructor.""" if PYTHON_VERSION < (3, 4): HTMLParser.__init__(self) else: super().__init__(convert_charrefs=True) + self.url = urlparse(url) self.generator = None + self.version = None + self._parsed_url = None + self.server = None + self.scriptpath = None + + def set_version(self, value): + """Set highest version.""" + if self.version and value < self.version: + return + + self.version = value + + def set_api_url(self, url): + """Set api_url.""" + url = url.split('.php', 1)[0] + (value, script_name) = url.rsplit('/', 1) + if script_name not in ('api', 'load', 'opensearch_desc'): + return + + if script_name == 'load': + self.set_version(MediaWikiVersion('1.17.0')) + if self._parsed_url: + # A Resource Loader link is less reliable than other links. + # Resource Loader can load resources from a different site. + # e.g. http://kino.skripov.com/index.php/$1 + # loads resources from http://megawiki.net/ + return + + new_parsed_url = urlparse(value) + if self._parsed_url: + assert new_parsed_url.path == self._parsed_url.path + + if not new_parsed_url.scheme or not new_parsed_url.netloc: + new_parsed_url = urlparse( + '{0}://{1}{2}'.format( + new_parsed_url.scheme or self.url.scheme, + new_parsed_url.netloc or self.url.netloc, + new_parsed_url.path)) + else: + if self._parsed_url: + # allow upgrades to https, but not downgrades + if self._parsed_url.scheme == 'https': + if new_parsed_url.scheme != self._parsed_url.scheme: + return + + # allow http://www.brickwiki.info/ vs http://brickwiki.info/ + if (new_parsed_url.netloc in self._parsed_url.netloc or + self._parsed_url.netloc in new_parsed_url.netloc): + return + + assert new_parsed_url == self._parsed_url, '{0} != {1}'.format( + self._parsed_url, new_parsed_url) + + self._parsed_url = new_parsed_url + self.server = '{0}://{1}'.format( + self._parsed_url.scheme, self._parsed_url.netloc) + self.scriptpath = self._parsed_url.path def handle_starttag(self, tag, attrs): + """Handle an opening tag.""" attrs = dict(attrs) if tag == "meta": if attrs.get('name') == 'generator': self.generator = attrs["content"] - if tag == "link": - if attrs.get('rel') == 'EditURI': - self.edituri = attrs["href"] + try: + self.version = MediaWikiVersion.from_generator( + self.generator) + except ValueError: + pass + elif tag == 'link' and 'rel' in attrs and 'href' in attrs: + if attrs['rel'] in ('EditURI', 'stylesheet', 'search'): + self.set_api_url(attrs['href']) + elif tag == 'script' and 'src' in attrs: + self.set_api_url(attrs['src']) diff --git a/tests/site_detect_tests.py b/tests/site_detect_tests.py index f13105f..bcad8cb 100644 --- a/tests/site_detect_tests.py +++ b/tests/site_detect_tests.py @@ -11,8 +11,9 @@ from requests.exceptions import Timeout +from pywikibot.exceptions import ServerError from pywikibot.site_detect import MWSite -from pywikibot.tools import PY2 +from pywikibot.tools import MediaWikiVersion, PY2 from tests.aspects import unittest, TestCase @@ -23,10 +24,6 @@ class TestWikiSiteDetection(TestCase): """Test Case for MediaWiki detection and site object creation.""" - - family = 'meta' - code = 'meta' - net = True def setUp(self): """Set up test.""" @@ -75,7 +72,7 @@ self.all += [url] try: site = MWSite(url) - except Timeout as e: + except (ServerError, Timeout) as e: self.skips[url] = e return except Exception as e: @@ -88,7 +85,7 @@ self.assertIsNone(site) else: self.assertIsInstance(site, result) - self.passes[url] = result + self.passes[url] = site except AssertionError as error: self.failures[url] = error @@ -102,15 +99,24 @@ def assertAllPass(self): """Assert that all urls were detected as a MediaWiki site.""" - self.assertEqual(len(self.passes), len(self.all) - len(self.skips)) - self.assertEqual(len(self.failures), 0) - self.assertEqual(len(self.errors), 0) + self.assertEqual(set(self.passes), set(self.all) - set(self.skips)) + self.assertEqual(self.failures, {}) + self.assertEqual(self.errors, {}) def assertAllError(self): """Assert that all urls were not detected as a MediaWiki site.""" - self.assertEqual(len(self.passes), 0) - self.assertEqual(len(self.failures), 0) - self.assertEqual(len(self.errors), len(self.all) - len(self.skips)) + self.assertEqual(self.passes, {}) + self.assertEqual(self.failures, {}) + self.assertEqual(set(self.errors), set(self.all) - set(self.skips)) + + +class InterWikiMapDetection(TestWikiSiteDetection): + + """Test all urls on the interwiki map.""" + + family = 'meta' + code = 'meta' + net = True def test_IWM(self): """Test the load_site method for MW sites on the IWM list.""" @@ -133,39 +139,85 @@ self.errors[url] = error else: try: - self.assertIsInstance(version, basestring) - self.assertRegex(version, r'^\d\.\d+.*') + self.assertIsInstance(version, MediaWikiVersion) self.passes[url] = site except AssertionError as error: print('failed to parse version of ' + url) self.failures[url] = error + +class SiteDetectionTestCase(TestWikiSiteDetection): + + """Test all urls on the interwiki map.""" + + net = True + def test_detect_site(self): """Test detection of MediaWiki sites.""" self.assertSite('http://botwiki.sno.cc/wiki/$1') - self.assertSite('http://glossary.reuters.com/index.php?title=$1') - self.assertSite('http://www.livepedia.gr/index.php?title=$1') self.assertSite('http://guildwars.wikia.com/wiki/$1') - self.assertSite('http://www.hrwiki.org/index.php/$1') + self.assertSite('http://www.hrwiki.org/index.php/$1') # v 1.15 self.assertSite('http://www.proofwiki.org/wiki/$1') self.assertSite( 'http://www.ck-wissen.de/ckwiki/index.php?title=$1') self.assertSite('http://en.citizendium.org/wiki/$1') self.assertSite( 'http://www.lojban.org/tiki/tiki-index.php?page=$1') - self.assertSite('http://www.EcoReality.org/wiki/$1') self.assertSite('http://www.wikichristian.org/index.php?title=$1') - self.assertSite('http://wikitree.org/index.php?title=$1') + self.assertSite('https://en.wikifur.com/wiki/$1') + self.assertSite('http://bluwiki.com/go/$1') + self.assertSite('http://kino.skripov.com/index.php/$1') + self.assertAllPass() + + def test_wikisophia(self): + """Test wikisophia.org which has redirect problems.""" + # /index.php?title=$1 reports 404, however a wiki exists there, + # but the API is also hidden. + self.assertNoSite('http://wikisophia.org/index.php?title=$1') + self.assertAllError() + + def test_pre_114_sites(self): + """Test pre 1.14 sites which should be detected as unsupported.""" + # v1.12 + self.assertNoSite('http://www.livepedia.gr/index.php?title=$1') + # v1.11 + self.assertNoSite('http://www.wikifon.org/$1') + self.assertNoSite('http://glossary.reuters.com/index.php?title=$1') + # v1.11, with no query module + self.assertNoSite('http://wikitree.org/index.php?title=$1') + # v1.9 + self.assertNoSite('http://www.wikinvest.com/$1') + self.assertAllError() + + def test_non_standard_version_sites(self): + """Test non-standard version string sites.""" + self.assertSite('https://wiki.gentoo.org/wiki/$1') + self.assertSite('http://wiki.arabeyes.org/$1') + self.assertSite('http://tfwiki.net/wiki/$1') self.assertAllPass() def test_detect_failure(self): """Test detection failure for MediaWiki sites with an API.""" - self.assertNoSite('https://en.wikifur.com/wiki/$1') + # SSL certificate verification fails + self.assertNoSite('http://hackerspaces.org/wiki/$1') + self.assertAllError() + + @unittest.expectedFailure + def test_api_hidden(self): + """Test MediaWiki sites with a hidden enabled API.""" # api.php is not available self.assertNoSite('http://wiki.animutationportal.com/index.php/$1') - # API is disabled + # HTML looks like it has an API, but redirect rules prevent access + self.assertNoSite('http://www.EcoReality.org/wiki/$1') + self.assertAllError() + + def test_api_disabled(self): + """Test MediaWiki sites without an enabled API.""" self.assertNoSite('http://wiki.linuxquestions.org/wiki/$1') - # offline + self.assertAllError() + + def test_offline_sites(self): + """Test offline sites.""" self.assertNoSite('http://seattlewiki.org/wiki/$1') self.assertAllError() @@ -181,7 +233,6 @@ def test_detect_nosite(self): """Test detection of non-wiki sites.""" - self.assertNoSite('http://bluwiki.com/go/$1') self.assertNoSite('http://www.imdb.com/name/nm$1/') self.assertNoSite('http://www.ecyrd.com/JSPWiki/Wiki.jsp?page=$1') self.assertNoSite('http://operawiki.info/$1') @@ -192,8 +243,13 @@ self.assertNoSite( 'http://www.merriam-webster.com/cgi-bin/dictionary?book=Dictionary&va=$1') self.assertNoSite('http://arxiv.org/abs/$1') + self.assertAllError() + + def test_musicbrainz_doc(self): + """Test http://musicbrainz.org/doc/ which has a page 'api.php'.""" + # Possible false positive caused by the existance of a page + # called http://musicbrainz.org/doc/api.php self.assertNoSite('http://musicbrainz.org/doc/$1') - self.assertNoSite('http://wiki.animutationportal.com/index.php/$1') self.assertAllError() diff --git a/tox.ini b/tox.ini index 728dc58..15ecea9 100644 --- a/tox.ini +++ b/tox.ini @@ -64,6 +64,7 @@ pywikibot/pagegenerators.py \ pywikibot/plural.py \ pywikibot/proofreadpage.py \ + pywikibot/site_detect.py \ pywikibot/textlib.py \ pywikibot/throttle.py \ pywikibot/titletranslate.py \ -- To view, visit https://gerrit.wikimedia.org/r/230512 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Iebadd7f782ab1b471d3de71fa1c5a52c0d6f1018 Gerrit-PatchSet: 16 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

8 years, 8 months

Passed: wikimedia/pywikibot-core#2763 (master - 3663c12)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #2763 Status: Passed Duration: 36 minutes and 49 seconds Commit: 3663c12 (master) Author: jenkins-bot Message: Merge "pagefromfile.py: -start argument conflict" View the changeset: https://github.com/wikimedia/pywikibot-core/compare/a2a6c800337a...3663c12c… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/78446777 -- You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications

8 years, 8 months

[Gerrit] pagefromfile.py: -start argument conflict - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: pagefromfile.py: -start argument conflict ...................................................................... pagefromfile.py: -start argument conflict Deprecated -start for -begin to avoid conflict with pagegenerators.py params, if pagegenerators will be used in future. Both parameters survive with same meaning, for backward compatibility. Start will be replaced completely in future. Bug: T100628 Change-Id: I84ab19e144199de7a7553b2674d10cfbf5fc0ab7 --- M scripts/pagefromfile.py 1 file changed, 11 insertions(+), 3 deletions(-) Approvals: XZise: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/pagefromfile.py b/scripts/pagefromfile.py index 1eda597..b1411f9 100755 --- a/scripts/pagefromfile.py +++ b/scripts/pagefromfile.py @@ -17,7 +17,8 @@ Specific arguments: --start:xxx Specify the text that marks the beginning of a page +-begin:xxx Specify the text that marks the beginning of a page +-start:xxx (deprecated) -end:xxx Specify the text that marks the end of a page -file:xxx Give the filename we are getting our material from (default: dict.txt) @@ -54,7 +55,7 @@ """ # # (C) Andre Engels, 2004 -# (C) Pywikibot team, 2005-2014 +# (C) Pywikibot team, 2005-2015 # # Distributed under the terms of the MIT license. # @@ -66,9 +67,11 @@ import os import re import codecs +from warnings import warn import pywikibot from pywikibot import config, Bot, i18n +from pywikibot.exceptions import ArgumentDeprecationWarning class NoTitle(Exception): @@ -278,8 +281,13 @@ notitle = False for arg in pywikibot.handle_args(args): - if arg.startswith("-start:"): + if arg.startswith('-start:'): pageStartMarker = arg[7:] + warn('-start param (text that marks the beginning) of a page has been ' + 'deprecated in favor of begin; make sure to use the updated param.', + ArgumentDeprecationWarning) + elif arg.startswith('-begin:'): + pageStartMarker = arg[len('-begin:'):] elif arg.startswith("-end:"): pageEndMarker = arg[5:] elif arg.startswith("-file:"): -- To view, visit https://gerrit.wikimedia.org/r/214717 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I84ab19e144199de7a7553b2674d10cfbf5fc0ab7 Gerrit-PatchSet: 7 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

8 years, 8 months

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits September 2015