jenkins-bot has submitted this change and it was merged.
Change subject: Add ability to ignore SSL certificate errors
......................................................................
Add ability to ignore SSL certificate errors
The following registered families have HTTPS,
but there are SSL certificate errors:
- lockwiki
- omegawiki
- vikidia
HTTPS is enabled for those families, with SSL cert errors ignored.
Bug: 68794
Change-Id: Ifd33bc2ac60a268f6fad2770c18ecd0ebc5a2b6d
---
M pywikibot/comms/http.py
M pywikibot/comms/threadedhttp.py
M pywikibot/families/lockwiki_family.py
M pywikibot/families/omegawiki_family.py
M pywikibot/families/vikidia_family.py
M pywikibot/family.py
6 files changed, 47 insertions(+), 2 deletions(-)
Approvals:
John Vandenberg: Looks good to me, but someone else must approve
XZise: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/comms/http.py b/pywikibot/comms/http.py
index f2f6eaa..69fb686 100644
--- a/pywikibot/comms/http.py
+++ b/pywikibot/comms/http.py
@@ -234,6 +234,9 @@
else:
host = site.hostname()
baseuri = urlparse.urljoin("%s://%s" % (proto, host), uri)
+
+ kwargs.setdefault("disable_ssl_certificate_validation",
+ site.ignore_certificate_error())
else:
baseuri = uri
diff --git a/pywikibot/comms/threadedhttp.py b/pywikibot/comms/threadedhttp.py
index 790190b..7d0b632 100644
--- a/pywikibot/comms/threadedhttp.py
+++ b/pywikibot/comms/threadedhttp.py
@@ -350,6 +350,11 @@
if item is None:
pywikibot.debug(u"Shutting down thread.", _logger)
return
+
+ # This needs to be set per request, however it is only used
+ # the first time the pooled connection is created.
+ self.http.disable_ssl_certificate_validation = \
+ item.kwargs.pop('disable_ssl_certificate_validation', False)
try:
item.data = self.http.request(*item.args, **item.kwargs)
finally:
@@ -459,7 +464,7 @@
self.response.get(k.lower(), None)
if k not in self.response:
return []
- #return self.response[k].split(re.compile(',\\s*'))
+ # return self.response[k].split(re.compile(',\\s*'))
# httplib2 joins multiple values for the same header
# using ','. but the netscape cookie format uses ','
diff --git a/pywikibot/families/lockwiki_family.py b/pywikibot/families/lockwiki_family.py
index 3b30fba..0a4d0a4 100644
--- a/pywikibot/families/lockwiki_family.py
+++ b/pywikibot/families/lockwiki_family.py
@@ -30,3 +30,11 @@
def nicepath(self, code):
"""Return the nice article path for this family."""
return "%s/" % self.path(self, code)
+
+ def protocol(self, code):
+ """Return https as the protocol for this family."""
+ return "https"
+
+ def ignore_certificate_error(self, code):
+ """Ignore certificate errors."""
+ return True # has a different domain in its certificate.
diff --git a/pywikibot/families/omegawiki_family.py b/pywikibot/families/omegawiki_family.py
index aa234c5..7661dac 100644
--- a/pywikibot/families/omegawiki_family.py
+++ b/pywikibot/families/omegawiki_family.py
@@ -40,3 +40,11 @@
def apipath(self, code):
"""Return the path to api.php for this family."""
return '/api.php'
+
+ def protocol(self, code):
+ """Return https as the protocol for this family."""
+ return "https"
+
+ def ignore_certificate_error(self, code):
+ """Ignore certificate errors."""
+ return True # has an expired certificate.
diff --git a/pywikibot/families/vikidia_family.py b/pywikibot/families/vikidia_family.py
index 4428479..e02e803 100644
--- a/pywikibot/families/vikidia_family.py
+++ b/pywikibot/families/vikidia_family.py
@@ -51,3 +51,11 @@
# Most wikis nowadays use UTF-8, but change this if yours uses
# a different encoding
return 'utf-8'
+
+ def protocol(self, code):
+ """Return https as the protocol for this family."""
+ return "https"
+
+ def ignore_certificate_error(self, code):
+ """Ignore certificate errors."""
+ return True # has self-signed certificate for a different domain.
diff --git a/pywikibot/family.py b/pywikibot/family.py
index d2e6a6d..9549657 100644
--- a/pywikibot/family.py
+++ b/pywikibot/family.py
@@ -917,7 +917,9 @@
# Methods
def protocol(self, code):
"""
- Can be overridden to return 'https'. Other protocols are not supported.
+ The protocol to use to connect to the site.
+
+ May be overridden to return 'https'. Other protocols are not supported.
@param code: language code
@type code: string
@@ -926,6 +928,17 @@
"""
return 'http'
+ def ignore_certificate_error(self, code):
+ """
+ Return whether a HTTPS certificate error should be ignored.
+
+ @param code: language code
+ @type code: string
+ @return: flag to allow access if certificate has an error.
+ @rtype: bool
+ """
+ return False
+
def hostname(self, code):
"""The hostname to use for standard http connections."""
return self.langs[code]
--
To view, visit https://gerrit.wikimedia.org/r/160189
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ifd33bc2ac60a268f6fad2770c18ecd0ebc5a2b6d
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: Access any mediawiki site without a family file
......................................................................
Access any mediawiki site without a family file
AutoFamily dynamically provides Family capabilities without a
special class file, using only the wiki URL. The URL may be
only the protocol and hostname, or a URL to the API php script.
pywikibot.site.Family uses AutoFamily if it finds a URL has been
provided instead of a file path.
To use a mediawiki site without a family file, add the following
to user-config.py
family_files['mhwp'] = 'https://mh.wikipedia.org/'
usernames['whwp']['*']=u'xyz'
or
family_files['w3c'] = 'https://www.w3.org/wiki/api.php'
Bug: 70795
Change-Id: I9ad83fe996bc240fb9d5f27b866a022fccd313c0
---
M pywikibot/family.py
M pywikibot/site.py
2 files changed, 41 insertions(+), 0 deletions(-)
Approvals:
XZise: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/family.py b/pywikibot/family.py
index d2e6a6d..ed6baf2 100644
--- a/pywikibot/family.py
+++ b/pywikibot/family.py
@@ -8,9 +8,15 @@
__version__ = '$Id$'
#
+import sys
import logging
import re
import collections
+
+if sys.version_info[0] == 2:
+ from urlparse import urlparse
+else:
+ from urllib.parse import urlparse
import pywikibot
from pywikibot import config2 as config
@@ -1091,4 +1097,30 @@
return ('commons', 'commons')
def protocol(self, code):
+ """Return 'https' as the protocol."""
return 'https'
+
+
+class AutoFamily(Family):
+
+ """Family that automatically loads the site configuration."""
+
+ def __init__(self, name, url, site=None):
+ """Constructor."""
+ super(AutoFamily, self).__init__()
+ self.name = name
+ self.url = urlparse(url)
+ self.langs = {
+ name: self.url.netloc
+ }
+
+ def protocol(self, code):
+ """Return the protocol of the URL."""
+ return self.url.scheme
+
+ def scriptpath(self, code):
+ """Extract the script path from the URL."""
+ if self.url.path.endswith('/api.php'):
+ return self.url.path[0:-8]
+ else:
+ return super(AutoFamily, self).scriptpath(code)
diff --git a/pywikibot/site.py b/pywikibot/site.py
index a5c1754..630a5aa 100644
--- a/pywikibot/site.py
+++ b/pywikibot/site.py
@@ -28,6 +28,7 @@
import pywikibot
from pywikibot import config
+from pywikibot.family import AutoFamily
from pywikibot.tools import itergroup, deprecated, deprecate_arg
from pywikibot.throttle import Throttle
from pywikibot.data import api
@@ -118,6 +119,14 @@
if fam in _families:
return _families[fam]
+ if fam in config.family_files:
+ family_file = config.family_files[fam]
+
+ if family_file.startswith('http://') or family_file.startswith('https://'):
+ myfamily = AutoFamily(fam, family_file)
+ _families[fam] = myfamily
+ return _families[fam]
+
try:
# Ignore warnings due to dots in family names.
import warnings
--
To view, visit https://gerrit.wikimedia.org/r/160207
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I9ad83fe996bc240fb9d5f27b866a022fccd313c0
Gerrit-PatchSet: 2
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: Nemo bis <federicoleva(a)tiscali.it>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: [FIX] Search: User pref independent and only srredirects on < 1.23
......................................................................
[FIX] Search: User pref independent and only srredirects on < 1.23
The search is independent of the user preferences, so they by
default always return from namespace 0.
In 1.23 the parameter 'srredirect' for 'list=search' was removed
because it always return redirects.
Change-Id: I7a9ad89a95ecd611acaca61b1604f95d3f000615
---
M pywikibot/site.py
M tests/site_tests.py
2 files changed, 4 insertions(+), 4 deletions(-)
Approvals:
Mpaa: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/site.py b/pywikibot/site.py
index a5c1754..238c507 100644
--- a/pywikibot/site.py
+++ b/pywikibot/site.py
@@ -3250,7 +3250,8 @@
@param namespaces: search only in these namespaces (defaults to 0)
@type namespaces: list of ints, or an empty list to signal all
namespaces
- @param getredirects: if True, include redirects in results
+ @param getredirects: if True, include redirects in results. Since
+ version MediaWiki 1.23 it will always return redirects.
@param content: if True, load the current content of each iterated page
(default False)
@@ -3268,7 +3269,7 @@
gsrsearch=searchstring, gsrwhat=where,
namespaces=namespaces, step=step,
total=total, g_content=content)
- if getredirects:
+ if getredirects and LV(self.version()) < LV('1.23'):
srgen.request["gsrredirects"] = ""
return srgen
diff --git a/tests/site_tests.py b/tests/site_tests.py
index 778d5e6..b2fb360 100644
--- a/tests/site_tests.py
+++ b/tests/site_tests.py
@@ -776,8 +776,7 @@
self.assertLessEqual(len(se), 100)
self.assertTrue(all(isinstance(hit, pywikibot.Page)
for hit in se))
- search_ns = [ns.id for ns in mysite.get_searched_namespaces(force=True)]
- self.assertTrue(all(hit.namespace() in search_ns for hit in se))
+ self.assertTrue(all(hit.namespace() == 0 for hit in se))
for hit in mysite.search("common", namespaces=4, total=5):
self.assertIsInstance(hit, pywikibot.Page)
self.assertEqual(hit.namespace(), 4)
--
To view, visit https://gerrit.wikimedia.org/r/160233
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I7a9ad89a95ecd611acaca61b1604f95d3f000615
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: merge two http.request calls in Request.submit
......................................................................
merge two http.request calls in Request.submit
Request.submit may submit using MIME or URL form.
Each approach invoked http.request, and then error handling
is performed in a uniform manner.
This patch moves the invocation out of each branch, so
changes to the http request parameters can be added in one
place rather than twice.
Change-Id: I47cbf80dac5b29f6d9548481815d47140aa0c575
---
M pywikibot/data/api.py
1 file changed, 8 insertions(+), 6 deletions(-)
Approvals:
Mpaa: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/data/api.py b/pywikibot/data/api.py
index 0e846ae..bb086fc 100644
--- a/pywikibot/data/api.py
+++ b/pywikibot/data/api.py
@@ -431,13 +431,15 @@
eoh = body.find(marker)
body = body[eoh + len(marker):]
# retrieve the headers from the MIME object
- mimehead = dict(list(container.items()))
- rawdata = http.request(self.site, uri, method="POST",
- headers=mimehead, body=body)
+ headers = dict(list(container.items()))
else:
- rawdata = http.request(self.site, uri, method="POST",
- headers={'Content-Type': 'application/x-www-form-urlencoded'},
- body=paramstring)
+ headers = {'Content-Type': 'application/x-www-form-urlencoded'}
+ body = paramstring
+
+ rawdata = http.request(
+ self.site, uri, method="POST",
+ headers=headers, body=body)
+
# import traceback
# traceback.print_stack()
# print rawdata
--
To view, visit https://gerrit.wikimedia.org/r/160190
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I47cbf80dac5b29f6d9548481815d47140aa0c575
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: No KeyError when the family does not exist
......................................................................
No KeyError when the family does not exist
If the user has a username in user-config.py for a family
which does not exist, pywikibot will not load, failing in config2.py
with a KeyError.
This change silently accepts unknown values for the three dicts
which are pre-populated with family names:
- usernames
- sysopnames
- disambiguation_comment
Change-Id: I614096aa2396f8a0794225da4ffad891f20bf6ab
---
M pywikibot/config2.py
1 file changed, 10 insertions(+), 5 deletions(-)
Approvals:
Mpaa: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/config2.py b/pywikibot/config2.py
index b021a05..0b04b5a 100644
--- a/pywikibot/config2.py
+++ b/pywikibot/config2.py
@@ -24,6 +24,7 @@
import os
import sys
+import collections
# Please keep _imported_modules in sync with the imports above
_imported_modules = ('os', 'sys')
@@ -71,9 +72,9 @@
# If you have a unique syop account for all languages of a family,
# you can use '*'
# sysopnames['myownwiki']['*'] = 'mySingleUsername'
-usernames = {}
-sysopnames = {}
-disambiguation_comment = {}
+usernames = collections.defaultdict(dict)
+sysopnames = collections.defaultdict(dict)
+disambiguation_comment = collections.defaultdict(dict)
# User agent format.
# For the meaning and more help in customization see:
@@ -220,6 +221,7 @@
if file_name.endswith("_family.py"):
family_name = file_name[:-len("_family.py")]
register_family_file(family_name, os.path.join(folder_path, file_name))
+
# Get the names of all known families, and initialize with empty dictionaries.
# ‘families/’ is a subdirectory of the directory in which config2.py is found.
@@ -722,8 +724,11 @@
_uc = {}
for _key, _val in _glv.items():
if isinstance(_val, dict):
- _uc[_key] = {}
- if len(_val.keys()) > 0:
+ if isinstance(_val, collections.defaultdict):
+ _uc[_key] = collections.defaultdict(dict)
+ else:
+ _uc[_key] = {}
+ if len(_val) > 0:
_uc[_key].update(_val)
else:
_uc[_key] = _val
--
To view, visit https://gerrit.wikimedia.org/r/160203
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I614096aa2396f8a0794225da4ffad891f20bf6ab
Gerrit-PatchSet: 2
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>