Pywikibot-commits

pywikibot-commits@lists.wikimedia.org

1 participants
13410 discussions

[Gerrit] Use the API to retrieve redirects to the category redirect t... - change (pywikibot/core)
by jenkins-bot (Code Review) 04 Oct '13

04 Oct '13

jenkins-bot has submitted this change and it was merged. Change subject: Use the API to retrieve redirects to the category redirect template (this time with caching) ...................................................................... Use the API to retrieve redirects to the category redirect template (this time with caching) Change-Id: I7af283b3b7a18bd713627cb5f1d8667773e3f3c6 --- M pywikibot/families/wikipedia_family.py M pywikibot/family.py 2 files changed, 24 insertions(+), 10 deletions(-) Approvals: Merlijn van Deen: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/families/wikipedia_family.py b/pywikibot/families/wikipedia_family.py index 0149a8e..67f3fd4 100644 --- a/pywikibot/families/wikipedia_family.py +++ b/pywikibot/families/wikipedia_family.py @@ -54,11 +54,7 @@ 'arz': (u'تحويل تصنيف',), 'cs': (u'Zastaralá kategorie',), 'da': (u'Kategoriomdirigering',), - 'en': (u'Category redirect', - u'Category Redirect', - u"Categoryredirect", - u'Catredirect', - u'Cat redirect',), + 'en': (u'Category redirect',), 'es': (u'Categoría redirigida',), 'eu': (u'Kategoria redirect',), 'fa': (u'رده بهتر', diff --git a/pywikibot/family.py b/pywikibot/family.py index 59484c9..a7c58d4 100644 --- a/pywikibot/family.py +++ b/pywikibot/family.py @@ -653,7 +653,6 @@ } # A list of category redirect template names in different languages - # Note: It *is* necessary to list template redirects here self.category_redirect_templates = { '_default': [] } @@ -858,15 +857,34 @@ % {'language_code': code}) def category_redirects(self, code, fallback="_default"): - if code in self.category_redirect_templates: - return self.category_redirect_templates[code] - elif fallback: - return self.category_redirect_templates[fallback] + if not hasattr(self, "_catredirtemplates") or code not in self._catredirtemplates: + self.get_cr_templates(code, fallback) + if code in self._catredirtemplates: + return self._catredirtemplates[code] else: raise KeyError( "ERROR: title for category redirect template in language '%s' unknown" % code) + def get_cr_templates(self, code, fallback): + if not hasattr(self, "_catredirtemplates"): + self._catredirtemplates = {} + if code in self.category_redirect_templates: + cr_template = self.category_redirect_templates[code][0] + else: + cr_template = self.category_redirect_templates[fallback][0] + # start with list of category redirect templates from family file + cr_page = pywikibot.Page(pywikibot.Site(code, self), + "Template:" + cr_template) + cr_list = list(self.category_redirect_templates[code]) + # retrieve all redirects to primary template from API, + # add any that are not already on the list + for t in cr_page.backlinks(filterRedirects=True, namespaces=10): + newtitle = t.title(withNamespace=False) + if newtitle not in cr_list: + cr_list.append(newtitle) + self._catredirtemplates[code] = cr_list + def disambig(self, code, fallback='_default'): if code in self.disambiguationTemplates: return self.disambiguationTemplates[code] -- To view, visit https://gerrit.wikimedia.org/r/85203 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I7af283b3b7a18bd713627cb5f1d8667773e3f3c6 Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Russell Blau <russblau(a)imapmail.org> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Legoktm <legoktm.wikipedia(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Russell Blau <russblau(a)imapmail.org> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] tests: Use unittest2 on python 2.6 - change (pywikibot/core)
by jenkins-bot (Code Review) 02 Oct '13

02 Oct '13

jenkins-bot has submitted this change and it was merged. Change subject: tests: Use unittest2 on python 2.6 ...................................................................... tests: Use unittest2 on python 2.6 unittest2 is a backport of python 2.7s unittest features to 2.6. Use it when running the tests that already use those features. Change-Id: I5332e9e0815475d4bb4147bb5c1ef0d0ddc5c3bd --- M setup.py M tests/api_tests.py M tests/dry_api_tests.py M tests/dry_site_tests.py M tests/i18n_tests.py M tests/page_tests.py M tests/pwb_tests.py M tests/site_tests.py M tests/textlib_tests.py M tests/ui_tests.py M tests/utils.py M tests/wikibase_tests.py 12 files changed, 61 insertions(+), 32 deletions(-) Approvals: Merlijn van Deen: Looks good to me, approved jenkins-bot: Verified diff --git a/setup.py b/setup.py index a38cfe8..b022d2f 100644 --- a/setup.py +++ b/setup.py @@ -17,10 +17,17 @@ from setuptools import setup, find_packages from setuptools.command import install +test_deps = [] + if sys.version_info[0] != 2: raise RuntimeError("ERROR: Pywikipediabot only runs under Python 2") elif sys.version_info[1] < 6: raise RuntimeError("ERROR: Pywikipediabot only runs under Python 2.6 or higher") +elif sys.version_info[1] == 6: + test_deps = ['unittest2'] + testcollector = "tests.utils.collector" +else: + testcollector = "tests" class pwb_install(install.install): @@ -47,7 +54,8 @@ dependency_links=[ 'https://git.wikimedia.org/zip/?r=pywikibot/externals/httplib2.git&format=gz…' ], - test_suite="tests", + test_suite=testcollector, + tests_require=test_deps, classifiers=[ 'License :: OSI Approved :: MIT License', 'Development Status :: 4 - Beta' diff --git a/tests/api_tests.py b/tests/api_tests.py index 7c21ff0..4437036 100644 --- a/tests/api_tests.py +++ b/tests/api_tests.py @@ -9,8 +9,7 @@ import datetime import pywikibot import pywikibot.data.api as api -import unittest -from utils import PywikibotTestCase +from utils import PywikibotTestCase, unittest mysite = pywikibot.Site('en', 'wikipedia') diff --git a/tests/dry_api_tests.py b/tests/dry_api_tests.py index cc18083..82437af 100644 --- a/tests/dry_api_tests.py +++ b/tests/dry_api_tests.py @@ -1,11 +1,11 @@ -import datetime -import pywikibot -from pywikibot.data.api import CachedRequest -import unittest - - -class DryAPITests(unittest.TestCase): - +import datetime +import pywikibot +from pywikibot.data.api import CachedRequest +from utils import unittest + + +class DryAPITests(unittest.TestCase): + def setUp(self): self.parms = {'site': pywikibot.Site('en'), 'action': 'query', diff --git a/tests/dry_site_tests.py b/tests/dry_site_tests.py index 8847345..61b6452 100644 --- a/tests/dry_site_tests.py +++ b/tests/dry_site_tests.py @@ -1,8 +1,9 @@ -import pywikibot -import unittest - - -class DrySite(pywikibot.site.APISite): +import pywikibot + +from utils import unittest + + +class DrySite(pywikibot.site.APISite): @property def userinfo(self): return self._userinfo diff --git a/tests/i18n_tests.py b/tests/i18n_tests.py index 8bef1e6..104f308 100644 --- a/tests/i18n_tests.py +++ b/tests/i18n_tests.py @@ -6,11 +6,12 @@ # __version__ = '$Id$' -import unittest import os from pywikibot import i18n import shutil +from utils import unittest + class TestTranslate(unittest.TestCase): def setUp(self): diff --git a/tests/page_tests.py b/tests/page_tests.py index 5a60678..d764f68 100644 --- a/tests/page_tests.py +++ b/tests/page_tests.py @@ -10,11 +10,10 @@ __version__ = '$Id$' -import unittest import pywikibot import pywikibot.page -from utils import PywikibotTestCase +from utils import PywikibotTestCase, unittest site = pywikibot.Site('en', 'wikipedia') mainpage = pywikibot.Page(pywikibot.page.Link("Main Page", site)) diff --git a/tests/pwb_tests.py b/tests/pwb_tests.py index bd30b33..51db60b 100644 --- a/tests/pwb_tests.py +++ b/tests/pwb_tests.py @@ -9,15 +9,18 @@ import os import sys import subprocess - -import unittest - import pywikibot + +from utils import unittest pypath = sys.executable basepath = os.path.split(os.path.split(__file__)[0])[0] pwbpath = os.path.join(basepath, 'pwb.py') testbasepath = os.path.join(basepath, 'tests', 'pwb') + + +def check_output(command): + return subprocess.Popen(command, stdout=subprocess.PIPE).communicate()[0] class TestPwb(unittest.TestCase): @@ -35,8 +38,8 @@ the environment we get when directly running a script.""" test = os.path.join(testbasepath, 'print_locals.py') - direct = subprocess.check_output([pypath, test]) - vpwb = subprocess.check_output([pypath, pwbpath, test]) + direct = check_output([pypath, test]) + vpwb = check_output([pypath, pwbpath, test]) self.assertEqual(direct, vpwb) if __name__ == "__main__": diff --git a/tests/site_tests.py b/tests/site_tests.py index 3958601..a3f6d81 100644 --- a/tests/site_tests.py +++ b/tests/site_tests.py @@ -10,11 +10,10 @@ __version__ = '$Id$' -import unittest import pywikibot import warnings from tests import patch_request, unpatch_request -from utils import PywikibotTestCase +from utils import PywikibotTestCase, unittest mysite = None mainpage = None @@ -35,7 +34,6 @@ def testBaseMethods(self): """Test cases for BaseSite methods""" - self.assertEqual(mysite.family.name, self.family) self.assertEqual(mysite.code, self.code) self.assertType(mysite.lang, basestring) diff --git a/tests/textlib_tests.py b/tests/textlib_tests.py index 3f504c0..52aa8e2 100644 --- a/tests/textlib_tests.py +++ b/tests/textlib_tests.py @@ -10,13 +10,14 @@ import mwparserfromhell except ImportError: mwparserfromhell = False -import unittest import codecs import os import pywikibot import pywikibot.textlib as textlib +from utils import unittest + files = {} dirname = os.path.join(os.path.dirname(__file__), "pages") diff --git a/tests/ui_tests.py b/tests/ui_tests.py index d4ba066..2f01410 100644 --- a/tests/ui_tests.py +++ b/tests/ui_tests.py @@ -9,13 +9,14 @@ # __version__ = '$Id$' -import unittest import cStringIO import StringIO import logging import os import sys +from utils import unittest + if os.name == "nt": from multiprocessing.managers import BaseManager import threading diff --git a/tests/utils.py b/tests/utils.py index adb31b7..bccf715 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -1,4 +1,23 @@ -import unittest +try: + # Unittest2 is a backport of python 2.7s unittest module to python 2.6 + # Trying to import unittest2 has to happen first because 2.6 does have a + # unittest module in the standard library but that doesn't support all the + # features of the one found in python 2.7, so importing unittest first and + # then trying to figure out if it supports the features used would mean + # checking the module contents etc. Just catching an ImportError once is + # much easier. + import unittest2 as unittest +except ImportError: + import unittest + + +def collector(): + # This test collector loads tests from the `tests` directory in files + # matching the pattern `*tests.py`. This gets used by `setup.py test` when + # running on Python 2.6 to use the unittest2 test runner instead of the + # unittest one. + return unittest.loader.defaultTestLoader.discover("tests", "*tests.py") + from tests import patch_request, unpatch_request diff --git a/tests/wikibase_tests.py b/tests/wikibase_tests.py index ef39990..eb653d8 100644 --- a/tests/wikibase_tests.py +++ b/tests/wikibase_tests.py @@ -10,11 +10,10 @@ __version__ = '$Id$' import os -import unittest import pywikibot import json -from utils import PywikibotTestCase +from utils import PywikibotTestCase, unittest site = pywikibot.Site('en', 'wikipedia') mainpage = pywikibot.Page(pywikibot.page.Link("Main Page", site)) -- To view, visit https://gerrit.wikimedia.org/r/86436 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I5332e9e0815475d4bb4147bb5c1ef0d0ddc5c3bd Gerrit-PatchSet: 4 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Mineo <themineo(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Legoktm <legoktm.wikipedia(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] Aligned help with implementation - change (pywikibot/core)
by jenkins-bot (Code Review) 02 Oct '13

02 Oct '13

jenkins-bot has submitted this change and it was merged. Change subject: Aligned help with implementation ...................................................................... Aligned help with implementation Change-Id: I32340b0ed4e6bf39a793ccb6b7288c02ed1b941c --- M pywikibot/config2.py 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: Merlijn van Deen: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/config2.py b/pywikibot/config2.py index ee62581..1085ddc 100644 --- a/pywikibot/config2.py +++ b/pywikibot/config2.py @@ -105,7 +105,8 @@ provided in this argument 2. If the user has a PYWIKIBOT2_DIR environment variable, use the value of it - 3. Use (and if necessary create) a 'pywikibot' folder (Windows) or + 3. Use (and if necessary create) a 'pywikibot' folder under + 'Application Data' or 'AppData\Roaming' (Windows) or '.pywikibot' directory (Unix and similar) under the user's home directory. -- To view, visit https://gerrit.wikimedia.org/r/87016 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I32340b0ed4e6bf39a793ccb6b7288c02ed1b941c Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Legoktm <legoktm.wikipedia(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] Enabling usage of Disambiguator - change (pywikibot/compat)
by jenkins-bot (Code Review) 01 Oct '13

01 Oct '13

jenkins-bot has submitted this change and it was merged. Change subject: Enabling usage of Disambiguator ...................................................................... Enabling usage of Disambiguator Per help page [1] and some examples [2] [3] when someone runs isDisambig method this code checks if the Disambiguator is enabled, By checking list of extensions in siteinfo, if the extension is not enabaled (like [4]) returns the old way but if exists (like [5]) uses API to check being disambiguation. For more info see the examples I showed in [2] and [3] It's prttey important, and please review as soon as possible because if the extension is enabled, the old way doens't work now. See the bug. [1]: https://www.mediawiki.org/wiki/Extension:Disambiguator [2]: https://en.wikipedia.org/w/api.php?action=query&titles=Aa%20River&prop=page… [3]: https://en.wikipedia.org/w/api.php?action=query&titles=Tehran&prop=pageprop… [4]: http://tools.wmflabs.org/wikitest-rtl/w/index.php?title=Special:Disambiguat… [5]: https://en.wikipedia.org/wiki/Special:DisambiguationPages Bug: 54480 Change-Id: Ie3c46671d888cf917640fc732a28cc827028d0de --- M wikipedia.py 1 file changed, 31 insertions(+), 6 deletions(-) Approvals: Merlijn van Deen: Looks good to me, approved jenkins-bot: Verified diff --git a/wikipedia.py b/wikipedia.py index a044026..15a40f2 100644 --- a/wikipedia.py +++ b/wikipedia.py @@ -1510,11 +1510,38 @@ def isImage(self): """Return True if this is an image description page, False otherwise.""" return self.namespace() == 6 - def isDisambig(self, get_Index=True): - """Return True if this is a disambiguation page, False otherwise. + """Return True if this is a disambiguation page, False otherwise.""" + if not hasattr(self, "_isDisambig"): + extensions = self._site.siteinfo('extensions') + namesofextensions = [] + for extension in extensions: + namesofextensions.append(extension['name']) + if not u'Disambiguator' in namesofextensions: + return self._isDisambig_disambiguationspage(get_Index) + else: + return self._isDisambig_disambiguator(get_Index) + else: + return self._isDisambig - Relies on the presence of specific templates, identified in + def _isDisambig_disambiguator(self, get_Index=True): + params = { + 'action' : 'query', + 'titles' : self.title(), + 'prop' : 'pageprops', + 'ppprop' : 'disambiguation' + } + data = query.GetData(params, self._site)['query']['pages'].values()[0] + self._isDisambig = False + if data.has_key(u'missing'): + raise NoPage('Page %s does not exist' % self.title(asLink=True)) + if data.has_key(u'pageprops'): + if data[u'pageprops'].has_key(u'disambiguation'): + self._isDisambig = True + return self._isDisambig + + def _isDisambig_disambiguationspage(self, get_Index=True): + """Relies on the presence of specific templates, identified in the Family file or on a wiki page, to identify disambiguation pages. @@ -1527,9 +1554,7 @@ which are given on en-wiki Template:Disambig is always assumed to be default, and will be - appended regardless of its existence. - - """ + appended regardless of its existence.""" if not hasattr(self, "_isDisambig"): if not hasattr(self._site, "_disambigtemplates"): try: -- To view, visit https://gerrit.wikimedia.org/r/86047 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ie3c46671d888cf917640fc732a28cc827028d0de Gerrit-PatchSet: 3 Gerrit-Project: pywikibot/compat Gerrit-Branch: master Gerrit-Owner: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: DrTrigon <dr.trigon(a)surfeu.ch> Gerrit-Reviewer: Legoktm <legoktm.wikipedia(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] Don't choke on 'removed' labels. - change (pywikibot/core)
by jenkins-bot (Code Review) 30 Sep '13

30 Sep '13

jenkins-bot has submitted this change and it was merged. Change subject: Don't choke on 'removed' labels. ...................................................................... Don't choke on 'removed' labels. A label will have 'removed' if the serializer recieved bad input like an empty string, so we should just skip that label Bug: 54767 Change-Id: Ibf4645edfaf25a7ed9afa56484e924bd855f24ed --- M pywikibot/page.py 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: Merlijn van Deen: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/page.py b/pywikibot/page.py index 84c88cc..b1f931a 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -2418,7 +2418,8 @@ self.labels = {} if 'labels' in self._content: for lang in self._content['labels']: - self.labels[lang] = self._content['labels'][lang]['value'] + if not 'removed' in self._content['labels'][lang]: # Bug 54767 + self.labels[lang] = self._content['labels'][lang]['value'] #descriptions self.descriptions = {} -- To view, visit https://gerrit.wikimedia.org/r/86442 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ibf4645edfaf25a7ed9afa56484e924bd855f24ed Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Legoktm <legoktm.wikipedia(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] Simplify logic - change (pywikibot/core)
by jenkins-bot (Code Review) 30 Sep '13

30 Sep '13

jenkins-bot has submitted this change and it was merged. Change subject: Simplify logic ...................................................................... Simplify logic Change-Id: I6df49a96da63d193422b906de17ed804046aa89e --- M scripts/flickrripper.py 1 file changed, 1 insertion(+), 4 deletions(-) Approvals: Merlijn van Deen: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/flickrripper.py b/scripts/flickrripper.py index 5d721ee..9b0c60d 100644 --- a/scripts/flickrripper.py +++ b/scripts/flickrripper.py @@ -392,10 +392,7 @@ """ Loop over a set of Flickr photos. """ result = [] retry = False - if not start_id: - found_start_id = True - else: - found_start_id = False + found_start_id = not start_id # http://www.flickr.com/services/api/flickr.groups.pools.getPhotos.html # Get the photos in a group -- To view, visit https://gerrit.wikimedia.org/r/86628 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I6df49a96da63d193422b906de17ed804046aa89e Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Legoktm <legoktm.wikipedia(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] PEP8-ify flickrripper.py - change (pywikibot/core)
by jenkins-bot (Code Review) 30 Sep '13

30 Sep '13

jenkins-bot has submitted this change and it was merged. Change subject: PEP8-ify flickrripper.py ...................................................................... PEP8-ify flickrripper.py Change-Id: Id33fac7d3bf4a309875e32e745632c49174d1cab --- M scripts/flickrripper.py 1 file changed, 96 insertions(+), 86 deletions(-) Approvals: Merlijn van Deen: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/flickrripper.py b/scripts/flickrripper.py index 5268d75..5d721ee 100644 --- a/scripts/flickrripper.py +++ b/scripts/flickrripper.py @@ -1,6 +1,6 @@ #!/usr/bin/python # -*- coding: utf-8 -*- -''' +""" Tool to copy a flickr stream to Commons # Get a set to work on (start with just a username). @@ -22,8 +22,7 @@ **Initial suggestion **User input *Filter the categories - -''' +""" # # (C) Multichill, 2009 # (C) Pywikipedia team, 2009-2013 @@ -47,24 +46,25 @@ from PIL import Image, ImageTk # see: http://www.pythonware.com/products/pil/ flickr_allowed_license = { - 0 : False, # All Rights Reserved - 1 : False, # Creative Commons Attribution-NonCommercial-ShareAlike License - 2 : False, # Creative Commons Attribution-NonCommercial License - 3 : False, # Creative Commons Attribution-NonCommercial-NoDerivs License - 4 : True, # Creative Commons Attribution License - 5 : True, # Creative Commons Attribution-ShareAlike License - 6 : False, # Creative Commons Attribution-NoDerivs License - 7 : True, # No known copyright restrictions - 8 : True, # United States Government Work + 0: False, # All Rights Reserved + 1: False, # Creative Commons Attribution-NonCommercial-ShareAlike License + 2: False, # Creative Commons Attribution-NonCommercial License + 3: False, # Creative Commons Attribution-NonCommercial-NoDerivs License + 4: True, # Creative Commons Attribution License + 5: True, # Creative Commons Attribution-ShareAlike License + 6: False, # Creative Commons Attribution-NoDerivs License + 7: True, # No known copyright restrictions + 8: True, # United States Government Work } -def getPhoto(flickr = None, photo_id = ''): - ''' + +def getPhoto(flickr=None, photo_id=''): + """ Get the photo info and the photo sizes so we can use these later on TODO: Add exception handling - ''' + """ gotPhoto = False while not gotPhoto: try: @@ -77,14 +77,15 @@ gotPhotos = False pywikibot.output(u'Flickr api problem, sleeping') time.sleep(30) - return (photoInfo, photoSizes) + return photoInfo, photoSizes -def isAllowedLicense(photoInfo = None): - ''' + +def isAllowedLicense(photoInfo=None): + """ Check if the image contains the right license TODO: Maybe add more licenses - ''' + """ license = photoInfo.find('photo').attrib['license'] if flickr_allowed_license[int(license)]: @@ -92,58 +93,64 @@ else: return False -def getPhotoUrl(photoSizes = None): - ''' + +def getPhotoUrl(photoSizes=None): + """ Get the url of the jpg file with the highest resolution - ''' + """ url = '' # The assumption is that the largest image is last for size in photoSizes.find('sizes').findall('size'): url = size.attrib['source'] return url -def downloadPhoto(photoUrl = ''): - ''' + +def downloadPhoto(photoUrl=''): + """ Download the photo and store it in a StrinIO.StringIO object. TODO: Add exception handling - ''' - imageFile=urllib.urlopen(photoUrl).read() + """ + imageFile = urllib.urlopen(photoUrl).read() return StringIO.StringIO(imageFile) + def findDuplicateImages(photo=None, site=pywikibot.getSite(u'commons', u'commons')): - ''' Takes the photo, calculates the SHA1 hash and asks the mediawiki api + """ Takes the photo, calculates the SHA1 hash and asks the mediawiki api for a list of duplicates. TODO: Add exception handling, fix site thing - ''' + """ hashObject = hashlib.sha1() hashObject.update(photo.getvalue()) return site.getFilesFromAnHash(base64.b16encode(hashObject.digest())) -def getTags(photoInfo = None): - ''' Get all the tags on a photo ''' + +def getTags(photoInfo=None): + """ Get all the tags on a photo """ result = [] for tag in photoInfo.find('photo').find('tags').findall('tag'): result.append(tag.text.lower()) return result -def getFlinfoDescription(photo_id = 0): - ''' + +def getFlinfoDescription(photo_id=0): + """ Get the description from http://wikipedia.ramselehof.de/flinfo.php TODO: Add exception handling, try a couple of times - ''' - parameters = urllib.urlencode({'id' : photo_id, 'raw' : 'on'}) + """ + parameters = urllib.urlencode({'id': photo_id, 'raw': 'on'}) rawDescription = urllib.urlopen( "http://wikipedia.ramselehof.de/flinfo.php?%s" % parameters).read() return rawDescription.decode('utf-8') + def getFilename(photoInfo=None, site=None, project=u'Flickr'): """ Build a good filename for the upload based on the username and the @@ -178,19 +185,19 @@ % (title, project, username)).exists(): i = 1 while True: - if (pywikibot.Page(site, u'File:%s - %s - %s (%d).jpg' - % (title, project, username, i)).exists()): + if pywikibot.Page(site, u'File:%s - %s - %s (%d).jpg' % (title, project, username, i)).exists(): i += 1 else: return u'%s - %s - %s (%d).jpg' % (title, project, username, i) else: return u'%s - %s - %s.jpg' % (title, project, username) + def cleanUpTitle(title): - ''' Clean up the title of a potential mediawiki page. Otherwise the title of + """ Clean up the title of a potential mediawiki page. Otherwise the title of the page might not be allowed by the software. - ''' + """ title = title.strip() title = re.sub(u"[<{\\[]", u"(", title) title = re.sub(u"[>}\\]]", u")", title) @@ -214,10 +221,10 @@ def buildDescription(flinfoDescription=u'', flickrreview=False, reviewer=u'', override=u'', addCategory=u'', removeCategories=False): - ''' Build the final description for the image. The description is based on + """ Build the final description for the image. The description is based on the info from flickrinfo and improved. - ''' + """ description = u'== {{int:filedesc}} ==\n%s' % flinfoDescription if removeCategories: description = pywikibot.removeCategoryLinks(description, @@ -243,14 +250,15 @@ description = description.replace(u'\r\n', u'\n') return description + def processPhoto(flickr=None, photo_id=u'', flickrreview=False, reviewer=u'', override=u'', addCategory=u'', removeCategories=False, autonomous=False): - ''' Process a single Flickr photo ''' + """ Process a single Flickr photo """ if photo_id: print photo_id (photoInfo, photoSizes) = getPhoto(flickr, photo_id) - if isAllowedLicense(photoInfo) or override: + if isAllowedLicense(photoInfo) or override: #Get the url of the largest photo photoUrl = getPhotoUrl(photoSizes) #Should download the photo only once @@ -297,42 +305,42 @@ class Tkdialog: - ''' The user dialog. ''' + """ The user dialog. """ def __init__(self, photoDescription, photo, filename): - self.root=Tk() + self.root = Tk() #"%dx%d%+d%+d" % (width, height, xoffset, yoffset) - self.root.geometry("%ix%i+10-10"%(config.tkhorsize, config.tkvertsize)) + self.root.geometry("%ix%i+10-10" % (config.tkhorsize, config.tkvertsize)) self.root.title(filename) self.photoDescription = photoDescription self.filename = filename self.photo = photo - self.skip=False - self.exit=False + self.skip = False + self.exit = False ## Init of the widgets # The image - self.image=self.getImage(self.photo, 800, 600) - self.imagePanel=Label(self.root, image=self.image) + self.image = self.getImage(self.photo, 800, 600) + self.imagePanel = Label(self.root, image=self.image) self.imagePanel.image = self.image # The filename - self.filenameLabel=Label(self.root,text=u"Suggested filename") - self.filenameField=Entry(self.root, width=100) + self.filenameLabel = Label(self.root, text=u"Suggested filename") + self.filenameField = Entry(self.root, width=100) self.filenameField.insert(END, filename) # The description - self.descriptionLabel=Label(self.root,text=u"Suggested description") - self.descriptionScrollbar=Scrollbar(self.root, orient=VERTICAL) - self.descriptionField=Text(self.root) + self.descriptionLabel = Label(self.root, text=u"Suggested description") + self.descriptionScrollbar = Scrollbar(self.root, orient=VERTICAL) + self.descriptionField = Text(self.root) self.descriptionField.insert(END, photoDescription) self.descriptionField.config(state=NORMAL, height=12, width=100, padx=0, pady=0, wrap=WORD, yscrollcommand=self.descriptionScrollbar.set) self.descriptionScrollbar.config(command=self.descriptionField.yview) # The buttons - self.okButton=Button(self.root, text="OK", command=self.okFile) - self.skipButton=Button(self.root, text="Skip", command=self.skipFile) + self.okButton = Button(self.root, text="OK", command=self.okFile) + self.skipButton = Button(self.root, text="Skip", command=self.skipFile) ## Start grid @@ -353,41 +361,41 @@ self.descriptionScrollbar.grid(row=14, column=5) def getImage(self, photo, width, height): - ''' Take the StringIO object and build an imageTK thumbnail ''' + """ Take the StringIO object and build an imageTK thumbnail """ image = Image.open(photo) image.thumbnail((width, height)) imageTk = ImageTk.PhotoImage(image) return imageTk def okFile(self): - ''' The user pressed the OK button. ''' - self.filename=self.filenameField.get() - self.photoDescription=self.descriptionField.get(0.0, END) + """ The user pressed the OK button. """ + self.filename = self.filenameField.get() + self.photoDescription = self.descriptionField.get(0.0, END) self.root.destroy() def skipFile(self): - ''' The user pressed the Skip button. ''' - self.skip=True + """ The user pressed the Skip button. """ + self.skip = True self.root.destroy() def run(self): - ''' Activate the dialog and return the new name and if the image is + """ Activate the dialog and return the new name and if the image is skipped. - ''' + """ self.root.mainloop() - return (self.photoDescription, self.filename, self.skip) + return self.photoDescription, self.filename, self.skip def getPhotos(flickr=None, user_id=u'', group_id=u'', photoset_id=u'', start_id='', end_id='', tags=u''): - ''' Loop over a set of Flickr photos. ''' + """ Loop over a set of Flickr photos. """ result = [] retry = False if not start_id: - found_start_id=True + found_start_id = True else: - found_start_id=False + found_start_id = False # http://www.flickr.com/services/api/flickr.groups.pools.getPhotos.html # Get the photos in a group @@ -405,12 +413,12 @@ for photo in flickr.groups_pools_getPhotos( group_id=group_id, user_id=user_id, tags=tags, per_page='100', page=i - ).find('photos').getchildren(): + ).find('photos').getchildren(): gotPhotos = True - if photo.attrib['id']==start_id: - found_start_id=True + if photo.attrib['id'] == start_id: + found_start_id = True if found_start_id: - if photo.attrib['id']==end_id: + if photo.attrib['id'] == end_id: pywikibot.output('Found end_id') return else: @@ -428,18 +436,18 @@ per_page='100', page='1') pages = photos.find('photoset').attrib['pages'] - for i in range(1, int(pages)+1): + for i in range(1, int(pages) + 1): gotPhotos = False while not gotPhotos: try: for photo in flickr.photosets_getPhotos( photoset_id=photoset_id, per_page='100', page=i - ).find('photoset').getchildren(): + ).find('photoset').getchildren(): gotPhotos = True - if photo.attrib['id']==start_id: - found_start_id=True + if photo.attrib['id'] == start_id: + found_start_id = True if found_start_id: - if photo.attrib['id']==end_id: + if photo.attrib['id'] == end_id: pywikibot.output('Found end_id') return else: @@ -457,16 +465,16 @@ per_page='100', page='1') pages = photos.find('photos').attrib['pages'] #flickrapi.exceptions.FlickrError - for i in range(1, int(pages)+1): + for i in range(1, int(pages) + 1): gotPhotos = False while not gotPhotos: try: for photo in flickr.people_getPublicPhotos( user_id=user_id, per_page='100', page=i - ).find('photos').getchildren(): + ).find('photos').getchildren(): gotPhotos = True if photo.attrib['id'] == start_id: - found_start_id=True + found_start_id = True if found_start_id: if photo.attrib['id'] == end_id: pywikibot.output('Found end_id') @@ -481,12 +489,13 @@ return + def usage(): - ''' + """ Print usage information TODO : Need more. - ''' + """ pywikibot.output( u"Flickrripper is a tool to transfer flickr photos to Wikimedia Commons") pywikibot.output(u"-group_id:<group_id>\n") @@ -494,6 +503,7 @@ pywikibot.output(u"-user_id:<user_id>\n") pywikibot.output(u"-tags:<tag>\n") return + def main(): site = pywikibot.getSite(u'commons', u'commons') @@ -509,19 +519,19 @@ if 'api_secret' in config.flickr and config.flickr['api_secret']: flickr = flickrapi.FlickrAPI(config.flickr['api_key'], config.flickr['api_secret']) (token, frob) = flickr.get_token_part_one(perms='read') - if not token: # The user still hasn't authorised this app yet, get_token_part_one() will have spawn a browser window + if not token: + # The user still hasn't authorised this app yet, get_token_part_one() will have spawn a browser window pywikibot.input("Press ENTER after you authorized this program") flickr.get_token_part_two((token, frob)) else: print 'Accessing public content only' flickr = flickrapi.FlickrAPI(config.flickr['api_key']) - group_id = u'' photoset_id = u'' user_id = u'' - start_id= u'' - end_id=u'' + start_id = u'' + end_id = u'' tags = u'' addCategory = u'' removeCategories = False -- To view, visit https://gerrit.wikimedia.org/r/86627 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id33fac7d3bf4a309875e32e745632c49174d1cab Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Legoktm <legoktm.wikipedia(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] Remove unused imports, re-order - change (pywikibot/core)
by jenkins-bot (Code Review) 30 Sep '13

30 Sep '13

jenkins-bot has submitted this change and it was merged. Change subject: Remove unused imports, re-order ...................................................................... Remove unused imports, re-order Change-Id: I2fe70fcf8cf1bafcbd6f1d40f9aa65ab6590c29c --- M scripts/flickrripper.py 1 file changed, 7 insertions(+), 4 deletions(-) Approvals: Merlijn van Deen: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/flickrripper.py b/scripts/flickrripper.py index 3e6f145..5268d75 100644 --- a/scripts/flickrripper.py +++ b/scripts/flickrripper.py @@ -32,14 +32,17 @@ # __version__ = '$Id$' -import sys, urllib, re, StringIO, hashlib, base64, time +import urllib +import re +import StringIO +import hashlib +import base64 +import time import pywikibot from pywikibot import config -from pywikibot.data import api -import imagerecat, upload +import upload import flickrapi # see: http://stuvel.eu/projects/flickrapi -import xml.etree.ElementTree from Tkinter import * from PIL import Image, ImageTk # see: http://www.pythonware.com/products/pil/ -- To view, visit https://gerrit.wikimedia.org/r/86626 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I2fe70fcf8cf1bafcbd6f1d40f9aa65ab6590c29c Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Legoktm <legoktm.wikipedia(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] Port flickrripper.py from compat - change (pywikibot/core)
by jenkins-bot (Code Review) 30 Sep '13

30 Sep '13

jenkins-bot has submitted this change and it was merged. Change subject: Port flickrripper.py from compat ...................................................................... Port flickrripper.py from compat Change-Id: I0e4b71f90d4690861ea58ba3bf754c1d4f49f1c5 --- A scripts/flickrripper.py 1 file changed, 620 insertions(+), 0 deletions(-) Approvals: Merlijn van Deen: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/flickrripper.py b/scripts/flickrripper.py new file mode 100644 index 0000000..3e6f145 --- /dev/null +++ b/scripts/flickrripper.py @@ -0,0 +1,620 @@ +#!/usr/bin/python +# -*- coding: utf-8 -*- +''' +Tool to copy a flickr stream to Commons + +# Get a set to work on (start with just a username). +# * Make it possible to delimit the set (from/to) +#For each image +#*Check the license +#*Check if it isn't already on Commons +#*Build suggested filename +#**Check for name collision and maybe alter it +#*Pull description from Flinfo +#*Show image and description to user +#**Add a nice hotcat lookalike for the adding of categories +#**Filter the categories +#*Upload the image + +Todo: +*Check if the image is already uploaded (SHA hash) +*Check and prevent filename collisions +**Initial suggestion +**User input +*Filter the categories + +''' +# +# (C) Multichill, 2009 +# (C) Pywikipedia team, 2009-2013 +# +# Distributed under the terms of the MIT license. +# +__version__ = '$Id$' + +import sys, urllib, re, StringIO, hashlib, base64, time +import pywikibot +from pywikibot import config +from pywikibot.data import api +import imagerecat, upload + +import flickrapi # see: http://stuvel.eu/projects/flickrapi +import xml.etree.ElementTree +from Tkinter import * +from PIL import Image, ImageTk # see: http://www.pythonware.com/products/pil/ + +flickr_allowed_license = { + 0 : False, # All Rights Reserved + 1 : False, # Creative Commons Attribution-NonCommercial-ShareAlike License + 2 : False, # Creative Commons Attribution-NonCommercial License + 3 : False, # Creative Commons Attribution-NonCommercial-NoDerivs License + 4 : True, # Creative Commons Attribution License + 5 : True, # Creative Commons Attribution-ShareAlike License + 6 : False, # Creative Commons Attribution-NoDerivs License + 7 : True, # No known copyright restrictions + 8 : True, # United States Government Work +} + +def getPhoto(flickr = None, photo_id = ''): + ''' + Get the photo info and the photo sizes so we can use these later on + + TODO: Add exception handling + + ''' + gotPhoto = False + while not gotPhoto: + try: + photoInfo = flickr.photos_getInfo(photo_id=photo_id) + #xml.etree.ElementTree.dump(photoInfo) + photoSizes = flickr.photos_getSizes(photo_id=photo_id) + #xml.etree.ElementTree.dump(photoSizes) + gotPhoto = True + except flickrapi.exceptions.FlickrError: + gotPhotos = False + pywikibot.output(u'Flickr api problem, sleeping') + time.sleep(30) + return (photoInfo, photoSizes) + +def isAllowedLicense(photoInfo = None): + ''' + Check if the image contains the right license + + TODO: Maybe add more licenses + ''' + + license = photoInfo.find('photo').attrib['license'] + if flickr_allowed_license[int(license)]: + return True + else: + return False + +def getPhotoUrl(photoSizes = None): + ''' + Get the url of the jpg file with the highest resolution + ''' + url = '' + # The assumption is that the largest image is last + for size in photoSizes.find('sizes').findall('size'): + url = size.attrib['source'] + return url + +def downloadPhoto(photoUrl = ''): + ''' + Download the photo and store it in a StrinIO.StringIO object. + + TODO: Add exception handling + + ''' + imageFile=urllib.urlopen(photoUrl).read() + return StringIO.StringIO(imageFile) + +def findDuplicateImages(photo=None, + site=pywikibot.getSite(u'commons', u'commons')): + ''' Takes the photo, calculates the SHA1 hash and asks the mediawiki api + for a list of duplicates. + + TODO: Add exception handling, fix site thing + + ''' + hashObject = hashlib.sha1() + hashObject.update(photo.getvalue()) + return site.getFilesFromAnHash(base64.b16encode(hashObject.digest())) + +def getTags(photoInfo = None): + ''' Get all the tags on a photo ''' + result = [] + for tag in photoInfo.find('photo').find('tags').findall('tag'): + result.append(tag.text.lower()) + + return result + +def getFlinfoDescription(photo_id = 0): + ''' + Get the description from http://wikipedia.ramselehof.de/flinfo.php + + TODO: Add exception handling, try a couple of times + ''' + parameters = urllib.urlencode({'id' : photo_id, 'raw' : 'on'}) + + rawDescription = urllib.urlopen( + "http://wikipedia.ramselehof.de/flinfo.php?%s" % parameters).read() + + return rawDescription.decode('utf-8') + +def getFilename(photoInfo=None, site=None, project=u'Flickr'): + """ Build a good filename for the upload based on the username and the + title. Prevents naming collisions. + + """ + if not site: + site = pywikibot.Site(u'commons', u'commons') + username = photoInfo.find('photo').find('owner').attrib['username'] + title = photoInfo.find('photo').find('title').text + if title: + title = cleanUpTitle(title) + + if not title: + #find the max length for a mw title + maxBytes = 240 - len(project.encode('utf-8')) \ + - len(username.encode('utf-8')) + description = photoInfo.find('photo').find('description').text + if description: + descBytes = len(description.encode('utf-8')) + if descBytes > maxBytes: + # maybe we cut more than needed, anyway we do it + items = max(min(len(description), maxBytes / 4), + len(description) - descBytes + maxBytes) + description = description[:items] + title = cleanUpTitle(description) + else: + title = u'' + # Should probably have the id of the photo as last resort. + + if pywikibot.Page(site, u'File:%s - %s - %s.jpg' + % (title, project, username)).exists(): + i = 1 + while True: + if (pywikibot.Page(site, u'File:%s - %s - %s (%d).jpg' + % (title, project, username, i)).exists()): + i += 1 + else: + return u'%s - %s - %s (%d).jpg' % (title, project, username, i) + else: + return u'%s - %s - %s.jpg' % (title, project, username) + +def cleanUpTitle(title): + ''' Clean up the title of a potential mediawiki page. Otherwise the title of + the page might not be allowed by the software. + + ''' + title = title.strip() + title = re.sub(u"[<{\\[]", u"(", title) + title = re.sub(u"[>}\\]]", u")", title) + title = re.sub(u"[ _]?\$!\$", u"", title) + title = re.sub(u",:[ _]", u", ", title) + title = re.sub(u"[;:][ _]", u", ", title) + title = re.sub(u"[\t\n ]+", u" ", title) + title = re.sub(u"[\r\n ]+", u" ", title) + title = re.sub(u"[\n]+", u"", title) + title = re.sub(u"[?!]([.\"]|$)", u"\\1", title) + title = re.sub(u"[&#%?!]", u"^", title) + title = re.sub(u"[;]", u",", title) + title = re.sub(u"[/+\\\\:]", u"-", title) + title = re.sub(u"--+", u"-", title) + title = re.sub(u",,+", u",", title) + title = re.sub(u"[-,^]([.]|$)", u"\\1", title) + title = title.replace(u" ", u"_") + title = title.strip(u"_") + return title + + +def buildDescription(flinfoDescription=u'', flickrreview=False, reviewer=u'', + override=u'', addCategory=u'', removeCategories=False): + ''' Build the final description for the image. The description is based on + the info from flickrinfo and improved. + + ''' + description = u'== {{int:filedesc}} ==\n%s' % flinfoDescription + if removeCategories: + description = pywikibot.removeCategoryLinks(description, + pywikibot.Site( + 'commons', 'commons')) + if override: + description = description.replace(u'{{cc-by-sa-2.0}}\n', u'') + description = description.replace(u'{{cc-by-2.0}}\n', u'') + description = description.replace(u'{{flickrreview}}\n', u'') + description = description.replace( + u'{{copyvio|Flickr, licensed as "All Rights Reserved" which is not a free license --~~~~}}\n', + u'') + description = description.replace(u'=={{int:license}}==', + u'=={{int:license}}==\n' + override) + elif flickrreview: + if reviewer: + description = description.replace(u'{{flickrreview}}', + u'{{flickrreview|' + reviewer + + '|{{subst:CURRENTYEAR}}-{{subst:CURRENTMONTH}}-{{subst:CURRENTDAY2}}}}') + if addCategory: + description = description.replace(u'{{subst:unc}}\n', u'') + description = description + u'\n[[Category:' + addCategory + ']]\n' + description = description.replace(u'\r\n', u'\n') + return description + +def processPhoto(flickr=None, photo_id=u'', flickrreview=False, reviewer=u'', + override=u'', addCategory=u'', removeCategories=False, + autonomous=False): + ''' Process a single Flickr photo ''' + if photo_id: + print photo_id + (photoInfo, photoSizes) = getPhoto(flickr, photo_id) + if isAllowedLicense(photoInfo) or override: + #Get the url of the largest photo + photoUrl = getPhotoUrl(photoSizes) + #Should download the photo only once + photo = downloadPhoto(photoUrl) + + #Don't upload duplicate images, should add override option + duplicates = findDuplicateImages(photo) + if duplicates: + pywikibot.output(u'Found duplicate image at %s' % duplicates.pop()) + else: + filename = getFilename(photoInfo) + flinfoDescription = getFlinfoDescription(photo_id) + photoDescription = buildDescription(flinfoDescription, + flickrreview, reviewer, + override, addCategory, + removeCategories) + #pywikibot.output(photoDescription) + if not autonomous: + (newPhotoDescription, newFilename, skip) = Tkdialog( + photoDescription, photo, filename).run() + else: + newPhotoDescription = photoDescription + newFilename = filename + skip = False + #pywikibot.output(newPhotoDescription) + #if (pywikibot.Page(title=u'File:'+ filename, site=pywikibot.getSite()).exists()): + # I should probably check if the hash is the same and if not upload it under a different name + #pywikibot.output(u'File:' + filename + u' already exists!') + #else: + #Do the actual upload + #Would be nice to check before I upload if the file is already at Commons + #Not that important for this program, but maybe for derived programs + if not skip: + bot = upload.UploadRobot(photoUrl, + description=newPhotoDescription, + useFilename=newFilename, + keepFilename=True, + verifyDescription=False) + bot.upload_image(debug=False) + return 1 + else: + pywikibot.output(u'Invalid license') + return 0 + + +class Tkdialog: + ''' The user dialog. ''' + def __init__(self, photoDescription, photo, filename): + self.root=Tk() + #"%dx%d%+d%+d" % (width, height, xoffset, yoffset) + self.root.geometry("%ix%i+10-10"%(config.tkhorsize, config.tkvertsize)) + + self.root.title(filename) + self.photoDescription = photoDescription + self.filename = filename + self.photo = photo + self.skip=False + self.exit=False + + ## Init of the widgets + # The image + self.image=self.getImage(self.photo, 800, 600) + self.imagePanel=Label(self.root, image=self.image) + + self.imagePanel.image = self.image + + # The filename + self.filenameLabel=Label(self.root,text=u"Suggested filename") + self.filenameField=Entry(self.root, width=100) + self.filenameField.insert(END, filename) + + # The description + self.descriptionLabel=Label(self.root,text=u"Suggested description") + self.descriptionScrollbar=Scrollbar(self.root, orient=VERTICAL) + self.descriptionField=Text(self.root) + self.descriptionField.insert(END, photoDescription) + self.descriptionField.config(state=NORMAL, height=12, width=100, padx=0, pady=0, wrap=WORD, yscrollcommand=self.descriptionScrollbar.set) + self.descriptionScrollbar.config(command=self.descriptionField.yview) + + # The buttons + self.okButton=Button(self.root, text="OK", command=self.okFile) + self.skipButton=Button(self.root, text="Skip", command=self.skipFile) + + ## Start grid + + # The image + self.imagePanel.grid(row=0, column=0, rowspan=11, columnspan=4) + + # The buttons + self.okButton.grid(row=11, column=1, rowspan=2) + self.skipButton.grid(row=11, column=2, rowspan=2) + + # The filename + self.filenameLabel.grid(row=13, column=0) + self.filenameField.grid(row=13, column=1, columnspan=3) + + # The description + self.descriptionLabel.grid(row=14, column=0) + self.descriptionField.grid(row=14, column=1, columnspan=3) + self.descriptionScrollbar.grid(row=14, column=5) + + def getImage(self, photo, width, height): + ''' Take the StringIO object and build an imageTK thumbnail ''' + image = Image.open(photo) + image.thumbnail((width, height)) + imageTk = ImageTk.PhotoImage(image) + return imageTk + + def okFile(self): + ''' The user pressed the OK button. ''' + self.filename=self.filenameField.get() + self.photoDescription=self.descriptionField.get(0.0, END) + self.root.destroy() + + def skipFile(self): + ''' The user pressed the Skip button. ''' + self.skip=True + self.root.destroy() + + def run(self): + ''' Activate the dialog and return the new name and if the image is + skipped. + + ''' + self.root.mainloop() + return (self.photoDescription, self.filename, self.skip) + + +def getPhotos(flickr=None, user_id=u'', group_id=u'', photoset_id=u'', + start_id='', end_id='', tags=u''): + ''' Loop over a set of Flickr photos. ''' + result = [] + retry = False + if not start_id: + found_start_id=True + else: + found_start_id=False + + # http://www.flickr.com/services/api/flickr.groups.pools.getPhotos.html + # Get the photos in a group + if group_id: + #First get the total number of photo's in the group + photos = flickr.groups_pools_getPhotos(group_id=group_id, + user_id=user_id, tags=tags, + per_page='100', page='1') + pages = photos.find('photos').attrib['pages'] + + for i in range(1, int(pages) + 1): + gotPhotos = False + while not gotPhotos: + try: + for photo in flickr.groups_pools_getPhotos( + group_id=group_id, user_id=user_id, tags=tags, + per_page='100', page=i + ).find('photos').getchildren(): + gotPhotos = True + if photo.attrib['id']==start_id: + found_start_id=True + if found_start_id: + if photo.attrib['id']==end_id: + pywikibot.output('Found end_id') + return + else: + yield photo.attrib['id'] + + except flickrapi.exceptions.FlickrError: + gotPhotos = False + pywikibot.output(u'Flickr api problem, sleeping') + time.sleep(30) + + # http://www.flickr.com/services/api/flickr.photosets.getPhotos.html + # Get the photos in a photoset + elif photoset_id: + photos = flickr.photosets_getPhotos(photoset_id=photoset_id, + per_page='100', page='1') + pages = photos.find('photoset').attrib['pages'] + + for i in range(1, int(pages)+1): + gotPhotos = False + while not gotPhotos: + try: + for photo in flickr.photosets_getPhotos( + photoset_id=photoset_id, per_page='100', page=i + ).find('photoset').getchildren(): + gotPhotos = True + if photo.attrib['id']==start_id: + found_start_id=True + if found_start_id: + if photo.attrib['id']==end_id: + pywikibot.output('Found end_id') + return + else: + yield photo.attrib['id'] + + except flickrapi.exceptions.FlickrError: + gotPhotos = False + pywikibot.output(u'Flickr api problem, sleeping') + time.sleep(30) + + # http://www.flickr.com/services/api/flickr.people.getPublicPhotos.html + # Get the (public) photos uploaded by a user + elif user_id: + photos = flickr.people_getPublicPhotos(user_id=user_id, + per_page='100', page='1') + pages = photos.find('photos').attrib['pages'] + #flickrapi.exceptions.FlickrError + for i in range(1, int(pages)+1): + gotPhotos = False + while not gotPhotos: + try: + for photo in flickr.people_getPublicPhotos( + user_id=user_id, per_page='100', page=i + ).find('photos').getchildren(): + gotPhotos = True + if photo.attrib['id'] == start_id: + found_start_id=True + if found_start_id: + if photo.attrib['id'] == end_id: + pywikibot.output('Found end_id') + return + else: + yield photo.attrib['id'] + + except flickrapi.exceptions.FlickrError: + gotPhotos = False + pywikibot.output(u'Flickr api problem, sleeping') + time.sleep(30) + + return + +def usage(): + ''' + Print usage information + + TODO : Need more. + ''' + pywikibot.output( + u"Flickrripper is a tool to transfer flickr photos to Wikimedia Commons") + pywikibot.output(u"-group_id:<group_id>\n") + pywikibot.output(u"-photoset_id:<photoset_id>\n") + pywikibot.output(u"-user_id:<user_id>\n") + pywikibot.output(u"-tags:<tag>\n") + return + +def main(): + site = pywikibot.getSite(u'commons', u'commons') + #imagerecat.initLists() + + #Get the api key + if not config.flickr['api_key']: + pywikibot.output('Flickr api key not found! Get yourself an api key') + pywikibot.output( + 'Any flickr user can get a key at http://www.flickr.com/services/api/keys/apply/') + return + + if 'api_secret' in config.flickr and config.flickr['api_secret']: + flickr = flickrapi.FlickrAPI(config.flickr['api_key'], config.flickr['api_secret']) + (token, frob) = flickr.get_token_part_one(perms='read') + if not token: # The user still hasn't authorised this app yet, get_token_part_one() will have spawn a browser window + pywikibot.input("Press ENTER after you authorized this program") + flickr.get_token_part_two((token, frob)) + else: + print 'Accessing public content only' + flickr = flickrapi.FlickrAPI(config.flickr['api_key']) + + + group_id = u'' + photoset_id = u'' + user_id = u'' + start_id= u'' + end_id=u'' + tags = u'' + addCategory = u'' + removeCategories = False + autonomous = False + totalPhotos = 0 + uploadedPhotos = 0 + + # Do we mark the images as reviewed right away? + if config.flickr['review']: + flickrreview = config.flickr['review'] + else: + flickrreview = False + + # Set the Flickr reviewer + if config.flickr['reviewer']: + reviewer = config.flickr['reviewer'] + elif 'commons' in config.sysopnames['commons']: + print config.sysopnames['commons'] + reviewer = config.sysopnames['commons']['commons'] + elif 'commons' in config.usernames['commons']: + reviewer = config.usernames['commons']['commons'] + else: + reviewer = u'' + + # Should be renamed to overrideLicense or something like that + override = u'' + for arg in pywikibot.handleArgs(): + if arg.startswith('-group_id'): + if len(arg) == 9: + group_id = pywikibot.input(u'What is the group_id of the pool?') + else: + group_id = arg[10:] + elif arg.startswith('-photoset_id'): + if len(arg) == 12: + photoset_id = pywikibot.input(u'What is the photoset_id?') + else: + photoset_id = arg[13:] + elif arg.startswith('-user_id'): + if len(arg) == 8: + user_id = pywikibot.input( + u'What is the user_id of the flickr user?') + else: + user_id = arg[9:] + elif arg.startswith('-start_id'): + if len(arg) == 9: + start_id = pywikibot.input( + u'What is the id of the photo you want to start at?') + else: + start_id = arg[10:] + elif arg.startswith('-end_id'): + if len(arg) == 7: + end_id = pywikibot.input( + u'What is the id of the photo you want to end at?') + else: + end_id = arg[8:] + elif arg.startswith('-tags'): + if len(arg) == 5: + tags = pywikibot.input( + u'What is the tag you want to filter out (currently only one supported)?') + else: + tags = arg[6:] + elif arg == '-flickrreview': + flickrreview = True + elif arg.startswith('-reviewer'): + if len(arg) == 9: + reviewer = pywikibot.input(u'Who is the reviewer?') + else: + reviewer = arg[10:] + elif arg.startswith('-override'): + if len(arg) == 9: + override = pywikibot.input(u'What is the override text?') + else: + override = arg[10:] + elif arg.startswith('-addcategory'): + if len(arg) == 12: + addCategory = pywikibot.input( + u'What category do you want to add?') + else: + addCategory = arg[13:] + elif arg == '-removecategories': + removeCategories = True + elif arg == '-autonomous': + autonomous = True + + if user_id or group_id or photoset_id: + for photo_id in getPhotos(flickr, user_id, group_id, photoset_id, + start_id, end_id, tags): + uploadedPhotos += processPhoto(flickr, photo_id, flickrreview, + reviewer, override, addCategory, + removeCategories, autonomous) + totalPhotos += 1 + else: + usage() + pywikibot.output(u'Finished running') + pywikibot.output(u'Total photos: ' + str(totalPhotos)) + pywikibot.output(u'Uploaded photos: ' + str(uploadedPhotos)) + +if __name__ == "__main__": + main() -- To view, visit https://gerrit.wikimedia.org/r/86625 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I0e4b71f90d4690861ea58ba3bf754c1d4f49f1c5 Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Legoktm <legoktm.wikipedia(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot

1 0

[Gerrit] Use hashlib library instead of deprecated md5 one - change (pywikibot/core)
by jenkins-bot (Code Review) 30 Sep '13

30 Sep '13

jenkins-bot has submitted this change and it was merged. Change subject: Use hashlib library instead of deprecated md5 one ...................................................................... Use hashlib library instead of deprecated md5 one Change-Id: Ib8fae9545ff1c5b05e54b0a86a1dd7d80a50361c --- M pywikibot/page.py 1 file changed, 4 insertions(+), 2 deletions(-) Approvals: Merlijn van Deen: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/page.py b/pywikibot/page.py index 84c88cc..f61dc0c 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -15,6 +15,7 @@ from pywikibot import config import pywikibot.site +import hashlib import htmlentitydefs import logging import re @@ -1645,10 +1646,11 @@ """Return image file's MD5 checksum.""" # FIXME: MD5 might be performed on incomplete file due to server disconnection # (see bug #1795683). - import md5 f = urllib.urlopen(self.fileUrl()) # TODO: check whether this needs a User-Agent header added - md5Checksum = md5.new(f.read()).hexdigest() + h = hashlib.md5() + h.update(f.read()) + md5Checksum = h.hexdigest() f.close() return md5Checksum -- To view, visit https://gerrit.wikimedia.org/r/86629 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ib8fae9545ff1c5b05e54b0a86a1dd7d80a50361c Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Legoktm <legoktm.wikipedia(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits