Pywikibot-commits March 2016

pywikibot-commits@lists.wikimedia.org

2 participants
151 discussions

Errored: wikimedia/pywikibot-core#3342 (master - 57d2283)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #3342 Status: Errored Duration: 1 hour, 6 minutes, and 55 seconds Commit: 57d2283 (master) Author: jenkins-bot Message: Merge "[bugfix] Enable empty plural variant" View the changeset: https://github.com/wikimedia/pywikibot-core/compare/85e76819e5be...57d2283c… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/117111948 -- You can configure recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications

8 years, 1 month

[Gerrit] [bugfix] Enable empty plural variant - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: [bugfix] Enable empty plural variant ...................................................................... [bugfix] Enable empty plural variant - regex for variants does not match empty plural variant. - Some tests added. Bug: T125402 Bug: T99057 Change-Id: I1e77cbe96b6b8861d05dc0c83d3cde1b636b54fa --- M pywikibot/i18n.py M tests/i18n_tests.py 2 files changed, 18 insertions(+), 2 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/i18n.py b/pywikibot/i18n.py index 5739837..f529aba 100644 --- a/pywikibot/i18n.py +++ b/pywikibot/i18n.py @@ -17,7 +17,7 @@ messages. See L{twntranslate} for more information on the messages. """ # -# (C) Pywikibot team, 2004-2015 +# (C) Pywikibot team, 2004-2016 # # Distributed under the terms of the MIT license. # @@ -357,7 +357,9 @@ plural_entries = [] specific_entries = {} - for number, plural in re.findall(r'\|?(?: *(\d+) *= *)?([^|]+)', + # A plural entry can not start at the end of the variants list, + # and must end with | or the end of the variants list. + for number, plural in re.findall(r'(?!$)(?: *(\d+) *= *)?(.*?)(?:\||$)', variants): if number: specific_entries[int(number)] = plural diff --git a/tests/i18n_tests.py b/tests/i18n_tests.py index 32f5f3a..22fd9b0 100644 --- a/tests/i18n_tests.py +++ b/tests/i18n_tests.py @@ -440,6 +440,20 @@ i18n._extract_plural('en', '{{PLURAL:foo|one|other}}', {'foo': 0}), 'other') + def test_empty_fields(self): + """Test default usage using a dict and no specific plurals.""" + self.assertEqual( + i18n._extract_plural('en', '{{PLURAL:foo||other}}', {'foo': 42}), + 'other') + self.assertEqual( + i18n._extract_plural('en', '{{PLURAL:foo||other}}', {'foo': 1}), + '') + self.assertEqual( + i18n._extract_plural('en', '{{PLURAL:foo|one|}}', {'foo': 1}), + 'one') + with self.assertRaises(IndexError): + i18n._extract_plural('en', '{{PLURAL:foo|one}}', {'foo': 0}) + def test_specific(self): """Test using a specific plural.""" self.assertEqual( -- To view, visit https://gerrit.wikimedia.org/r/278015 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I1e77cbe96b6b8861d05dc0c83d3cde1b636b54fa Gerrit-PatchSet: 7 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Beta16 <l.rabinelli(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Siebrand <siebrand(a)kitano.nl> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

8 years, 1 month

Errored: wikimedia/pywikibot-core#3341 (master - 85e7681)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #3341 Status: Errored Duration: 1 hour, 57 minutes, and 52 seconds Commit: 85e7681 (master) Author: jenkins-bot Message: Merge "Run a mysql query and yield results by row" View the changeset: https://github.com/wikimedia/pywikibot-core/compare/ce3e5e43d444...85e76819… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/117046644 -- You can configure recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications

8 years, 1 month

Errored: wikimedia/pywikibot-core#3340 (master - ce3e5e4)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #3340 Status: Errored Duration: 22 minutes and 15 seconds Commit: ce3e5e4 (master) Author: jenkins-bot Message: Merge "WikidataPageFromItemGenerator and performance" View the changeset: https://github.com/wikimedia/pywikibot-core/compare/0a7e4f560b9f...ce3e5e43… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/117034910 -- You can configure recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications

8 years, 1 month

[Gerrit] Run a mysql query and yield results by row - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Run a mysql query and yield results by row ...................................................................... Run a mysql query and yield results by row Allow execution of a mysql query and yield result row by row. Change pagegenerators.MySQLPageGenerator to use this function. Change-Id: I723ab533cabc0f59e5a0d7752c513b9ddb1f343c --- A pywikibot/data/mysql.py M pywikibot/pagegenerators.py 2 files changed, 91 insertions(+), 31 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/data/mysql.py b/pywikibot/data/mysql.py new file mode 100644 index 0000000..afe5993 --- /dev/null +++ b/pywikibot/data/mysql.py @@ -0,0 +1,78 @@ +# -*- coding: utf-8 -*- +"""Miscellaneous helper functions for mysql queries.""" +# +# (C) Pywikibot team, 2016 +# +# Distributed under the terms of the MIT license. +# +from __future__ import absolute_import, print_function, unicode_literals +__version__ = '$Id$' + +# Requires oursql <https://pythonhosted.org/oursql/> or +# MySQLdb <https://sourceforge.net/projects/mysql-python/> +try: + import oursql as mysqldb +except ImportError: + import MySQLdb as mysqldb + +import pywikibot + +from pywikibot import config2 as config + + +def mysql_query(query, params=(), dbname=None, encoding='utf-8', verbose=None): + """ + Yield rows from a MySQL query. + + An example query that yields all ns0 pages might look like:: + + SELECT + page_namespace, + page_title, + FROM page + WHERE page_namespace = 0; + + @param query: MySQL query to execute + @type query: str + @param params: input parametes for the query, if needed + @type params: tuple + @param dbname: db name + @type dbname: str + @param encoding: encoding used by the database + @type encoding: str + @param verbose: if True, print query to be executed; + if None, config.verbose_output will be used. + @type verbose: None or bool + @return: generator which yield tuples + """ + if verbose is None: + verbose = config.verbose_output + + if config.db_connect_file is None: + conn = mysqldb.connect(config.db_hostname, + db=config.db_name_format.format(dbname), + user=config.db_username, + passwd=config.db_password, + port=config.db_port) + else: + conn = mysqldb.connect(config.db_hostname, + db=config.db_name_format.format(dbname), + read_default_file=config.db_connect_file, + port=config.db_port) + + cursor = conn.cursor() + if verbose: + pywikibot.output('Executing query:\n%s' % query) + query = query.encode(encoding) + params = tuple(p.encode(encoding) for p in params) + + if params: + cursor.execute(query, params) + else: + cursor.execute(query) + + for row in cursor: + yield row + + cursor.close() + conn.close() diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py index 8981b9d..0e0b1f3 100644 --- a/pywikibot/pagegenerators.py +++ b/pywikibot/pagegenerators.py @@ -2424,7 +2424,7 @@ yield page -def MySQLPageGenerator(query, site=None): +def MySQLPageGenerator(query, site=None, verbose=None): """ Yield a list of pages based on a MySQL query. @@ -2437,43 +2437,25 @@ FROM page WHERE page_namespace = 0; - Requires oursql <https://pythonhosted.org/oursql/> or - MySQLdb <https://sourceforge.net/projects/mysql-python/> - @param query: MySQL query to execute @param site: Site object @type site: L{pywikibot.site.BaseSite} - @return: iterator of pywikibot.Page + @param verbose: if True, print query to be executed; + if None, config.verbose_output will be used. + @type verbose: None or bool + @return: generator which yield pywikibot.Page """ - try: - import oursql as mysqldb - except ImportError: - import MySQLdb as mysqldb + from pywikibot.data import mysql + if site is None: site = pywikibot.Site() - if config.db_connect_file is None: - conn = mysqldb.connect(config.db_hostname, - db=config.db_name_format.format(site.dbName()), - user=config.db_username, - passwd=config.db_password, - port=config.db_port) - else: - conn = mysqldb.connect(config.db_hostname, - db=config.db_name_format.format(site.dbName()), - read_default_file=config.db_connect_file, - port=config.db_port) - - cursor = conn.cursor() - pywikibot.output(u'Executing query:\n%s' % query) - query = query.encode(site.encoding()) - cursor.execute(query) - while True: - try: - namespaceNumber, pageName = cursor.fetchone() - except TypeError: - # Limit reached or no more results - break + row_gen = mysql.mysql_query(query, + dbname=site.dbName(), + encoding=site.encoding(), + verbose=verbose) + for row in row_gen: + namespaceNumber, pageName = row if pageName: # Namespace Dict only supports int namespace = site.namespace(int(namespaceNumber)) -- To view, visit https://gerrit.wikimedia.org/r/275300 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I723ab533cabc0f59e5a0d7752c513b9ddb1f343c Gerrit-PatchSet: 6 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Eranroz <eranroz89(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

8 years, 1 month

[Gerrit] WikidataPageFromItemGenerator and performance - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: WikidataPageFromItemGenerator and performance ...................................................................... WikidataPageFromItemGenerator and performance A new WikidataPageFromItemGenerator provides a convinent way to translate batch of item pages (e.g data items from repository) to site pages. WikidataQueryPageGenerator and WikidataSPARQLPageGenerator requests for each page its sitelink, which requires many web requests and is very inefficent. Instead we use a batch query requesting only the data we need (sitelinks). Bug: T129556 Change-Id: I515c7135b7c2f8b9851c82a189abdd102dd562ee --- M pywikibot/pagegenerators.py 1 file changed, 39 insertions(+), 22 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py index 6a59a1d..58d67b4 100644 --- a/pywikibot/pagegenerators.py +++ b/pywikibot/pagegenerators.py @@ -40,6 +40,7 @@ deprecated_args, redirect_func, issue_deprecation_warning, + itergroup, DequeGenerator, intersect_generators, IteratorNextMixin, @@ -2593,6 +2594,31 @@ yield pywikibot.Page(pywikibot.Link(fd(month, day), site)) +def WikidataPageFromItemGenerator(gen, site): + """Generate pages from site based on sitelinks of item pages. + + @param gen: generator of L{pywikibot.ItemPage} + @param site: Site for generator results. + @type site: L{pywikibot.site.BaseSite} + + """ + repo = site.data_repository() + for sublist in itergroup(gen, 50): + req = {'ids': [item.id for item in sublist], + 'sitefilter': site.dbName(), + 'action': 'wbgetentities', + 'props': 'sitelinks'} + + wbrequest = repo._simple_request(**req) + wbdata = wbrequest.submit() + entities = (item for item in wbdata['entities'].values() if + 'sitelinks' in item and site.dbName() in item['sitelinks']) + sitelinks = (item['sitelinks'][site.dbName()]['title'] + for item in entities) + for sitelink in sitelinks: + yield pywikibot.Page(site, sitelink) + + def WikidataQueryPageGenerator(query, site=None): """Generate pages that result from the given WikidataQuery. @@ -2606,24 +2632,22 @@ if site is None: site = pywikibot.Site() repo = site.data_repository() + is_repo = isinstance(site, pywikibot.site.DataSite) + if not is_repo: + # limit the results to those with sitelinks to target site + query += ' link[%s]' % site.dbName() wd_queryset = wdquery.QuerySet(query) wd_query = wdquery.WikidataQuery(cacheMaxAge=0) data = wd_query.query(wd_queryset) - pywikibot.output(u'retrieved %d items' % data[u'status'][u'items']) - for item in data[u'items']: - page = pywikibot.ItemPage(repo, u'Q{0}'.format(item)) - if isinstance(site, pywikibot.site.DataSite): - yield page - continue + items_pages = (pywikibot.ItemPage(repo, 'Q{0}'.format(item)) + for item in data[u'items']) + if is_repo: + return items_pages - try: - link = page.getSitelink(site) - except pywikibot.NoPage: - continue - yield pywikibot.Page(pywikibot.Link(link, site)) + return WikidataPageFromItemGenerator(items_pages, site) def WikidataSPARQLPageGenerator(query, site=None, item_name='item', endpoint=None): @@ -2644,18 +2668,11 @@ query_object = sparql.SparqlQuery(endpoint=endpoint) data = query_object.get_items(query, item_name=item_name) + items_pages = (pywikibot.ItemPage(repo, item) for item in data) + if isinstance(site, pywikibot.site.DataSite): + return items_pages - for item in data: - page = pywikibot.ItemPage(repo, item) - if isinstance(site, pywikibot.site.DataSite): - yield page - continue - - try: - link = page.getSitelink(site) - except pywikibot.NoPage: - continue - yield pywikibot.Page(pywikibot.Link(link, site)) + return WikidataPageFromItemGenerator(items_pages, site) def WikibaseSearchItemPageGenerator(text, language=None, total=None, site=None): -- To view, visit https://gerrit.wikimedia.org/r/276562 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I515c7135b7c2f8b9851c82a189abdd102dd562ee Gerrit-PatchSet: 5 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Eranroz <eranroz89(a)gmail.com> Gerrit-Reviewer: Eranroz <eranroz89(a)gmail.com> Gerrit-Reviewer: FelixReimann <felix(a)fex-it.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

8 years, 1 month

Errored: wikimedia/pywikibot-core#3339 (master - 0a7e4f5)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #3339 Status: Errored Duration: 55 minutes and 59 seconds Commit: 0a7e4f5 (master) Author: John Vandenberg Message: tests.utils.execute: accept locale 'C' or missing As per f7ecddb, for the test suite Bug: T129406 Change-Id: If097b88c44a9726958511ec77ce57b90db37d00d View the changeset: https://github.com/wikimedia/pywikibot-core/compare/055d5f05bfe4...0a7e4f56… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/117004986 -- You can configure recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications

8 years, 1 month

[Gerrit] tests.utils.execute: accept locale 'C' or missing - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: tests.utils.execute: accept locale 'C' or missing ...................................................................... tests.utils.execute: accept locale 'C' or missing As per f7ecddb, for the test suite Bug: T129406 Change-Id: If097b88c44a9726958511ec77ce57b90db37d00d --- M tests/utils.py 1 file changed, 5 insertions(+), 1 deletion(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/tests/utils.py b/tests/utils.py index bc5cfa2..9c09991 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -806,7 +806,11 @@ # A complete locale string needs to be created, so the country code # is guessed, however it is discarded when loading config. if config.userinterface_lang: - current_locale = locale.getdefaultlocale()[0].split('.')[0] + current_locale = locale.getdefaultlocale()[0] + if current_locale in [None, 'C']: + current_locale = 'en' + else: + current_locale = current_locale.split('.')[0] locale_prefix = str(config.userinterface_lang + '_') if not current_locale.startswith(locale_prefix): -- To view, visit https://gerrit.wikimedia.org/r/278300 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: If097b88c44a9726958511ec77ce57b90db37d00d Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

8 years, 1 month

Errored: wikimedia/pywikibot-core#3338 (master - 055d5f0)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #3338 Status: Errored Duration: 1 hour, 32 minutes, and 23 seconds Commit: 055d5f0 (master) Author: John Vandenberg Message: Disable other languages for musicbrainz wiki wiki.musicbrainz.org is reporting that it has languages 'ab' and 'lb', when these are actually interwiki links not interlanguage links. Bug: T130381 Change-Id: I4761396f4c4ede1cb2dd90cfb9cfac56220b8b9a View the changeset: https://github.com/wikimedia/pywikibot-core/compare/0d2272818175...055d5f05… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/116963725 -- You can configure recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications

8 years, 1 month

[Gerrit] Disable other languages for musicbrainz wiki - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Disable other languages for musicbrainz wiki ...................................................................... Disable other languages for musicbrainz wiki wiki.musicbrainz.org is reporting that it has languages 'ab' and 'lb', when these are actually interwiki links not interlanguage links. Bug: T130381 Change-Id: I4761396f4c4ede1cb2dd90cfb9cfac56220b8b9a --- M .travis.yml 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/.travis.yml b/.travis.yml index 81b9956..90cbacb 100644 --- a/.travis.yml +++ b/.travis.yml @@ -50,7 +50,7 @@ - mkdir ~/.pywikibot - - python pwb.py generate_family_file 'https://wiki.musicbrainz.org/' musicbrainz + - python pwb.py generate_family_file 'https://wiki.musicbrainz.org/' musicbrainz 'n' - if [[ $FAMILY == 'wpbeta' ]]; then python -m generate_family_file 'http://'$LANGUAGE'.wikipedia.beta.wmflabs.org/' 'wpbeta' 'n' ; fi -- To view, visit https://gerrit.wikimedia.org/r/278309 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I4761396f4c4ede1cb2dd90cfb9cfac56220b8b9a Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

8 years, 1 month

← Newer
1
2
3
4
5
6
7
8
9
...
16
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits March 2016