I found the CPU time consuming behaviour on other scripts top e.g. while running checkimages.py. I fear it comes from logging but I am unable to investigate in that matter at the moment.
---
** [bugs:#1646] interwiki.py CPU occupation too high (hogging)**
**Status:** open
**Created:** Wed Jul 24, 2013 02:20 PM UTC by André Malafaya Baptista
**Last Updated:** Sat Aug 17, 2013 11:48 PM UTC
**Owner:** nobody
Since some time ago (I think less than 2 months), running interwiki.py hogs one CPU after running for some time. As I have four CPUs, I get a constant 25% CPU occupation by python.exe. The console output gets also sluggish.
Example:
interwiki.py -lang:nl -family:wiktionary -auto -async -cleanup -pt:1 -start:Category:!
Pywikipedia trunk/pywikipedia/ (r11780, 2013/07/20, 11:39:29, ok)
Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)]
config-settings:
use_api = True
use_api_login = True
---
Sent from sourceforge.net because Pywikipedia-bugs(a)lists.wikimedia.org is subscribed to https://sourceforge.net/p/pywikipediabot/bugs/
To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/pywikipediabot/admin/bugs/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.
Getting 60 pages from wiktionary:fr...
[*1 CPU is at 100%]
[*WiFi manually disconnected: network down]
[*Around 1 minute elapses till the next console log]
No changes needed on page [[*****]]
No changes needed on page [[*****]]
No changes needed on page [[*****]]
Getting 15 pages from wiktionary:ro...
ERROR: URLError: <urlopen error [Errno 11001] getaddrinfo failed>
WARNING: Could not open 'http://ro.wiktionary.org/w/index.php?title=Special:Expo
rt&useskin=monobook'. Maybe the server or your connection is down. Retrying in 1 minutes...
[*1 CPU occupation drops to 0% while waiting for retry]
As it is visible in this log, it's the processing of pages that is taking too long and occupying too much CPU. Because I disconnected the WiFi, the bot was not in the process of page retrieval anymore. So what makes it take so long to process whatever it has to process after those 60 pages are retrieved? BTW, I had set -array:50 to make sure not too many pages are being processed.
Also, this only happens after running for a couple of hours. In the beginning of the bot run, these symptoms do not exist. Memory occupation seems normal at all times.
---
** [bugs:#1646] interwiki.py CPU occupation too high (hogging)**
**Status:** open
**Created:** Wed Jul 24, 2013 02:20 PM UTC by André Malafaya Baptista
**Last Updated:** Fri Aug 16, 2013 11:26 AM UTC
**Owner:** nobody
Since some time ago (I think less than 2 months), running interwiki.py hogs one CPU after running for some time. As I have four CPUs, I get a constant 25% CPU occupation by python.exe. The console output gets also sluggish.
Example:
interwiki.py -lang:nl -family:wiktionary -auto -async -cleanup -pt:1 -start:Category:!
Pywikipedia trunk/pywikipedia/ (r11780, 2013/07/20, 11:39:29, ok)
Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)]
config-settings:
use_api = True
use_api_login = True
---
Sent from sourceforge.net because Pywikipedia-bugs(a)lists.wikimedia.org is subscribed to https://sourceforge.net/p/pywikipediabot/bugs/
To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/pywikipediabot/admin/bugs/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.
From my latest observation today, I see the normal occasional CPU spikes little after starting interwiki.py. As time passes, these become more and more frequent (or of longer duration). After some 30 minutes, the CPU spike is basically continuous, thus yielding a 100% occupation in one CPU.
---
** [bugs:#1646] interwiki.py CPU occupation too high (hogging)**
**Status:** open
**Created:** Wed Jul 24, 2013 02:20 PM UTC by André Malafaya Baptista
**Last Updated:** Thu Jul 25, 2013 10:34 PM UTC
**Owner:** nobody
Since some time ago (I think less than 2 months), running interwiki.py hogs one CPU after running for some time. As I have four CPUs, I get a constant 25% CPU occupation by python.exe. The console output gets also sluggish.
Example:
interwiki.py -lang:nl -family:wiktionary -auto -async -cleanup -pt:1 -start:Category:!
Pywikipedia trunk/pywikipedia/ (r11780, 2013/07/20, 11:39:29, ok)
Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)]
config-settings:
use_api = True
use_api_login = True
---
Sent from sourceforge.net because Pywikipedia-bugs(a)lists.wikimedia.org is subscribed to https://sourceforge.net/p/pywikipediabot/bugs/
To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/pywikipediabot/admin/bugs/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.
Fixed in:
Change-Id: I7cf2bc30853896b18618ed54ffc08368146871fd
by:
Kunal Mehta
in:
Aug 07 2013
---
** [bugs:#1652] imagecopy.py doesn't work**
**Status:** closed-fixed
**Created:** Thu Aug 01, 2013 05:22 PM UTC by rubin16
**Last Updated:** Thu Aug 01, 2013 05:22 PM UTC
**Owner:** nobody
this script was using CommonsHelper on Toolserver for file description generation: toolserver.org/~magnus/commonshelper.php
As now CommonsHelper is moved to Labs, is it possible to update script to make it work with labs?
https://tools.wmflabs.org/commonshelper/
Ru.wikipedia will be really grateful to you, as Labs CommonsHelper is working correctly for cyrillic wikipedias, and Toolserver copy was mixing it for years (https://jira.toolserver.org/browse/TS-923 and https://jira.toolserver.org/browse/MAGNUS-218)
---
Sent from sourceforge.net because Pywikipedia-bugs(a)lists.wikimedia.org is subscribed to https://sourceforge.net/p/pywikipediabot/bugs/
To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/pywikipediabot/admin/bugs/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.
https://gerrit.wikimedia.org/r/#/c/78525/
---
** [bugs:#1647] Unicode error with existing page title**
**Status:** pending-fixed
**Created:** Sun Jul 28, 2013 07:54 AM UTC by Adrián Chaves Fernández
**Last Updated:** Sun Jul 28, 2013 07:54 AM UTC
**Owner:** nobody
I have written the following script:
# -*- coding: utf-8 -*-
import pywikibot
site = pywikibot.Site("gl", "wiktionary")
page = pywikibot.Page(site, u"𐌰𐌽𐌳𐌰𐌿𐍂𐌰")
print page.get()
It fails with the following output:
[gallaecio@afonso fontes]$ python2 test.py
Traceback (most recent call last):
File "test.py", line 7, in <module>
print page.get()
File "/usr/lib/python2.7/site-packages/pywikibot/__init__.py", line 249, in wrapper
return method(*__args, **__kw)
File "/usr/lib/python2.7/site-packages/pywikibot/__init__.py", line 249, in wrapper
return method(*__args, **__kw)
File "/usr/lib/python2.7/site-packages/pywikibot/page.py", line 278, in get
self._getInternals(sysop)
File "/usr/lib/python2.7/site-packages/pywikibot/page.py", line 302, in _getInternals
self.site.loadrevisions(self, getText=True, sysop=sysop)
File "/usr/lib/python2.7/site-packages/pywikibot/page.py", line 96, in site
return self._link.site
File "/usr/lib/python2.7/site-packages/pywikibot/page.py", line 3080, in site
self.parse()
File "/usr/lib/python2.7/site-packages/pywikibot/page.py", line 3037, in parse
u"contains illegal char(s) '%s'" % m.group(0))
pywikibot.exceptions.InvalidTitle: contains illegal char(s) '𐌰'
I’m using the **rewrite** branch, built today (2013-07-28).
---
Sent from sourceforge.net because Pywikipedia-bugs(a)lists.wikimedia.org is subscribed to https://sourceforge.net/p/pywikipediabot/bugs/
To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/pywikipediabot/admin/bugs/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.
- **status**: open --> pending-fixed
---
** [bugs:#1647] Unicode error with existing page title**
**Status:** pending-fixed
**Created:** Sun Jul 28, 2013 07:54 AM UTC by Adrián Chaves Fernández
**Last Updated:** Sun Jul 28, 2013 07:54 AM UTC
**Owner:** nobody
I have written the following script:
# -*- coding: utf-8 -*-
import pywikibot
site = pywikibot.Site("gl", "wiktionary")
page = pywikibot.Page(site, u"𐌰𐌽𐌳𐌰𐌿𐍂𐌰")
print page.get()
It fails with the following output:
[gallaecio@afonso fontes]$ python2 test.py
Traceback (most recent call last):
File "test.py", line 7, in <module>
print page.get()
File "/usr/lib/python2.7/site-packages/pywikibot/__init__.py", line 249, in wrapper
return method(*__args, **__kw)
File "/usr/lib/python2.7/site-packages/pywikibot/__init__.py", line 249, in wrapper
return method(*__args, **__kw)
File "/usr/lib/python2.7/site-packages/pywikibot/page.py", line 278, in get
self._getInternals(sysop)
File "/usr/lib/python2.7/site-packages/pywikibot/page.py", line 302, in _getInternals
self.site.loadrevisions(self, getText=True, sysop=sysop)
File "/usr/lib/python2.7/site-packages/pywikibot/page.py", line 96, in site
return self._link.site
File "/usr/lib/python2.7/site-packages/pywikibot/page.py", line 3080, in site
self.parse()
File "/usr/lib/python2.7/site-packages/pywikibot/page.py", line 3037, in parse
u"contains illegal char(s) '%s'" % m.group(0))
pywikibot.exceptions.InvalidTitle: contains illegal char(s) '𐌰'
I’m using the **rewrite** branch, built today (2013-07-28).
---
Sent from sourceforge.net because Pywikipedia-bugs(a)lists.wikimedia.org is subscribed to https://sourceforge.net/p/pywikipediabot/bugs/
To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/pywikipediabot/admin/bugs/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.
edit: it should be rewrite and not trunk
---
** [bugs:#1653] Pagegenerator WikidataItemGenerator does not work with RefferingPageGenerator**
**Status:** open
**Created:** Sat Aug 10, 2013 11:36 PM UTC by Sk!d
**Last Updated:** Sat Aug 10, 2013 11:36 PM UTC
**Owner:** nobody
There must be a bug in Pagegenrators this code does not work:
referredPage = pywikibot.page.PropertyPage(pywikibot.Site().data_repository(), "Property:P21")
pagegenrator = pywikibot.pagegenerators.WikidataItemGenerator(pywikibot.pagegenerators.ReferringPageGenerator(referredPage, withTemplateInclusion=False, content=False))
if you iterate over pagegenerator you get the stacktrace:
File "X\core\wikidatascripts\itemfix.py", line 34, in <module>
if int(item.title()[1:]) <176:
File "X\core\pywikibot\page.py", line 2264, in title
self._link._text = self.getID()
File "X\core\pywikibot\page.py", line 2357, in getID
self.get(force=force)
File "X\core\pywikibot\page.py", line 2486, in get
super(ItemPage, self).get(force=force, *args)
File "X\core\pywikibot\page.py", line 2317, in get
data = self.repo.loadcontent(self.__defined_by(), *args)
File "X\core\pywikibot\site.py", line 3373, in loadcontent
data = req.submit()
File "X\core\pywikibot\data\api.py", line 393, in submit
raise APIError(code, info, **result["error"])
pywikibot.data.api.APIError: param-missing: Either provide the item "ids" or pairs of "sites" and "titles" for corresponding pages
a work around is by iterating over referredPage.getReferences()
---
Sent from sourceforge.net because Pywikipedia-bugs(a)lists.wikimedia.org is subscribed to https://sourceforge.net/p/pywikipediabot/bugs/
To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/pywikipediabot/admin/bugs/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.
---
** [bugs:#1653] Pagegenerator WikidataItemGenerator does not work with RefferingPageGenerator**
**Status:** open
**Created:** Sat Aug 10, 2013 11:36 PM UTC by Sk!d
**Last Updated:** Sat Aug 10, 2013 11:36 PM UTC
**Owner:** nobody
There must be a bug in Pagegenrators this code does not work:
referredPage = pywikibot.page.PropertyPage(pywikibot.Site().data_repository(), "Property:P21")
pagegenrator = pywikibot.pagegenerators.WikidataItemGenerator(pywikibot.pagegenerators.ReferringPageGenerator(referredPage, withTemplateInclusion=False, content=False))
if you iterate over pagegenerator you get the stacktrace:
File "X\core\wikidatascripts\itemfix.py", line 34, in <module>
if int(item.title()[1:]) <176:
File "X\core\pywikibot\page.py", line 2264, in title
self._link._text = self.getID()
File "X\core\pywikibot\page.py", line 2357, in getID
self.get(force=force)
File "X\core\pywikibot\page.py", line 2486, in get
super(ItemPage, self).get(force=force, *args)
File "X\core\pywikibot\page.py", line 2317, in get
data = self.repo.loadcontent(self.__defined_by(), *args)
File "X\core\pywikibot\site.py", line 3373, in loadcontent
data = req.submit()
File "X\core\pywikibot\data\api.py", line 393, in submit
raise APIError(code, info, **result["error"])
pywikibot.data.api.APIError: param-missing: Either provide the item "ids" or pairs of "sites" and "titles" for corresponding pages
a work around is by iterating over referredPage.getReferences()
---
Sent from sourceforge.net because Pywikipedia-bugs(a)lists.wikimedia.org is subscribed to https://sourceforge.net/p/pywikipediabot/bugs/
To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/pywikipediabot/admin/bugs/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.
Merged by xqt.
---
** [bugs:#1651] create_user_files does not support usernames with apostrophes**
**Status:** closed-fixed
**Created:** Sun Jul 28, 2013 08:07 PM UTC by Merlijn S. van Deen
**Last Updated:** Wed Aug 07, 2013 03:19 AM UTC
**Owner:** Legoktm
If a username contains an apostrophe, it isn't escaped, which results in
username = 'Namewith'Apos'trophe's''
instead of
username = 'Namewith\'Apos\'trophe\'s\''
---
Sent from sourceforge.net because Pywikipedia-bugs(a)lists.wikimedia.org is subscribed to https://sourceforge.net/p/pywikipediabot/bugs/
To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/pywikipediabot/admin/bugs/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.