Feature Requests item #1898442, was opened at 2008-02-21 00:50
Message generated for change (Comment added) made by leogregianin
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603141&aid=1898442&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
>Status: Closed
Priority: 5
Private: No
Submitted By: Uberfuzzy (uberfuzzy)
>Assigned to: Leonardo Gregianin (leogregianin)
Summary: change hardcoded lag wait time
Initial Comment:
would it be possible to change the default lag wait time to be config'able instead of hard coded at 5 seconds?
move that 5 out to config.py
and add a -lag:# command to override it on a per run basis (similar to -pt)
wikipedia.py, line approx 1302
output(u"Pausing 5 seconds due to database server lag.")
time.sleep(5)
----------------------------------------------------------------------
>Comment By: Leonardo Gregianin (leogregianin)
Date: 2008-07-18 22:34
Message:
Logged In: YES
user_id=1136737
Originator: NO
# server lag; Mediawiki recommends waiting 5 seconds and retrying
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603141&aid=1898442&group_…
Patches item #1903837, was opened at 2008-02-28 07:35
Message generated for change (Comment added) made by leogregianin
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1903837&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
>Status: Closed
>Resolution: Accepted
Priority: 5
Private: No
Submitted By: Purodha B Blissenbach (purodha)
Assigned to: Nobody/Anonymous (nobody)
Summary: interwiki.py - decription enhancement.
Initial Comment:
Attached are some textual modifications to clarify and correct the help shown by interwiki.py. The program itself is not affected, only descriptive texts.
----------------------------------------------------------------------
>Comment By: Leonardo Gregianin (leogregianin)
Date: 2008-07-18 22:03
Message:
Logged In: YES
user_id=1136737
Originator: NO
fixed in r5732
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1903837&group_…
Revision: 5732
Author: leogregianin
Date: 2008-07-19 02:02:53 +0000 (Sat, 19 Jul 2008)
Log Message:
-----------
documentation patch 1903837 by purodha
Modified Paths:
--------------
trunk/pywikipedia/interwiki.py
Modified: trunk/pywikipedia/interwiki.py
===================================================================
--- trunk/pywikipedia/interwiki.py 2008-07-19 01:42:32 UTC (rev 5731)
+++ trunk/pywikipedia/interwiki.py 2008-07-19 02:02:53 UTC (rev 5732)
@@ -63,12 +63,13 @@
-bracket only work on pages that have (in the home language) parenthesis
in their title. All other pages are skipped.
+ (note: without ending colon)
-skipfile: used as -skipfile:filename, skip all links mentioned in
the given file. This does not work with -number!
- -skipauto: use to skip all pages that can be translated automatically,
- like dates, centuries, months, etc.
+ -skipauto use to skip all pages that can be translated automatically,
+ like dates, centuries, months, etc. (note: without ending colon)
These arguments are useful to provide hints to the bot:
@@ -92,8 +93,9 @@
for the format, one can for example give "en:something" or
"20:" as hint.
- -same: looks over all 'serious' languages for the same title.
+ -same looks over all 'serious' languages for the same title.
-same is equivalent to -hint:all:
+ (note: without ending colon)
-wiktionary: similar to -same, but will ONLY accept names that are
identical to the original. Also, if the title is not
@@ -113,30 +115,34 @@
of the page always, rather than doing so only when being
asked for (by typing '?'). Only useful in combination
with a hint-asking option like -untranslated, -askhints
- or -untranslatedonly
+ or -untranslatedonly. (note: without ending colon)
-noauto: Do not use the automatic translation feature for years and
dates, only use found links and hints.
These arguments define how much user confirmation is required:
- -autonomous: run automatically, do not ask any questions. If a question
+ -autonomous run automatically, do not ask any questions. If a question
to an operator is needed, write the name of the page
to autonomous_problems.dat and continue on the next page.
+ (note: without ending colon)
- -confirm: ask for confirmation before any page is changed on the
+ -confirm ask for confirmation before any page is changed on the
live wiki. Without this argument, additions and
unambiguous modifications are made without confirmation.
+ (note: without ending colon)
- -force: do not ask permission to make "controversial" changes,
+ -force do not ask permission to make "controversial" changes,
like removing a language because none of the found
alternatives actually exists.
+ (note: without ending colon)
- -select: ask for each link whether it should be include before
+ -select ask for each link whether it should be include before
changing any page. This is useful if you want to remove
invalid interwiki and if you do multiple hints of which
some might be correct and others incorrect. Combining
-select and -confirm is possible, but seems like overkill.
+ (note: without ending colon)
These arguments specify in which way the bot should follow interwiki links:
@@ -156,16 +162,17 @@
-ignorefile: similar to -ignore, except that the pages are taken from
the given file instead of the command line.
- -localright: do not follow interwiki from other pages than the starting
+ -localright do not follow interwiki from other pages than the starting
page. (Warning! Should be used very sparingly, only when
you are sure you have first gotten the interwiki on the
- starting page exactly right)
+ starting page exactly right).
+ (note: without ending colon)
The following arguments are only important for users who have accounts for
multiple languages, and specify on which sites the bot should modify pages:
-localonly only work on the local wiki, not on other wikis in the family
- I have a login at
+ I have a login at. (note: without ending colon)
-limittwo only update two pages - one in the local wiki (if loged-in),
and one in the top available one.
@@ -173,6 +180,7 @@
this option will make sure that only local and de: (larger)
site is updated. This option is useful to quickly set two way
links without updating all of wiki's sites.
+ (note: without ending colon)
-whenneeded works like limittwo, but other languages are changed in the
following cases:
@@ -183,7 +191,7 @@
Optionally, -whenneeded can be given an additional number
(for example -whenneeded:3), in which case other languages
will be changed if there are that number or more links to
- change or add
+ change or add. (note: without ending colon)
The following arguments influence how many pages the bot works on at once:
-array: The number of pages the bot tries to be working on at once.
Patches item #2001978, was opened at 2008-06-24 15:12
Message generated for change (Settings changed) made by leogregianin
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603140&aid=2001978&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
>Status: Closed
>Resolution: Accepted
Priority: 5
Private: No
Submitted By: Alex S.H. Lin (lin4h)
Assigned to: Nobody/Anonymous (nobody)
Summary: update for zh locale, and others
Initial Comment:
redirect.py : Make speedy deletion for broken redirects i18n.
family.py : add as,hu info
wikinews_family.py : add cs
wikibooks_family.py : modify hu
wiktionary_family.py: modify hu and add obsolete info
wikipedia_family.py : add as
wikiquote_family.py : modify ko,hu and add obsolete info
interwiki.py : modify zh
selflink.py, protect.py,imagetransfer.py, welcome.py, table2wiki.py, movepages.py : add zh or ja locale.
----------------------------------------------------------------------
Comment By: Shinjiman (shinjiman)
Date: 2008-07-09 04:57
Message:
Logged In: YES
user_id=1259779
Originator: NO
The patch was applied as r5698 on the Wikimedia SVN repo.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603140&aid=2001978&group_…
Revision: 5731
Author: leogregianin
Date: 2008-07-19 01:42:32 +0000 (Sat, 19 Jul 2008)
Log Message:
-----------
more docum
Modified Paths:
--------------
trunk/pywikipedia/wikipediatools.py
Modified: trunk/pywikipedia/wikipediatools.py
===================================================================
--- trunk/pywikipedia/wikipediatools.py 2008-07-18 18:44:19 UTC (rev 5730)
+++ trunk/pywikipedia/wikipediatools.py 2008-07-19 01:42:32 UTC (rev 5731)
@@ -17,6 +17,9 @@
user-config.py file, use this directory as the base
4. If all else fails, use the directory from which this module was
loaded.
+ 5. If the user-config.py file is not found, another will be created
+ in the current directory, following in the footsteps of project,
+ language and bot username.
"""
for arg in sys.argv[1:]:
Revision: 5730
Author: russblau
Date: 2008-07-18 18:44:19 +0000 (Fri, 18 Jul 2008)
Log Message:
-----------
add crossnamespace (for compatibility with old family files, not currently used here)
Modified Paths:
--------------
branches/rewrite/pywikibot/family.py
Modified: branches/rewrite/pywikibot/family.py
===================================================================
--- branches/rewrite/pywikibot/family.py 2008-07-18 18:43:32 UTC (rev 5729)
+++ branches/rewrite/pywikibot/family.py 2008-07-18 18:44:19 UTC (rev 5730)
@@ -1992,6 +1992,8 @@
},
}
+ self.crossnamespace = {}
+
# letters that can follow a wikilink and are regarded as part of this link
# This depends on the linktrail setting in LanguageXx.php and on
# [[MediaWiki:Linktrail]].
Revision: 5729
Author: russblau
Date: 2008-07-18 18:43:32 +0000 (Fri, 18 Jul 2008)
Log Message:
-----------
add very basic thread pool class
Modified Paths:
--------------
branches/rewrite/pywikibot/tools.py
Modified: branches/rewrite/pywikibot/tools.py
===================================================================
--- branches/rewrite/pywikibot/tools.py 2008-07-18 18:43:04 UTC (rev 5728)
+++ branches/rewrite/pywikibot/tools.py 2008-07-18 18:43:32 UTC (rev 5729)
@@ -7,7 +7,7 @@
#
__version__ = '$Id: $'
-
+import sys
import threading
import time
import Queue
@@ -123,7 +123,50 @@
if group:
yield group
-
+
+class ThreadList(list):
+ """A simple threadpool class to limit the number of simultaneous threads.
+
+ Any threading.Thread object can be added to the pool using the append()
+ method. If the maximum number of simultaneous threads has not been reached,
+ the Thread object will be started immediately; if not, the append() call
+ will block until the thread is able to start.
+
+ >>> pool = ThreadList(limit=10)
+ >>> def work():
+ ... time.sleep(1)
+ ...
+ >>> for x in xrange(20):
+ ... pool.append(threading.Thread(target=work))
+ ...
+
+ """
+ def __init__(self, limit=sys.maxint, *args):
+ self.limit = limit
+ list.__init__(self, *args)
+ for item in list(self):
+ if not isinstance(threading.Thread, item):
+ raise TypeError("Cannot add '%s' to ThreadList" % type(item))
+
+ def active_count(self):
+ """Return the number of alive threads, and delete all non-alive ones."""
+ count = 0
+ for item in list(self):
+ if item.isAlive():
+ count += 1
+ else:
+ self.remove(item)
+ return count
+
+ def append(self, thd):
+ if not isinstance(thd, threading.Thread):
+ raise TypeError("Cannot append '%s' to ThreadList" % type(thd))
+ while self.active_count() >= self.limit:
+ time.sleep(2)
+ list.append(self, thd)
+ thd.start()
+
+
if __name__ == "__main__":
def _test():
import doctest
Revision: 5728
Author: russblau
Date: 2008-07-18 18:43:04 +0000 (Fri, 18 Jul 2008)
Log Message:
-----------
Update isDisambig() logic
Modified Paths:
--------------
branches/rewrite/pywikibot/page.py
Modified: branches/rewrite/pywikibot/page.py
===================================================================
--- branches/rewrite/pywikibot/page.py 2008-07-17 22:03:06 UTC (rev 5727)
+++ branches/rewrite/pywikibot/page.py 2008-07-18 18:43:04 UTC (rev 5728)
@@ -457,20 +457,37 @@
def isDisambig(self):
"""Return True if this is a disambiguation page, False otherwise.
- Relies on the presence of specific templates, identified in the Family
- file, to identify disambiguation pages.
+ Relies on the presence of specific templates, identified in
+ the Family file or on a wiki page, to identify disambiguation
+ pages.
+ By default, loads a list of template names from the Family file;
+ if the value in the Family file is None, looks for the list on
+ [[MediaWiki:Disambiguationspage]].
+
"""
- if not hasattr(self, '_isDisambig'):
- locdis = self.site().family.disambig(self.site().code)
- for template in self.templates():
- tn = template.title(withNamespace=False)
- if tn in locdis:
- _isDisambig = True
+ if not hasattr(self, "_isDisambig"):
+ if not hasattr(self.site(), "_disambigtemplates"):
+ self.site()._disambigtemplates = \
+ self.site().family.disambig(self.site().code)
+ if self.site()._disambigtemplates is None:
+ try:
+ disambigpages = Page(self.site(),
+ "MediaWiki:Disambiguationspage")
+ self.site()._disambigtemplates = [
+ link.title(withNamespace=False)
+ for link in disambigpages.linkedPages()
+ if link.namespace() == 10
+ ]
+ except NoPage:
+ self.site()._disambigtemplates = ['Disambig']
+ for t in self.templates():
+ if t.title(withNamespace=False) in self.site()._disambigtemplates:
+ self._isDisambig = True
break
else:
- _isDisambig = False
- return _isDisambig
+ self._isDisambig = False
+ return self._isDisambig
def getReferences(self, follow_redirects=True, withTemplateInclusion=True,
onlyTemplateInclusion=False, redirectsOnly=False,
Bugs item #2021398, was opened at 2008-07-18 07:17
Message generated for change (Comment added) made by russblau
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2021398&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: interwiki
Group: None
>Status: Closed
>Resolution: Invalid
Priority: 5
Private: No
Submitted By: Purodha B Blissenbach (purodha)
Assigned to: Nobody/Anonymous (nobody)
Summary: Pywikipedia: TypeError: argument of type 'NoneType' is not i
Initial Comment:
python /home/.../interwiki.py -v -initialredirect -familiy:wikipedia -lang:ksh Edelweipirate
Checked for running processes. 1 processes currently running, including the current process.
Pywikipediabot (r5706 (wikipedia.py), Jul 10 2008, 05:46:47)
Python 2.5.2 (r252:60911, May 28 2008, 19:19:25)
[GCC 4.2.4 (Debian 4.2.4-1)]
Retrieving mediawiki messages from Special:Allmessages
WARNING: No character set found.
Not a wiki page.
Missing main page name
Getting 1 pages from wikipedia:ksh...
Not a wiki page.
[[Edelweipirate]]: [[ksh:Edelweipirate]] gives new interwiki [[ru:?????? ??????????]]
[[Edelweipirate]]: [[ksh:Edelweipirate]] gives new interwiki [[en:Edelweiss Pirates]]
[[Edelweipirate]]: [[ksh:Edelweipirate]] gives new interwiki [[fr:Pirates Edelweiss]]
[[Edelweipirate]]: [[ksh:Edelweipirate]] gives new interwiki [[de:Edelweipiraten]]
[[Edelweipirate]]: [[ksh:Edelweipirate]] gives new interwiki [[sv:Edelweisspiraten]]
[[Edelweipirate]]: [[ksh:Edelweipirate]] gives new interwiki [[pt:Piratas de Edelweiss]]
Getting 1 pages from wikipedia:ru...
Not a wiki page.
Getting 1 pages from wikipedia:en...
Not a wiki page.
Dump ksh (wikipedia) saved
Traceback (most recent call last):
File "/home/purodha/pywikipedia/interwiki.py", line 1755, in <module>
bot.run()
File "/home/purodha/pywikipedia/interwiki.py", line 1492, in run
self.queryStep()
File "/home/purodha/pywikipedia/interwiki.py", line 1466, in queryStep
self.oneQuery()
File "/home/purodha/pywikipedia/interwiki.py", line 1462, in oneQuery
subject.workDone(self)
File "/home/purodha/pywikipedia/interwiki.py", line 796, in workDone
(skip, alternativePage) = self.disambigMismatch(page)
File "/home/purodha/pywikipedia/interwiki.py", line 671, in disambigMismatch
elif not self.originPage.isDisambig() and page.isDisambig():
File "/home/purodha/pywikipedia/wikipedia.py", line 991, in isDisambig
foo = self.templates()
File "/home/purodha/pywikipedia/wikipedia.py", line 1701, in templates
return [template for (template, param) in self.templatesWithParams(get_redirect=get_redirect)]
File "/home/purodha/pywikipedia/wikipedia.py", line 1769, in templatesWithParams
name in self.site().family.disambig(self.site().lang):
TypeError: argument of type 'NoneType' is not iterable
~$
----------------------------------------------------------------------
>Comment By: Russell Blau (russblau)
Date: 2008-07-18 09:43
Message:
Logged In: YES
user_id=855050
Originator: NO
Your wikipedia.py is out of date. Update to the latest version
(apparently you updated your family file, but not your wikipedia.py).
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2021398&group_…