pywikibot August 2008

pywikibot@lists.wikimedia.org

29 participants
206 discussions

[Pywikipedia-l] [ pywikipediabot-Bugs-2079760 ] Periods converted to percent signs in section links

by SourceForge.net

Bugs item #2079760, was opened at 2008-08-27 23:30 Message generated for change (Comment added) made by jeremybaron You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2079760&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: interwiki Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Mikko Silvonen (silvonen) Assigned to: Nobody/Anonymous (nobody) Summary: Periods converted to percent signs in section links Initial Comment: Why did my interwiki.py edit http://en.wikipedia.org/w/index.php?title=1st_Belorussian_Front&diff=234605… convert the link [[de:Zentralfront#1._Wei.C3.9Frussische_Front]] to [[de:Zentralfront#1% Weirussische Front]]? The correct decoded link would be: [[de:Zentralfront#1. Weirussische Front]] C:\svn\pywikipedia>python version.py Pywikipedia [http] trunk/pywikipedia (r5854, Aug 27 2008, 21:32:58) Python 2.5.1 (r251:54863, May 1 2007, 17:47:05) [MSC v.1310 32 bit (Intel)] ---------------------------------------------------------------------- Comment By: Jeremy Baron (jeremybaron) Date: 2008-08-28 00:35 Message: Logged In: YES user_id=1669658 Originator: NO I don't know the anchor encoding MediaWiki uses too well but I think this fixes it. (patch below because I see no obvious way to attach. I know there is a way, maybe I don't have sufficient privs.) Rudimentary tests (before and after patch application): In [7]: import wikipedia Checked for running processes. 1 processes currently running, including the current process. In [8]: sectionlinktests = ('de:Zentralfront#1._Wei.C3.9Frussische_Front','a#.41.29');sectionlinktester = lambda x: wikipedia.Page(wikipedia.getSite(),x).aslink() In [9]: [(x,sectionlinktester(x)) for x in sectionlinktests] Out[9]: [('de:Zentralfront#1._Wei.C3.9Frussische_Front', u'[[de:Zentralfront#1% Wei\xdfrussische Front]]'), ('a#.41.29', u'[[A#A)]]')] In [10]: reload(wikipedia) Checked for running processes. 2 processes currently running, including the current process. Out[10]: <module 'wikipedia' from '/Users/jeremy/sandbox/mediawiki/pywikipediabot/pywikipedia/wikipedia.py'> In [11]: [(x,sectionlinktester(x)) for x in sectionlinktests] Out[11]: [('de:Zentralfront#1._Wei.C3.9Frussische_Front', u'[[de:Zentralfront#1. Wei\xdfrussische Front]]'), ('a#.41.29', u'[[A#A)]]')] patch: Index: pywikipedia/wikipedia.py =================================================================== --- pywikipedia/wikipedia.py (revision 5855) +++ pywikipedia/wikipedia.py (working copy) @@ -228,6 +228,7 @@ Rwatchlist = re.compile(r"<input tabindex='[\d]+' type='checkbox' " r"name='wpWatchthis' checked='checked'") Rlink = re.compile(r'\[\[(?P<title>[^\]\|\[]*)(\|[^\]]*)?\]\]') +resectiondecode = re.compile(r".(?=[0-9a-f]{2})",re.I) class Page(object): @@ -526,7 +527,7 @@ """ section = self._section if section and decode: - section = section.replace('.', '%') + section = resectiondecode.sub('%',section) section = url2unicode(section, self._site) if not underscore: section = section.replace('_', ' ') btw, sourceforge strips all kinds of things out of bugspam, not just german chars :-/ ---------------------------------------------------------------------- Comment By: Jeremy Baron (jeremybaron) Date: 2008-08-28 00:34 Message: Logged In: YES user_id=1669658 Originator: NO I don't know the anchor encoding MediaWiki uses too well but I think this fixes it. (patch below because I see no obvious way to attach. I know there is a way, maybe I don't have sufficient privs.) Rudimentary tests (before and after patch application): In [7]: import wikipedia Checked for running processes. 1 processes currently running, including the current process. In [8]: sectionlinktests = ('de:Zentralfront#1._Wei.C3.9Frussische_Front','a#.41.29');sectionlinktester = lambda x: wikipedia.Page(wikipedia.getSite(),x).aslink() In [9]: [(x,sectionlinktester(x)) for x in sectionlinktests] Out[9]: [('de:Zentralfront#1._Wei.C3.9Frussische_Front', u'[[de:Zentralfront#1% Wei\xdfrussische Front]]'), ('a#.41.29', u'[[A#A)]]')] In [10]: reload(wikipedia) Checked for running processes. 2 processes currently running, including the current process. Out[10]: <module 'wikipedia' from '/Users/jeremy/sandbox/mediawiki/pywikipediabot/pywikipedia/wikipedia.py'> In [11]: [(x,sectionlinktester(x)) for x in sectionlinktests] Out[11]: [('de:Zentralfront#1._Wei.C3.9Frussische_Front', u'[[de:Zentralfront#1. Wei\xdfrussische Front]]'), ('a#.41.29', u'[[A#A)]]')] patch: Index: pywikipedia/wikipedia.py =================================================================== --- pywikipedia/wikipedia.py (revision 5855) +++ pywikipedia/wikipedia.py (working copy) @@ -228,6 +228,7 @@ Rwatchlist = re.compile(r"<input tabindex='[\d]+' type='checkbox' " r"name='wpWatchthis' checked='checked'") Rlink = re.compile(r'\[\[(?P<title>[^\]\|\[]*)(\|[^\]]*)?\]\]') +resectiondecode = re.compile(r".(?=[0-9a-f]{2})",re.I) class Page(object): @@ -526,7 +527,7 @@ """ section = self._section if section and decode: - section = section.replace('.', '%') + section = resectiondecode.sub('%',section) section = url2unicode(section, self._site) if not underscore: section = section.replace('_', ' ') btw, sourceforge strips all kinds of things out of bugspam, not just german chars :-/ ---------------------------------------------------------------------- Comment By: Mikko Silvonen (silvonen) Date: 2008-08-27 23:38 Message: Logged In: YES user_id=127947 Originator: YES Ouch, the SourceForge email system removes the German sharp s character from the messages. See this issue on the web for the correct links. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2079760&group_…

15 years, 7 months

[Pywikipedia-l] [ pywikipediabot-Bugs-2079760 ] Periods converted to percent signs in section links

by SourceForge.net

15 years, 7 months

[Pywikipedia-l] [ pywikipediabot-Bugs-2079760 ] Periods converted to percent signs in section links

by SourceForge.net

Bugs item #2079760, was opened at 2008-08-28 06:30 Message generated for change (Comment added) made by silvonen You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2079760&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: interwiki Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Mikko Silvonen (silvonen) Assigned to: Nobody/Anonymous (nobody) Summary: Periods converted to percent signs in section links Initial Comment: Why did my interwiki.py edit http://en.wikipedia.org/w/index.php?title=1st_Belorussian_Front&diff=234605… convert the link [[de:Zentralfront#1._Wei.C3.9Frussische_Front]] to [[de:Zentralfront#1% Weirussische Front]]? The correct decoded link would be: [[de:Zentralfront#1. Weirussische Front]] C:\svn\pywikipedia>python version.py Pywikipedia [http] trunk/pywikipedia (r5854, Aug 27 2008, 21:32:58) Python 2.5.1 (r251:54863, May 1 2007, 17:47:05) [MSC v.1310 32 bit (Intel)] ---------------------------------------------------------------------- >Comment By: Mikko Silvonen (silvonen) Date: 2008-08-28 06:38 Message: Logged In: YES user_id=127947 Originator: YES Ouch, the SourceForge email system removes the German sharp s character from the messages. See this issue on the web for the correct links. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2079760&group_…

15 years, 7 months

[Pywikipedia-l] [ pywikipediabot-Bugs-2079760 ] Periods converted to percent signs in section links

by SourceForge.net

Bugs item #2079760, was opened at 2008-08-28 06:30 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2079760&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: interwiki Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Mikko Silvonen (silvonen) Assigned to: Nobody/Anonymous (nobody) Summary: Periods converted to percent signs in section links Initial Comment: Why did my interwiki.py edit http://en.wikipedia.org/w/index.php?title=1st_Belorussian_Front&diff=234605… convert the link [[de:Zentralfront#1._Wei.C3.9Frussische_Front]] to [[de:Zentralfront#1% Weirussische Front]]? The correct decoded link would be: [[de:Zentralfront#1. Weirussische Front]] C:\svn\pywikipedia>python version.py Pywikipedia [http] trunk/pywikipedia (r5854, Aug 27 2008, 21:32:58) Python 2.5.1 (r251:54863, May 1 2007, 17:47:05) [MSC v.1310 32 bit (Intel)] ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2079760&group_…

15 years, 7 months

[Pywikipedia-l] SVN: [5855] branches/rewrite/pywikibot/site.py

by russblau＠svn.wikimedia.org

Revision: 5855 Author: russblau Date: 2008-08-27 22:06:03 +0000 (Wed, 27 Aug 2008) Log Message: ----------- bug fixes Modified Paths: -------------- branches/rewrite/pywikibot/site.py Modified: branches/rewrite/pywikibot/site.py =================================================================== --- branches/rewrite/pywikibot/site.py 2008-08-27 21:32:58 UTC (rev 5854) +++ branches/rewrite/pywikibot/site.py 2008-08-27 22:06:03 UTC (rev 5855) @@ -525,7 +525,8 @@ """Load page info from api and save in page attributes""" title = page.title(withSection=False) query = api.PropertyGenerator("info", site=self, - titles=title.encode(self.encoding())) + titles=title.encode(self.encoding()), + inprop="protection") for pageitem in query: if pageitem['title'] != title: raise Error( @@ -572,7 +573,7 @@ if not hasattr(page, "_redir"): self.loadpageinfo(page) if not page._redir: - raise pywikibot.IsNotRedirectPage + raise pywikibot.IsNotRedirectPage(page.title()) title = page.title(withSection=False) query = api.Request(site=self, action="query", property="info", inprop="protection|talkid|subjectid", @@ -730,13 +731,14 @@ or "nonredirects" return eigen - def pagereferences(self, page, followRedirects, filterRedirects, - withTemplateInclusion, onlyTemplateInclusion): + def pagereferences(self, page, followRedirects=False, filterRedirects=None, + withTemplateInclusion=True, onlyTemplateInclusion=False): """Convenience method combining pagebacklinks and page_embeddedin.""" + #TODO Warn about deprecated arguments if onlyTemplateInclusion: return self.page_embeddedin(page) if not withTemplateInclusion: - return self.pagebacklinks(page, follow_redirects) + return self.pagebacklinks(page, followRedirects) import itertools return itertools.chain(self.pagebacklinks( page, followRedirects, filterRedirects),

15 years, 7 months

[Pywikipedia-l] SVN: [5854] trunk/pywikipedia/censure.py

by balasyum＠svn.wikimedia.org

Revision: 5854 Author: balasyum Date: 2008-08-27 21:32:58 +0000 (Wed, 27 Aug 2008) Log Message: ----------- Changing 'hu' logPage, as it has been moved on the wiki. Modified Paths: -------------- trunk/pywikipedia/censure.py Modified: trunk/pywikipedia/censure.py =================================================================== --- trunk/pywikipedia/censure.py 2008-08-27 18:38:16 UTC (rev 5853) +++ trunk/pywikipedia/censure.py 2008-08-27 21:32:58 UTC (rev 5854) @@ -18,7 +18,7 @@ # The page, where the bot logs to logPages = { - 'hu': u'Wikipédia:Cenzúra', + 'hu': u'Wikipédia:Potenciálisan vandalizmus áldozatául esett szócikkek', } # To add a new language, create or find the bad word page

15 years, 7 months

[Pywikipedia-l] SVN: [5853] trunk/pywikipedia/category_redirect.py

by russblau＠svn.wikimedia.org

Revision: 5853 Author: russblau Date: 2008-08-27 18:38:16 +0000 (Wed, 27 Aug 2008) Log Message: ----------- By request, leave all protected page edit requests on a single page, not individual talk pages Modified Paths: -------------- trunk/pywikipedia/category_redirect.py Modified: trunk/pywikipedia/category_redirect.py =================================================================== --- trunk/pywikipedia/category_redirect.py 2008-08-27 14:44:46 UTC (rev 5852) +++ trunk/pywikipedia/category_redirect.py 2008-08-27 18:38:16 UTC (rev 5853) @@ -79,6 +79,7 @@ self.catprefix = self.site.namespace(14)+":" self.result_queue = Queue.Queue() self.log_text = [] + self.edit_requests = [] # Localization: @@ -168,21 +169,21 @@ 'no': u"Bot for vedlikehold av kategoriomdirigeringer", } - self.talk_notification = { - '_default': u""" -== Category link == + self.edit_request_text = { + '_default': u"""\ {{editprotected}} -* This protected page has been detected in [[%(oldcat)s]], but that category \ -has been redirected to [[%(newcat)s]]. Please update the category link. \ +The following protected pages have been detected as requiring updates to \ +category links: +%s --~~~~ -""", +""" } - self.talk_notification_comment = { - '_default': - u"Robot: Category-redirect notification on protected page", + self.edit_request_item = { + '_default': u"* %s is in %s, which is a redirect to %s" } + def change_category(self, article, oldCat, newCat, comment=None, sortKey=None): """Given an article in category oldCat, moves it to category newCat. @@ -208,34 +209,9 @@ u'Skipping %s because of edit conflict' % article.aslink()) except wikipedia.LockedPage: wikipedia.output(u'Skipping locked page %s' % article.aslink()) - if not article.isTalkPage() and article.namespace() != 2: - # no messages on user pages or non-talk pages - talkpage = article.toggleTalkPage() - try: - talktext = talkpage.get() - except wikipedia.IsRedirectPage: - return False - except wikipedia.NoPage: - talktext = u"" - talktext = talktext + wikipedia.translate( - self.site.lang, - self.talk_notification) % { - 'oldcat': oldCat.aslink(textlink=True), - 'newcat': newCat.aslink(textlink=True)} - try: - talkpage.put(talktext, - wikipedia.translate( - self.site.lang, - self.talk_notification_comment), - minorEdit=False) - wikipedia.output( - u"Left protected page notification on %s" - % talkpage.aslink()) - except wikipedia.PageNotSaved: - wikipedia.output( - u"Protected page notification on %s failed" - % talkpage.aslink()) - + self.edit_requests.append((article.aslink(), + oldCat.aslink(textlink=True), + newCat.aslink(textlink=True))) except wikipedia.SpamfilterError, error: wikipedia.output( u'Changing page %s blocked by spam filter (URL=%s)' @@ -244,6 +220,9 @@ wikipedia.output( u"Page %s not saved; sysop privileges required." % article.aslink()) + self.edit_requests.append((article.aslink(), + oldCat.aslink(textlink=True), + newCat.aslink(textlink=True))) except wikipedia.PageNotSaved, error: wikipedia.output(u"Saving page %s failed: %s" % (article.aslink(), error.message)) @@ -379,17 +358,17 @@ other_words = self.site.redirect() if other_words: redirect_magicwords.extend(other_words) - problems = [] - problem_page = wikipedia.Page(self.site, - u"User:%(user)s/category redirect problems" % locals()) l = time.localtime() today = "%04d-%02d-%02d" % l[:3] log_page = wikipedia.Page(self.site, u"User:%(user)s/category redirect logs/%(today)s" % locals()) - + problem_page = wikipedia.Page(self.site, + u"User:%(user)s/category redirect problems" % locals()) + edit_request_page = wikipedia.Page(self.site, + u"User:%(user)s/category edit requests" % locals()) datafile = wikipedia.config.datafilepath( "%s-catmovebot-data" % self.site.dbName()) try: @@ -631,6 +610,9 @@ self.maint_comment)) log_page.put("\n".join(self.log_text)) problem_page.put("\n".join(problems)) + editrequest_page.put(self.edit_request_text + % "\n".join((self.edit_request_item % item) + for item in self.edit_requests)) def main(*args):

15 years, 7 months

[Pywikipedia-l] SVN: [5852] trunk/pywikipedia

by balasyum＠svn.wikimedia.org

Revision: 5852 Author: balasyum Date: 2008-08-27 14:44:46 +0000 (Wed, 27 Aug 2008) Log Message: ----------- Adding maintainer.py the feature to censure recent changes for bad words, other little modifications. Modified Paths: -------------- trunk/pywikipedia/maintainer.py trunk/pywikipedia/rciw.py Added Paths: ----------- trunk/pywikipedia/censure.py Added: trunk/pywikipedia/censure.py =================================================================== --- trunk/pywikipedia/censure.py (rev 0) +++ trunk/pywikipedia/censure.py 2008-08-27 14:44:46 UTC (rev 5852) @@ -0,0 +1,114 @@ +#! /usr/bin/env python +# -*- coding: utf-8 -*- + +"""Bad word checker bot +Should not be run manually/directly, but automatically by maintainer.py +Warning: experimental software, use at your own risk +""" + +__version__ = '$Id$' + +# Author: Balasyum +# http://hu.wikipedia.org/wiki/User:Balasyum + +import wikipedia +import sys +import thread + +# The page, where the bot logs to + +logPages = { + 'hu': u'Wikipédia:Cenzúra', + } + +# To add a new language, create or find the bad word page +# similarly to the 'hu' one (one word per line, starting with <pre> and ending with </pre> lines), +# and add to the badWordList lines below. + +badWordList = { + 'hu': u'User:Cenzúrabot/lista', + } + +site = wikipedia.getSite() +if not badWordList.has_key(site.language()) or not logPages.has_key(site.language()): + wikipedia.output('Error: your language isn\'t supported, see the source code for further details') + sys.exit(1) +ownWordPage = wikipedia.Page(site, badWordList[site.language()]) +try: + ownWordList = ownWordPage.get(get_redirect = True) +except wikipedia.NoPage: + wikipedia.output('Error: the page containing the bad word list of your language doesn\'t exist') + sys.exit(1) +ownWordList = ownWordList.split('\n') +del ownWordList[0] +del ownWordList[len(ownWordList) - 1] + +def seekbpos(str1, str2): + i = 0 + while i < len(str1): + if str1[i] != str2[i]: + return i + i += 1 + return i + +def seekepos(str1, str2, bpos): + i1 = len(str1) - 1 + i2 = len(str2) - 1 + while i1 > -1 and i2 > -1: + if i1 == bpos: + return i2 + elif i1 < bpos or str1[i1] != str2[i2]: + return i2 + 1 + i1 -= 1 + i2 -= 1 + return -1 + +def checkPage(title, onlyLastDiff = False): + if title == logPages[site.language()]: + return + wikipedia.output('Checking ' + title + ' for bad word list') + page = wikipedia.Page(site, title) + try: + text = page.get() + if onlyLastDiff: + oldver = page.getOldVersion(page.previousRevision()) + if len(text) > len(oldver): + bpos = seekbpos(oldver, text) + epos = seekepos(oldver, text, bpos) + diff = text[bpos:epos] + text = diff + except wikipedia.NoPage: + wikipedia.output('Page ' + title + ' doesn\'t exist, skipping') + return + except wikipedia.IsRedirectPage: + wikipedia.output('Page ' + title + ' is a redirect, skipping') + return + + report = False + wordsIn = [] + for badWord in ownWordList: + if text.find(' ' + badWord + ' ') != -1: + wordsIn.append(badWord) + report = True + if report: + logPage = wikipedia.Page(site, logPages[site.language()]) + try: + log = logPage.get() + except: + pass + wikipedia.output(title + ' matches the bad word list') + log = '* [' + page.permalink()+ ' ' + title + '] - ' + ' '.join(wordsIn) + '\n' + log + logPage.put(log, title) + else: + wikipedia.output(title + ' doesn\'t match any of the bad word list') + +def main(): + wikipedia.output('Warning: this script should not be run manually/directly, but automatically by maintainer.py') + if len(sys.argv) == 1: + wikipedia.output("Usage: censure.py <article title>") + sys.exit(1) + del sys.argv[0] + checkPage(' '.join(sys.argv).decode('utf-8')) + +if __name__ == "__main__": + main() Modified: trunk/pywikipedia/maintainer.py =================================================================== --- trunk/pywikipedia/maintainer.py 2008-08-27 13:41:53 UTC (rev 5851) +++ trunk/pywikipedia/maintainer.py 2008-08-27 14:44:46 UTC (rev 5852) @@ -3,37 +3,81 @@ """ A wiki-maintainer script that shares tasks between workers, requires no intervention. -Note: the script requires the Python IRC library http://python-irclib.sourceforge.net/ +This script requires the Python IRC library http://python-irclib.sourceforge.net/ + +Warning: experimental software, use at your own risk """ __version__ = '$Id$' # Author: Balasyum # http://hu.wikipedia.org/wiki/User:Balasyum -# License : LGPL from ircbot import SingleServerIRCBot from irclib import nm_to_n import random import wikipedia +import thread import threading import time import rciw +import censure ver = 1 +site = wikipedia.getSite() +site.forceLogin() + +class rcFeeder(SingleServerIRCBot): + def __init__(self, channel, nickname, server, port=6667): + SingleServerIRCBot.__init__(self, [(server, port)], nickname, nickname) + self.channel = channel + self.rcbot = rciw.IWRCBot(site) + self.tasks = [] + + def on_nicknameinuse(self, c, e): + c.nick(c.get_nickname() + "_") + + def on_welcome(self, c, e): + c.join(self.channel) + + def on_privmsg(self, c, e): + pass + + def on_pubmsg(self, c, e): + try: + msg = unicode(e.arguments()[0],'utf-8') + except UnicodeDecodeError: + return + name = msg[8:msg.find(u'14',9)] + if 'rciw' in self.tasks: + self.rcbot.addQueue(name) + if 'censure' in self.tasks: + thread.start_new_thread(censure.checkPage, (name, True)) + + def on_dccmsg(self, c, e): + pass + + def on_dccchat(self, c, e): + pass + + def on_quit(self, e, cmd): + pass + class MaintcontBot(SingleServerIRCBot): def __init__(self, nickname, server, port=6667): SingleServerIRCBot.__init__(self, [(server, port)], nickname, nickname) - self.rct = threading.Thread(target=rciw.main) - self.rct.setDaemon(True) - self.rciwrunning = False + feederThread = threading.Thread(target=self.feederBot) + feederThread.setDaemon(True) + feederThread.start() + def feederBot(self): + self.feed = rcFeeder('#' + site.language() + '.' + site.family.name, site.loggedInAs(), "irc.wikimedia.org") + self.feed.start() + def on_nicknameinuse(self, c, e): c.nick("mainter" + str(random.randrange(100, 999))) def on_welcome(self, c, e): - site = wikipedia.getSite() - site.forceLogin() self.connection.privmsg("maintcont", "workerjoin " + site.language() + '.' + site.family.name + ' ' + str(ver)) def on_privmsg(self, c, e): @@ -43,17 +87,9 @@ do = cmd.split() if do[0] == "accepted": print "Joined the network" - t = threading.Thread(target=self.activator) - t.setDaemon(True) - t.start() + thread.start_new_thread(self.activator,()) elif do[0] == "tasklist" and len(do) > 1: - tasks = do[1].split('|') - if 'rciw' in do[1]: - self.rct.start() - self.rciwrunning = True - if (not 'rciw' in do[1]) and self.rciwrunning: - self.rct.join(0) - self.rciwrunning = False + self.feed.tasks = do[1].split('|') def on_dccmsg(self, c, e): pass @@ -66,9 +102,17 @@ self.connection.privmsg("maintcont", "active") time.sleep(10) -def main(): - bot = MaintcontBot("mainter" + str(random.randrange(100, 999)), "irc.freenode.net") - bot.start() +class Maintainer: + def __init__(self): + controllThread = threading.Thread(target=self.controllBot) + controllThread.setDaemon(True) + controllThread.start() + while True: + raw_input() + def controllBot(self): + bot = MaintcontBot("mainter" + str(random.randrange(100, 999)), "irc.freenode.net") + bot.start() + if __name__ == "__main__": - main() \ No newline at end of file + Maintainer() Modified: trunk/pywikipedia/rciw.py =================================================================== --- trunk/pywikipedia/rciw.py 2008-08-27 13:41:53 UTC (rev 5851) +++ trunk/pywikipedia/rciw.py 2008-08-27 14:44:46 UTC (rev 5852) @@ -4,6 +4,8 @@ A simple IRC script to check for Recent Changes through IRC, and to check for interwikis in those recently modified articles. +Can not be run manually/directly, but automatically by maintainer.py + In use on hu:, not sure if this scales well on a large wiki such as en: (Depending on the edit rate, the number of IW threads could grow continuously without ever decreasing) @@ -12,6 +14,7 @@ -safe Does not handle the same page more than once in a session +Warning: experimental software, use at your own risk """ __version__ = '$Id$' @@ -19,8 +22,6 @@ # http://hu.wikipedia.org/wiki/User:Kisbes # License : GFDL -from ircbot import SingleServerIRCBot -from irclib import nm_to_n, nm_to_h, irc_lower, ip_numstr_to_quad, ip_quad_to_numstr import interwiki import threading import re @@ -28,10 +29,8 @@ import time from Queue import Queue -class IWRCBot(SingleServerIRCBot): - def __init__(self, site, channel, nickname, server, port, safe): - SingleServerIRCBot.__init__(self, [(server, port)], nickname, nickname) - self.channel = channel +class IWRCBot(): + def __init__(self, site, safe = True): self.other_ns = re.compile(u'14\[\[07(' + u'|'.join(site.namespaces()) + u')') interwiki.globalvar.autonomous = True self.site = site @@ -52,27 +51,9 @@ bot.queryStep() self.queue.task_done() - def join(self): - self.queue.join() - - def on_nicknameinuse(self, c, e): - c.nick(c.get_nickname() + "_") - - def on_welcome(self, c, e): - c.join(self.channel) - - def on_privmsg(self, c, e): - pass - - def on_pubmsg(self, c, e): - try: - msg = unicode(e.arguments()[0],'utf-8') - except UnicodeDecodeError: + def addQueue(self, name): + if self.other_ns.match(name): return - if self.other_ns.match(msg): - return - - name = msg[8:msg.find(u'14',9)] if self.safe: if name in self.processed: return @@ -82,35 +63,8 @@ # it is a simple atomic append(), no need to acquire a semaphore self.queue.put_nowait(page) - def on_dccmsg(self, c, e): - pass - - def on_dccchat(self, c, e): - pass - - def do_command(self, e, cmd): - pass - - def on_quit(self, e, cmd): - pass - def main(): - safe = False - for arg in wikipedia.handleArgs(): - if arg == 'safe': - safe = True - site = wikipedia.getSite() - site.forceLogin() - chan = '#' + site.language() + '.' + site.family.name - bot = IWRCBot(site, chan, site.loggedInAs(), "irc.wikimedia.org", 6667, safe) - try: - bot.start() - except: - # Quit IRC - bot.disconnect() - # Join the IW threads - bot.join() - raise + wikipedia.output('Warning: this script can not be run manually/directly, but automatically by maintainer.py') if __name__ == "__main__": main()

15 years, 7 months

[Pywikipedia-l] SVN: [5851] trunk/pywikipedia/category_redirect.py

by russblau＠svn.wikimedia.org

Revision: 5851 Author: russblau Date: 2008-08-27 13:41:53 +0000 (Wed, 27 Aug 2008) Log Message: ----------- * Use API to identify whether redirected category has any contents to move * Eliminate "-often" option, no longer needed because of this improvement * Eliminate multi-threading, no longer needed * Other bug-fixes Modified Paths: -------------- trunk/pywikipedia/category_redirect.py Modified: trunk/pywikipedia/category_redirect.py =================================================================== --- trunk/pywikipedia/category_redirect.py 2008-08-27 10:25:38 UTC (rev 5850) +++ trunk/pywikipedia/category_redirect.py 2008-08-27 13:41:53 UTC (rev 5851) @@ -3,12 +3,6 @@ Usage: category-redirect.py [options] -The following command-line options can be used with this bot: - --often Only scan those redirected categories that have been - identified as often-populated (only useful if the site - has such a category defined) - """ __version__ = '$Id$' @@ -26,6 +20,7 @@ class APIError(Exception): """The wiki API returned an error message.""" + def __init__(self, errordict): """Save error dict returned by MW API.""" self.errors = errordict @@ -78,9 +73,12 @@ class CategoryRedirectBot(object): - def __init__(self, often): - self.cooldown = 6 # days - self.often = often + def __init__(self): + self.cooldown = 7 # days + self.site = wikipedia.getSite() + self.catprefix = self.site.namespace(14)+":" + self.result_queue = Queue.Queue() + self.log_text = [] # Localization: @@ -94,17 +92,10 @@ 'simple': "Category:Category redirects", }, 'commons': { - 'commons': "Category:Non-empty category redirects" + 'commons': "Category:Category redirects" } } - # Category that contains frequently-used redirected category pages - self.often_redirect_cat = { - 'wikipedia': { - 'en': "Category:Often-populated Wikipedia category redirects", - }, - } - # List of all templates that are used to mark category redirects # (put the most preferred form first) self.redir_templates = { @@ -203,9 +194,7 @@ fixed due to protection. """ - cats = article.categories(get_redirect=True) - - oldtext = article.get(get_redirect=True) + oldtext = article.get(get_redirect=True, force=True) newtext = wikipedia.replaceCategoryInPlace(oldtext, oldCat, newCat) if newtext == oldtext: wikipedia.output( @@ -219,7 +208,7 @@ u'Skipping %s because of edit conflict' % article.aslink()) except wikipedia.LockedPage: wikipedia.output(u'Skipping locked page %s' % article.aslink()) - if not article.isTalkPage and article.namespace != 2: + if not article.isTalkPage() and article.namespace() != 2: # no messages on user pages or non-talk pages talkpage = article.toggleTalkPage() try: @@ -272,9 +261,9 @@ # Move articles found, moved = 0, 0 for result in self.query_results(list="categorymembers", - cmtitle=oldCat.title(), - cmprop="title|sortkey", - cmlimit="max"): + cmtitle=oldCat.title(), + cmprop="title|sortkey", + cmlimit="max"): found += len(result['categorymembers']) for item in result['categorymembers']: article = wikipedia.Page(self.site, item['title']) @@ -301,12 +290,6 @@ if found: wikipedia.output(u"%s: %s found, %s moved" % (oldCat.title(), found, moved)) - #Dummy edit to refresh the page, shouldn't show up in any logs. - try: - oldCat.put(oldCat.get()) - except: - self.log_text.append(u'* Dummy edit at %s failed' - % oldCat.aslink(textlink=True)) self.result_queue.put((oldCatTitle, found, moved)) return except wikipedia.ServerError: @@ -391,11 +374,6 @@ def run(self): """Run the bot""" - self.site = wikipedia.getSite() - self.catprefix = self.site.namespace(14)+":" - self.result_queue = Queue.Queue() - self.log_text = [] - user = self.site.loggedInAs() redirect_magicwords = ["redirect"] other_words = self.site.redirect() @@ -427,25 +405,22 @@ template_list = self.redir_templates[self.site.family.name ][self.site.lang] # regex to match soft category redirects - template_regexes = [ - re.compile( -ur"{{\s*(?:%(prefix)s\s*:\s*)?%(template)s\s*\|(\s*%(catns)s\s*:\s*)?([^}]+)}}" - % {'prefix': self.site.namespace(10).lower(), - 'template': item.replace(" ", "[ _]+"), - 'catns': self.site.namespace(14)}, - re.I) - for item in template_list - ] + # note that any templates containing optional "category:" are + # incorrect and will be fixed by the bot template_regex = re.compile( -ur"{{\s*(?:%(prefix)s\s*:\s*)?(?:%(template)s)\s*\|(\s*%(catns)s\s*:\s*)?([^}]+)}}" - % {'prefix': self.site.namespace(10).lower(), - 'template': "|".join(item.replace(" ", "[ _]+") - for item in template_list), - 'catns': self.site.namespace(14)}, - re.I) + ur"""{{\s*(?:%(prefix)s\s*:\s*)? # optional "template:" + (?:%(template)s)\s*\| # catredir template name + (\s*%(catns)s\s*:\s*)? # optional "category:" + ([^|}]+) # redirect target cat + (?:\|[^|}]*)*}} # optional arguments 2+, ignored + """ % {'prefix': self.site.namespace(10).lower(), + 'template': "|".join(item.replace(" ", "[ _]+") + for item in template_list), + 'catns': self.site.namespace(14)}, + re.I|re.X) # regex to match hard redirects to category pages catredir_regex = re.compile( -ur'\s*#(?:%(redir)s)\s*:?\s*\[\[\s*:?%(catns)s\s*:(.*?)\]\]\s*' + ur'\s*#(?:%(redir)s)\s*:?\s*\[\[\s*:?%(catns)s\s*:(.*?)\]\]\s*' % {'redir': "|".join(redirect_magicwords), 'catns': self.site.namespace(14)}, re.I) @@ -458,10 +433,10 @@ # with an appropriate template comment = wikipedia.translate(self.site.lang, self.redir_comment) for result in self.query_results(list='allpages', - apnamespace='14', # Category: - apfrom='!', - apfilterredir='redirects', - aplimit='max'): + apnamespace='14', # Category: + apfrom='!', + apfilterredir='redirects', + aplimit='max'): gen = (wikipedia.Page(self.site, page_item['title']) for page_item in result['allpages']) for page in pagegenerators.PreloadingGenerator(gen, 120): @@ -508,35 +483,32 @@ u'eilimit': 'max', u'format': 'json' } - counts = {} - destmap = {} - catmap = {} - catlist = [] - catpages = [] - if self.often: - target = self.often_redirect_cat[self.site.family.name - ][self.site.lang] - else: - target = self.cat_redirect_cat[self.site.family.name - ][self.site.lang] + counts, destmap, catmap = {}, {}, {} + catlist, catpages, nonemptypages = [], [], [] + target = self.cat_redirect_cat[self.site.family.name][self.site.lang] - # get and preload all members of the category-redirect category + # get a list of all members of the category-redirect category for result in self.query_results(generator=u'categorymembers', - gcmtitle=target, - gcmnamespace=u'14', # CATEGORY - gcmlimit=u'max', - prop='info'): + gcmtitle=target, + gcmnamespace=u'14', # CATEGORY + gcmlimit=u'max', + prop='info|categoryinfo'): for catdata in result['pages'].values(): - catpages.append(wikipedia.Page(self.site, catdata['title'])) + thispage = wikipedia.Page(self.site, catdata['title']) + catpages.append(thispage) + if 'categoryinfo' in catdata \ + and catdata['categoryinfo']['size'] != "0": + # save those categories that have contents + nonemptypages.append(thispage) + # preload the category pages for redirected categories wikipedia.output(u"") wikipedia.output(u"Preloading %s category redirect pages" % len(catpages)) - thread_limit = int(math.log(len(catpages), 2)) for cat in pagegenerators.PreloadingGenerator(catpages, 120): cat_title = cat.titleWithoutNamespace() - if "Wikipedia category redirect" in cat_title: -# self.log_text.append("* Ignoring %s%s" % (self.catprefix, cat_title)) + if "category redirect" in cat_title: + self.log_text.append("* Ignoring %s%s" % (self.catprefix, cat_title)) continue try: text = cat.get(get_redirect=True) @@ -610,20 +582,28 @@ except wikipedia.Error, e: self.log_text.append("** Failed: %s" % str(e)) + # only scan those pages that have contents (nonemptypages) + # and that haven't been removed from catlist as broken redirects + cats_to_empty = set(catlist) & set(nonemptypages) wikipedia.output(u"") wikipedia.output(u"Moving pages out of %s redirected categories." - % len(catlist)) - threadpool = ThreadList(limit=thread_limit) + % len(cats_to_empty)) + thread_limit = int(math.log(len(cats_to_empty), 8) + 1) + threadpool = ThreadList(limit=1) # temporarily disabling multi-threads - for cat in catlist: + for cat in cats_to_empty: cat_title = cat.titleWithoutNamespace() if not self.readyToEdit(cat): counts[cat_title] = None + self.log_text.append( + u"* Skipping %s; in cooldown period." + % cat.aslink(textlink=True)) continue - threadpool.append(threading.Thread(target=self.move_contents, - args=(cat_title, catmap[cat]), - kwargs=dict(editSummary=comment))) - while len(counts) < len(catlist): + threadpool.append( + threading.Thread(target=self.move_contents, + args=(cat_title, catmap[cat]), + kwargs=dict(editSummary=comment))) + while len(counts) < len(cats_to_empty): title, found, moved = self.result_queue.get() if found is None: self.log_text.append( @@ -636,10 +616,9 @@ % (self.catprefix, title, found, moved)) counts[title] = found - if not self.often: - for cat in record.keys(): - if cat not in counts.keys(): - del record[cat] + for cat in record.keys(): + if cat not in counts.keys(): + del record[cat] for cat in counts.keys(): if counts[cat] is not None: if counts[cat]: @@ -657,17 +636,12 @@ def main(*args): try: a = wikipedia.handleArgs(*args) - if "-often" in a: - a.remove("-often") - often = True - else: - often = False if len(a) == 1: raise RuntimeError('Unrecognized argument "%s"' % a[0]) elif a: raise RuntimeError('Unrecognized arguments: ' + " ".join(('"%s"' % arg) for arg in a)) - bot = CategoryRedirectBot(often) + bot = CategoryRedirectBot() bot.run() finally: wikipedia.stopme()

15 years, 7 months

[Pywikipedia-l] SVN: [5850] trunk/pywikipedia/category_redirect.py

by russblau＠svn.wikimedia.org

Revision: 5850 Author: russblau Date: 2008-08-27 10:25:38 +0000 (Wed, 27 Aug 2008) Log Message: ----------- bugfix Modified Paths: -------------- trunk/pywikipedia/category_redirect.py Modified: trunk/pywikipedia/category_redirect.py =================================================================== --- trunk/pywikipedia/category_redirect.py 2008-08-26 23:35:30 UTC (rev 5849) +++ trunk/pywikipedia/category_redirect.py 2008-08-27 10:25:38 UTC (rev 5850) @@ -223,13 +223,11 @@ # no messages on user pages or non-talk pages talkpage = article.toggleTalkPage() try: - talktext = talk.get() + talktext = talkpage.get() except wikipedia.IsRedirectPage: return False except wikipedia.NoPage: talktext = u"" - if not talk.isTalkPage(): - return False talktext = talktext + wikipedia.translate( self.site.lang, self.talk_notification) % {

15 years, 8 months

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

pywikibot August 2008