Pywikipedia-l August 2008

pywikipedia-l@lists.wikimedia.org

29 participants
206 discussions

SVN: [5845] trunk/pywikipedia/category_redirect.py

by russblau＠svn.wikimedia.org

Revision: 5845 Author: russblau Date: 2008-08-24 18:20:21 +0000 (Sun, 24 Aug 2008) Log Message: ----------- Two more localizable strings Modified Paths: -------------- trunk/pywikipedia/category_redirect.py Modified: trunk/pywikipedia/category_redirect.py =================================================================== --- trunk/pywikipedia/category_redirect.py 2008-08-24 17:48:31 UTC (rev 5844) +++ trunk/pywikipedia/category_redirect.py 2008-08-24 18:20:21 UTC (rev 5845) @@ -22,6 +22,7 @@ import time from datetime import datetime, timedelta + class APIError(Exception): """The wiki API returned an error message.""" def __init__(self, errordict): @@ -175,6 +176,21 @@ 'no': u"Bot for vedlikehold av kategoriomdirigeringer", } + self.talk_notification = { + '_default': u""" +== Category link == +{{editprotected}} +* This protected page has been detected in [[%(oldcat)s]], but that category \ +has been redirected to [[%(newcat)s]]. Please update the category link. \ +--~~~~ +""", + } + + self.talk_notification_comment = { + '_default': + u"Robot: Category-redirect notification on protected page", + } + def change_category(self, article, oldCat, newCat, comment=None, sortKey=None): """Given an article in category oldCat, moves it to category newCat. @@ -213,17 +229,17 @@ talktext = u"" if not talk.isTalkPage(): return False - talktext = talktext + u""" -== Category link == -{{editprotected}} -* This protected page has been detected in [[%s]], but that category has \ -been redirected to [[%s]]. Please update the category link. --~~~~ -""" % (oldCat.aslink(textlink=True), newCat.aslink(textlink=True)) - # NEEDS LOCALIZATION + talktext = talktext + wikipedia.translate( + self.site.lang, + self.talk_notification) % { + 'oldcat': oldCat.aslink(textlink=True), + 'newcat': newCat.aslink(textlink=True)} try: talkpage.put(talktext, - u"Robot: Category-redirect notification on protected page", - minorEdit=False) # NEEDS LOCALIZATION + wikipedia.translate( + self.site.lang, + self.talk_notification_comment), + minorEdit=False) wikipedia.output( u"Left protected page notification on %s" % talkpage.aslink())

15 years, 8 months

[ pywikipediabot-Patches-2070928 ] category_redirect.py - update for ja locale

by SourceForge.net

Patches item #2070928, was opened at 2008-08-24 08:03 Message generated for change (Comment added) made by cosoleto You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=2070928&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Nightshadow28 (nightshadow28) Assigned to: Nobody/Anonymous (nobody) Summary: category_redirect.py - update for ja locale Initial Comment: update for ja locale. I attached a diff file on sourceforge. * category_redirect.py Thanks. ---------------------------------------------------------------------- >Comment By: Francesco Cosoleto (cosoleto) Date: 2008-08-24 19:51 Message: Logged In: YES user_id=181280 Originator: NO Thanks to you and sorry for inconvenience. ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 19:30 Message: Logged In: YES user_id=1971526 Originator: YES I did it. Thanks. https://jira.toolserver.org/browse/PYWP-8 ---------------------------------------------------------------------- Comment By: Francesco Cosoleto (cosoleto) Date: 2008-08-24 17:20 Message: Logged In: YES user_id=181280 Originator: NO https://jira.toolserver.org/secure/CreateIssue!default.jspa https://jira.toolserver.org/secure/AttachFile!default.jspa?id=NEWID ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 15:25 Message: Logged In: YES user_id=1971526 Originator: YES Um, the site can not attach a file for issue, can't it? (I could not find the function on create-issue page) In this item, we need to share a file written in non-English text... ---------------------------------------------------------------------- Comment By: Francesco Cosoleto (cosoleto) Date: 2008-08-24 12:56 Message: Logged In: YES user_id=181280 Originator: NO If SF doesn't work, you can use an alternative bug tracker hosted by Toolserver: https://jira.toolserver.org/browse/PYWP ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 08:50 Message: Logged In: YES user_id=1971526 Originator: YES Thanks. I try to attach a dummy file. And if it fails, I will report into the item about detail. ---------------------------------------------------------------------- Comment By: NicDumZ Nicolas Dumazet (nicdumz) Date: 2008-08-24 08:40 Message: Logged In: YES user_id=1963242 Originator: NO I think this is a sourceforge bug, see https://sourceforge.net/tracker/index.php?func=detail&aid=2070259&group_id=… Trying to attach a test file... ---------------------------------------------------------------------- Comment By: NicDumZ Nicolas Dumazet (nicdumz) Date: 2008-08-24 08:40 Message: Logged In: YES user_id=1963242 Originator: NO I think this is a sourceforge bug, see ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 08:15 Message: Logged In: YES user_id=1971526 Originator: YES Retrying... Why attaching fails? ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 08:05 Message: Logged In: YES user_id=1971526 Originator: YES Attached a patch file again. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=2070928&group_…

15 years, 8 months

[JIRA] Created: (PYWP-8) category_redirect.py - update for ja locale

by euphoritechnica (JIRA)

Please do not reply to this email, it was automatically generated. If you would like to comment on the issue, visit this URL: https://jira.toolserver.org/browse/PYWP-8 category_redirect.py - update for ja locale ------------------------------------------- Key: PYWP-8 URL: https://jira.toolserver.org/browse/PYWP-8 Project: pywikipedia Issue Type: Improvement Reporter: euphoritechnica category_redirect.py - update for ja locale. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.toolserver.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

15 years, 8 months

SVN: [5844] trunk/pywikipedia/category_redirect.py

by cosoleto＠svn.wikimedia.org

Revision: 5844 Author: cosoleto Date: 2008-08-24 17:48:31 +0000 (Sun, 24 Aug 2008) Log Message: ----------- Applied patch #2070928, ja translations for category_redirect.py by Nightshadow28 Modified Paths: -------------- trunk/pywikipedia/category_redirect.py Modified: trunk/pywikipedia/category_redirect.py =================================================================== --- trunk/pywikipedia/category_redirect.py 2008-08-24 11:59:21 UTC (rev 5843) +++ trunk/pywikipedia/category_redirect.py 2008-08-24 17:48:31 UTC (rev 5844) @@ -1,654 +1,662 @@ -# -*- coding: utf-8 -*- -"""This bot will move pages out of redirected categories - -Usage: category-redirect.py [options] - -The following command-line options can be used with this bot: - --often Only scan those redirected categories that have been - identified as often-populated (only useful if the site - has such a category defined) - -""" - -import wikipedia, catlib -import pagegenerators -import simplejson -import cPickle -import math -import re -import sys, traceback -import threading, Queue -import time -from datetime import datetime, timedelta - -class APIError(Exception): - """The wiki API returned an error message.""" - def __init__(self, errordict): - """Save error dict returned by MW API.""" - self.errors = errordict - - def __str__(self): - return "%(code)s: %(info)s" % self.errors - - -class ThreadList(list): - """A simple threadpool class to limit the number of simultaneous threads. - - Any threading.Thread object can be added to the pool using the append() - method. If the maximum number of simultaneous threads has not been - reached, the Thread object will be started immediately; if not, the - append() call will block until the thread is able to start. - - >>> pool = ThreadList(limit=10) - >>> def work(): - ... time.sleep(1) - ... - >>> for x in xrange(20): - ... pool.append(threading.Thread(target=work)) - ... - - """ - def __init__(self, limit=sys.maxint, *args): - self.limit = limit - list.__init__(self, *args) - for item in list(self): - if not isinstance(threading.Thread, item): - raise TypeError("Cannot add '%s' to ThreadList" % type(item)) - - def active_count(self): - """Return the number of alive threads, and delete all non-alive ones.""" - count = 0 - for item in list(self): - if item.isAlive(): - count += 1 - else: - self.remove(item) - return count - - def append(self, thd): - if not isinstance(thd, threading.Thread): - raise TypeError("Cannot append '%s' to ThreadList" % type(thd)) - while self.active_count() >= self.limit: - time.sleep(2) - list.append(self, thd) - thd.start() - - -class CategoryRedirectBot(object): - def __init__(self, often): - self.cooldown = 6 # days - self.often = often - - # Localization: - - # Category that contains all redirected category pages - self.cat_redirect_cat = { - 'wikipedia': { - 'en': "Category:Wikipedia category redirects", - 'ar': "تصنيف:تحويلات تصنيفات ويكيبيديا", - 'no': "Kategori:Wikipedia omdirigertekategorier", - 'simple': "Category:Category redirects", - }, - 'commons': { - 'commons': "Category:Non-empty category redirects" - } - } - - # Category that contains frequently-used redirected category pages - self.often_redirect_cat = { - 'wikipedia': { - 'en': "Category:Often-populated Wikipedia category redirects", - }, - } - - # List of all templates that are used to mark category redirects - # (put the most preferred form first) - self.redir_templates = { - 'wikipedia': { - 'en': ("Category redirect", - "Category redirect3", - "Categoryredirect", - "Empty category", - "CR", - "Catredirect", - "Emptycat", - "Emptycategory", - "Empty cat", - "Seecat",), - 'ar': ("تحويل تصنيف", - "Category redirect", - "تحويلة تصنيف",), - 'no': ("Kategoriomdirigering",), - 'simple': ("Category redirect", - "Catredirect"), - }, - 'commons': { - 'commons': (u'Category redirect', - u'Categoryredirect', - u'See cat', - u'Seecat', - u'Catredirect', - u'Cat redirect', - u'CatRed', - u'Catredir',), - } - } - - self.move_comment = { - '_default': -u"Robot: moving pages out of redirected category", - 'ar': -u"روبوت: نقل الصفحات من تصنيف محول", - 'no': -u"Robot: Flytter sider ut av omdirigeringskategori", - 'commons': -u'Robot: Changing category link (following [[Template:Category redirect|category redirect]])' - } - - self.redir_comment = { - '_default': -u"Robot: adding category redirect template for maintenance", - 'ar': -u"روبوت: إضافة قالب تحويل تصنيف للصيانة", - 'no': -u"Robot: Legger til vedlikeholdsmal for kategoriomdirigering", - } - - self.dbl_redir_comment = { - '_default': u"Robot: fixing double-redirect", - 'ar': u"روبوت: تصليح تحويلة مزدوجة", - 'no': u"Robot: Ordner doble omdirigeringer", - } - - self.maint_comment = { - '_default': u"Category redirect maintenance bot", - 'ar': u"بوت صيانة تحويل التصنيف", - 'no': u"Bot for vedlikehold av kategoriomdirigeringer", - } - - def change_category(self, article, oldCat, newCat, comment=None, - sortKey=None): - """Given an article in category oldCat, moves it to category newCat. - Moves subcategories of oldCat as well. oldCat and newCat should be - Category objects. If newCat is None, the category will be removed. - - This is a copy of portions of catlib.change_category() with the - added capability to post a talk page message on pages that cannot be - fixed due to protection. - - """ - cats = article.categories(get_redirect=True) - - oldtext = article.get(get_redirect=True) - newtext = wikipedia.replaceCategoryInPlace(oldtext, oldCat, newCat) - if newtext == oldtext: - wikipedia.output( - u'No changes in made in page %s.' % article.aslink()) - return False - try: - article.put(newtext, comment) - return True - except wikipedia.EditConflict: - wikipedia.output( - u'Skipping %s because of edit conflict' % article.aslink()) - except wikipedia.LockedPage: - wikipedia.output(u'Skipping locked page %s' % article.aslink()) - if not article.isTalkPage and article.namespace != 2: - # no messages on user pages or non-talk pages - talkpage = article.toggleTalkPage() - try: - talktext = talk.get() - except wikipedia.IsRedirectPage: - return False - except wikipedia.NoPage: - talktext = u"" - if not talk.isTalkPage(): - return False - talktext = talktext + u""" -== Category link == -{{editprotected}} -* This protected page has been detected in [[%s]], but that category has \ -been redirected to [[%s]]. Please update the category link. --~~~~ -""" % (oldCat.aslink(textlink=True), newCat.aslink(textlink=True)) - # NEEDS LOCALIZATION - try: - talkpage.put(talktext, - u"Robot: Category-redirect notification on protected page", - minorEdit=False) # NEEDS LOCALIZATION - wikipedia.output( - u"Left protected page notification on %s" - % talkpage.aslink()) - except wikipedia.PageNotSaved: - wikipedia.output( - u"Protected page notification on %s failed" - % talkpage.aslink()) - - except wikipedia.SpamfilterError, error: - wikipedia.output( - u'Changing page %s blocked by spam filter (URL=%s)' - % (article.aslink(), error.url)) - except wikipedia.NoUsername: - wikipedia.output( - u"Page %s not saved; sysop privileges required." - % article.aslink()) - except wikipedia.PageNotSaved, error: - wikipedia.output(u"Saving page %s failed: %s" - % (article.aslink(), error.message)) - return False - - def move_contents(self, oldCatTitle, newCatTitle, editSummary): - """The worker function that moves pages out of oldCat into newCat""" - while True: - try: - oldCat = catlib.Category(self.site, - self.catprefix + oldCatTitle) - newCat = catlib.Category(self.site, - self.catprefix + newCatTitle) - - # Move articles - found, moved = 0, 0 - for result in self.query_results(list="categorymembers", - cmtitle=oldCat.title(), - cmprop="title|sortkey", - cmlimit="max"): - found += len(result['categorymembers']) - for item in result['categorymembers']: - article = wikipedia.Page(self.site, item['title']) - changed = self.change_category(article, oldCat, newCat, - comment=editSummary) - if changed: moved += 1 - - # pass 2: look for template doc pages - for result in self.query_results(list="categorymembers", - cmtitle=oldCat.title(), - cmprop="title|sortkey", - cmnamespace="10", - cmlimit="max"): - for item in result['categorymembers']: - doc = wikipedia.Page(self.site, item['title']+"/doc") - try: - old_text = doc.get() - except wikipedia.Error: - continue - changed = self.change_category(article, oldCat, newCat, - comment=editSummary) - if changed: moved += 1 - - if found: - wikipedia.output(u"%s: %s found, %s moved" - % (oldCat.title(), found, moved)) - #Dummy edit to refresh the page, shouldn't show up in any logs. - try: - oldCat.put(oldCat.get()) - except: - self.log_text.append(u'* Dummy edit at %s failed' - % oldCat.aslink(textlink=True)) - self.result_queue.put((oldCatTitle, found, moved)) - return - except wikipedia.ServerError: - wikipedia.output(u"Server error: retrying in 5 seconds...") - time.sleep(5) - continue - except: - self.result_queue.put((oldCatTitle, None, None)) - raise - - def readyToEdit(self, cat): - """Return True if cat not edited during cooldown period, else False.""" - dateformat ="%Y%m%d%H%M%S" - today = datetime.now() - deadline = today + timedelta(days=-self.cooldown) - if cat.editTime() is None: - raise RuntimeError - return (deadline.strftime(dateformat) > cat.editTime()) - - def query_results(self, **data): - """Iterate results from API action=query, using data as parameters.""" - addr = self.site.apipath() - querydata = {'action': 'query', - 'format': 'json', - 'maxlag': str(wikipedia.config.maxlag)} - querydata.update(data) - if not querydata.has_key("action")\ - or not querydata['action'] == 'query': - raise ValueError( - "query_results: 'action' set to value other than 'query'" - ) - waited = 0 - while True: - response, data = self.site.postForm(addr, querydata) - if data.startswith(u"unknown_action"): - e = {'code': data[:14], 'info': data[16:]} - raise APIError(e) - try: - result = simplejson.loads(data) - except ValueError: - # if the result isn't valid JSON, there must be a server - # problem. Wait a few seconds and try again - # TODO: warn user; if the server is down, this could - # cause an infinite loop - wikipedia.output("Invalid API response received; retrying...") - time.sleep(5) - continue - if type(result) is dict and result.has_key("error"): - if result['error']['code'] == "maxlag": - print "Pausing due to server lag.\r", - time.sleep(5) - waited += 5 - if waited % 30 == 0: - wikipedia.output( - u"(Waited %i seconds due to server lag.)" - % waited) - continue - else: - # raise error - raise APIError(result['error']) - waited = 0 - if type(result) is list: - # query returned no results - return - assert type(result) is dict, \ - "Unexpected result of type '%s' received." % type(result) - assert result.has_key("query"), \ - "No 'query' response found, result keys = %s" % result.keys() - yield result['query'] - if result.has_key("query-continue"): - assert len(result['query-continue'].keys()) == 1, \ - "More than one query-continue key returned: %s" \ - % result['query-continue'].keys() - query_type = result['query-continue'].keys()[0] - assert (query_type in querydata.keys() - or query_type in querydata.values()), \ - "Site returned unknown query-continue type '%s'"\ - % query_type - querydata.update(result['query-continue'][query_type]) - else: - return - - def run(self): - """Run the bot""" - self.site = wikipedia.getSite() - self.catprefix = self.site.namespace(14)+":" - self.result_queue = Queue.Queue() - self.log_text = [] - - user = self.site.loggedInAs() - redirect_magicwords = ["redirect"] - other_words = self.site.redirect() - if other_words: - redirect_magicwords.extend(other_words) - - problems = [] - - problem_page = wikipedia.Page(self.site, - u"User:%(user)s/category redirect problems" % locals()) - l = time.localtime() - today = "%04d-%02d-%02d" % l[:3] - log_page = wikipedia.Page(self.site, - u"User:%(user)s/category redirect logs/%(today)s" - % locals()) - - datafile = wikipedia.config.datafilepath( - "%s-catmovebot-data" % self.site.dbName()) - try: - inp = open(datafile, "rb") - record = cPickle.load(inp) - inp.close() - except IOError: - record = {} - if record: - cPickle.dump(record, open(datafile + ".bak", "wb")) - - # Set up regexes for later scanning - template_list = self.redir_templates[self.site.family.name - ][self.site.lang] - # regex to match soft category redirects - template_regexes = [ - re.compile( -ur"{{\s*(?:%(prefix)s\s*:\s*)?%(template)s\s*\|(\s*%(catns)s\s*:\s*)?([^}]+)}}" - % {'prefix': self.site.namespace(10).lower(), - 'template': item.replace(" ", "[ _]+"), - 'catns': self.site.namespace(14)}, - re.I) - for item in template_list - ] - template_regex = re.compile( -ur"{{\s*(?:%(prefix)s\s*:\s*)?(?:%(template)s)\s*\|(\s*%(catns)s\s*:\s*)?([^}]+)}}" - % {'prefix': self.site.namespace(10).lower(), - 'template': "|".join(item.replace(" ", "[ _]+") - for item in template_list), - 'catns': self.site.namespace(14)}, - re.I) - # regex to match hard redirects to category pages - catredir_regex = re.compile( -ur'\s*#(?:%(redir)s)\s*:?\s*\[\[\s*:?%(catns)s\s*:(.*?)\]\]\s*' - % {'redir': "|".join(redirect_magicwords), - 'catns': self.site.namespace(14)}, - re.I) - # regex to match all other hard redirects - redir_regex = re.compile(ur"(?i)\s*#(?:%s)\s*:?\s*\[\[(.*?)\]\]" - % "|".join(redirect_magicwords), - re.I) - - # check for hard-redirected categories that are not already marked - # with an appropriate template - comment = wikipedia.translate(self.site.lang, self.redir_comment) - for result in self.query_results(list='allpages', - apnamespace='14', # Category: - apfrom='!', - apfilterredir='redirects', - aplimit='max'): - gen = (wikipedia.Page(self.site, page_item['title']) - for page_item in result['allpages']) - for page in pagegenerators.PreloadingGenerator(gen, 120): - text = page.get(get_redirect=True) - if re.search(template_regex, text): - # this is already a soft-redirect, so skip it (for now) - continue - m = catredir_regex.match(text) - if m: - # this is a hard-redirect to a category page - newtext = (u"{{%(template)s|%(cat)s}}" - % {'cat': m.group(1), - 'template': template_list[0]}) - try: - page.put(newtext, comment, minorEdit=True) - self.log_text.append(u"* Added {{tl|%s}} to %s" - % (template_list[0], - page.aslink(textlink=True))) - except wikipedia.Error, e: - self.log_text.append( - u"* Failed to add {{tl|%s}} to %s (%s)" - % (template_list[0], - page.aslink(textlink=True), - e)) - else: - r = redir_regex.match(text) - if r: - problems.append( - u"# %s is a hard redirect to [[:%s]]" - % (page.aslink(textlink=True), - r.group(1))) - else: - problems.append( - u"# %s is a hard redirect; unable to extract target." - % page.aslink(textlink=True)) - - wikipedia.output("Done checking hard-redirect category pages.") - - comment = wikipedia.translate(self.site.lang, self.move_comment) - scan_data = { - u'action': 'query', - u'list': 'embeddedin', - u'einamespace': '14', # Category: - u'eilimit': 'max', - u'format': 'json' - } - counts = {} - destmap = {} - catmap = {} - catlist = [] - catpages = [] - if self.often: - target = self.often_redirect_cat[self.site.family.name - ][self.site.lang] - else: - target = self.cat_redirect_cat[self.site.family.name - ][self.site.lang] - - # get and preload all members of the category-redirect category - for result in self.query_results(generator=u'categorymembers', - gcmtitle=target, - gcmnamespace=u'14', # CATEGORY - gcmlimit=u'max', - prop='info'): - for catdata in result['pages'].values(): - catpages.append(wikipedia.Page(self.site, catdata['title'])) - - wikipedia.output(u"") - wikipedia.output(u"Preloading %s category redirect pages" - % len(catpages)) - thread_limit = int(math.log(len(catpages), 2)) - for cat in pagegenerators.PreloadingGenerator(catpages, 120): - cat_title = cat.titleWithoutNamespace() - if "Wikipedia category redirect" in cat_title: -# self.log_text.append("* Ignoring %s%s" % (self.catprefix, cat_title)) - continue - try: - text = cat.get(get_redirect=True) - except wikipedia.Error: - self.log_text.append(u"* Could not load %s%s; ignoring" - % (self.catprefix, cat_title)) - continue - match = template_regex.search(text) - if match is None: - self.log_text.append(u"* False positive: %s" % cat_title) - continue - catlist.append(cat) - destination = match.group(2) - target = catlib.Category(self.site, self.catprefix+destination) - destmap.setdefault(target, []).append(cat) - catmap[cat] = destination - if match.group(1): - # category redirect target starts with "Category:" - fix it - text = text[ :match.start(1)] + text[match.end(1): ] - cat.put(text, - u"Robot: fixing category redirect parameter format") - self.log_text.append( - u"* Removed category prefix from parameter in %s" - % cat.aslink(textlink=True)) - - wikipedia.output(u"") - wikipedia.output(u"Checking %s destination categories" % len(destmap)) - for dest in pagegenerators.PreloadingGenerator(destmap.keys(), 120): - if not dest.exists(): - for d in destmap[dest]: - problems.append("# %s redirects to %s" - % (d.aslink(textlink=True), - dest.aslink(textlink=True))) - catlist.remove(d) - # do a null edit on d to make it appear in the - # "needs repair" category (if this wiki has one) - try: - d.put(d.get(get_redirect=True)) - except: - pass - if dest in catlist: - for d in destmap[dest]: - # is catmap[dest] also a redirect? - newcat = catlib.Category(self.site, - self.catprefix+catmap[dest]) - while newcat in catlist: - if newcat == d or newcat == dest: - self.log_text.append(u"* Redirect loop from %s" - % newcat.aslink(textlink=True)) - break - newcat = catlib.Category(self.site, - self.catprefix+catmap[newcat]) - else: - self.log_text.append( - u"* Fixed double-redirect: %s -> %s -> %s" - % (d.aslink(textlink=True), - dest.aslink(textlink=True), - newcat.aslink(textlink=True))) - oldtext = d.get(get_redirect=True) - # remove the old redirect from the old text, - # leaving behind any non-redirect text - oldtext = template_regex.sub("", oldtext) - newtext = (u"{{category redirect|%(ncat)s}}" - % {'ncat': newcat.titleWithoutNamespace()}) - newtext = newtext + oldtext.strip() - try: - d.put(newtext, - wikipedia.translate(self.site.lang, - self.dbl_redir_comment), - minorEdit=True) - except wikipedia.Error, e: - self.log_text.append("** Failed: %s" % str(e)) - - wikipedia.output(u"") - wikipedia.output(u"Moving pages out of %s redirected categories." - % len(catlist)) - threadpool = ThreadList(limit=thread_limit) - - for cat in catlist: - cat_title = cat.titleWithoutNamespace() - if not self.readyToEdit(cat): - counts[cat_title] = None - continue - threadpool.append(threading.Thread(target=self.move_contents, - args=(cat_title, catmap[cat]), - kwargs=dict(editSummary=comment))) - while len(counts) < len(catlist): - title, found, moved = self.result_queue.get() - if found is None: - self.log_text.append( - u"* [[:%s%s]]: error in move_contents thread" - % (self.catprefix, title)) - else: - if found: - self.log_text.append( - u"* [[:%s%s]]: %d found, %d moved" - % (self.catprefix, title, found, moved)) - counts[title] = found - - if not self.often: - for cat in record.keys(): - if cat not in counts.keys(): - del record[cat] - for cat in counts.keys(): - if counts[cat] is not None: - if counts[cat]: - record.setdefault(cat, {})[today] = counts[cat] - else: - record.setdefault(cat, {}) - cPickle.dump(record, open(datafile, "wb")) - - wikipedia.setAction(wikipedia.translate(self.site.lang, - self.maint_comment)) - log_page.put("\n".join(self.log_text)) - problem_page.put("\n".join(problems)) - - -def main(*args): - try: - a = wikipedia.handleArgs(*args) - if "-often" in a: - a.remove("-often") - often = True - else: - often = False - if len(a) == 1: - raise RuntimeError('Unrecognized argument "%s"' % a[0]) - elif a: - raise RuntimeError('Unrecognized arguments: ' + - " ".join(('"%s"' % arg) for arg in a)) - bot = CategoryRedirectBot(often) - bot.run() - finally: - wikipedia.stopme() - - -if __name__ == "__main__": - main() +# -*- coding: utf-8 -*- +"""This bot will move pages out of redirected categories + +Usage: category-redirect.py [options] + +The following command-line options can be used with this bot: + +-often Only scan those redirected categories that have been + identified as often-populated (only useful if the site + has such a category defined) + +""" + +import wikipedia, catlib +import pagegenerators +import simplejson +import cPickle +import math +import re +import sys, traceback +import threading, Queue +import time +from datetime import datetime, timedelta + +class APIError(Exception): + """The wiki API returned an error message.""" + def __init__(self, errordict): + """Save error dict returned by MW API.""" + self.errors = errordict + + def __str__(self): + return "%(code)s: %(info)s" % self.errors + + +class ThreadList(list): + """A simple threadpool class to limit the number of simultaneous threads. + + Any threading.Thread object can be added to the pool using the append() + method. If the maximum number of simultaneous threads has not been + reached, the Thread object will be started immediately; if not, the + append() call will block until the thread is able to start. + + >>> pool = ThreadList(limit=10) + >>> def work(): + ... time.sleep(1) + ... + >>> for x in xrange(20): + ... pool.append(threading.Thread(target=work)) + ... + + """ + def __init__(self, limit=sys.maxint, *args): + self.limit = limit + list.__init__(self, *args) + for item in list(self): + if not isinstance(threading.Thread, item): + raise TypeError("Cannot add '%s' to ThreadList" % type(item)) + + def active_count(self): + """Return the number of alive threads, and delete all non-alive ones.""" + count = 0 + for item in list(self): + if item.isAlive(): + count += 1 + else: + self.remove(item) + return count + + def append(self, thd): + if not isinstance(thd, threading.Thread): + raise TypeError("Cannot append '%s' to ThreadList" % type(thd)) + while self.active_count() >= self.limit: + time.sleep(2) + list.append(self, thd) + thd.start() + + +class CategoryRedirectBot(object): + def __init__(self, often): + self.cooldown = 6 # days + self.often = often + + # Localization: + + # Category that contains all redirected category pages + self.cat_redirect_cat = { + 'wikipedia': { + 'en': "Category:Wikipedia category redirects", + 'ar': "تصنيف:تحويلات تصنيفات ويكيبيديا", + 'no': "Kategori:Wikipedia omdirigertekategorier", + 'ja': "Category:移行中のカテゴリ", + 'simple': "Category:Category redirects", + }, + 'commons': { + 'commons': "Category:Non-empty category redirects" + } + } + + # Category that contains frequently-used redirected category pages + self.often_redirect_cat = { + 'wikipedia': { + 'en': "Category:Often-populated Wikipedia category redirects", + }, + } + + # List of all templates that are used to mark category redirects + # (put the most preferred form first) + self.redir_templates = { + 'wikipedia': { + 'en': ("Category redirect", + "Category redirect3", + "Categoryredirect", + "Empty category", + "CR", + "Catredirect", + "Emptycat", + "Emptycategory", + "Empty cat", + "Seecat",), + 'ar': ("تحويل تصنيف", + "Category redirect", + "تحويلة تصنيف",), + 'no': ("Kategoriomdirigering",), + 'ja': ("Category redirect",), + 'simple': ("Category redirect", + "Catredirect"), + }, + 'commons': { + 'commons': (u'Category redirect', + u'Categoryredirect', + u'See cat', + u'Seecat', + u'Catredirect', + u'Cat redirect', + u'CatRed', + u'Catredir',), + } + } + + self.move_comment = { + '_default': +u"Robot: moving pages out of redirected category", + 'ar': +u"روبوت: نقل الصفحات من تصنيف محول", + 'no': +u"Robot: Flytter sider ut av omdirigeringskategori", + 'ja': +u"ロボットによる: 移行中のカテゴリからのカテゴリ変更", + 'commons': +u'Robot: Changing category link (following [[Template:Category redirect|category redirect]])' + } + + self.redir_comment = { + '_default': +u"Robot: adding category redirect template for maintenance", + 'ar': +u"روبوت: إضافة قالب تحويل تصنيف للصيانة", + 'ja': +u"ロボットによる: 移行中のカテゴリとしてタグ付け", + 'no': +u"Robot: Legger til vedlikeholdsmal for kategoriomdirigering", + } + + self.dbl_redir_comment = { + '_default': u"Robot: fixing double-redirect", + 'ar': u"روبوت: تصليح تحويلة مزدوجة", + 'ja': u"ロボットによる: 二重リダイレクト修正", + 'no': u"Robot: Ordner doble omdirigeringer", + } + + self.maint_comment = { + '_default': u"Category redirect maintenance bot", + 'ar': u"بوت صيانة تحويل التصنيف", + 'ja': u"移行中のカテゴリのメンテナンス・ボット", + 'no': u"Bot for vedlikehold av kategoriomdirigeringer", + } + + def change_category(self, article, oldCat, newCat, comment=None, + sortKey=None): + """Given an article in category oldCat, moves it to category newCat. + Moves subcategories of oldCat as well. oldCat and newCat should be + Category objects. If newCat is None, the category will be removed. + + This is a copy of portions of catlib.change_category() with the + added capability to post a talk page message on pages that cannot be + fixed due to protection. + + """ + cats = article.categories(get_redirect=True) + + oldtext = article.get(get_redirect=True) + newtext = wikipedia.replaceCategoryInPlace(oldtext, oldCat, newCat) + if newtext == oldtext: + wikipedia.output( + u'No changes in made in page %s.' % article.aslink()) + return False + try: + article.put(newtext, comment) + return True + except wikipedia.EditConflict: + wikipedia.output( + u'Skipping %s because of edit conflict' % article.aslink()) + except wikipedia.LockedPage: + wikipedia.output(u'Skipping locked page %s' % article.aslink()) + if not article.isTalkPage and article.namespace != 2: + # no messages on user pages or non-talk pages + talkpage = article.toggleTalkPage() + try: + talktext = talk.get() + except wikipedia.IsRedirectPage: + return False + except wikipedia.NoPage: + talktext = u"" + if not talk.isTalkPage(): + return False + talktext = talktext + u""" +== Category link == +{{editprotected}} +* This protected page has been detected in [[%s]], but that category has \ +been redirected to [[%s]]. Please update the category link. --~~~~ +""" % (oldCat.aslink(textlink=True), newCat.aslink(textlink=True)) + # NEEDS LOCALIZATION + try: + talkpage.put(talktext, + u"Robot: Category-redirect notification on protected page", + minorEdit=False) # NEEDS LOCALIZATION + wikipedia.output( + u"Left protected page notification on %s" + % talkpage.aslink()) + except wikipedia.PageNotSaved: + wikipedia.output( + u"Protected page notification on %s failed" + % talkpage.aslink()) + + except wikipedia.SpamfilterError, error: + wikipedia.output( + u'Changing page %s blocked by spam filter (URL=%s)' + % (article.aslink(), error.url)) + except wikipedia.NoUsername: + wikipedia.output( + u"Page %s not saved; sysop privileges required." + % article.aslink()) + except wikipedia.PageNotSaved, error: + wikipedia.output(u"Saving page %s failed: %s" + % (article.aslink(), error.message)) + return False + + def move_contents(self, oldCatTitle, newCatTitle, editSummary): + """The worker function that moves pages out of oldCat into newCat""" + while True: + try: + oldCat = catlib.Category(self.site, + self.catprefix + oldCatTitle) + newCat = catlib.Category(self.site, + self.catprefix + newCatTitle) + + # Move articles + found, moved = 0, 0 + for result in self.query_results(list="categorymembers", + cmtitle=oldCat.title(), + cmprop="title|sortkey", + cmlimit="max"): + found += len(result['categorymembers']) + for item in result['categorymembers']: + article = wikipedia.Page(self.site, item['title']) + changed = self.change_category(article, oldCat, newCat, + comment=editSummary) + if changed: moved += 1 + + # pass 2: look for template doc pages + for result in self.query_results(list="categorymembers", + cmtitle=oldCat.title(), + cmprop="title|sortkey", + cmnamespace="10", + cmlimit="max"): + for item in result['categorymembers']: + doc = wikipedia.Page(self.site, item['title']+"/doc") + try: + old_text = doc.get() + except wikipedia.Error: + continue + changed = self.change_category(article, oldCat, newCat, + comment=editSummary) + if changed: moved += 1 + + if found: + wikipedia.output(u"%s: %s found, %s moved" + % (oldCat.title(), found, moved)) + #Dummy edit to refresh the page, shouldn't show up in any logs. + try: + oldCat.put(oldCat.get()) + except: + self.log_text.append(u'* Dummy edit at %s failed' + % oldCat.aslink(textlink=True)) + self.result_queue.put((oldCatTitle, found, moved)) + return + except wikipedia.ServerError: + wikipedia.output(u"Server error: retrying in 5 seconds...") + time.sleep(5) + continue + except: + self.result_queue.put((oldCatTitle, None, None)) + raise + + def readyToEdit(self, cat): + """Return True if cat not edited during cooldown period, else False.""" + dateformat ="%Y%m%d%H%M%S" + today = datetime.now() + deadline = today + timedelta(days=-self.cooldown) + if cat.editTime() is None: + raise RuntimeError + return (deadline.strftime(dateformat) > cat.editTime()) + + def query_results(self, **data): + """Iterate results from API action=query, using data as parameters.""" + addr = self.site.apipath() + querydata = {'action': 'query', + 'format': 'json', + 'maxlag': str(wikipedia.config.maxlag)} + querydata.update(data) + if not querydata.has_key("action")\ + or not querydata['action'] == 'query': + raise ValueError( + "query_results: 'action' set to value other than 'query'" + ) + waited = 0 + while True: + response, data = self.site.postForm(addr, querydata) + if data.startswith(u"unknown_action"): + e = {'code': data[:14], 'info': data[16:]} + raise APIError(e) + try: + result = simplejson.loads(data) + except ValueError: + # if the result isn't valid JSON, there must be a server + # problem. Wait a few seconds and try again + # TODO: warn user; if the server is down, this could + # cause an infinite loop + wikipedia.output("Invalid API response received; retrying...") + time.sleep(5) + continue + if type(result) is dict and result.has_key("error"): + if result['error']['code'] == "maxlag": + print "Pausing due to server lag.\r", + time.sleep(5) + waited += 5 + if waited % 30 == 0: + wikipedia.output( + u"(Waited %i seconds due to server lag.)" + % waited) + continue + else: + # raise error + raise APIError(result['error']) + waited = 0 + if type(result) is list: + # query returned no results + return + assert type(result) is dict, \ + "Unexpected result of type '%s' received." % type(result) + assert result.has_key("query"), \ + "No 'query' response found, result keys = %s" % result.keys() + yield result['query'] + if result.has_key("query-continue"): + assert len(result['query-continue'].keys()) == 1, \ + "More than one query-continue key returned: %s" \ + % result['query-continue'].keys() + query_type = result['query-continue'].keys()[0] + assert (query_type in querydata.keys() + or query_type in querydata.values()), \ + "Site returned unknown query-continue type '%s'"\ + % query_type + querydata.update(result['query-continue'][query_type]) + else: + return + + def run(self): + """Run the bot""" + self.site = wikipedia.getSite() + self.catprefix = self.site.namespace(14)+":" + self.result_queue = Queue.Queue() + self.log_text = [] + + user = self.site.loggedInAs() + redirect_magicwords = ["redirect"] + other_words = self.site.redirect() + if other_words: + redirect_magicwords.extend(other_words) + + problems = [] + + problem_page = wikipedia.Page(self.site, + u"User:%(user)s/category redirect problems" % locals()) + l = time.localtime() + today = "%04d-%02d-%02d" % l[:3] + log_page = wikipedia.Page(self.site, + u"User:%(user)s/category redirect logs/%(today)s" + % locals()) + + datafile = wikipedia.config.datafilepath( + "%s-catmovebot-data" % self.site.dbName()) + try: + inp = open(datafile, "rb") + record = cPickle.load(inp) + inp.close() + except IOError: + record = {} + if record: + cPickle.dump(record, open(datafile + ".bak", "wb")) + + # Set up regexes for later scanning + template_list = self.redir_templates[self.site.family.name + ][self.site.lang] + # regex to match soft category redirects + template_regexes = [ + re.compile( +ur"{{\s*(?:%(prefix)s\s*:\s*)?%(template)s\s*\|(\s*%(catns)s\s*:\s*)?([^}]+)}}" + % {'prefix': self.site.namespace(10).lower(), + 'template': item.replace(" ", "[ _]+"), + 'catns': self.site.namespace(14)}, + re.I) + for item in template_list + ] + template_regex = re.compile( +ur"{{\s*(?:%(prefix)s\s*:\s*)?(?:%(template)s)\s*\|(\s*%(catns)s\s*:\s*)?([^}]+)}}" + % {'prefix': self.site.namespace(10).lower(), + 'template': "|".join(item.replace(" ", "[ _]+") + for item in template_list), + 'catns': self.site.namespace(14)}, + re.I) + # regex to match hard redirects to category pages + catredir_regex = re.compile( +ur'\s*#(?:%(redir)s)\s*:?\s*\[\[\s*:?%(catns)s\s*:(.*?)\]\]\s*' + % {'redir': "|".join(redirect_magicwords), + 'catns': self.site.namespace(14)}, + re.I) + # regex to match all other hard redirects + redir_regex = re.compile(ur"(?i)\s*#(?:%s)\s*:?\s*\[\[(.*?)\]\]" + % "|".join(redirect_magicwords), + re.I) + + # check for hard-redirected categories that are not already marked + # with an appropriate template + comment = wikipedia.translate(self.site.lang, self.redir_comment) + for result in self.query_results(list='allpages', + apnamespace='14', # Category: + apfrom='!', + apfilterredir='redirects', + aplimit='max'): + gen = (wikipedia.Page(self.site, page_item['title']) + for page_item in result['allpages']) + for page in pagegenerators.PreloadingGenerator(gen, 120): + text = page.get(get_redirect=True) + if re.search(template_regex, text): + # this is already a soft-redirect, so skip it (for now) + continue + m = catredir_regex.match(text) + if m: + # this is a hard-redirect to a category page + newtext = (u"{{%(template)s|%(cat)s}}" + % {'cat': m.group(1), + 'template': template_list[0]}) + try: + page.put(newtext, comment, minorEdit=True) + self.log_text.append(u"* Added {{tl|%s}} to %s" + % (template_list[0], + page.aslink(textlink=True))) + except wikipedia.Error, e: + self.log_text.append( + u"* Failed to add {{tl|%s}} to %s (%s)" + % (template_list[0], + page.aslink(textlink=True), + e)) + else: + r = redir_regex.match(text) + if r: + problems.append( + u"# %s is a hard redirect to [[:%s]]" + % (page.aslink(textlink=True), + r.group(1))) + else: + problems.append( + u"# %s is a hard redirect; unable to extract target." + % page.aslink(textlink=True)) + + wikipedia.output("Done checking hard-redirect category pages.") + + comment = wikipedia.translate(self.site.lang, self.move_comment) + scan_data = { + u'action': 'query', + u'list': 'embeddedin', + u'einamespace': '14', # Category: + u'eilimit': 'max', + u'format': 'json' + } + counts = {} + destmap = {} + catmap = {} + catlist = [] + catpages = [] + if self.often: + target = self.often_redirect_cat[self.site.family.name + ][self.site.lang] + else: + target = self.cat_redirect_cat[self.site.family.name + ][self.site.lang] + + # get and preload all members of the category-redirect category + for result in self.query_results(generator=u'categorymembers', + gcmtitle=target, + gcmnamespace=u'14', # CATEGORY + gcmlimit=u'max', + prop='info'): + for catdata in result['pages'].values(): + catpages.append(wikipedia.Page(self.site, catdata['title'])) + + wikipedia.output(u"") + wikipedia.output(u"Preloading %s category redirect pages" + % len(catpages)) + thread_limit = int(math.log(len(catpages), 2)) + for cat in pagegenerators.PreloadingGenerator(catpages, 120): + cat_title = cat.titleWithoutNamespace() + if "Wikipedia category redirect" in cat_title: +# self.log_text.append("* Ignoring %s%s" % (self.catprefix, cat_title)) + continue + try: + text = cat.get(get_redirect=True) + except wikipedia.Error: + self.log_text.append(u"* Could not load %s%s; ignoring" + % (self.catprefix, cat_title)) + continue + match = template_regex.search(text) + if match is None: + self.log_text.append(u"* False positive: %s" % cat_title) + continue + catlist.append(cat) + destination = match.group(2) + target = catlib.Category(self.site, self.catprefix+destination) + destmap.setdefault(target, []).append(cat) + catmap[cat] = destination + if match.group(1): + # category redirect target starts with "Category:" - fix it + text = text[ :match.start(1)] + text[match.end(1): ] + cat.put(text, + u"Robot: fixing category redirect parameter format") + self.log_text.append( + u"* Removed category prefix from parameter in %s" + % cat.aslink(textlink=True)) + + wikipedia.output(u"") + wikipedia.output(u"Checking %s destination categories" % len(destmap)) + for dest in pagegenerators.PreloadingGenerator(destmap.keys(), 120): + if not dest.exists(): + for d in destmap[dest]: + problems.append("# %s redirects to %s" + % (d.aslink(textlink=True), + dest.aslink(textlink=True))) + catlist.remove(d) + # do a null edit on d to make it appear in the + # "needs repair" category (if this wiki has one) + try: + d.put(d.get(get_redirect=True)) + except: + pass + if dest in catlist: + for d in destmap[dest]: + # is catmap[dest] also a redirect? + newcat = catlib.Category(self.site, + self.catprefix+catmap[dest]) + while newcat in catlist: + if newcat == d or newcat == dest: + self.log_text.append(u"* Redirect loop from %s" + % newcat.aslink(textlink=True)) + break + newcat = catlib.Category(self.site, + self.catprefix+catmap[newcat]) + else: + self.log_text.append( + u"* Fixed double-redirect: %s -> %s -> %s" + % (d.aslink(textlink=True), + dest.aslink(textlink=True), + newcat.aslink(textlink=True))) + oldtext = d.get(get_redirect=True) + # remove the old redirect from the old text, + # leaving behind any non-redirect text + oldtext = template_regex.sub("", oldtext) + newtext = (u"{{category redirect|%(ncat)s}}" + % {'ncat': newcat.titleWithoutNamespace()}) + newtext = newtext + oldtext.strip() + try: + d.put(newtext, + wikipedia.translate(self.site.lang, + self.dbl_redir_comment), + minorEdit=True) + except wikipedia.Error, e: + self.log_text.append("** Failed: %s" % str(e)) + + wikipedia.output(u"") + wikipedia.output(u"Moving pages out of %s redirected categories." + % len(catlist)) + threadpool = ThreadList(limit=thread_limit) + + for cat in catlist: + cat_title = cat.titleWithoutNamespace() + if not self.readyToEdit(cat): + counts[cat_title] = None + continue + threadpool.append(threading.Thread(target=self.move_contents, + args=(cat_title, catmap[cat]), + kwargs=dict(editSummary=comment))) + while len(counts) < len(catlist): + title, found, moved = self.result_queue.get() + if found is None: + self.log_text.append( + u"* [[:%s%s]]: error in move_contents thread" + % (self.catprefix, title)) + else: + if found: + self.log_text.append( + u"* [[:%s%s]]: %d found, %d moved" + % (self.catprefix, title, found, moved)) + counts[title] = found + + if not self.often: + for cat in record.keys(): + if cat not in counts.keys(): + del record[cat] + for cat in counts.keys(): + if counts[cat] is not None: + if counts[cat]: + record.setdefault(cat, {})[today] = counts[cat] + else: + record.setdefault(cat, {}) + cPickle.dump(record, open(datafile, "wb")) + + wikipedia.setAction(wikipedia.translate(self.site.lang, + self.maint_comment)) + log_page.put("\n".join(self.log_text)) + problem_page.put("\n".join(problems)) + + +def main(*args): + try: + a = wikipedia.handleArgs(*args) + if "-often" in a: + a.remove("-often") + often = True + else: + often = False + if len(a) == 1: + raise RuntimeError('Unrecognized argument "%s"' % a[0]) + elif a: + raise RuntimeError('Unrecognized arguments: ' + + " ".join(('"%s"' % arg) for arg in a)) + bot = CategoryRedirectBot(often) + bot.run() + finally: + wikipedia.stopme() + + +if __name__ == "__main__": + main()

15 years, 8 months

[ pywikipediabot-Patches-2070928 ] category_redirect.py - update for ja locale

by SourceForge.net

Patches item #2070928, was opened at 2008-08-24 06:03 Message generated for change (Comment added) made by nightshadow28 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=2070928&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nightshadow28 (nightshadow28) Assigned to: Nobody/Anonymous (nobody) Summary: category_redirect.py - update for ja locale Initial Comment: update for ja locale. I attached a diff file on sourceforge. * category_redirect.py Thanks. ---------------------------------------------------------------------- >Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 17:30 Message: Logged In: YES user_id=1971526 Originator: YES I did it. Thanks. https://jira.toolserver.org/browse/PYWP-8 ---------------------------------------------------------------------- Comment By: Francesco Cosoleto (cosoleto) Date: 2008-08-24 15:20 Message: Logged In: YES user_id=181280 Originator: NO https://jira.toolserver.org/secure/CreateIssue!default.jspa https://jira.toolserver.org/secure/AttachFile!default.jspa?id=NEWID ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 13:25 Message: Logged In: YES user_id=1971526 Originator: YES Um, the site can not attach a file for issue, can't it? (I could not find the function on create-issue page) In this item, we need to share a file written in non-English text... ---------------------------------------------------------------------- Comment By: Francesco Cosoleto (cosoleto) Date: 2008-08-24 10:56 Message: Logged In: YES user_id=181280 Originator: NO If SF doesn't work, you can use an alternative bug tracker hosted by Toolserver: https://jira.toolserver.org/browse/PYWP ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 06:50 Message: Logged In: YES user_id=1971526 Originator: YES Thanks. I try to attach a dummy file. And if it fails, I will report into the item about detail. ---------------------------------------------------------------------- Comment By: NicDumZ Nicolas Dumazet (nicdumz) Date: 2008-08-24 06:40 Message: Logged In: YES user_id=1963242 Originator: NO I think this is a sourceforge bug, see https://sourceforge.net/tracker/index.php?func=detail&aid=2070259&group_id=… Trying to attach a test file... ---------------------------------------------------------------------- Comment By: NicDumZ Nicolas Dumazet (nicdumz) Date: 2008-08-24 06:40 Message: Logged In: YES user_id=1963242 Originator: NO I think this is a sourceforge bug, see ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 06:15 Message: Logged In: YES user_id=1971526 Originator: YES Retrying... Why attaching fails? ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 06:05 Message: Logged In: YES user_id=1971526 Originator: YES Attached a patch file again. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=2070928&group_…

15 years, 8 months

[ pywikipediabot-Patches-2070928 ] category_redirect.py - update for ja locale

by SourceForge.net

Patches item #2070928, was opened at 2008-08-24 08:03 Message generated for change (Comment added) made by cosoleto You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=2070928&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nightshadow28 (nightshadow28) Assigned to: Nobody/Anonymous (nobody) Summary: category_redirect.py - update for ja locale Initial Comment: update for ja locale. I attached a diff file on sourceforge. * category_redirect.py Thanks. ---------------------------------------------------------------------- >Comment By: Francesco Cosoleto (cosoleto) Date: 2008-08-24 17:20 Message: Logged In: YES user_id=181280 Originator: NO https://jira.toolserver.org/secure/CreateIssue!default.jspa https://jira.toolserver.org/secure/AttachFile!default.jspa?id=NEWID ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 15:25 Message: Logged In: YES user_id=1971526 Originator: YES Um, the site can not attach a file for issue, can't it? (I could not find the function on create-issue page) In this item, we need to share a file written in non-English text... ---------------------------------------------------------------------- Comment By: Francesco Cosoleto (cosoleto) Date: 2008-08-24 12:56 Message: Logged In: YES user_id=181280 Originator: NO If SF doesn't work, you can use an alternative bug tracker hosted by Toolserver: https://jira.toolserver.org/browse/PYWP ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 08:50 Message: Logged In: YES user_id=1971526 Originator: YES Thanks. I try to attach a dummy file. And if it fails, I will report into the item about detail. ---------------------------------------------------------------------- Comment By: NicDumZ — Nicolas Dumazet (nicdumz) Date: 2008-08-24 08:40 Message: Logged In: YES user_id=1963242 Originator: NO I think this is a sourceforge bug, see https://sourceforge.net/tracker/index.php?func=detail&aid=2070259&group_id=… Trying to attach a test file... ---------------------------------------------------------------------- Comment By: NicDumZ — Nicolas Dumazet (nicdumz) Date: 2008-08-24 08:40 Message: Logged In: YES user_id=1963242 Originator: NO I think this is a sourceforge bug, see ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 08:15 Message: Logged In: YES user_id=1971526 Originator: YES Retrying... Why attaching fails? ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 08:05 Message: Logged In: YES user_id=1971526 Originator: YES Attached a patch file again. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=2070928&group_…

15 years, 8 months

[ pywikipediabot-Patches-2070928 ] category_redirect.py - update for ja locale

by SourceForge.net

Patches item #2070928, was opened at 2008-08-24 06:03 Message generated for change (Comment added) made by nightshadow28 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=2070928&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nightshadow28 (nightshadow28) Assigned to: Nobody/Anonymous (nobody) Summary: category_redirect.py - update for ja locale Initial Comment: update for ja locale. I attached a diff file on sourceforge. * category_redirect.py Thanks. ---------------------------------------------------------------------- >Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 13:25 Message: Logged In: YES user_id=1971526 Originator: YES Um, the site can not attach a file for issue, can't it? (I could not find the function on create-issue page) In this item, we need to share a file written in non-English text... ---------------------------------------------------------------------- Comment By: Francesco Cosoleto (cosoleto) Date: 2008-08-24 10:56 Message: Logged In: YES user_id=181280 Originator: NO If SF doesn't work, you can use an alternative bug tracker hosted by Toolserver: https://jira.toolserver.org/browse/PYWP ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 06:50 Message: Logged In: YES user_id=1971526 Originator: YES Thanks. I try to attach a dummy file. And if it fails, I will report into the item about detail. ---------------------------------------------------------------------- Comment By: NicDumZ Nicolas Dumazet (nicdumz) Date: 2008-08-24 06:40 Message: Logged In: YES user_id=1963242 Originator: NO I think this is a sourceforge bug, see https://sourceforge.net/tracker/index.php?func=detail&aid=2070259&group_id=… Trying to attach a test file... ---------------------------------------------------------------------- Comment By: NicDumZ Nicolas Dumazet (nicdumz) Date: 2008-08-24 06:40 Message: Logged In: YES user_id=1963242 Originator: NO I think this is a sourceforge bug, see ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 06:15 Message: Logged In: YES user_id=1971526 Originator: YES Retrying... Why attaching fails? ---------------------------------------------------------------------- Comment By: Nightshadow28 (nightshadow28) Date: 2008-08-24 06:05 Message: Logged In: YES user_id=1971526 Originator: YES Attached a patch file again. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=2070928&group_…

15 years, 8 months

SVN: [5843] branches/rewrite/pywikibot/site.py

by russblau＠svn.wikimedia.org

Revision: 5843 Author: russblau Date: 2008-08-24 11:59:21 +0000 (Sun, 24 Aug 2008) Log Message: ----------- missing namespace Modified Paths: -------------- branches/rewrite/pywikibot/site.py Modified: branches/rewrite/pywikibot/site.py =================================================================== --- branches/rewrite/pywikibot/site.py 2008-08-24 11:53:26 UTC (rev 5842) +++ branches/rewrite/pywikibot/site.py 2008-08-24 11:59:21 UTC (rev 5843) @@ -1483,7 +1483,7 @@ if start < end: raise Error( "usercontribs: start must be later than end with reverse=False") - ucgen = ListGenerator("usercontribs", site=self, + ucgen = api.ListGenerator("usercontribs", site=self, ucprop="ids|title|timestamp|comment|flags") if user: ucgen.request["ucuser"] = user

15 years, 8 months

Re: [Pywikipedia-l] replace.py - how to skip pages where a

by maks

Hello! It seems you have to program new bot script by youself. >Is it possible to do it this way?: > > 1. Recognize the pattern where the term is already wikilinked, and then > skip that page; and then > 2. If the wikilinked term doesn't already exist, *then* perform the > search and replace operation. > >If so, how could I do that? Do I need a different bot from replace.py? Maksim

15 years, 8 months

SVN: [5842] trunk/pywikipedia/CONTENTS

by russblau＠svn.wikimedia.org

Revision: 5842 Author: russblau Date: 2008-08-24 11:53:26 +0000 (Sun, 24 Aug 2008) Log Message: ----------- cleanup (grammar, spelling, break long lines); pywikibot software can be used on sites other than Wikipedia! Modified Paths: -------------- trunk/pywikipedia/CONTENTS Modified: trunk/pywikipedia/CONTENTS =================================================================== --- trunk/pywikipedia/CONTENTS 2008-08-24 11:08:22 UTC (rev 5841) +++ trunk/pywikipedia/CONTENTS 2008-08-24 11:53:26 UTC (rev 5842) @@ -1,5 +1,5 @@ -This is a package to build robots for MediaWiki wikis like Wikipedia. Some example robots are -included. +This is a package to build robots for MediaWiki wikis like Wikipedia. Some +example robots are included. ======================================================================= PLEASE DO NOT PLAY WITH THIS PACKAGE. These programs can actually @@ -31,32 +31,38 @@ software interface. pagegenerators.py : Generator pages. userlib.py : Library to work with users, their pages and talk pages. -BeautifulSoup.py : is a Python HTML/XML parser designed for quick turnaround - projects like screen-scraping. See more: +BeautifulSoup.py : is a Python HTML/XML parser designed for quick + turnaround projects like screen-scraping. See + more: http://www.crummy.com/software/BeautifulSoup === Utilities === basic.py : Is a template from which simple bots can be made. -checkusage.py : Provides a way for users of the Wikimedia toolserver to check the - use of images from Commons on other Wikimedia wikis. -extract_wikilinks.py : Two bots to get all linked-to Wikipedia pages from an - HTML-file. They differ in their output: extract_names - gives bare names (can be used for solve_disambiguation.py, - table2wiki.py or windows-chars.py), extract_wikilinks - gives them in interwiki-link format (can be used for +checkusage.py : Provides a way for users of the Wikimedia toolserver + to check the use of images from Commons on + other Wikimedia wikis. +extract_wikilinks.py : Two bots to get all linked-to wiki pages from an + HTML-file. They differ in their output: + extract_names gives bare names (can be used for + solve_disambiguation.py, table2wiki.py or + windows-chars.py), extract_wikilinks gives them in + interwiki-link format (can be used for interwiki.py) followlive.py : Periodically grab the list of new articles and analyze them. If the article is too short, a menu will let you easily add a template. -get.py : Script to gets a page and writes its contents to standard output. -login.py : Log in to an account on your "home" Wikipedia. +get.py : Script to get a page and write its contents to standard + output. +login.py : Log in to an account on your "home" wiki. splitwarning.py : split an interwiki.log file into warning files for each - separate language. suggestion: Zip the created files up, - put them somewhere on the internet, and send an - announcement of the location on the robot mailinglist. + separate language. suggestion: Zip the created + files up, put them somewhere on the internet, and + send an announcement of the location on the robot + mailinglist. test.py : Check whether you are logged in. -testfamily.py : Check whether you are logged in all known languages in a family. +testfamily.py : Check whether you are logged in all known languages + in a family. xmltest.py : Read an XML file (e.g. the sax_parse_bug.txt sometimes created by interwiki.py), and if it contains an error, show a stacktrace with the location of the error. @@ -65,60 +71,69 @@ detailed infortion on possiblities. sqldump.py : Extract information from local cur SQL dump files, like the ones at http://download.wikimedia.org -rcsort.py : A tool to see the recentchanges ordered by user instead of by date. +rcsort.py : A tool to see the recentchanges ordered by user instead + of by date. threadpool.py : xmlreader.py : watchlist.py : Allows access to the bot account's watchlist. -wikicomserver.py : This library allows the use of the pywikipediabot directly - from COM-aware applications. +wikicomserver.py : This library allows the use of the pywikipediabot + directly from COM-aware applications. === Robots === capitalize_redirects.py: Script to create a redirect of capitalize articles. -casechecker.py : Script to enumerate all pages in the wikipedia and find all titles - with mixed latin and cyrilic alphabets. +casechecker.py : Script to enumerate all pages in the wikipedia and + find all titles with mixed Latin and Cyrillic + alphabets. category.py : add a category link to all pages mentioned on a page, change or remove category tags +category_redirect.py : Maintain category redirects and replace links to + redirected categories. catall.py : Add or change categories on a number of pages. -catmove.pl : Need Perl programming language for takes a list of category - moves or removes to make and uses category.py. +catmove.pl : Need Perl programming language for this; takes a list + of category moves or removes to make and uses + category.py. clean_sandbox.py : This bot makes the cleaned of the page of tests. -commons_link.py : This robot include commons template to linking Commons and - your wiki project. -copyright.py : This robot check copyright text in Google, Yahoo! and Live Search. +commons_link.py : This robot include commons template to linking Commons + and your wiki project. +copyright.py : This robot check copyright text in Google, Yahoo! and + Live Search. cosmetic_changes.py : Can do slight modifications to a wiki page source code such that the code looks cleaner. delete.py : This script can be used to delete pages en masse. disambredir.py : Changing redirect names in disambiguation pages. featured.py : A robot to check feature articles. -fixes.py : This is not a bot, perform one of the predefined replacements - tasks, used for "replace.py -fix:replacement". -image.py : This script can be used to change one image to another or - remove an image entirely. -imagetransfer.py : Given a Wikipedia page, check the interwiki links - for images, and let the user choose among them for - images to upload -inline_images.py : This bot looks for images that are linked inline (i.e., they - are hosted from an external server and hotlinked). +fixes.py : This is not a bot, perform one of the predefined + replacements tasks, used for "replace.py + -fix:replacement". +image.py : This script can be used to change one image to another + or remove an image entirely. +imagetransfer.py : Given a wiki page, check the interwiki links for + images, and let the user choose among them for + images to upload. +inline_images.py : This bot looks for images that are linked inline + (i.e., they are hosted from an external server and + hotlinked). interwiki.py : A robot to check interwiki links on all pages (or - a range of pages) of a Wikipedia. + a range of pages) of a wiki. interwiki_graph.py : Possible create graph with interwiki.py. imageharvest.py : Bot for getting multiple images from an external site. -isbn.py : Bot for converts all ISBN-10 codes to the ISBN-13 format. +isbn.py : Bot to convert all ISBN-10 codes to the ISBN-13 + format. makecat.py : Given an existing or new category, find pages for that category. movepages.py : Bot page moves to another title. -nowcommons.py : This bot can be deleted images with NowCommons template. -pagefromfile.py : This bot takes its input from a file that contains a number of - pages to be put on the wiki. -piper.py : Pipes article text through external program(s) on STDIN and collects - its STDOUT which is used as the new article text of it differs from the - original. +nowcommons.py : This bot can delete images with NowCommons template. +pagefromfile.py : This bot takes its input from a file that contains a + number of pages to be put on the wiki. +piper.py : Pipes article text through external program(s) on + STDIN and collects its STDOUT which is used as the + new article text if it differs from the original. redirect.py : Fix double redirects and broken redirects. Note: solve_disambiguation also has functions which treat redirects. -refcheck.py : This script checks references to see if they are properly - formatted. +refcheck.py : This script checks references to see if they are + properly formatted. replace.py : Search articles for a text and replace it by another text. Both text are set in two configurable text files. The bot can either work on a set of given @@ -127,10 +142,11 @@ selflink.py : This bot goes over multiple pages of the home wiki, searches for selflinks, and allows removing them. solve_disambiguation.py: Interactive robot doing disambiguation. -speedy_delete.py : This bot load a list of pages from the category of candidates - for speedy deletion and give the user an interactive prompt to decide - whether each should be deleted or not. -spellcheck.py : This bot spellchecks Wikipedia pages. +speedy_delete.py : This bot load a list of pages from the category of + candidates for speedy deletion and give the + user an interactive prompt to decide whether + each should be deleted or not. +spellcheck.py : This bot spellchecks wiki pages. standardize_interwiki.py:A robot that downloads a page, and reformats the interwiki links in a standard way (i.e. move all of them to the bottom or the top, with the same @@ -146,13 +162,13 @@ unlink.py : This bot unlinks a page on every page that links to it. unusedfiles.py : Bot appends some text to all unused images and other text to the respective uploaders. -upload.py : upload an image to Wikipedia. +upload.py : upload an image to a wiki. us-states.py : A robot to add redirects to cities for US state abbreviations. warnfile.py : A robot that parses a warning file created by - interwiki.py on another Wikipedia language, - and implements the suggested changes without - verifying them. + interwiki.py on another language wiki, and + implements the suggested changes without verifying + them. weblinkchecker.py : Check if external links are still working. welcome.py : Script to welcome new users. windows_chars.py : Change characters that are not part of Latin-1 into @@ -182,19 +198,17 @@ simplejson : A simple, fast, extensible JSON encoder and decoder used by query.py. spelling : Contains dictionaries for spellcheck.py. -userinterfaces : Contains Tkinter, WxPython, terminal and transliteration - interfaces user choose in user-config.py +userinterfaces : Contains Tkinter, WxPython, terminal and + transliteration interfaces user choose in + user-config.py watchlists : Information retrieved by watchlist.py will be stored here. wiktionary : Contains script to used for Wiktionary project. - === Unit tests === wiktionarytest.py : Unit tests for wiktionary.py - - External software can be used with PyWikipediaBot: * Win32com library for use with wikicomserver.py * Pydot, Pyparsing and Graphviz for use with interwiki_graph.py @@ -209,7 +223,6 @@ * python-celementtree (recommended if you use XML dumps) * python-tkinter (optional, used by some experimental GUI stuff) - More precise information, and a list of the options that are available for the various programs, can be retrieved by running the bot with the -help parameter, e.g.

15 years, 8 months

← Newer
1
2
3
4
5
6
7
8
...
21
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

Pywikipedia-l August 2008