Pywikipedia-l August 2007

pywikipedia-l@lists.wikimedia.org

26 participants
318 discussions

[ pywikipediabot-Bugs-1771889 ] Problems with namespaces in wikipedia.py
by SourceForge.net 13 Aug '07

13 Aug '07

Bugs item #1771889, was opened at 2007-08-10 17:20 Message generated for change (Comment added) made by falk_steinhauer You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1771889&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Falk Steinhauer (falk_steinhauer) Assigned to: Nobody/Anonymous (nobody) Summary: Problems with namespaces in wikipedia.py Initial Comment: I am using snapshot 2007-06-19: In our wiki we are using title prefixes for articles that are not in german. They are Fr: (French) and En: (Englisch). One of our French articles marks the end of a subarticle of [[Special:All Pages]] (see here: http://www.wiki-aventurica.de/index.php?title=Spezial:Alle_Seiten) If I am using commandline option -start:! the script runs into a recursion. After Fr:xxxx is yielded the script whishes to continue with article xxxx, which is in my case alphabetically before Fr:xxxx. You can see, that this leads to a recursion. If xxxx is after Fr:xxxx, some articles might be skipped. I detected the reponsible line of code: wikipedia.py line 3504 # save the last hit, so that we know where to continue when we # finished all articles on the current page. Append a '!' so that # we don't yield a page twice. start = Page(self,hit).titleWithoutNamespace() + '!' Maybe this can also be fixed in titleWithoutNamespace() Is it necessary to cut off the namespace? ---------------------------------------------------------------------- >Comment By: Falk Steinhauer (falk_steinhauer) Date: 2007-08-13 22:54 Message: Logged In: YES user_id=1810075 Originator: YES I don't have these problems with the actual release. That's why I stepped back. We worked around the initial problem within our wiki. ---------------------------------------------------------------------- Comment By: Daniel Herding (wikipedian) Date: 2007-08-13 10:20 Message: Logged In: YES user_id=880694 Originator: NO The timeouts are a way to reduce database server load during peak times. See: http://www.mediawiki.org/wiki/Manual:Maxlag_parameter Maybe your server is generally a bit slow, so try to increase the maxlag parameter in your user-config.py, for example: maxlag = 10 ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2007-08-11 11:48 Message: Logged In: NO I stepped back to snapshot 2007-06-19 because of several problems with nightly build 2007-08-10 08:39:28. With this version my scripts were not able to change pages with wikipedia.Page.put(). Server timeout was reported frequently, but the server was not down. ---------------------------------------------------------------------- Comment By: Falk Steinhauer (falk_steinhauer) Date: 2007-08-10 23:34 Message: Logged In: YES user_id=1810075 Originator: YES Something is still disturbing. Our language prefixes are still cut off. But so such pages cannot be found in namespace 0. ---------------------------------------------------------------------- Comment By: Falk Steinhauer (falk_steinhauer) Date: 2007-08-10 23:26 Message: Logged In: YES user_id=1810075 Originator: YES Thanks now it works. Some strange thing is, that no redirects are yielded, but parameter includeredirects of AllpagesPageGenerator() is default set to True. ---------------------------------------------------------------------- Comment By: Merlijn S. van Deen (valhallasw) Date: 2007-08-10 18:55 Message: Logged In: YES user_id=687283 Originator: NO Strange, as these prefixes should not be interpreted as namespaces. For now, please update to SVN or the latest nightly ( http://tools.wikimedia.de/~valhallasw/pywiki/ ), and test if the issue still exists. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1771889&group_…

1 0

SVN: [4040] trunk/pywikipedia
by btongminh＠svn.wikimedia.org 13 Aug '07

13 Aug '07

Revision: 4040 Author: btongminh Date: 2007-08-13 21:41:28 +0000 (Mon, 13 Aug 2007) Log Message: ----------- Fix stuff for reporting feature. Modified Paths: -------------- trunk/pywikipedia/delinker.py trunk/pywikipedia/delinker.txt trunk/pywikipedia/image_replacer.py Modified: trunk/pywikipedia/delinker.py =================================================================== --- trunk/pywikipedia/delinker.py 2007-08-13 21:05:24 UTC (rev 4039) +++ trunk/pywikipedia/delinker.py 2007-08-13 21:41:28 UTC (rev 4040) @@ -452,6 +452,8 @@ # Pass the usage to the Delinker pool along with other arguments self.CommonsDelinker.Delinkers.append((image, usage_domains, timestamp, admin, reason, replacement)) + elif replacement: + self.CommonsDelinker.Loggers.append((timestamp, image, replacement)) def do(self, args): try: Modified: trunk/pywikipedia/delinker.txt =================================================================== --- trunk/pywikipedia/delinker.txt 2007-08-13 21:05:24 UTC (rev 4039) +++ trunk/pywikipedia/delinker.txt 2007-08-13 21:41:28 UTC (rev 4040) @@ -245,6 +245,10 @@ CommonsDelinker['clean_list'] = False CommonsDelinker['disallowed_replacements'] = [(r'\.png$', r'\.svg$')] +CommonsDelinker['replacer_report_replacements'] = False +CommonsDelinker['replacer_report_template'] = 'universally replaced' + + ## SQL connection information. # Database engine to use. Currently supported: MySQL, sqlite3. # Global delinker requires MySQL. Modified: trunk/pywikipedia/image_replacer.py =================================================================== --- trunk/pywikipedia/image_replacer.py 2007-08-13 21:05:24 UTC (rev 4039) +++ trunk/pywikipedia/image_replacer.py 2007-08-13 21:41:28 UTC (rev 4040) @@ -31,9 +31,8 @@ def site_prefix(site): if site.lang == site.family.name: return site.lang - # TODO: fix - #if (site.lang, site.family.name) == ('-', 'wikisource'): - # return 'wikisource' + if (site.lang, site.family.name) == ('-', 'wikisource'): + return 'oldwikisource' return '%s:%s' % (site.family.name, site.lang) class Replacer(object): @@ -52,7 +51,7 @@ self.first_revision = 0 if self.config.get('replacer_report_replacements', False): - self.reporters = threadpool.ThreadPool(self.reporter) + self.reporters = threadpool.ThreadPool(Reporter) self.reporters.add_thread(self.site, self.config) @@ -113,8 +112,8 @@ for timestamp, user, text in revisions[1:]: if replacement.group(0) in text and user != username: db_time = db_timestamp(timestamp) - if db_time < self.first_revision or not revision: - self.first_revision = db_time + if db_time < self.first_revision or not self.first_revision: + self.first_revision = int(db_time) return (db_time, strip_image(replacement.group(1)), strip_image(replacement.group(2)), user, replacement.group(3)) @@ -137,7 +136,7 @@ for old_image, new_image, user, comment in finished_images: self.cursor.execute("""SELECT wiki, namespace, page_title FROM %s WHERE img = %%s AND status <> 'ok'""" % - self.config['delinker_table'], (old_image, )) + self.config['log_table'], (old_image, )) not_ok = list(self.cursor) self.reporters.append((old_image, new_image, user, @@ -161,11 +160,11 @@ return True class Reporter(threadpool.Thread): - def __init__(self, site, config): + def __init__(self, pool, site, config): self.site = site self.config = config - threadpool.Thread.__init__(self) + threadpool.Thread.__init__(self, pool) def do(self, (old_image, new_image, user, comment, not_ok)): not_ok_items = [] for wiki, namespace, page_title in not_ok:

1 0

SVN: [4039] trunk/pywikipedia
by btongminh＠svn.wikimedia.org 13 Aug '07

13 Aug '07

Revision: 4039 Author: btongminh Date: 2007-08-13 21:05:24 +0000 (Mon, 13 Aug 2007) Log Message: ----------- Cosmetic fixes; Documentation; New reporting feature. Modified Paths: -------------- trunk/pywikipedia/delinker.py trunk/pywikipedia/delinker.txt trunk/pywikipedia/image_replacer.py Modified: trunk/pywikipedia/delinker.py =================================================================== --- trunk/pywikipedia/delinker.py 2007-08-13 19:50:49 UTC (rev 4038) +++ trunk/pywikipedia/delinker.py 2007-08-13 21:05:24 UTC (rev 4039) @@ -317,7 +317,7 @@ self.summaries[type] = {} if domain in self.summaries[type]: if (time.time() - self.summaries[type][domain][1]) < \ - self.CommonsDelinker.config['summary_cache']: + self.CommonsDelinker.config['summary_cache']: # Return cached result return self.summaries[type][domain][0] Modified: trunk/pywikipedia/delinker.txt =================================================================== --- trunk/pywikipedia/delinker.txt 2007-08-13 19:50:49 UTC (rev 4038) +++ trunk/pywikipedia/delinker.txt 2007-08-13 21:05:24 UTC (rev 4039) @@ -106,6 +106,13 @@ actually has edit permissions to the list. * ''disallowed_replacements = [(r'\.png$', r'\.svg$')]'': List of regular expressions of refused replacements. + +==== Reporting replacements ==== +The replacer can insert a report on replaced images. +* ''replacer_report_replacements = False'': Set to True to enable reporting. +* ''replacer_report_template = universally replaced'': The template to insert. The + template will be called with the following parameters: ''new_image, user, comment, + not_ok''. === SQL settings === * ''sql_engine = "mysql"'': Database engine to use. Currently supported: @@ -132,6 +139,7 @@ status ENUM('ok', 'skipped', 'failed'), newimg VARBINARY(255) ); + CREATE TABLE replacer ( id INT NOT NULL AUTO_INCREMENT, timestamp VARBINARY(14), @@ -146,7 +154,7 @@ ); </code> -==== Edit and debugging settings ==== +=== Edit and debugging settings === * ''save_diff = False'': Save all changes to a diff. Create a directory diff/ before running. * ''edit = True'': Actually edit to the wiki. Modified: trunk/pywikipedia/image_replacer.py =================================================================== --- trunk/pywikipedia/image_replacer.py 2007-08-13 19:50:49 UTC (rev 4038) +++ trunk/pywikipedia/image_replacer.py 2007-08-13 21:05:24 UTC (rev 4039) @@ -12,8 +12,9 @@ __version__ = '$Id$' import config, wikipedia import re, time +import threadpool -from delinker import wait_callback, output, connect_database +from delinker import wait_callback, output, connect_database, family def mw_timestamp(ts): return '%s%s%s%s-%s%s-%s%sT%s%s:%s%s:%s%sZ' % tuple(ts) @@ -27,6 +28,14 @@ img = img[0].upper() + img[1:] return img.strip() +def site_prefix(site): + if site.lang == site.family.name: + return site.lang + # TODO: fix + #if (site.lang, site.family.name) == ('-', 'wikisource'): + # return 'wikisource' + return '%s:%s' % (site.family.name, site.lang) + class Replacer(object): def __init__(self): self.config = config.CommonsDelinker @@ -41,6 +50,12 @@ self.database = connect_database() self.cursor = self.database.cursor() + self.first_revision = 0 + if self.config.get('replacer_report_replacements', False): + self.reporters = threadpool.ThreadPool(self.reporter) + self.reporters.add_thread(self.site, self.config) + + def read_replace_log(self): # FIXME: Make sqlite3 compatible insert = """INSERT INTO %s (timestamp, old_image, new_image, @@ -97,18 +112,44 @@ for timestamp, user, text in revisions[1:]: if replacement.group(0) in text and user != username: - return (db_timestamp(timestamp), - strip_image(replacement.group(1)), + db_time = db_timestamp(timestamp) + if db_time < self.first_revision or not revision: + self.first_revision = db_time + return (db_time, strip_image(replacement.group(1)), strip_image(replacement.group(2)), user, replacement.group(3)) output('Warning! Could not find out who did %s' % \ repr(replacement.group(0)), False) return + + def read_finished_replacements(self): + self.cursor.execute('START TRANSACTION WITH CONSISTENT SNAPSHOT') + self.cursor.execute("""SELECT old_image, new_image, user, comment FROM + %s WHERE status = 'done' AND timestamp >= %i""" % \ + (self.config['replacer_table'], self.first_revision)) + finished_images = list(self.cursor) + self.cursor.execute("""UPDATE %s SET status = 'reported' + WHERE status = 'done' AND timestamp >= %i""" % \ + (self.config['replacer_table'], self.first_revision)) + self.cursor.commit() + + for old_image, new_image, user, comment in finished_images: + self.cursor.execute("""SELECT wiki, namespace, page_title + FROM %s WHERE img = %%s AND status <> 'ok'""" % + self.config['delinker_table'], (old_image, )) + not_ok = list(self.cursor) + self.reporters.append((old_image, new_image, user, + comment, not_ok)) + + def start(self): while True: self.read_replace_log() + if self.config.get('replacer_report_replacements', False): + self.read_finished_replacements() + # Replacer should not loop as often as delinker time.sleep(self.config['timeout'] * 2) @@ -119,6 +160,36 @@ return False return True +class Reporter(threadpool.Thread): + def __init__(self, site, config): + self.site = site + self.config = config + + threadpool.Thread.__init__(self) + def do(self, (old_image, new_image, user, comment, not_ok)): + not_ok_items = [] + for wiki, namespace, page_title in not_ok: + site = family(wiki) + if unicode(site) == unicode(self.site): + title = u'%s:%s' % (site.namespace(namespace), page_title) + else: + title = u'%s:%s:%s' % (site_prefix(site), + site.namespace(namespace), page_title) + not_ok_items.append(title) + + page = wikipedia.Page(self.site, u'Image:' + old_image) + text = page.get() + template = u'{{%s|new_image=%s|user=%s|comment=%s|not_ok=%}}' % \ + (self.config['replacer_report_template'], + new_image, user, comment, + self.config.get('replacer_report_seperator', u', ').join(not_ok)) + page.put(u'%s\n%s' % (template, text), + comment = u'This image has been replaced by ' + new_image) + + output(u'Reporting replacement of %s by %s to %s' % \ + (old_image, new_image)) + + if __name__ == '__main__': import sys, cgitb try:

1 0

SVN: [4038] trunk/pywikipedia/interwiki.py
by wikipedian＠svn.wikimedia.org 13 Aug '07

13 Aug '07

Revision: 4038 Author: wikipedian Date: 2007-08-13 19:50:49 +0000 (Mon, 13 Aug 2007) Log Message: ----------- docu Modified Paths: -------------- trunk/pywikipedia/interwiki.py Modified: trunk/pywikipedia/interwiki.py =================================================================== --- trunk/pywikipedia/interwiki.py 2007-08-13 19:47:50 UTC (rev 4037) +++ trunk/pywikipedia/interwiki.py 2007-08-13 19:50:49 UTC (rev 4038) @@ -1076,6 +1076,7 @@ reporting of missing backlinks for pages we already fixed """ + # use sets because searching an element is faster than in lists expectedPages = set(new.values()) expectedSites = set([page.site() for page in expectedPages]) try: @@ -1086,6 +1087,8 @@ except wikipedia.NoPage: wikipedia.output(u"WARNING: Page %s does no longer exist?!" % page.title()) break + # To speed things up, create a dictionary which maps sites to pages. + # This assumes that there is only one interwiki link per language. linkedPagesDict = {} for linkedPage in linkedPages: linkedPagesDict[linkedPage.site()] = linkedPage

1 0

SVN: [4037] trunk/pywikipedia/interwiki.py
by wikipedian＠svn.wikimedia.org 13 Aug '07

13 Aug '07

Revision: 4037 Author: wikipedian Date: 2007-08-13 19:47:50 +0000 (Mon, 13 Aug 2007) Log Message: ----------- Sped up backlinks report generation. By making use of dictionaries and sets, decreased complexity from O(n^3) to O(n^2). For example, the backlinks report for python interwiki.py -lang:de Indien -localonly is now generated in 26 seconds, instead of the 190 seconds that were needed before. Modified Paths: -------------- trunk/pywikipedia/interwiki.py Modified: trunk/pywikipedia/interwiki.py =================================================================== --- trunk/pywikipedia/interwiki.py 2007-08-13 19:41:35 UTC (rev 4036) +++ trunk/pywikipedia/interwiki.py 2007-08-13 19:47:50 UTC (rev 4037) @@ -1076,34 +1076,33 @@ reporting of missing backlinks for pages we already fixed """ + expectedPages = set(new.values()) + expectedSites = set([page.site() for page in expectedPages]) try: for site, page in new.iteritems(): if site not in updatedSites and not page.section(): - shouldlink = new.values() try: - linked = page.interwiki() + linkedPages = set(page.interwiki()) except wikipedia.NoPage: wikipedia.output(u"WARNING: Page %s does no longer exist?!" % page.title()) break - for xpage in shouldlink: - if xpage != page and not xpage in linked: - for l in linked: - if l.site() == xpage.site(): - wikipedia.output(u"WARNING: %s: %s does not link to %s but to %s" % (page.site().family.name, page.aslink(True), xpage.aslink(True), l.aslink(True))) - break - else: - wikipedia.output(u"WARNING: %s: %s does not link to %s" % (page.site().family.name, page.aslink(True), xpage.aslink(True))) + linkedPagesDict = {} + for linkedPage in linkedPages: + linkedPagesDict[linkedPage.site()] = linkedPage + for expectedPage in expectedPages: + if expectedPage != page and expectedPage not in linkedPages: + try: + linkedPage = linkedPagesDict[expectedPage.site()] + wikipedia.output(u"WARNING: %s: %s does not link to %s but to %s" % (page.site().family.name, page.aslink(True), expectedPage.aslink(True), linkedPage.aslink(True))) + except KeyError: + wikipedia.output(u"WARNING: %s: %s does not link to %s" % (page.site().family.name, page.aslink(True), expectedPage.aslink(True))) # Check for superfluous links - for xpage in linked: - if not xpage in shouldlink: + for linkedPage in linkedPages: + if linkedPage not in expectedPages: # Check whether there is an alternative page on that language. - for l in shouldlink: - if l.site() == xpage.site(): - # Already reported above. - break - else: - # New warning - wikipedia.output(u"WARNING: %s: %s links to incorrect %s" % (page.site().family.name, page.aslink(True), xpage.aslink(True))) + # In this case, it was already reported above. + if linkedPage.site() not in expectedSites: + wikipedia.output(u"WARNING: %s: %s links to incorrect %s" % (page.site().family.name, page.aslink(True), linkedPage.aslink(True))) except (socket.error, IOError): wikipedia.output(u'ERROR: could not report backlinks')

1 0

SVN: [4036] trunk/pywikipedia
by btongminh＠svn.wikimedia.org 13 Aug '07

13 Aug '07

Revision: 4036 Author: btongminh Date: 2007-08-13 19:41:35 +0000 (Mon, 13 Aug 2007) Log Message: ----------- Norwegian translations (Jhs). Only checked for syntax errors, but should not break anything. Modified Paths: -------------- trunk/pywikipedia/clean_sandbox.py trunk/pywikipedia/featured.py trunk/pywikipedia/interwiki.py trunk/pywikipedia/movepages.py trunk/pywikipedia/pagefromfile.py trunk/pywikipedia/redirect.py trunk/pywikipedia/replace.py trunk/pywikipedia/solve_disambiguation.py trunk/pywikipedia/standardize_interwiki.py trunk/pywikipedia/table2wiki.py trunk/pywikipedia/template.py trunk/pywikipedia/weblinkchecker.py Modified: trunk/pywikipedia/clean_sandbox.py =================================================================== --- trunk/pywikipedia/clean_sandbox.py 2007-08-13 19:10:17 UTC (rev 4035) +++ trunk/pywikipedia/clean_sandbox.py 2007-08-13 19:41:35 UTC (rev 4036) @@ -29,6 +29,7 @@ 'de': u'{{Bitte erst NACH dieser Zeile schreiben! (Begrüßungskasten)}}\r\n', 'en': u'{{Please leave this line alone (sandbox heading)}}\n ', 'nl': u'{{subst:Wikipedia:Zandbak/schoon zand}}', + 'no': u'{{Sandkasse}}\n}}', 'pl': u'{{Prosimy - NIE ZMIENIAJ, NIE KASUJ, NIE PRZENOŚ tej linijki - pisz niżej}}', 'pt': u'{{página de testes}}\r\n', 'commons': u'{{Sandbox}}\n' @@ -39,6 +40,7 @@ 'de': u'Bot: Setze Seite zurück.', 'en': u'Robot: This page will automatically be cleaned.', 'nl': u'Robot: Automatisch voorzien van schoon zand.', + 'no': u'bot: Rydder sandkassa.', 'pl': u'Robot czyści brudnopis', 'pt': u'Bot: Limpeza da página de testes', } @@ -48,6 +50,7 @@ 'de': u'Wikipedia:Spielwiese', 'en': u'Wikipedia:Sandbox', 'nl': u'Wikipedia:Zandbak', + 'no': u'Wikipedia:Sandkasse', 'pl': u'Wikipedia:Brudnopis', 'pt': u'Wikipedia:Página de testes', 'commons': u'Commons:Sandbox' Modified: trunk/pywikipedia/featured.py =================================================================== --- trunk/pywikipedia/featured.py 2007-08-13 19:10:17 UTC (rev 4035) +++ trunk/pywikipedia/featured.py 2007-08-13 19:41:35 UTC (rev 4036) @@ -49,6 +49,7 @@ 'ka': u'რჩეული სტატიის ბმული გვერდისათვის [[%s:%s]]', 'lt': u'Pavyzdinis straipsnis [[%s:%s]]', 'nl': u'Etalage-artikel link voor [[%s:%s]]', + 'no': u'bot: [[%s:%s]] er en utmerka artikkel', 'pl': u'Link do artykułu wyróżnionego [[%s:%s]]', 'pt': u'Ligando artigos destacados para [[%s:%s]]', 'sr': u'Међувики за изабране чланке за [[%s:%s]]', Modified: trunk/pywikipedia/interwiki.py =================================================================== --- trunk/pywikipedia/interwiki.py 2007-08-13 19:10:17 UTC (rev 4035) +++ trunk/pywikipedia/interwiki.py 2007-08-13 19:41:35 UTC (rev 4036) @@ -292,7 +292,7 @@ 'nds-nl': (u'bot', u'derbie', u'derof', u'aanders'), 'nl': (u'robot ', u'Erbij', u'Eraf', u'Anders'), 'nn': (u'robot ', u'la til', u'fjerna', u'endra'), - 'no': (u'robot ', u'Tilføyer', u'Fjerner', u'Endrer'), + 'no': (u'robot ', u'legger til', u'fjerner', u'endrer'), 'os': (u'Робот', u'баххæст кодта', u'Баивта', u'Аиуварс'), 'pl': (u'robot ', u'dodaje', u'usuwa', u'poprawia'), 'pms': (u'ël trigomiro ', u'a gionta', u'a modìfica', u'a gava'), Modified: trunk/pywikipedia/movepages.py =================================================================== --- trunk/pywikipedia/movepages.py 2007-08-13 19:10:17 UTC (rev 4035) +++ trunk/pywikipedia/movepages.py 2007-08-13 19:41:35 UTC (rev 4036) @@ -55,6 +55,7 @@ 'fr': u'Page renommée par bot', 'he': u'העברת דף באמצעות בוט', 'nl': u'Paginatitel gewijzigd door robot', + 'no': u'bot: Flytter side', 'pl': u'Przeniesienie artykułu przez robota', 'pt': u'Página movida por bot', } @@ -66,6 +67,7 @@ 'el': u'Διαγραφή σελίδων με bot', 'fr': u'Page supprimée par bot', 'nl': u'Pagina verwijderd door robot', + 'no': u'bot: Sletter side', 'pl': u'Usunięcie artykułu przez robota', 'pt': u'Página apagada por bot', } Modified: trunk/pywikipedia/pagefromfile.py =================================================================== --- trunk/pywikipedia/pagefromfile.py 2007-08-13 19:10:17 UTC (rev 4035) +++ trunk/pywikipedia/pagefromfile.py 2007-08-13 19:41:35 UTC (rev 4036) @@ -62,6 +62,7 @@ 'it': u'Caricamento automatico', 'ksh': u'Automatesch aanjelaat', 'nl': u'Geautomatiseerde import', + 'no': u'bot: Automatisk import', 'pl': u'Automatyczny import artykułów', 'pt': u'Importação automática de artigos' } @@ -76,6 +77,7 @@ 'it': u'aggiungo in cima', 'ksh': u'Automatesch füürjesaz', 'nl': u'bovenaan toegevoegd', + 'no': u'legger til øverst', 'pl': u'dodaj na górze', 'pt': u'adicionado no topo' } @@ -89,6 +91,7 @@ 'it': u'aggiungo in fondo', 'ksh': u'Automatesch aanjehange', 'nl': u'onderaan toegevoegd', + 'no': u'legger til nederst', 'pl': u'dodaj na dole', 'pt': u'adicionando no fim' } @@ -102,6 +105,7 @@ 'it': u'sovrascritto il testo esistente', 'ksh': u'Automatesch ußjetuusch', 'nl': u'bestaande tekst overschreven', + 'no': u'erstatter eksisterende tekst', 'pl': u'aktualny tekst nadpisany', 'pt': u'sobrescrever texto' } Modified: trunk/pywikipedia/redirect.py =================================================================== --- trunk/pywikipedia/redirect.py 2007-08-13 19:10:17 UTC (rev 4035) +++ trunk/pywikipedia/redirect.py 2007-08-13 19:41:35 UTC (rev 4036) @@ -1,4 +1,4 @@ -# -*- coding: utf-8 -*- +# -*- coding: utf-8 -*- """ Script to resolve double redirects, and to delete broken redirects. Requires access to MediaWiki's maintenance pages or to a XML dump file. Delete function requires @@ -54,6 +54,7 @@ 'ksh':u'Bot: Dubbel Ömlëijdong fottjemaat', 'lt':u'robotas: Taisomas dvigubas peradresavimas', 'nl':u'Robot: Dubbele doorverwijzing gecorrigeerd', + 'no':u'bot: Retter dobbel omdirigering', 'pl':u'Robot naprawia podwójne przekierowanie', 'pt':u'Bot: Corrigido duplo redirecionamento', 'ru':u'Робот: исправление двойного перенаправления', Modified: trunk/pywikipedia/replace.py =================================================================== --- trunk/pywikipedia/replace.py 2007-08-13 19:10:17 UTC (rev 4035) +++ trunk/pywikipedia/replace.py 2007-08-13 19:41:35 UTC (rev 4036) @@ -118,6 +118,7 @@ 'lt':u'robotas: Automatinis teksto keitimas %s', 'nds':u'Bot: Text automaatsch utwesselt: %s', 'nl':u'Bot: automatisch tekst vervangen %s', + 'no':u'bot: Autometisk teksterstatning: %s', 'pl':u'Robot automatycznie zamienia tekst %s', 'pt':u'Bot: Mudança automática %s', 'sr':u'Бот: Аутоматска замена текста %s', Modified: trunk/pywikipedia/solve_disambiguation.py =================================================================== --- trunk/pywikipedia/solve_disambiguation.py 2007-08-13 19:10:17 UTC (rev 4035) +++ trunk/pywikipedia/solve_disambiguation.py 2007-08-13 19:41:35 UTC (rev 4036) @@ -89,6 +89,7 @@ 'it': u'Sistemazione automatica della disambigua: %s', 'lt': u'Nuorodų į nukrepiamąjį straipsnį keitimas: %s', 'nl': u'Robot-geholpen doorverwijzing: %s', + 'no': u'bot: Retter lenke til peker: %s', 'pl': u'Wspomagane przez robota ujednoznacznienie: %s', 'pt': u'Desambiguação assistida por bot: %s', 'ru': u'Разрешение значений с помощью бота: %s', @@ -108,6 +109,7 @@ 'it': u'Sistemazione automatica del redirect: %s', 'lt': u'Nuorodų į peradresavimo straipsnį keitimas: %s', 'nl': u'Robot-geholpen redirect-oplossing: %s', + 'no': u'bot: Endrer omdirigeringslenke: %s', 'pl': u'Wspomagane przez robota ujednoznacznienie: %s', 'pt': u'Desambiguação assistida por bot: %s', 'ru': u'Разрешение значений с помощью бота: %s', @@ -125,6 +127,7 @@ 'it': u'%s_(disambigua)', 'lt': u'%s_(reikšmės)', 'nl': u'%s_(doorverwijspagina)', + 'no': u'%s_(peker)', 'pl': u'%s_(ujednoznacznienie)', 'pt': u'%s_(desambiguação)', 'he': u'%s_(פירושונים)', Modified: trunk/pywikipedia/standardize_interwiki.py =================================================================== --- trunk/pywikipedia/standardize_interwiki.py 2007-08-13 19:10:17 UTC (rev 4035) +++ trunk/pywikipedia/standardize_interwiki.py 2007-08-13 19:41:35 UTC (rev 4036) @@ -23,6 +23,7 @@ # The summary that the Bot will use. comment = { 'en':u'Robot: interwiki standardization', + 'no':u'bot: Språklenkestandardisering', 'it':u'Bot: Standardizzo interwiki', } site = wikipedia.getSite() Modified: trunk/pywikipedia/table2wiki.py =================================================================== --- trunk/pywikipedia/table2wiki.py 2007-08-13 19:10:17 UTC (rev 4035) +++ trunk/pywikipedia/table2wiki.py 2007-08-13 19:41:35 UTC (rev 4036) @@ -60,6 +60,7 @@ 'ia':u'Robot controlate: Syntaxe del tabella cambiate de HTML a Wiki', 'lt':u'kontroliuojamas robotas: atnaujinta lentelės sintaksė', 'nl':u'Tabel gewijzigd van HTML- naar Wikisyntax', + 'no':u'bot: Konverter tabellsyntaks', 'pl':u'Kontrolowany przez użytkownika robot poprawia składnię tabeli', 'pt':u'Bot: Sintaxe da tabela HTML para Wiki atualizada', } @@ -72,6 +73,7 @@ 'ia':u'Robot controlate: Syntaxe del tabella cambiate - %d advertimento!', 'lt':u'kontroliuojamas robotas: atnaujinta lentelės sintaksė - %d įspėjimas!', 'nl':u'Tabel gewijzigd van HTML- naar Wikisyntax - %d waarschuwing!', + 'no':u'bot: Konverterer tabellsyntaks – %d advarsel!', 'pl':u'Kontrolowany przez użytkownika robot poprawia składnię tabeli - %d ostrzeżenie!', 'pt':u'Bot: Sintaxe da tabela HTML para Wiki atualizada - %d aviso', } @@ -84,6 +86,7 @@ 'ia':u'Robot controlate: Syntaxe del tabella cambiate - %d advertimentos!', 'lt':u'kontroliuojamas robotas: atnaujinta lentelės sintaksė - %d įspėjimai!', 'nl':u'Tabel gewijzigd van HTML- naar Wikisyntax - %d waarschuwingen!', + 'no':u'bot: Konverterer tabellsyntaks – %d advarsler!', 'pl':u'Kontrolowany przez użytkownika robot poprawia składnię tabeli - %d ostrzeżeń!', 'pt':u'Bot: Sintaxe da tabela HTML para Wiki atualizada - %d avisos', } Modified: trunk/pywikipedia/template.py =================================================================== --- trunk/pywikipedia/template.py 2007-08-13 19:10:17 UTC (rev 4035) +++ trunk/pywikipedia/template.py 2007-08-13 19:41:35 UTC (rev 4036) @@ -155,6 +155,7 @@ 'hu':u'Robot: Sablon csere: %s', 'lt':u'robotas: Keičiamas šablonas: %s', 'nds':u'Bot: Vörlaag utwesselt: %s', + 'no':u'bot: Endrer mal: %s', 'pl':u'Robot zmienia szablon: %s', 'pt':u'Bot: Alterando predefinição: %s', 'sr':u'Бот: Измена шаблона: %s', @@ -167,6 +168,7 @@ 'fr':u'Bot: Modifie modèles %s', 'lt':u'robotas: Keičiami šablonai: %s', 'nds':u'Bot: Vörlagen utwesselt: %s', + 'no':u'bot: Endrer maler: %s', 'pl':u'Robot zmienia szablony: %s', 'pt':u'Bot: Alterando predefinição: %s', } @@ -181,6 +183,7 @@ 'hu':u'Robot: Sablon eltávolítása: %s', 'lt':u'robotas: Šalinamas šablonas: %s', 'nds':u'Bot: Vörlaag rut: %s', + 'no':u'bot: Fjerner mal: %s', 'pl':u'Robot usuwa szablon: %s', 'pt':u'Bot: Removendo predefinição: %s', 'sr':u'Бот: Уклањање шаблона: %s', @@ -193,6 +196,7 @@ 'fr':u'Bot: Enlève modèles : %s', 'lt':u'robotas: Šalinami šablonai: %s', 'nds':u'Bot: Vörlagen rut: %s', + 'no':u'bot: Fjerner maler: %s', 'pl':u'Robot usuwa szablony: %s', 'pt':u'Bot: Removendo predefinição: %s', } @@ -204,6 +208,7 @@ 'fr':u'Bot: Remplace modèle : %s', 'he':u'רובוט: מכליל תבנית בקוד הדף: %s', 'nds':u'Bot: Vörlaag in Text övernahmen: %s', + 'no':u'bot: Erstatter mal: %s', 'pl':u'Robot podmienia szablon: %s', 'pt':u'Bot: Substituindo predefinição: %s', } @@ -214,6 +219,7 @@ 'fr':u'Bot: Remplace modèles : %s', 'he':u'רובוט: מכליל תבניות בקוד הדף: %s', 'nds':u'Bot: Vörlagen in Text övernahmen: %s', + 'no':u'bot: Erstatter maler: %s', 'pl':u'Robot podmienia szablony: %s', 'pt':u'Bot: Substituindo predefinição: %s', } Modified: trunk/pywikipedia/weblinkchecker.py =================================================================== --- trunk/pywikipedia/weblinkchecker.py 2007-08-13 19:10:17 UTC (rev 4035) +++ trunk/pywikipedia/weblinkchecker.py 2007-08-13 19:41:35 UTC (rev 4036) @@ -75,6 +75,7 @@ 'ia': u'Robot: Reporto de un ligamine externe non functionante', 'nds': u'Lenk-Bot: Weblenk geiht nich mehr', 'nl': u'Robot: Melding (tijdelijk) onbereikbare externe link', + 'no': u'bot: Rapporter død eksternlenke', 'pl': u'Robot zgłasza niedostępny link zewnętrzny', 'pt': u'Bot: Link externo não funcionando', 'sr': u'Бот: Пријављивање непостојећих спољашњих повезница', @@ -91,6 +92,7 @@ 'ia': u'== Ligamine defuncte ==\n\nDurante plure sessiones automatic, le robot ha constatate que le sequente ligamine externe non es disponibile. Per favor confirma que le ligamine de facto es defuncte, e in caso de si, repara o elimina lo!\n\n%s\n%s--~~~~', 'nds': u'== Weblenk geiht nich mehr ==\n\nDe Bot hett en poor Mal al versöcht, disse Siet optoropen un kunn dor nich bikamen. Schall man een nakieken, wat de Siet noch dor is un den Lenk richten oder rutnehmen.\n\n%s\n%s--~~~~', 'nl': u'== Dode link ==\nTijdens enkele automatische controles bleek de onderstaande externe link onbereikbaar. Controleer alstublieft of de link inderdaad onbereikbaar is. Verwijder deze tekst alstublieft na een succesvolle controle of na het verwijderen of corrigeren van de externe link.\n\n%s\n%s--~~~~[[Categorie:Wikipedia:Onbereikbare externe link]]', + 'no': u'{{subst:Bruker:JhsBot/Død lenke}}\n\n%s\n%s~~~~\n\n{{ødelagt lenke}}', 'pl': u'== Martwy link ==\n\nW czasie kilku automatycznych przebiegów bota, poniższy link zewnętrzny był niedostępny. Proszę sprawdzić czy odnośnik jest faktycznie niedziałający i ewentualnie go usunąć.\n\n%s\n%s--~~~~', 'pt': u'== Link quebrado ==\n\nFoi checado os links externos deste artigo por vários minutos. Alguém verifique por favor se a ligação estiver fora do ar e tente arrumá-lo ou removê-la!\n\n%s\n --~~~~ ', 'sr': u'== Покварене спољашње повезнице ==\n\nТоком неколико аутоматски провера, бот је пронашао покварене спољашње повезнице. Молимо вас проверите да ли је повезница добра, поправите је или је уклоните!\n\n%s\n%s--~~~~', @@ -100,6 +102,7 @@ 'de': u'Die Webseite wurde vom Internet Archive gespeichert. Bitte verlinke gegebenenfalls eine geeignete archivierte Version: [%s]. ', 'en': u'\nThe web page has been saved by the Internet Archive. Please consider linking to an appropriate archived version: [%s]. ', 'nl': u'\nDeze website is bewaard in het Internet Archief. Overweeg te linken naar een gearchiveerde pagina: [%s]. ', + 'no': u'\nDenne nettsiden er lagra i Internet Archive. Vurder om lenka kan endres til å peke til en av de arkiverte versjonene: [%s]. ', 'pt': u'Esta página web foi gravada na Internet Archive. Por favor considere o link para a versão arquivada: [%s]. ', } @@ -109,6 +112,7 @@ re.compile('.*[\./(a)]example.org(/.*)?'), # reserved for documentation re.compile('.*[\./(a)]gso.gbv.de(/.*)?'), # bot somehow can't handle their redirects re.compile('.*[\./(a)]berlinonline.de(/.*)?'), # a de: user wants to fix them by hand and doesn't want them to be deleted, see [[de:Benutzer:BLueFiSH.as/BZ]]. + re.compile('.*[\./(a)]bodo.kommune.no(/.*)?'), # bot can't handle their redirects ] def weblinksIn(text, withoutBracketed = False, onlyBracketed = False):

1 0

SVN: [4035] trunk/pywikipedia/delinker.txt
by btongminh＠svn.wikimedia.org 13 Aug '07

13 Aug '07

Revision: 4035 Author: btongminh Date: 2007-08-13 19:10:17 +0000 (Mon, 13 Aug 2007) Log Message: ----------- SQL table layout. Modified Paths: -------------- trunk/pywikipedia/delinker.txt Modified: trunk/pywikipedia/delinker.txt =================================================================== --- trunk/pywikipedia/delinker.txt 2007-08-13 15:05:15 UTC (rev 4034) +++ trunk/pywikipedia/delinker.txt 2007-08-13 19:10:17 UTC (rev 4035) @@ -64,7 +64,7 @@ First setup the dictionary ''CommonsDelinker'', by adding to the config: CommonsDelinker = {} -==== General settings ==== +=== General settings === * ''timeout = 60'': A general timeout, used for fetching the log and other timeouts. Set to 60 for medium sized wikis, such as English Wikipedia, and 60-120 for smaller wikis such as German Wikipedia. Note that during @@ -88,7 +88,7 @@ GLOBALLY WITHOUT CONSULTING BRYAN AND SIEBRAND. Thank you. * ''no_sysop = True'': Disable delinking as sysop. -==== Delinker settings ==== +=== Delinker settings === Those variables only need to be set if the delinker is enabled. * ''delink_wait = 600'': The time to wait after deletion before the image is delinked. @@ -96,7 +96,7 @@ summary, the file is not delinked. * ''summary_cache = 3600'': Time before on-wiki settings are updated. -==== Replacer settings ==== +=== Replacer settings === Those variables only need to be set if the replacer is enabled. * ''replace_template = "replace image"'': The template for to command replacement. @@ -107,7 +107,7 @@ * ''disallowed_replacements = [(r'\.png$', r'\.svg$')]'': List of regular expressions of refused replacements. -==== SQL settings ==== +=== SQL settings === * ''sql_engine = "mysql"'': Database engine to use. Currently supported: MySQL. Support for sqlite3 is planned. The Global delinker requires MySQL. * ''sql_config = {\ @@ -121,6 +121,31 @@ * ''replacer_table = "database.replacer"'': The database.table for the replacer. Only required if the replacer is activated. +==== SQL table layout ==== +<code lang="sql"> +CREATE TABLE delinker ( + timestamp CHAR(14), + img VARBINARY(255), + wiki VARBINARY(255), + page_title VARBINARY(255), + namespace INT, + status ENUM('ok', 'skipped', 'failed'), + newimg VARBINARY(255) +); +CREATE TABLE replacer ( + id INT NOT NULL AUTO_INCREMENT, + timestamp VARBINARY(14), + old_image VARBINARY(255), + new_image VARBINARY(255), + status ENUM('pending', 'ok', 'refused', 'done'), + user VARBINARY(255), + comment VARBINARY(255), + + PRIMARY KEY(id), + INDEX(status) +); +</code> + ==== Edit and debugging settings ==== * ''save_diff = False'': Save all changes to a diff. Create a directory diff/ before running.

1 0

-start:!
by Francesco Cosoleto 13 Aug '07

13 Aug '07

Hi, What do you think about add an invite to use dump file / -xml when a pywiki user does something like: > python replace.py -start:! a b (of course, 'a' is surely a very rare text always) or: > python replace.py -start:Image:! c d helpful? Regards, Francesco Cosoleto

1 0

SVN: [4034] trunk/pywikipedia/welcome.py
by wikipedian＠svn.wikimedia.org 13 Aug '07

13 Aug '07

Revision: 4034 Author: wikipedian Date: 2007-08-13 15:05:15 +0000 (Mon, 13 Aug 2007) Log Message: ----------- Applied changes by Filnik (new -savedata parameter). It still runs, but I haven't tested if it still runs properly; if it doesn't, blame Filnik. Especially, there was broken indentation near the end of the file (date formatting code), there were tabs instead of spaces. I hope I got it right. Modified Paths: -------------- trunk/pywikipedia/welcome.py Modified: trunk/pywikipedia/welcome.py =================================================================== --- trunk/pywikipedia/welcome.py 2007-08-13 10:19:19 UTC (rev 4033) +++ trunk/pywikipedia/welcome.py 2007-08-13 15:05:15 UTC (rev 4034) @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/python # -*- coding: utf-8 -*- """ Script to welcome new users. This script works out of the box for Wikis that @@ -72,6 +72,9 @@ -random Use a random signature, taking the signatures from a wiki page (for istruction, see below). + -savedata This feature saves the random signature index to allow to + continue to welcome with the last signature used. + ********************************* GUIDE *********************************** Report, Bad and white list guide: @@ -152,23 +155,25 @@ __version__ = '$Id: welcome.py,v 1.4 2007/04/14 18:05:42 siebrand Exp$' # -import wikipedia, string -import time, re, config -import urllib -import locale +import wikipedia, config, string, locale +import time, re, cPickle, os, urllib + locale.setlocale(locale.LC_ALL,'') -number = 1 # number of edits that an user required to be welcomed -numberlog = 15 # number of users that are required to add the log :) -limit = 50 # number of users that the bot load to check -offset_variable = 0 # number of newest users to skip each run -recursive = True # define if the Bot is recursive or not -time_variable = 3600 # how much time (sec.) the bot sleeps before restart -log_variable = True # create the welcome log or not -ask = False # should bot ask to add username to bad-username list -filter_wp = False # check if the username is ok or not -sign = ' ~~~~' # default signature -random = False # should signature be random or not +number = 1 # number of edits that an user required to be welcomed +numberlog = 15 # number of users that are required to add the log :) +limit = 50 # number of users that the bot load to check +offset_variable = 0 # number of newest users to skip each run +recursive = True # define if the Bot is recursive or not +time_variable = 3600 # how much time (sec.) the bot sleeps before restart +log_variable = True # create the welcome log or not +ask = False # should bot ask to add username to bad-username list +filter_wp = False # check if the username is ok or not +sign = ' --~~~~' # default signature +random = False # should signature be random or not +savedata = False # should save the signature index or not +filename = 'welcome.data' # file where is stored the random signature index +directory = str(os.getcwd()) # Script users the class wikipedia.translate() to find the right # page/user/summary/etc so the need to specify language and project have @@ -510,6 +515,8 @@ ask = True elif arg == '-filter': filter_wp = True + elif arg == '-savedata': + savedata = True elif arg == '-random': random = True elif arg.startswith('-limit'): @@ -559,13 +566,17 @@ welcomer = u'{{subst:Utente:Filnik/Benve|nome={{subst:PAGENAME}}}} %s' welcomed_users = list() - number_user = 0 + if savedata == True and os.path.exists(directory + '/' + filename): + f = file(filename) + number_user = cPickle.load(f) + else: + number_user = 0 # Use try and finally, to put the wikipedia.stopme() always at the end of the code. try: # Here there is the main loop. while True: if filter_wp == True: - # A standard list of bad username components (you can change/delate it in your project...) [ i divide the list into two to make it smaller...] + # A standard list of bad username components (you can change/delate it in your project...) [ i divide the list into three to make it smaller...] elencoaf = [' ano', ' anus', 'anal ', 'babies', 'baldracca', 'balle', 'bastardo', 'bestiali', 'bestiale', 'bastarda', 'b.i.t.c.h.', 'bitch', 'boobie', 'bordello', 'breast', 'cacata', 'cacca', 'cachapera', 'cagata', @@ -642,7 +653,7 @@ if random == True: try: wikipedia.output(u'Loading random signatures...') - signList = defineSign(wsite,signPageTitle) + signList = defineSign(wsite, signPageTitle) except wikipedia.NoPage: wikipedia.output(u'The list with signatures is not available... Using default signature...') random = False @@ -766,10 +777,10 @@ # If recursive, don't exit, repeat after one hour. if recursive == True: waitstr = unicode(time_variable) - if locale.getlocale()[1]: - strfstr = unicode(time.strftime(u"%d %b %Y %H:%M:%S (UTC)", time.gmtime()), locale.getlocale()[1]) - else: - strfstr = unicode(time.strftime(u"%d %b %Y %H:%M:%S (UTC)", time.gmtime())) + if locale.getlocale()[1]: + strfstr = unicode(time.strftime(u"%d %b %Y %H:%M:%S (UTC)", time.gmtime()), locale.getlocale()[1]) + else: + strfstr = unicode(time.strftime(u"%d %b %Y %H:%M:%S (UTC)", time.gmtime())) wikipedia.output(u'Sleeping %s seconds before rerun. %s' % (waitstr, strfstr)) time.sleep(time_variable) # If not recursive, break. @@ -777,4 +788,9 @@ wikipedia.output(u'Stop!') break finally: - wikipedia.stopme() + if random == True: + if savedata == True: + f = file(filename, 'w') + cPickle.dump(number_user, f) + f.close() + wikipedia.stopme() \ No newline at end of file

1 0

Fix of translation weblinkchecker
by Ward De Ridder 13 Aug '07

13 Aug '07

Is there anyone who can publish the file in the attachment? I don't know why but I am not able to upload to SVN (but I can download the newer versions). Tips are welcome. Summary: Fix wrong Dutch translation Thanks Ward

1 0

← Newer
1
...
16
17
18
19
20
21
22
...
32
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

Pywikipedia-l August 2007