jenkins-bot has submitted this change and it was merged.
Change subject: weblinkchecker.py: add the ability to run the script with XML dump
......................................................................
weblinkchecker.py: add the ability to run the script with XML dump
Like reflinks.py
Bug: 55039
Change-Id: I7ba4f460897316ae1f5cbcca0080f8c3262d9abf
---
M weblinkchecker.py
1 file changed, 66 insertions(+), 0 deletions(-)
Approvals:
Ladsgroup: Looks good to me, approved
jenkins-bot: Verified
diff --git a/weblinkchecker.py b/weblinkchecker.py
index 71a0c1b..eafd7ed 100644
--- a/weblinkchecker.py
+++ b/weblinkchecker.py
@@ -36,6 +36,11 @@
-namespace Only process templates in the namespace with the given number or
name. This parameter may be used multiple times.
+-xml Should be used instead of a simple page fetching method from
+ pagegenerators.py for performance and load issues
+
+-xmlstart Page to start with when using an XML dump
+
-ignore HTTP return codes to ignore. Can be provided several times :
-ignore:401 -ignore:500
@@ -109,6 +114,7 @@
from pywikibot import i18n
import config
import pagegenerators
+import xmlreader
import pywikibot.weblib
docuReplacements = {
@@ -173,6 +179,47 @@
yield m.group('url')
else:
yield m.group('urlb')
+
+
+class XmlDumpPageGenerator:
+ """Xml generator that yiels pages containing a web link"""
+
+ def __init__(self, xmlFilename, xmlStart, namespaces):
+ self.xmlStart = xmlStart
+ self.namespaces = namespaces
+ self.skipping = bool(xmlStart)
+ self.site = pywikibot.getSite()
+
+ dump = xmlreader.XmlDump(xmlFilename)
+ self.parser = dump.parse()
+
+ def __iter__(self):
+ return self
+
+ def next(self):
+ try:
+ for entry in self.parser:
+ if self.skipping:
+ if entry.title != self.xmlStart:
+ continue
+ self.skipping = False
+ page = pywikibot.Page(self.site, entry.title)
+ if not self.namespaces == []:
+ if page.namespace() not in self.namespaces:
+ continue
+ found = False
+ for url in weblinksIn(entry.text):
+ found = True
+ if found:
+ return page
+ except KeyboardInterrupt:
+ try:
+ if not self.skipping:
+ pywikibot.output(
+ u'To resume, use "-xmlstart:%s" on the command line.'
+ % entry.title)
+ except NameError:
+ pass
class LinkChecker(object):
@@ -754,6 +801,7 @@
def main():
gen = None
singlePageTitle = []
+ xmlFilename = None
# Which namespaces should be processed?
# default to [] which means all namespaces will be processed
namespaces = []
@@ -780,6 +828,17 @@
HTTPignore.append(int(arg[8:]))
elif arg.startswith('-day:'):
day = int(arg[5:])
+ elif arg.startswith('-xmlstart'):
+ if len(arg) == 9:
+ xmlStart = pywikibot.input(
+ u'Please enter the dumped article to start with:')
+ else:
+ xmlStart = arg[10:]
+ elif arg.startswith('-xml'):
+ if len(arg) == 4:
+ xmlFilename = i18n.input('pywikibot-enter-xml-filename')
+ else:
+ xmlFilename = arg[5:]
else:
if not genFactory.handleArg(arg):
singlePageTitle.append(arg)
@@ -789,6 +848,13 @@
page = pywikibot.Page(pywikibot.getSite(), singlePageTitle)
gen = iter([page])
+ if xmlFilename:
+ try:
+ xmlStart
+ except NameError:
+ xmlStart = None
+ gen = XmlDumpPageGenerator(xmlFilename, xmlStart, namespaces)
+
if not gen:
gen = genFactory.getCombinedGenerator()
if gen:
--
To view, visit https://gerrit.wikimedia.org/r/96272
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I7ba4f460897316ae1f5cbcca0080f8c3262d9abf
Gerrit-PatchSet: 5
Gerrit-Project: pywikibot/compat
Gerrit-Branch: master
Gerrit-Owner: Beta16 <l.rabinelli(a)gmail.com>
Gerrit-Reviewer: Beta16 <l.rabinelli(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Legoktm <legoktm.wikipedia(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: Fix the order of the argument for Link() used in titletranslate
......................................................................
Fix the order of the argument for Link() used in titletranslate
Change-Id: Ied74dc60fc7018e51785acfdf8b71a6c4269b8e6
---
M pywikibot/titletranslate.py
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Ladsgroup: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/titletranslate.py b/pywikibot/titletranslate.py
index f3feac9..656a5ea 100644
--- a/pywikibot/titletranslate.py
+++ b/pywikibot/titletranslate.py
@@ -70,7 +70,7 @@
for newcode in codes:
if newcode in site.languages():
if newcode != site.code:
- x = pywikibot.Link(site.getSite(code=newcode), newname)
+ x = pywikibot.Link(newname, site.getSite(code=newcode))
if x not in result:
result.append(x)
else:
--
To view, visit https://gerrit.wikimedia.org/r/122100
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ied74dc60fc7018e51785acfdf8b71a6c4269b8e6
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Whym <whym(a)whym.org>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: Use getCategoryLinks when renaming categories.
......................................................................
Use getCategoryLinks when renaming categories.
This allows us to skip categories included from translcuded templates. This is a potential fix for bug 58084.
Change-Id: I05fa90523f718f09de83d03aab3bd9d031e7cbbb
---
M pywikibot/page.py
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Ladsgroup: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/page.py b/pywikibot/page.py
index d5ac219..b18b875 100644
--- a/pywikibot/page.py
+++ b/pywikibot/page.py
@@ -1494,7 +1494,7 @@
# get list of Category objects the article is in and remove possible
# duplicates
cats = []
- for cat in self.categories(get_redirect=True):
+ for cat in pywikibot.textlib.getCategoryLinks(self.text):
if cat not in cats:
cats.append(cat)
--
To view, visit https://gerrit.wikimedia.org/r/123434
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I05fa90523f718f09de83d03aab3bd9d031e7cbbb
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Gerrit Patch Uploader <gerritpatchuploader(a)gmail.com>
Gerrit-Reviewer: Gerrit Patch Uploader <gerritpatchuploader(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Strainu <wiki(a)strainu.ro>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: Show the number of old threads that are pending for archiving
......................................................................
Show the number of old threads that are pending for archiving
Change-Id: I88a69df5a5841c3725edb66c15cc855c45cd8bee
---
M scripts/archivebot.py
1 file changed, 4 insertions(+), 3 deletions(-)
Approvals:
Ladsgroup: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/archivebot.py b/scripts/archivebot.py
index c970c5f..23a5a30 100644
--- a/scripts/archivebot.py
+++ b/scripts/archivebot.py
@@ -426,11 +426,12 @@
if not self.Page.botMayEdit():
return
whys = self.analyzePage()
- if self.archivedThreads < int(self.get('minthreadstoarchive', 2)):
+ mintoarchive = int(self.get('minthreadstoarchive', 2))
+ if self.archivedThreads < mintoarchive:
# We might not want to archive a measly few threads
# (lowers edit frequency)
- pywikibot.output(u'There are only %d Threads. Skipping'
- % self.archivedThreads)
+ pywikibot.output(u'Only %d (< %d) threads are old enough. Skipping'
+ % (self.archivedThreads, mintoarchive))
return
if whys:
pywikibot.output(u'Archiving %d thread(s).' % self.archivedThreads)
--
To view, visit https://gerrit.wikimedia.org/r/122102
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I88a69df5a5841c3725edb66c15cc855c45cd8bee
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Whym <whym(a)whym.org>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: jenkins-bot <>