September 2011 - pywikibot - lists.wikimedia.org

[Pywikipedia-l] Urlencoded section titles
by Bináris 13 Sep '18

13 Sep '18

Happy Monday, There are strange people who make such links (kindof urlencoded?): [[Második világháború#Partrasz.C3.A1ll.C3.A1s Szic.C3.ADli.C3.A1ban .28Huskey hadm.C5.B1velet.29|Huskey hadműveletben]] So the section title must have been copied from the URL. Do we have a ready tool to fix these? -- Bináris

3 11

[Pywikipedia-l] Replace.py should save -- please comment
by Bináris 26 Feb '12

26 Feb '12

Hi! My old problem is that repalce.py can't write the pages to work on into a file on my disk. I have used a modificated version for years that does no changes but writes the title of the involved pages to a subpage on Wikipedia in automated mode, and then I can make the replacements from that page much more quickly than directly from dump or living Wikipedia. This is slow and generates a plenty of dummy edits. In other words, replace.py has a tool to get the titles from a file (-file) or from a wikipage (-links), but has no tool to generate this file. Now I am ready to rewrite it. This way we can start it and the bot will find all the possible articles to work on and save the titles without editing Wikipedia (and without artificial delay), meanwhile we can have the lunch or run a marathon or sleep. Then we make the replacements from this with -file. My idea is that replace.py should have two new parameters: -save writes the results into a new file instead of editing articles. It overwrites existing file without notice. -saveappend writes into a file or appends to the existing one. OR: -save writes and appends (primary mode) -savenew writes and overwrites The help is here: http://docs.python.org/howto/unicode.html#reading-and-writing-unicode-data So we have to import codecs. My script is: articles=codecs.open('cikkek.txt','a',encoding='utf-8') ... tutuzuzu=u'# %s\n' %page.aslink() <-- needs rewrite to the new syntax articles.write(unicode(tutuzuzu)) <-- needs further testing, if nicode() is really needed articles.flush() It works fine except '\n' is a unix-styled newline that has to be converted by lfcr.py in order to make it readable with notepad.exe. This is with constant filename, that should be developed to get from command line. Your opinions before I begin? -- Bináris

5 12

[Pywikipedia-l] SourceForge passwords
by Bináris 20 Feb '12

20 Feb '12

Have you also got an e-mail with subject "SourceForge.net passwords reset"? Is that authentic? -- Bináris

10 28

[Pywikipedia-l] How to read special pages?
by Bináris 17 Feb '12

17 Feb '12

I want to read a special page with Page.get(). The message is: File "C:\Program Files\Pywikipedia\wikipedia.py", line 601, in get raise NoPage('%s is in the Special namespace!' % self.aslink()) pywikibot.exceptions.NoPage What is the solution? -- Bináris

7 21

[Pywikipedia-l] SVN access request
by Bináris 07 Feb '12

07 Feb '12

Hi, I would like to get an SVN access and some help to start. I need it mainly for inserting and maintaining TOCbot that is under preparation (it has worked in huwiki for several months and is now being internationalized). Information about TOCbot: http://hu.wikipedia.org/wiki/Szerkeszt%C5%91:Bin%C3%A1ris/TOCbot Description, user guide and bot owners' guide and a collection of examples is ready as well as an auxilary script, while the main script is not yet public. It will soon be published for test and may need much care in the first time. I would also like to take part in maintenance of replace.py for what I worked a lot already. At the moment I am interested only in trunk version. My SF page: http://sourceforge.net/users/binbot/ -- I don't know how to list all my contributions, here appears a part of them since May 22, but there are much more. I have also been active on mailing list in the past years. Please support and give me technical help to use the system. -- Bináris

3 8

Re: [Pywikipedia-l] [Pywikipedia-svn] SVN: [9196] trunk/pywikipedia
by Merlijn van Deen 19 Jan '12

19 Jan '12

Whoo! Great work :-) Tests always are good contributions :-) On a sidenote - is there a reason you're implementing these in 'trunk' and not in 'rewrite'? Of course, these contributions are very welcome in the trunk, but I still think it would be good to push the rewrite branch. Best regards, Merlijn On 24 April 2011 07:41, <jayvdb(a)svn.wikimedia.org> wrote: > http://www.mediawiki.org/wiki/Special:Code/pywikipedia/9196 > > Revision: 9196 > Author: jayvdb > Date: 2011-04-24 05:40:59 +0000 (Sun, 24 Apr 2011) > Log Message: > ----------- > Allow lists of Page and User objects to be interogated > > Modified Paths: > -------------- > trunk/pywikipedia/query.py > trunk/pywikipedia/tests/test_query.py > > Modified: trunk/pywikipedia/query.py > =================================================================== > --- trunk/pywikipedia/query.py 2011-04-24 04:23:12 UTC (rev 9195) > +++ trunk/pywikipedia/query.py 2011-04-24 05:40:59 UTC (rev 9196) > @@ -263,10 +263,21 @@ > > encList = '' > # items may not have one symbol - '|' > - for l in list: > - if type(l) == str and u'|' in l: > - raise wikipedia.Error("item '%s' contains '|' symbol" % l ) > - encList += ToUtf8(l) + u'|' > + for item in list: > + if isinstance(item,basestring): > + if u'|' in item: > + raise wikipedia.Error(u"item '%s' contains '|' symbol" % > item ) > + encList += ToUtf8(item) + u'|' > + elif isinstance(item,wikipedia.Page): > + encList += ToUtf8(item.title()) + u'|' > + elif item.__class__.__name__ == 'User': > + # delay loading this until it is needed > + import userlib > + encList += ToUtf8(item.name()) + u'|' > + else: > + raise wikipedia.Error(u'unknown item class %s' % > item.__class__.__name__) > + > + # strip trailing '|' before returning > return encList[:-1] > > def ToUtf8(s): > > Modified: trunk/pywikipedia/tests/test_query.py > =================================================================== > --- trunk/pywikipedia/tests/test_query.py 2011-04-24 04:23:12 UTC > (rev 9195) > +++ trunk/pywikipedia/tests/test_query.py 2011-04-24 05:40:59 UTC > (rev 9196) > @@ -7,6 +7,8 @@ > import unittest > import tests.test_pywiki > > +import wikipedia as pywikibot > +import catlib, userlib > import query > > > @@ -74,5 +76,72 @@ > ]} > self.assertEqualQueryResult(params, expectedresult) > > + def test_titles_Page(self): > + params = { > + 'action': 'query', > + 'list': 'users', > + 'usprop': ['registration'], > + 'ususers': [pywikibot.Page(self.site, u'Example'), > + pywikibot.Page(self.site, u'Example2')], > + } > + expectedresult = {u'users': [ > + { > + u'userid': 215131, > + u'name': u'Example', > + u'registration': u'2005-03-19T00:17:19Z' > + }, > + { > + u'userid': 5176706, > + u'name': u'Example2', > + u'registration': u'2007-08-26T02:13:33Z' > + }, > + ]} > + self.assertEqualQueryResult(params, expectedresult) > + > + def test_titles_User(self): > + params = { > + 'action': 'query', > + 'list': 'users', > + 'usprop': ['registration'], > + 'ususers': [userlib.User(self.site, u'Example'), > + userlib.User(self.site, u'Example2')], > + } > + expectedresult = {u'users': [ > + { > + u'userid': 215131, > + u'name': u'Example', > + u'registration': u'2005-03-19T00:17:19Z' > + }, > + { > + u'userid': 5176706, > + u'name': u'Example2', > + u'registration': u'2007-08-26T02:13:33Z' > + }, > + ]} > + self.assertEqualQueryResult(params, expectedresult) > + > + def test_titles_Category(self): > + params = { > + 'action': 'query', > + 'prop': 'revisions', > + 'rvprop': ['ids', 'timestamp', 'user'], > + 'rvdir': 'newer', > + 'rvlimit': 1, > + 'titles': [catlib.Category(self.site, > u'Category:Categories')], > + } > + expectedresult = {u'pages': {u'794823': > + { > + u'ns': 14, > + u'pageid': 794823, > + u'revisions': [{ > + u'revid': 4494485, > + u'user': u'SEWilco', > + u'timestamp': u'2004-07-07T18:45:50Z', > + }], > + u'title': u'Category:Categories', > + }, > + }} > + self.assertEqualQueryResult(params, expectedresult) > + > if __name__ == "__main__": > unittest.main() > > > _______________________________________________ > Pywikipedia-svn mailing list > Pywikipedia-svn(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/pywikipedia-svn >

8 13

[Pywikipedia-l] rewrite branch
by John Vandenberg 22 Dec '11

22 Dec '11

On Mon, Apr 25, 2011 at 7:49 AM, Merlijn van Deen <valhallasw(a)arctus.nl> wrote: > Whoo! Great work :-) Tests always are good contributions :-) Thanks ;-) I agree. > On a sidenote - is there a reason you're implementing these in 'trunk' and > not in 'rewrite'? Of course, these contributions are very welcome in the > trunk, but I still think it would be good to push the rewrite branch. I'm working off trunk because it is trunk. I'd assumed that the rewrite branch was a single-purpose branch to rewrite something, and that it would be merged back when it is stable. Is it stable? Is there any documentation on what the plans are for the rewrite branch? Is there a roadmap to finish it? Is see now that the rewrite branch has more unit tests, but more are needed. Is there a need to create a backwards compatibility layer? Or, is everyone except me using the rewrite branch? ;-) -- John Vandenberg

3 2

[Pywikipedia-l] Please approve me for commit access
by MJB 30 Oct '11

30 Oct '11

Hello , Tim Starling told me that i can ask here to approve me for commit access, i am working always with python and as i am working in Persian community, my codes is about working on Persian Wikis you can see a summary here <http://fa.wikinews.org/wiki/%DA%A9%D8%A7%D8%B1%D8%A8%D8%B1:Mjbmrbot#Robot_T…> and i would like to work on localization and some minor edits that would be helpful, here is example of my codes : # -*- coding: utf-8 -*- import wikipedia, urllib, re, os, datetime, calendar from xml.dom import minidom from time import strftime Lang="fa" Family="wikinews" RR = (u'??????', u'?????', u'????', u'?????', u'??', u'????', u'?????', u'???', u'???????', u'?????', u'??????', u'??????') RR2 = (u'January', u'February', u'March', u'April', u'May', u'June', u'July', u'August', u'September', u'October', u'November', u'December') RRfr = (u'janvier', u'février', u'mars', u'avril', u'mai', u'juin', u'juillet', u'août', u'septembre', u'octobre', u'novembre', u'décembre') RRfr2 = (u'Janvier', u'Février', u'Mars', u'Avril', u'Mai', u'Juin', u'Juillet', u'Août', u'Septembre', u'Octobre', u'Novembre', u'Décembre') RR3 = (31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31) MMM = int(strftime("%m")) YYY = strftime("%Y") YYY2 = strftime("%Y") YYY = YYY.replace(u'1', u'1') YYY = YYY.replace(u'2', u'2') YYY = YYY.replace(u'3', u'3') YYY = YYY.replace(u'4', u'4') YYY = YYY.replace(u'5', u'5') YYY = YYY.replace(u'6', u'6') YYY = YYY.replace(u'7', u'7') YYY = YYY.replace(u'8', u'8') YYY = YYY.replace(u'9', u'9') YYY = YYY.replace(u'0', u'0') site = wikipedia.getSite(Lang, Family) FF = "" for i in range(1, RR3[MMM-1] + 1): site = wikipedia.getSite(Lang, Family) DD = str(i) DD2 = str(i) DDfr = str(i) if (DDfr==u'1'): DDfr = u"1er" DDfr2 = str(i) if (len(DDfr2)==1): DDfr2 = u'0' + DDfr2 DD = DD.replace(u'1', u'1') DD = DD.replace(u'2', u'2') DD = DD.replace(u'3', u'3') DD = DD.replace(u'4', u'4') DD = DD.replace(u'5', u'5') DD = DD.replace(u'6', u'6') DD = DD.replace(u'7', u'7') DD = DD.replace(u'8', u'8') DD = DD.replace(u'9', u'9') DD = DD.replace(u'0', u'0') FF = FF + u"== [[/" + DD + u'|' + DD + u' ' + RR[MMM-1] + u"]] ==\n{{/" + DD + u'}}\n\n' pagename = (u'???:' + DD + u' ' + RR[MMM-1] + u' ' + YYY) page = wikipedia.Page(site, pagename) AA = (u'[[???:' + RR[MMM-1] + u' ' + YYY + u']]\n[[???:' + DD + u' ' + RR[MMM-1] + u']]\n\n[[en:Category:' + RR2[MMM-1] + u' ' + DD2 + u', ' + YYY2 + u']]\n[[fr:Catégorie:' + DDfr + u' ' + RRfr[MMM-1] + u' ' + YYY2 + u']]') CCC = (u'????: ????? ???????? ??? ' + RR[MMM-1] + u' ' + YYY) wikipedia.output(u"Loading %s..." % pagename) try: text = page.get() except wikipedia.NoPage: print "Page doesn't exist, creating it ..." page.put(AA, comment=CCC, watchArticle = None, minorEdit = False) pagename = (u'???:' + DD + u' ' + RR[MMM-1]) page = wikipedia.Page(site, pagename) AA = (u'{{' + RR[MMM-1] + u'}}\n{{?????????|{{????????}}}}\n{{??????????-???|' + DD2 + u' ' + RR2[MMM-1] + u'}}\n\n[[en:Category:' + RR2[MMM-1] + u' ' + DD2 + u']]\n[[fr:Catégorie:' + DDfr + u' ' + RRfr[MMM-1] + u']]') CCC = (u'????: ????? ??????? ??? ' + RR[MMM-1]) wikipedia.output(u"Loading %s..." % pagename) try: text = page.get() except wikipedia.NoPage: print "Page doesn't exist, creating it ..." page.put(AA, comment=CCC, watchArticle = None, minorEdit = False) pagename = (u'????????:' + YYY + u'/' + RR[MMM-1] + u'/' + DD) page = wikipedia.Page(site, pagename) AA = (u'<onlyinclude>\n<DynamicPageList>\ncategory=????????\ncategory=' + DD + u' ' + RR[MMM-1] + u' ' + YYY + u'\nsuppresserrors=true\nstablepages=only\n</DynamicPageList>\n</onlyinclude>\n\n[[en:Wikinews:' + YYY2 + u'/' + RR2[MMM-1] + u'/' + DD2 + u']]\n[[fr:Wikinews:' + YYY2 + u'/' + RRfr[MMM-1] + u'/' + DDfr2 + u']]') CCC = (u'????: ????? ???????? ??? ' + RR[MMM-1] + u' ' + YYY) wikipedia.output(u"Loading %s..." % pagename) try: text = page.get() except wikipedia.NoPage: print "Page doesn't exist, creating it ..." page.put(AA, comment=CCC, watchArticle = None, minorEdit = False) pagename = (u'????????:' + YYY + u'/' + RR[MMM-1]) page = wikipedia.Page(site, pagename) FF = FF + u'[[???:' + RR[MMM-1] + u' ' + YYY + ']]\n\n[[en:Wikinews:' + YYY2 + u'/' + RR2[MMM-1] + u']]\n[[fr:Wikinews:' + YYY2 + u'/' + RRfr[MMM-1] + ']]' CCC = (u'????: ????? ????? ????? ?? ??? ' + RR[MMM-1] + u' ' + YYY) wikipedia.output(u"Loading %s..." % pagename) try: text = page.get() except wikipedia.NoPage: print "Page doesn't exist, creating it ..." page.put(FF, comment=CCC, watchArticle = None, minorEdit = False) pagename = (u'???:' + RR[MMM-1] + u' ' + YYY) page = wikipedia.Page(site, pagename) AA = (u'[[???:' + RR[MMM-1] + u']]\n[[???:'+ YYY + u']]\n\n[[en:Category:' + RR2[MMM-1] + u' ' + YYY2 + u']]\n[[fr:Catégorie:' + RRfr2[MMM-1] + u' ' + YYY2 + u']]') CCC = (u'????: ????? ???? ??? ' + RR[MMM-1] + u' ' + YYY) try: text = page.get() except wikipedia.NoPage: print "Page doesn't exist, creating it ..." page.put(AA, comment=CCC, watchArticle = None, minorEdit = False) pagename = (u'???:' + YYY) page = wikipedia.Page(site, pagename) AA = (u'{{???????????|' + YYY2 + u'}}\n\n[[???:????? ?? ???? ?????]]\n\n[[en:Category:' + YYY2 + u']]\n[[fr:Catégorie:' + YYY2 + u']]') CCC = (u'????: ????? ???? ??? ' + YYY) try: text = page.get() except wikipedia.NoPage: print "Page doesn't exist, creating it ..." page.put(AA, comment=CCC, watchArticle = None, minorEdit = False) This is a simple code that create date category pages and templates, my user name in wmf projects is Mjbmr and i have a bot with Mjbmrbot username that it have a global flag, I hope you accept my request, Regards -- Mjbmr

7 9

[Pywikipedia-l] Running the rewrite branch as developer/power-user [windows]
by Merlijn van Deen 09 Oct '11

09 Oct '11

Hello all, *As several people have mentioned they had trouble starting with the rewrite branch, I decided to do a step-by-step log of installing the rewrite in a way that is good for developing -- this means you are able to edit the framework files, while not inflicting any changes on other users (or other bots you run!) of the system. By using setup.py develop, edits you make to the framework will immediately be used (no need to setup.py install them), but only within the virtualenv.* *This is the windows version of my earlier email* * * I do not run python on windows, so this is a tutorial that starts with installing python. It's a bit rougher than the unix one, as I did not want to spend too much time on it. 1. Install python 2.7 http://python.org/ftp/python/2.7.1/python-2.7.1.msi<http://python.org/download/> (do *not* use the 64-bit version, due to http://bugs.python.org/issue6792 ) 2. Install Setuptools http://pypi.python.org/pypi/setuptools#files 3. Install Virtualenv start/run: cmd c:\Python27\Scripts\easy_install.exe virtualenv 4. create a virtualenv for pwb C:\Users\valhallasw>c:\Python27\Scripts\virtualenv.exe pywikibot New python executable in pywikibot\Scripts\python.exe Installing setuptools.....................done. 5. Go to C:\Users\valhallasw\pywikibot and use tortoisesvn to get the rewrite 6. create a shortcut to cmd /k c:\users\valhallasw\pywikibot\scripts\activate.bat with working path C:\Users\valhallasw\pywikibot\rewrite 7. Use the shortcut. You now have a new cmd.exe window 8. python setup.py develop Your default user directory is "C:\Users\valhallasw\AppData\Roaming\pywikibot" How to proceed? ([K]eep [c]hange) change, to c:\users\valhallasw\pywikibot\conf\ Answer 'y' to the warning prompt (not 'yes') Do you want to copy files: y [note: I copied my unix user-config.py to c:\users\valhallasw\pywikibot] Path to existing wikipedia.py? C:\Users\valhallasw\pywikibot NOTE: user-config.py already exists in the directory Create user-fixes.py file? ([y]es, [N]o) n (pywikibot) C:\Users\valhallasw\pywikibot\rewrite>echo SET PYWIKIBOT2_DIR=c:\users\valhallasw\pywikibot\conf>> ..\Scripts\activate.bat (DON'T put a space between f and >>!) Close the window, and 9. Use the shortcut from (7) again You should now have a cmd.exe with a working pywikibot setup! (pywikibot) C:\Users\valhallasw\pywikibot\rewrite\scripts>python touch.py Gebruiker:Valhallasw Retrieving 1 pages from wikipedia:nl. Page [[Gebruiker:Valhallasw]] saved NOTE: you *must* use 'python' in front of the script name, or python will not find the pywikibot directory. Good luck! Merlijn

2 1

Re: [Pywikipedia-l] [Wikitech-l] serious interwiki.py issues on MW 1.18 wikis
by Merlijn van Deen 30 Sep '11

30 Sep '11

Hi Ariel and Andre, On Fri, Sep 30, 2011 at 9:39 AM, Ariel T. Glenn <ariel(a)wikimedia.org>wrote: > Out of curiosity... If the new revisions of one of these badly edited > pages are deleted, leaving the top revision as the one just before the > bad iw bot edit, does a rerun of the bot on the page fail? On Fri, Sep 30, 2011 at 11:13 AM, Andre Engels <andreengels(a)gmail.com> wrote: > I deleted the page [[nl:Blankenbach]], then restored the 2 versions before > the problematic bot edit. When now I look at the page, instead of the page > content I get: (...) Using this undeleted version, and running interwiki.py, gives the expected result: valhallasw@dorthonion:~/src/pywikipedia/trunk$ python interwiki.py -page:Blankenbach NOTE: Number of pages queued is 0, trying to add 60 more. Getting 1 pages from wikipedia:nl... WARNING: Family file wikipedia contains version number 1.17wmf1, but it should be 1.18wmf1 NOTE: [[nl:Blankenbach]] does not exist. Skipping. This also happens for running it from dewiki (python interwiki.py -lang:de -page:Blankenbach%20%28Begriffskl%C3%A4rung%29) or running as 'full-auto' bot (python interwiki.py -all -async -cleanup -log -auto -ns:0 -start:Blankenbach). Special:Export acts like the page just does not exist (http://nl.wikipedia.org/w/index.php?title=Speciaal:Exporteren&useskin=monob… shows page Blanzac but not Blankenbach) api.php also more or less does the expected thing: http://nl.wikipedia.org/w/api.php?action=query&prop=revisions&titles=Blanke… - that is, unless you supply rvlimit=1: http://nl.wikipedia.org/w/api.php?action=query&prop=revisions&titles=Blanke… However, none of them seem to return an empty page - and playing around with pywikipediabot does not allow be to get an empty page (depending on settings, it can either be the result on the edit page (page.get(), use_api=False / screen scraping), a pywikibot.exceptions.NoPage exception (PreloadingGenerator / wikipedia.getall, which uses Special:Export) or the correct page text (page.get(), use_api=True). Anyway, thanks a huge heap for trying this (and for everyone, for thinking about it). Unfortunately, I won't have much time this weekend to debug -- hopefully some other pwb developer has. Best regards, and thanks again, Merlijn P.S. On 30 September 2011 11:12, Max Semenik <maxsem.wiki(a)gmail.com> wrote: > So you screen-scrape? No surprise it breaks. Why? For example, due to > protocol-relative URLs. Or some other changes to HTML output. Why not just > use API? No, most of pywikipedia has been adapted to the api and/or special:export, which, imo, is just an 'old' mediawiki api. Keep in mind interwiki.py is old (2003!), and pywikipedia initally was an extension of the interwiki bot. Thus, there could very well be some code that is seldom used which still uses screen scraping. And actually, in practice, screen scraping worked pretty well.

1 0

[Pywikipedia-l] Fwd: [Wikitech-l] serious interwiki.py issues on MW 1.18 wikis
by Andre Engels 30 Sep '11

30 Sep '11

It seems somewhere along the way pywikipedia-l got removed, therefore forwarding this message, which might be interesting: The problem seems to not be bot-specific, if it happened that a 'normal' user got to the page before a bot, he would have seen something very unusual too. André Engels ---------- Forwarded message ---------- From: Andre Engels <andreengels(a)gmail.com> Date: Fri, Sep 30, 2011 at 11:13 AM Subject: Re: [Wikitech-l] serious interwiki.py issues on MW 1.18 wikis To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org> On Fri, Sep 30, 2011 at 9:39 AM, Ariel T. Glenn <ariel(a)wikimedia.org> wrote: > Out of curiosity... If the new revisions of one of these badly edited > pages are deleted, leaving the top revision as the one just before the > bad iw bot edit, does a rerun of the bot on the page fail? > I did a test, and the result was very interesting, which might point to the cause of this bug: I deleted the page [[nl:Blankenbach]], then restored the 2 versions before the problematic bot edit. When now I look at the page, instead of the page content I get: In de database is geen inhoud aangetroffen voor de pagina met . Dit kan voorkomen als u een verouderde verwijzing naar het verschil tussen twee versies van een pagina volgt of een versie opvraagt die is verwijderd. Als dit niet het geval is, hebt u wellicht een fout in de software gevonden. Maak hiervan melding bij een systeembeheerder<http://nl.wikipedia.org/wiki/Speciaal:Gebruikerslijst/sysop>van Wikipedia en vermeld daarbij de URL van deze pagina. Going to the specific version that after the deletion-and-partial-restore should be the newest ( http://nl.wikipedia.org/w/index.php?title=Blankenbach&oldid=10676248), it claims that there is a newer version, but going to the newer version or the newest version, I get the abovementioned message again. As an extra test, I did the delete-then-restore-some-versions-but-not-the-most-recent action with another page (http://nl.wikipedia.org/wiki/Gebruiker:Andre_Engels/Test), and there I found no such problem. From this I conclude that the bug has not been caused by that process, but that for some reason the page had a wrong (or empty) version number for its 'most recent' version, or something like that. -- André Engels, andreengels(a)gmail.com -- André Engels, andreengels(a)gmail.com

1 0

Re: [Pywikipedia-l] [Wikitech-l] serious interwiki.py issues on MW 1.18 wikis
by Andre Engels 30 Sep '11

30 Sep '11

On Fri, Sep 30, 2011 at 11:12 AM, Max Semenik <maxsem.wiki(a)gmail.com> wrote: > On Fri, Sep 30, 2011 at 12:56 PM, Andre Engels <andreengels(a)gmail.com > >wrote: > > > > > The interwiki links are retrieved from page content. The page content has > > been received through a call to Special:Export. > > > > > > > I.e. would receiving no content (from the bot POV) produce that > behavior? > > > > > > > Yes, the only reasonable explanation seems to be that the bot interprets > > what it gets from the server as an empty page. > > > > So you screen-scrape? No surprise it breaks. Why? For example, due to > protocol-relative URLs. Or some other changes to HTML output. Why not just > use API? > Basically, because most of the core functionality comes from before the API came into existence. At least, that would be my explanation. -- André Engels, andreengels(a)gmail.com

1 0

[Pywikipedia-l] serious interwiki.py issues on MW 1.18 wikis
by Merlijn van Deen 29 Sep '11

29 Sep '11

Hello to both the wikitech and pywikipedia lists -- please keep both informed when replying. Thanks. A few days ago, we - the pywikipedia developers - received alarming reports of interwiki bots removing content from pages. This does not seem to happen often, and we have not been able to reproduce the conditions in which this happens. However, the common denominator is the fact it seems to be happening only on the wikipedia's that run MediaWiki 1.18 wikis. As such, I think this topic might be relevant for wikitech-l, too. In addition, there is no-one in the pywikipedia team with a clear idea of why this is happening. As such, we would appreciate any ideas. 1. What happens? Essentially, the interwiki bot does its job, retrieves the graph and determines the correct interwiki links. It should then add it to the page, but instead, /only/ the interwiki links are stored. For example: http://nl.wikipedia.org/w/index.php?title=Blankenbach&diff=next&oldid=10676… http://eo.wikipedia.org/w/index.php?title=Anton%C3%ADn_Kl%C3%A1%C5%A1tersk%… http://simple.wikipedia.org/w/index.php?title=Mettau%2C_Switzerland&action=… 2. Why does this happen? This is unclear. On the one hand, interwiki.py is somewhat black magic: none of the current developers intimately knows its workings. On the other hand, the bug is not reproducible: running it on the exact same page with the exact same page text does not result in a cleared page. It could very well be something like broken network error handling - but mainly, we have no idea. Did anything change in Special:Export (which is still used in interwiki.py) or the API which might cause something like this? I couldn't find anything in the release notes. 3. Reasons for relating it to MW 1.18 To find out on which wikis this problem happens, I used a quick-and-dirty heuristic: select rc_comment, rc_cur_time, rc_user, rc_namespace, rc_title, rc_old_len, rc_new_len from recentchanges left join user_groups on ug_user=rc_user where rc_new_len < rc_old_len * 0.1 and ug_group = 'bot' and rc_namespace=0 limit 10 /* SLOW OK */; This is a slow query (~30s for nlwiki_p on the toolserver), but it gives some interesting results: nlwiki: 9 rows, all broken interwiki bots eowiki: 25 rows, all interwiki bots simplewiki: 3 rows, of which 2 are interwiki bots dewiki: 0 rows using rc_old_len * 0.3: 14 rows, all double redirect fixes frwiki: 9 rows, but *none* from interwiki bots (all edits are by the same antivandalism bot) itwiki: 0 rows ptwiki: 0 rows All ideas and hints are very welcome. Hopefully we will be able to solve this before tuesday... Best regards, Merlijn van Deen

1 0

Re: [Pywikipedia-l] [Pywikipedia-announce] [BUG NOTICE] interwiki.py removes content from pages, cause unknown
by info＠gno.de 28 Sep '11

28 Sep '11

Seems Page.get() was empty. Unfortunately I cannot help debugging, I am on holydays for few days. xqt ----- Ursprungliche Nachricht ----- Von: Maarten Dammers Gesendet: Dienstag, 27. September 2011 22:53 An: pywikipedia-announce(a)lists.wikimedia.org Betreff: [Pywikipedia-l] [Pywikipedia-announce] [BUG NOTICE] interwiki.py removes content from pages, cause unknown Hello all, This is a notice to all active interwiki bot owners. Today, we have received two worrying bug reports of interwiki bots, removing content from pages: http://nl.wikipedia.org/w/index.php?title=Blankenbach&diff=prev&oldid=27331… http://simple.wikipedia.org/w/index.php?title=Mettau%2C_Switzerland&action=… Unfortunately, it is not possible to reproduce this bug by running interwiki.py on the mentioned pages. As such, we have no idea what this problem was caused by. HiW-bot (nlwiki edit) was running r9558 at the time of the edit. We would like your help in tracing the cause of this bug. If possible, see if your bot has made any strange changes, and report them in the bug thread for this problem: https://sourceforge.net/tracker/index.php?func=detail&aid=3414669&group_id=… This might be related to the MediaWiki 1.18 deploy, see http://blog.wikimedia.org/2011/09/16/mediawiki118iscomin/ for more information Thank you for your cooperation and understanding, The pywikipediabot developers _______________________________________________ Pywikipedia-announce mailing list Pywikipedia-announce(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-announce _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

2 1

[Pywikipedia-l] [Pywikipedia-announce] [BUG NOTICE] interwiki.py removes content from pages, cause unknown
by Maarten Dammers 28 Sep '11

28 Sep '11

Hello all, This is a notice to all active interwiki bot owners. Today, we have received two worrying bug reports of interwiki bots, removing content from pages: http://nl.wikipedia.org/w/index.php?title=Blankenbach&diff=prev&oldid=27331… http://simple.wikipedia.org/w/index.php?title=Mettau%2C_Switzerland&action=… Unfortunately, it is not possible to reproduce this bug by running interwiki.py on the mentioned pages. As such, we have no idea what this problem was caused by. HiW-bot (nlwiki edit) was running r9558 at the time of the edit. We would like your help in tracing the cause of this bug. If possible, see if your bot has made any strange changes, and report them in the bug thread for this problem: https://sourceforge.net/tracker/index.php?func=detail&aid=3414669&group_id=… This might be related to the MediaWiki 1.18 deploy, see http://blog.wikimedia.org/2011/09/16/mediawiki118iscomin/ for more information Thank you for your cooperation and understanding, The pywikipediabot developers _______________________________________________ Pywikipedia-announce mailing list Pywikipedia-announce(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-announce

2 1

Re: [Pywikipedia-l] Problem
by majid 20 Sep '11

20 Sep '11

Thank you for your attention Information user-config. And I will send you the script used. Please check the files posted, please solve this problem .............................................. On 18 September 2011 13:51, majid <magidred(a)yahoo.com> wrote: I've never editing interwiki command run in a long time I have not been successful robot, Please check the problem First of all, it's annoying the bot is not doing what it should be doing. However, it's hard for other people to understand what is going on with your current report. Could you please send the following information: * what you are trying to do (e.g. 'fixing interwiki links on enwiki, srwiki, itwiki') * which command (with which parameters) you are using for that * the result you are expecting * the real result, and the text outputted by the script and then, to help debugging the issue: * the output of python version.py * the family you are working on (i.e. wikipedia, wikibooks, etc). If it's a custom family, please also post the family file. * your user-config.py Thanks! Best, Merlijn -----Inline Attachment Follows----- _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

2 2

[Pywikipedia-l] Problem
by majid 18 Sep '11

18 Sep '11

hi Since the grant account toolserver to the I robot، I've never editing interwiki command run in a long time I have not been successful robot, Please check the problem

2 1

[Pywikipedia-l] imageharvest gets wrong url
by estrangeiro 15 Sep '11

15 Sep '11

Hi! First, I do realize that the problem that I'm gonna show you following is probably easy to fix, but, well, I'm not so in to python yet. Second, it appears to me that it is a problem that could affect more people than me, since the task I'm trying to execute is not that uncommon, I think, so if it were fixed it would help... more than one. =) Here is the deal: I'm trying to upload all the images from this page: <http://pt.wikipedia.org/wiki/Portal:Anarquia/Acervo_de_imagens> to another wiki website. The bot seems to recognize the images, but instead of go to "pt.wikipedia.org/Ficheiro:name-of-image.xxx" where they actually are, he retypes the given url of the portal and add the "ficheiro:name-of-image.xxx" part. For example, the first image of the archive must be found at <http://pt.wikipedia.org/wiki/Ficheiro:Cutters1.preview.jpg>, but instead the bot tries to upload from <http://pt.wikipedia.org/wiki/Portal:Anarquia//wiki/ficheiro:cutters1.previe…>, what obviously does not work. Here is the result in my terminal: "HTTP Error 404: Not Found" I'm wondering what should I do to fix it, which script to edit and what change in it, since I am a Python newbie. Thank you! * "Ficheiro" is the portuguese equivalent to "file", in this case

2 5

[Pywikipedia-l] Botlist broken
by Dr. Trigon 15 Sep '11

15 Sep '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello all Today DrTrigonBot complained about a "HTTPError: HTTP Error 400: Bad Request" that was originating from botlist.py. I was able to track this down to the point where "タチコマ robot" should be used in an URL as offset (line 96) which caused the error when puting in this bot name. As I can see a urllib.quote(...) should solve the problem as done in the attached file. Could please someone verify this (if there is time) and then commit this change to SVN. Btw: I was not able to find something similar to "urllib.quote" in the bot framework so I had to import urllib. Thanks a lot and Greetings DrTrigon -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk5xrygACgkQAXWvBxzBrDAxLwCfWtikjvv88d2Z+F49q37OcHtP axcAniISk6qypZdT3WxlbAJPxfo9x4zn =DTud -----END PGP SIGNATURE-----

1 0

[Pywikipedia-l] Database replicas
by Kudu 14 Sep '11

14 Sep '11

Can pywikipedia be used with database replicas à la Toolserver? Thanks in advance, ~K

2 1

[Pywikipedia-l] misspelling.py for rewrite branch
by Daniel Herding 13 Sep '11

13 Sep '11

Hi all, I have just installed the rewrite branch, and I have adapted the misspelling.py script for it. As I'm waiting for my new SSH public key to get added to the SVN server, could someone else please commit it to the rewrite branch for me? I tested it on de: and en:, but it probably wouldn't be a bad idea to check my changes. After all, it's my first adaptation to the rewrite branch, and I haven't done PyWikiBot development for quite some time. Daniel

1 0

[Pywikipedia-l] replace.py error
by Kudu 09 Sep '11

09 Sep '11

node:pywikipedia kudu$ python replace.py -transcludes:sometemplate -regex 'someregex' 'somereplacement' unicode test: triggers problem #3081100 Getting references to [[Template:sometemplate]] via API... Getting 1 pages from wiki:wiki... Traceback (most recent call last): File "replace.py", line 825, in <module> main() File "replace.py", line 816, in main bot.run() File "replace.py", line 399, in run new_text = self.doReplacements(new_text) File "replace.py", line 342, in doReplacements allowoverlap=self.allowoverlap) File "~/pywikipedia/pywikibot/textlib.py", line 175, in replaceExcept match.group(groupID) + \ TypeError: coercing to Unicode: need string or buffer, NoneType found ~K

2 3

Re: [Pywikipedia-l] [Pywikipedia-svn] SVN: [9448] trunk/pywikipedia/pywikibot/i18n.py
by info＠gno.de 04 Sep '11

04 Sep '11

Hi Merlijn, I've seen it during discussion with Nikerabbit on tw.net (look at the first LQ-threat according pywikipedia) and I am working on a pythonic version of the plural-gettext.txt. I am using lambda function for that like this one (for mk-wiki): plural = lambda n: 0 if (n == 1 or n%10 == 1) else 1 I think this is the easiest way. But we have a remaining problem found on table2wiki. Now we have 3 messages. One message is used if we have no warning, and we have two messages either for one or a lot of warnings. I guess we could keep the first "table2wiki-no-warning" message and merge the two messages "table2wiki-one-warning" and "table2wiki-warnings" using {{Plural:}} tag. It is also possible to use the first message every time and append the warning message if needed. Greetings ----- Original Nachricht ---- Von: Merlijn van Deen <valhallasw(a)arctus.nl> An: pywikipedia-l(a)lists.wikimedia.org Datum: 25.08.2011 19:46 Betreff: Re: [Pywikipedia-l] [Pywikipedia-svn] SVN: [9448] trunk/pywikipedia/pywikibot/i18n.py > Hello xqt, > > On 21 August 2011 15:49, <xqt(a)svn.wikimedia.org> wrote: > > > + value. At the moment, we have only one plural_func = x: x!= 1 yet. > > Multiple > > + PLURAL tags are not supported (yet). > > > > Good to see you're working on plural support. Have you seen > http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/Translate/data/pl > ural-gettext.txt?view=markup > , > which lists the plural possibilities for each language. Unfortunately, > python does not support the a ? b : c notation, so it's not possible to > copy the code 1-on-1. > > Alternatively, there is > http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/Translate/data/pl > ural-cldr.yaml?view=markup > , > but that is fairly unreadable IMO. > > Good luck! > > Best, > Merlijn > > > -------------------------------- > > _______________________________________________ > Pywikipedia-l mailing list > Pywikipedia-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l > Arcor empfiehlt: Mal über die Karriere nachdenken! Wissenswertes und Nützliches finden Sie hierzu unter http://www.arcor.de/content/finanzen_job/job_karriere/bewerbung_karriere/ra…

3 9

Re: [Pywikipedia-l] [Pywikipedia-svn] SVN: [9494] trunk/pywikipedia/wikipedia.py
by Dr. Trigon 04 Sep '11

04 Sep '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 If this is also an issue with section detection within pages you could (if you like) also consider to use the code given in 'getSections' [1]... [1] https://fisheye.toolserver.org/browse/drtrigon/pywikipedia/dtbext/dtbext_wi… Greetings DrTrigon Am 03.09.2011 13:58, schrieb xqt(a)svn.wikimedia.org: > http://www.mediawiki.org/wiki/Special:Code/pywikipedia/9494 > > Revision: 9494 Author: xqt Date: 2011-09-03 11:58:48 +0000 > (Sat, 03 Sep 2011) Log Message: ----------- reverrevert r3147 due > to bug #2989218; check for italic code in headings.TODO: use a > better regex to find it. > > Modified Paths: -------------- trunk/pywikipedia/wikipedia.py > > Modified: trunk/pywikipedia/wikipedia.py > =================================================================== > > - --- trunk/pywikipedia/wikipedia.py 2011-09-03 11:17:47 UTC (rev 9493) > +++ trunk/pywikipedia/wikipedia.py 2011-09-03 11:58:48 UTC (rev > 9494) @@ -66,7 +66,6 @@ within a non-wiki-markup section of text > decodeEsperantoX: decode Esperanto text using the x convention. > encodeEsperantoX: convert wikitext to the Esperanto x-encoding. - > sectionencode: encode text for use as a section title in > wiki-links. findmarker(text, startwith, append): return a string > which is not part of text expandmarker(text, marker, separator): > return marker string expanded @@ -654,7 +653,7 @@ self._contents = > contents hn = self.section() if hn: - m = > re.search("=+ *%s *=+" % hn, self._contents) + m > = re.search("=+[ ']*%s[ ']*=+" % hn, self._contents) if verbose and > not m: output(u"WARNING: Section does not exist: %s" % > self.aslink(forceInterwiki = True)) # Store any exceptions for > later reference @@ -779,8 +778,8 @@ else: raise > IsRedirectPage(redirtarget) if self.section(): - # TODO: > What the hell is this? Docu please. - m = > re.search("\.3D\_*(\.27\.27+)?(\.5B\.5B)?\_*%s\_*(\.5B\.5B)?(\.27\.27+)?\_*\.3D" > % re.escape(self.section()), > sectionencode(pageInfo['revisions'][0]['*'],self.site().encoding())) > > + m = re.search("=+[ ']*%s[ ']*=+" % re.escape(self.section()), > + pageInfo['revisions'][0]['*']) if not > m: try: self._getexception @@ -920,8 +919,8 @@ else: raise > IsRedirectPage(redirtarget) if self.section(): - # TODO: > What the hell is this? Docu please. - m = > re.search("\.3D\_*(\.27\.27+)?(\.5B\.5B)?\_*%s\_*(\.5B\.5B)?(\.27\.27+)?\_*\.3D" > % re.escape(self.section()), > sectionencode(text,self.site().encoding())) + m = > re.search("=+[ ']*%s[ ']*=+" % re.escape(self.section()), + > text) if not m: try: self._getexception @@ -4140,8 +4139,7 @@ > page2._startTime = time.strftime('%Y%m%d%H%M%S', time.gmtime()) if > section: - m = > re.search("\.3D\_*(\.27\.27+)?(\.5B\.5B)?\_*%s\_*(\.5B\.5B)?(\.27\.27+)?\_*\.3D" > > - - % re.escape(section), sectionencode(text,page2.site().encoding())) > + m = re.search("=+[ ']*%s[ ']*=+" % > re.escape(section), text) if not m: try: page2._getexception @@ > -4302,7 +4300,7 @@ # Use the data loading time. page2._startTime = > time.strftime('%Y%m%d%H%M%S', time.gmtime()) if section: - > m = > re.search("\.3D\_*(\.27\.27+)?(\.5B\.5B)?\_*%s\_*(\.5B\.5B)?(\.27\.27+)?\_*\.3D" > % re.escape(section), sectionencode(text,page2.site().encoding())) > + m = re.search("=+[ ']*%s[ ']*=+" % > re.escape(section), text) if not m: try: page2._getexception @@ > -4531,10 +4529,6 @@ break return text > > -def sectionencode(text, encoding): - """Encode text so that it > can be used as a section title in wiki-links.""" - return > urllib.quote(text.replace(" > ","_").encode(encoding)).replace("%",".") - ######## Unicode > library functions ######## > > def UnicodeToAsciiHtml(s): > > > _______________________________________________ Pywikipedia-svn > mailing list Pywikipedia-svn(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/pywikipedia-svn > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk5ijR8ACgkQAXWvBxzBrDBNNQCgve2/z/SUa3bUNd625ibUKG/G sEMAn2/LtRfr9kvdV1UX+aVKL9MQZwl8 =9anJ -----END PGP SIGNATURE-----

2 2

Re: [Pywikipedia-l] 'wiki' as synonym for 'wikipedia' in family?
by info＠gno.de 01 Sep '11

01 Sep '11

Hi DrTrigon, "wiki" is reserved by Family.known_families. You know you get the related dbName of a site just by Site.dbName() which returns "dewiki_p" for "wikipedia:de". Thereby for historic reasons, the databases are called xxwiki instead of xxwikipedia for Wikipedias. You are using side effects doing your dbname2wikilink() conversion and I am sure you are able to modificate this stuff with stuff.replace("wiki:", "wikipedia:") if needed. In other words (your wrote me) "There should be one-- and preferably only one --__obvious__ way to do it" and btw "Although that way may not be obvious at first unless you're Dutch" (again: PEP20, The Zen of Python) Sali ;) xqt ----- Original Nachricht ---- Von: "Dr. Trigon" <dr.trigon(a)surfeu.ch> An: pywikipedia-l(a)lists.wikimedia.org Datum: 30.08.2011 19:02 Betreff: Re: [Pywikipedia-l] 'wiki' as synonym for 'wikipedia' in family? > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > "wiki" is being use as the interwiki link to the original wiki, > > WikiWikiWeb at http://c2.com/cgi/wiki, so you shouldn't use that > > for anything related to Wikipedia. You are getting it returned > > since Wikipedias can use it as an interwiki link to the > > WikiWikiWeb, i.e. [[:wiki:WelcomeVisitors]] becomes > > http://c2.com/cgi/wiki?WelcomeVisitors > > So this means essentially 'wiki' in toolserver DB (like e.g. 'dewiki') > does not refer to the same as 'wiki' in pywikipediabot and interwiki... > (strange...) > > But would also explain this 'inconsistency'... ;) > > Thanks and Greetings > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAk5dF68ACgkQAXWvBxzBrDCi8ACg0YV5W6uEDDY61xcgxB9qsKq8 > 8KsAn0kHTE6IwqZZxZI8Lb/9Dk6E7ciP > =dKzI > -----END PGP SIGNATURE----- > > _______________________________________________ > Pywikipedia-l mailing list > Pywikipedia-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l >

2 2

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

pywikibot September 2011