pywikibot August 2013

pywikibot@lists.wikimedia.org

21 participants
38 discussions

[Pywikipedia-l] Repo structure
by Dr. Trigon 29 Aug '13

29 Aug '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello all As I was creating new repos, I did a look at [1]: pywikibot pywikibot/compat pywikibot/core pywikibot/externals pywikibot/externals/httplib2 pywikibot/i18n pywikibot/sf-export pywikibot/spelling [1] https://gerrit.wikimedia.org/r/#/admin/projects/?filter=pywikibot So in my opinion WE HAVE TO BE VERY CAREFUL NOT TO MESS UP THE CLEAN STRUCTURE here! At the moment everything is still ok - but we might have to write down some rules like e.g. external code belong to pywikibot/externals/... all other code is maintained by us. (By the way how is this supposed to work? *wonder*) I also requested a personal repo for me and my bot called pywikibot/bots/drtrigonbot I would strongly encourage or even force to use this or a similar repo structure in order to keep everything nice and clean. We just have to agree on something BEFORE we create a mess. The structure given in [2] is e.g. out-dated now and needs and update. [2] https://www.mediawiki.org/wiki/Git/Conversion/pywikipedia Greetings DrTrigon -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlIczWgACgkQAXWvBxzBrDDw3gCfbyZY2hiWa8692AWpLmChkrXU 1poAn1x9SHMEyJDtA014EV0eqqtqb5yz =ZAmR -----END PGP SIGNATURE-----

1 1

[Pywikipedia-l] BeautifulSoup
by Amir Ladsgroup 28 Aug '13

28 Aug '13

Hello folks We had some discussions in the IRC channel about BS. There are so many problems about using BS in different systems. So many people don't want to run patch.exe and so many people are using servers for running bots and don't have permission to do that. My suggestion is using the old system (an internal folder as an external like i18n but I don't think we need to make another external) but if you think we can do something else I'll be glad to discuss Please keep in mind compatibility is one of the most important issues of PWB Best -- Amir

9 27

[Pywikipedia-l] Create additional repositories
by Dr. Trigon 27 Aug '13

27 Aug '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello all! Hello amir! I have created and maintained some other additional SVN repositiories on TS in the past [1]. [1] https://fisheye.toolserver.org/browse/drtrigon/externals Since it is not possible to include SVN repos into GIT as submodule (or else?) I fear I have to convert them (by svn2git?) to GIT repos in order to re-integrate them again into pywikibot/compat. Would it be possible to create those repos on gerrit in pywikibot too? E.g.: pywikibot/haartraining pywikibot/opencv pywikibot/pycolorname pywikibot/archive(s) (for .tar.gz files - other suggestions welcome) in order to copy them or pywikibot/other or pywikibot/thirdparty as general repos to create several such additional codes (also for future). Though later is not optimal since it can be included just as 1 submodule only. Thanks a lot for your thought and help Greetins DrTrigon -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlH4420ACgkQAXWvBxzBrDC5KwCghIA4WXfFogvMW9DZA69bfP47 Yg8An2YpPRTR+ANrqzApHw2cbPRMHtmU =BYOr -----END PGP SIGNATURE-----

3 10

Re: [Pywikipedia-l] Time lag between wikidata writes
by Chinmay Naik 23 Aug '13

23 Aug '13

Hi, I am Chinmay Naik, a Gsoc intern and operator of ProteinBoxBot(runs on pywikipedia - formerly rewrite branch). When i run the bot, there exists a sleep time of around 10 s (7s average) between two successive writes. The bot will handle around 40,000 wikidata items corresponding to gene wiki articles from http://en.wikipedia.org/wiki/Category:Human_proteins . Also , i will be uploading around 30 (claims + sources) for each wikidata item. If u take a look at the recent edits ( https://www.wikidata.org/w/index.php?title=Special:Contributions/ProteinBox…), it takes around 5 mins to completely write all claims+sources to each wikidata item. This is a very large time lag and will hugely affect deployment. Is there any way to reduce this sleep time? Any pointers on this would be helpful. Thanks, Chinmay

4 4

[Pywikipedia-l] ascii encode error in upload.py
by AugurNZ 22 Aug '13

22 Aug '13

Hi, I was pointed here from #pywikipediabot on Freenode IRC. I am using Pywikipediabot (trunk) revision 11781 under Ubuntu 12.04. I'm having no success using the upload.py script. It keeps failing with the following error message... ---------- augur@ubuntu:/home/augur/pywikipedia# python upload.py -family:tropical -keep -filename:Pwbicon.png -noverify Pwbicon.png "The Pywikipediabot icon" No handlers could be found for logger "pywiki" Reading file Pwbicon.png The suggested description is: The Pywikipediabot icon Uploading file to tropical:en via API.... ERROR: UnicodeDecodeError: 'ascii' codec can't decode byte 0x89 in position 654: ordinal not in range(128) WARNING: Could not open 'http://www.tropicalwikis.com/w/api.php'. Maybe the server or your connection is down. Retrying in 1 minutes... ---------- I can perform other actions on www.tropicalwikis.com with my bot, such as using login.py, replace.py etc. I have also tried using a different -family: and logging in to www.thetestwiki.org (where I also have a valid bot account) and tried uploading the image file there, with the same result. I'm not a python expert, in fact I'm barely a python novice. I came across this information which may be relevant though... http://wiki.python.org/moin/UnicodeDecodeError Thanks. AugurNZ ps. sorry if this is a repeat of a previous message. I wasn't registered on the list when I first sent this, but I am now.

6 7

Re: [Pywikipedia-l] ascii encode error in upload.py
by AugurNZ 21 Aug '13

21 Aug '13

The file I'm trying to upload is the *.png thumbnail of the Pywikipediabot logo from this page... http://www.mediawiki.org/wiki/File:Pwb_icon.svg The thumbnail is called http://upload.wikimedia.org/wikipedia/commons/thumb/d/d2/Pwb_icon.svg/128px… but I have shortened the filename, as per the example below, to "Pwbicon.png" (without the quotes). So the filename should not contain any non-ASCII chars. As for the file itself, I don't know, and wouldn't even know how to find out. ;) > For me it seems that you have an HTTP error on uploading and the error > message contains a non-ASCII character. This is an old problem, nobody > could solve it by this time. > Does your uploaded file contain none-ASCII chars? > > > > 2013/8/21 AugurNZ <augurnz(a)orcon.net.nz> > >> Hi, I was pointed here from #pywikipediabot on Freenode IRC. I am using >> Pywikipediabot (trunk) revision 11781 under Ubuntu 12.04. I'm having no >> success using the upload.py script. It keeps failing with the following >> error message... >> >> ---------- >> augur@ubuntu:/home/augur/pywikipedia# python upload.py -family:tropical >> -keep -filename:Pwbicon.png -noverify Pwbicon.png "The Pywikipediabot >> icon" >> No handlers could be found for logger "pywiki" >> Reading file Pwbicon.png >> The suggested description is: >> The Pywikipediabot icon >> Uploading file to tropical:en via API.... >> ERROR: UnicodeDecodeError: 'ascii' codec can't decode byte 0x89 in >> position 654: ordinal not in range(128) >> WARNING: Could not open 'http://www.tropicalwikis.com/w/api.php'. Maybe >> the server or >> your connection is down. Retrying in 1 minutes... >> ---------- >> >> I can perform other actions on www.tropicalwikis.com with my bot, such >> as >> using login.py, replace.py etc. I have also tried using a different >> -family: and logging in to www.thetestwiki.org (where I also have a >> valid >> bot account) and tried uploading the image file there, with the same >> result. >> >> I'm not a python expert, in fact I'm barely a python novice. I came >> across >> this information which may be relevant though... >> http://wiki.python.org/moin/UnicodeDecodeError >> >> Thanks. >> AugurNZ >> >> ps. sorry if this is a repeat of a previous message. I wasn't registered >> on the list when I first sent this, but I am now. >> >> >> _______________________________________________ >> Pywikipedia-l mailing list >> Pywikipedia-l(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l >> > > > > -- > Bináris >

2 4

[Pywikipedia-l] A basic pywikipedia version
by Alex Brollo 20 Aug '13

20 Aug '13

While first comparing piwikipedia and wikitools, I have been astonished to see how much light and simple are scripts to run a bot and "do basic things", ti: reading and writing pages, access to API, and so on. I'll try - consider that I'm far from skilled or "professional" - to prune off from pywikipedia any script but the basic ones, covered by wikitools, just to have some personal fun; but I encourage developers to do something similar, writing a real "pywikipedia core" containing nothing but really needed modules to run basics with personal scripts. I presume that the result will exclude 95% or more of present pwykipedia content. Alex brollo

2 2

Re: [Pywikipedia-l] About creating a bot
by Maarten Dammers 19 Aug '13

19 Aug '13

Op 20-8-2013 3:01, Chi Hong Lee schreef: > I hope that someone may help to teach me how to create a bot You could start by not sending the email to -owner address, but to the actual list and to read the manual at https://www.mediawiki.org/wiki/Pywikipediabot :-) Maarten

1 0

Re: [Pywikipedia-l] Question about externals and patches
by info＠gno.de 19 Aug '13

19 Aug '13

There is a remaining bug. Try the following e.g. from idle: >>> import Wikipedia you are asked to install patch.exe. If you decline this request you are asked a second time. If declined a second time the script raises a fatal error while importing beautifullsoup. Beautifullsoup is a central part if the screen scraping compat framework and we should not prohibit using the framework for a cosmetic patch. Please give us back that part as e.g. externals or a library script. Anyway it is not a good idea to ask for installation (twice) and fall into a fatal error if you answer is NO. There should be alternatives to patching the working copy (e.g. nightly dump etc.) Greetings xqt ----- Ursprüngliche Nachricht ----- Von: Dr. Trigon Gesendet: 03.07.2013 20:14 An: pywikipedia-l(a)lists.wikimedia.org Betreff: Re: [Pywikipedia-l] Question about externals and patches -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello to all (again)! Quit some time has elapsed since my first mail. Today I tested successfully 'patch.exe' [1] in win and in linux wine. This is another one than mentioned before, but/and as I can see works quite well even with LF (unix-style) line endings. [1] http://gnuwin32.sourceforge.net/packages/patch.htm Since I will commit these changes now the time has come to test it on other machines also. After that was successful too I will remove BeautifulSoup.py from externals because it will be downloaded and patched automatically. In order to test this on your windows machine you can e.g.: 1.) create a directory, e.g. '[...]/Desktop/patch-test' 2.) download and extract 'patch.exe' from http://downloads.sourceforge.net/project/gnuwin32/patch/2.5.9-7/patch-2.5.9… to this directory 3.) download and extract 'BeautifulSoup.py' from https://pypi.python.org/packages/source/B/BeautifulSoup/BeautifulSoup-3.2.0… to this directory 4.) download the patch 'patch-BeautifulSoup' from http://svn.wikimedia.org/viewvc/pywikipedia/trunk/pywikipedia/externals/pat… to this directory 5.) open 'cmd', cd to '[...]/Desktop/patch-test' and enter "patch.exe - -p0 -d [...]/Desktop/patch-test/ -i [...]/Desktop/patch-test/patch-BeautifulSoup --binary" 6.) check the output for error messages and compare the resulting 'BeautifulSoup.py' to the one in your pywikipedia/externals directory, they should be exactly the same! IF NOT AND/OR ERRORS OCCURED PLEASE REPORT THIS TO ME! Thanks a lot to everybody helping and Greetings DrTrigon On 25.05.2013 11:40, Dr. Trigon wrote: > Hello all! > > Following issue: I am up re-organizing the whole "externals" part > in trunk as you might have recognized already. In fact this is done > now with the single exception of a generic patching system, e.g. > needed for BeautifulSoup.py. (As usual) this is no problem under > linux, but becomes a major issue under win. > > The mechanism I want to use is the well known diff-patch duo. > Therefore a "patch" executable/binary (OR python script) is needed > (for every OS). While this is kind of "built-in" in linux, win > needs extra-attention. This is what I found so far: > > * http://sourceforge.net/projects/unxutils/ The executables do only > depend on the Microsoft C-runtime (msvcrt.dll) and not on an > emulation layer like that provided by Cygwin tools - windows ONLY > not multi OS > > * http://python-patch.googlecode.com/svn/trunk/patch.py Python > script therefore multi OS - but does not support the full > diff-patch "command set" e.g. cannot create new files > > * https://code.google.com/p/google-diff-match-patch/ Not a > command-line tool like "patch" but a python library/module. Multi > OS. > > So I am stuck here and need some further knowledge, experience and > personal preferences from your side in order to make a good > decicion. > > In my opinion we should also keep further os (than just linux, > mac, win) in our mind, becuase they are very close to what we > already have too. > > Thanks for any help and Greetings DrTrigon > > _______________________________________________ Pywikipedia-l > mailing list Pywikipedia-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHUacQACgkQAXWvBxzBrDCwOACgst3qYcS1YTqZvUvtmy8+2TOD jVkAnjBYVcMCNLSY5zBBFVG7LGg9pTDD =fDqI -----END PGP SIGNATURE----- _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

5 13

[Pywikipedia-l] Making installation of PWB a little bit more user-freindly
by Amir Ladsgroup 18 Aug '13

18 Aug '13

As you probably know We had a workshop http://wikimania2013.wikimedia.org/wiki/DevCamp/Schedule on Wikimania's DevCamp about using PWB for people who are interested. It went very well thanks Legoktm (a BIG THANK YOU) but It was a very big problem that installation of PWB is not user-friendly (and It's worse when people try to install rewrite instead of trunk) The main cause as far as we faced during the workshop (on different computers) is dependencies. Maarten (multichill) has suggested we change setup.py and generate_user_config.py and make a auto-generated list of needed dependencies (based on OS or other things) and ask user when he/she wants to install that which one you need! and install it right away. another suggestion (my idea) is an example: When someone doesn't install i18n submodule. Codes breaks (error that says "import i18n, there is no module named "i18n"" or something like that) but we have to catch this error and ask a question and ask do you want to install i18n submodule? and maybe a user doesn't want to install it (just wants to run it in English WP) We have to let the user run the code. Best -- Amir

4 12

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

pywikibot August 2013