Hello folks We had some discussions in the IRC channel about BS. There are so many problems about using BS in different systems. So many people don't want to run patch.exe and so many people are using servers for running bots and don't have permission to do that.
My suggestion is using the old system (an internal folder as an external like i18n but I don't think we need to make another external) but if you think we can do something else I'll be glad to discuss
Please keep in mind compatibility is one of the most important issues of PWB
Best
2013/8/19 Amir Ladsgroup ladsgroup@gmail.com
Please keep in mind compatibility is one of the most important issues of PWB
Maxi support for this thought. Thank you.
I didn't know at all was Beautifulsoup does; reading doc for the first time, I see that's a xml/parser. Is it superior to pyquery? When I needed xml parsing, I found pyquery simple and so similar to jQuery that I didn't found any problem while using it.
Alex brollo
2013/8/19 Bináris wikiposta@gmail.com
2013/8/19 Amir Ladsgroup ladsgroup@gmail.com
Please keep in mind compatibility is one of the most important issues of PWB
Maxi support for this thought. Thank you.
Pywikipedia-l mailing list Pywikipedia-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
Hi Alex,
Op 19-8-2013 15:08, Alex Brollo schreef:
I didn't know at all was Beautifulsoup does; reading doc for the first time, I see that's a xml/parser. Is it superior to pyquery? When I needed xml parsing, I found pyquery simple and so similar to jQuery that I didn't found any problem while using it.
Beautifulsoup takes crappy formatted html/xml/*ML, doesn't choke and outputs it properly formatted.
Can someone please tell me why we need to patch Beautifulsou? Can't we just fix it upstream?
Maarten
2013/8/19 Maarten Dammers maarten@mdammers.nl
Hi Alex,
Op 19-8-2013 15:08, Alex Brollo schreef:
I didn't know at all was Beautifulsoup does; reading doc for the first
time, I see that's a xml/parser. Is it superior to pyquery? When I needed xml parsing, I found pyquery simple and so similar to jQuery that I didn't found any problem while using it.
Beautifulsoup takes crappy formatted html/xml/*ML, doesn't choke and outputs it properly formatted.
Can someone please tell me why we need to patch Beautifulsou? Can't we just fix it upstream?
Maarten
Thanks Maarten, so BS is one more module that I don't need. Really I use pywikipedia in a personal, exotic way - running only some basic scripts, an approach similar to wikitools. I'll try to remove from my compat folder any unused script - I suppose that the result will be surprising. :-) PyQuery is interesting since it's so similar to jQuery - a big advantage for me; but I see that BS is something deeply different.
Alex
2013/8/20 Alex Brollo alex.brollo@gmail.com
Thanks Maarten, so BS is one more module that I don't need. Really I use pywikipedia in a personal, exotic way - running only some basic scripts, an approach similar to wikitools.
This is an important point. Now I found the real words for this situation. Let me quote Douglas Adams' *The Restaurant at the End of the Universe:*
"Transtellar Cruise Lines would like to apologize to passengers for the continuing delay to this flight. We are currently awaiting the loading of our complement of small lemon-soaked paper napkins for your comfort, refreshment and hygiene during the journey. Meanwhile we thank you for your patience. The cabin crew will shortly be serving coffee and biscuits again." [...] "Yes," said the voice again, "there has been a delay. Passengers are to be kept temporarily in suspended animation, for their comfort and convenience. Coffee and biscuits are being served every year, after which passengers are returned to suspended animation for their continued comfort and convenience. Departure will take place when the flight stores are complete. We apologize for the delay."
This is how he decribes a 900 year delay of a regular spaceflight. Sometimes I feel that forcing users of Pywikibot to make exotic updates for BS when they only want to use basic tools is this kind of case. Excuse me for that. I have already run this exe, this philosophical detour is for future only.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
"Transtellar Cruise Lines would like to apologize to passengers for the continuing delay to this flight. We are currently awaiting the loading of our complement of small lemon-soaked paper napkins for your comfort, refreshment and hygiene during the journey. Meanwhile we thank you for your patience. The cabin crew will shortly be serving coffee and biscuits again." [...] "Yes," said the voice again, "there has been a delay. Passengers are to be kept temporarily in suspended animation, for their comfort and convenience. Coffee and biscuits are being served every year, after which passengers are returned to suspended animation for their continued comfort and convenience. Departure will take place when the flight stores are complete. We apologize for the delay."
There are 2 others, namely;
"[...] to summarize the summary of the summary: People are a problem."
and
"We applogize for the incovienience."
(in fact there a millions of billions of them... I REALLY miss DNA... ;)) anyway - back to business...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Yes BS is a a very useful tool indeed. It is used in several places, also some of my scripts - it is often a fast, simple and easy solution.
We had some discussions in the IRC channel about BS. There are so many problems about using BS in different systems. So many people don't want to run patch.exe and so many people are using servers for running bots and don't have permission to do that.
I understand that we had alread several discussion about it indeed. A hard thing for me is to help or solve the situation because I never got any feadback WHY running patch.exe is such a big issue? I just always heard "I don't like it." - ok, but WHY? It is not dangerous at all AND it is a standard tool, in fact it is the base on which git works...
My suggestion is using the old system (an internal folder as an external like i18n but I don't think we need to make another external) but if you think we can do something else I'll be glad to discuss
What do you exactly mean with "old system"? Do you want to create a repo 'externals' and just drop BS there in order to use as submodule? Or what was the concept?
Please keep in mind compatibility is one of the most important issues of PWB
Indeed! The current solution (using patch) is compatible with linux/mac (since patch is in the OS distro there) and with win through patch.exe. In fact win in the main problem here... ;)) Any other important OS that I missed? (and does not have patch like all/most? unix)
And may be I should also mention that not only BS is an issue in this context, there are some other modules also - I sent the link with the list in a previous mail.
I would be VERY happy to finally find a solution here!
Greetings and all the best! DrTrigon
On Mon, Aug 19, 2013 at 4:34 PM, Dr. Trigon dr.trigon@surfeu.ch wrote:
We had some discussions in the IRC channel about BS. There are so many problems about using BS in different systems. So many people don't want to run patch.exe and so many people are using servers for running bots and don't have permission to do that.
I understand that we had alread several discussion about it indeed. A hard thing for me is to help or solve the situation because I never got any feadback WHY running patch.exe is such a big issue? I just always heard "I don't like it." - ok, but WHY? It is not dangerous at all AND it is a standard tool, in fact it is the base on which git works...
One thing that people probably don't like is that it's a completely non-standard workflow. Standard ways of modifying BeautifulSoup include: 1) fixing the upstream distribution, 2) subclassing & overiding behavior, or 3) creating a private fork. What exactly are you patching? The one patch that I found pointed to in the list archives
Bundling dependencies (ie httplib2 in externals) is pretty non-standard too. Why not declare the dependencies and let `pip install` resolve them at installation time. Dependency isolation and private non-system installs can be achieved using virtualenv or a similar tool.
That's how most other Python projects work. Is there a reason it won't work here?
Tom
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I will make it more general since it does not really apply to BS anymore - see my last mail.
One thing that people probably don't like is that it's a completely non-standard workflow. Standard ways of modifying BeautifulSoup include: 1) fixing the upstream distribution, 2) subclassing & overiding behavior, or 3) creating a private fork. What exactly are you patching? The one patch that I found pointed to in the list archives
So regarding the patches, most of them are little modifications that are needed in order to use them locally from externals folder without the need to install them into the full system - because if there would have been installed there (e.g. by OS package management, or user interaction), there would be no need to install them. As mentioned already several times this was just the simplest and minimal intrusive solution I could think of - and should be easy to understand.
So 1) and 3) would be nice solutions but create too much workload for me to maintain. 2) is what I actually try to do - but it need some changes for some packages.
Bundling dependencies (ie httplib2 in externals) is pretty non-standard too. Why not declare the dependencies and let `pip install` resolve them at installation time. Dependency isolation and private non-system installs can be achieved using virtualenv or a similar tool.
Regarding pip - we are talking about compat which does not need to be installed - so pip and installation in general would be something completely new. Despite the fact that core has also a mode in which it can be used without installing as module. Regarding virtualenv - I am a newbee here - can you explain of what use it could be in this case?
That's how most other Python projects work. Is there a reason it won't work here?
In fact - I oriented myself at another python project too "VisTrails" but this might not have been the best decision. Was the first python code I found that was able to install it's own deps. But it did not need to patch some of them... ;)
Greetings and thanks!
2013/8/19 Dr. Trigon dr.trigon@surfeu.ch
I understand that we had alread several discussion about it indeed. A hard thing for me is to help or solve the situation because I never got any feadback WHY running patch.exe is such a big issue? I just always heard "I don't like it." - ok, but WHY?
But I already told you. It is an exe in an open source project, with no explanation why it is needed and where the source code may be seen. You just get into your face: "run this exe!!!" This is enough to not like it.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
There is a very long message in before that now there will be some additional code installed. And then you get asked whether you want to do this or not. So I do not understand... (if there is a message missing this is a bug) ...what do you have in mind? Could you formulate a message for me please? Or is the actual message just unclear?
And the exe is also from an open source project, the source is freely available. Or does this not matter?!
Please help me to understand your wishes. My are, taht until now I still need patch and thus I added patch.exe for compatibility with windows users.
At the moment patch.exe should not be executed for BS anymore. So as I see it should not get executed anymore for more ore less all users, except those running my and may be some other (exotic) bot scripts.
Thanks for your feedback and Greetings
On 19.08.2013 23:28, Bináris wrote:
2013/8/19 Dr. Trigon <dr.trigon@surfeu.ch mailto:dr.trigon@surfeu.ch>
I understand that we had alread several discussion about it indeed. A hard thing for me is to help or solve the situation because I never got any feadback WHY running patch.exe is such a big issue? I just always heard "I don't like it." - ok, but WHY?
But I already told you. It is an exe in an open source project, with no explanation why it is needed and where the source code may be seen. You just get into your face: "run this exe!!!" This is enough to not like it.
_______________________________________________ Pywikipedia-l mailing list Pywikipedia-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
2013/8/19 Dr. Trigon dr.trigon@surfeu.ch
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
There is a very long message in before that now there will be some additional code installed. And then you get asked whether you want to do this or not. So I do not understand... (if there is a message missing this is a bug) ...what do you have in mind? Could you formulate a message for me please?
One more line: "See http://... for details."
Or is the actual message just unclear?
Obviously it is in the above mentioned questions.
And the exe is also from an open source project, the source is freely available. Or does this not matter?!
But the user does not get the proper information about that. How could they know it? Proper information would exceed the limits of the screen, that's why I think a web page as an appropriate solution. There this can be written (what is the source of the exe, what it exactly does, where is the source etc.).
An exe is not necessarily of the devil, but a more detailed explanation is a friendly thing instead "you have to run this exe, yes or no?"
B.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 20.08.2013 03:40, Bináris wrote:
Ok that sound like a good solution - in fact we should enhance the "Installation Manual" [1] and explain this there. The link should point to the manual, what do you think? Some time back you had something similar for transition to py2.7.? do you know what URL you had in use there?
[1] http://www.mediawiki.org/wiki/Manual:Pywikipediabot/Installation#Dependencie...
What about a README file instead of an URL? Was there a final decision made whether to docu things locally in the repo (REDAME, doxygen, ...) vs. online www.mediawiki.org ?
Just trust me... ;)) ...no - honestly you should not... ;))
Exactly! Good thing... so we will manage to converge against a good solution sometime! Cool!
Greetings DrTrigon
On Aug 20, 2013, at 5:15 AM, "Dr. Trigon" dr.trigon@surfeu.ch wrote:
What about a README file instead of an URL? Was there a final decision made whether to docu things locally in the repo (REDAME, doxygen, ...) vs. online www.mediawiki.org ?
The README file for pywikipedia IS A BLOODY JOKE!!!!
It provides ZERO information.
I've been building code on the net since it was the ARPAnet and the overall structure of pywikipedia simply flies in the face of conventions.
Were it not for the CONTENTS file, anyone downloading the code for the first time would be utterly lost.
The "docs" directory is a similar joke. If the instructions contained within it are so straight forward, why doesn't the "Installation script" do it automatically? ... oh, that's right, there IS NO installation script!!!
I first started trying to use pywikipedia on a private MediaWiki ... where "private" according to your definitions means NOT part of wikipedia.com. I downloaded it from the SVN at the end of June, just before you began converting to GIT.
Being used to things like Perl and MacPorts, getting pywikipedia working has been a painful experience.
I am a retired Unix SysAdmin, however, when I retired in 2003, Python was not yet a widely used language (and git had not yet been invented).
Your installation instructions still do not acknowledge the fact that "generate_family_file.py" is BROKEN.... that you CANNOT have a "user-config.py" file if you want to run it. "... Before you attempt to create one using the instructions below execute python generate_family_file.py. If it succeeds, ..." It CAN NOT succeed if you have followed the instructions up to that point, especially the step immediately before it where you are instructed to build "user-config.py" http://www.mediawiki.org/wiki/Manual:Pywikipediabot/Use_on_third-party_wikis
The page: http://www.mediawiki.org/wiki/Manual:Pywikipediabot/Installation is simply an instruction on using GIT. I've already updated the page: http://www.mediawiki.org/wiki/Manual:Pywikipediabot/Mac to reflect "Mountain Lion," and to reflect this problem with the instructions.
I realize that I'm in a severe minority here, being both a Unix and OSX (i.e. Mac) person in a Windows based world, but I can live with the Windows distortions as I have been doing for the past 30 years. That is one reason why I have not had anything to say perviously... that plus the fact that I just barely have pywikipedia working for the MediaWiki I'm working on -- LOTRO-wiki.com.
But if you are going to package this software to look like a "correct" (i.e. conventional) Internet download, you need to fix things like the README file!
I realize also that from a developers point of view, maintaining a WIKI is a PAIN !!!! it is much easier to stay inside one's text editor (or whatever tool is being used to write and modify the code with) and to update things like the "CONTENTS" and "README" files. But that is a choice you have made... transferring your documentation from "standard" internet style to Wiki style. However, that choice implies much more work is necessary to keep the two in sync.
One last point -- since you have these assorted "configure" scripts which MUST be run before one can begin using the product -- why not have an "install" script, which automatically runs them as needed. ... of course to do that, you will first need to fix "generate_family_file.py".
(I'll update the OSX instructions once I figure out how to use GIT. Fortunately, Apple includes the GIT client as part of OSX. )
T.T.F.N. William H. Magill # iMac11,3 Core i7 [2.93GHz - 8 GB 1067MHz] OS X 10.8.4 # MacBook Pro4.1 Core 2 Duo [2.5GHz - 4GB 667] OS X 10.6.8 # Macmini6,1 Intel Core i5 [2.5 Ghz - 4GB 1600MHz] OS X 10.8.3
magill@icloud.com magill@mac.com whmagill@gmail.com
On 20/08/2013 16:07, William H. Magill wrote:
On Aug 20, 2013, at 5:15 AM, "Dr. Trigon" dr.trigon@surfeu.ch wrote:
What about a README file instead of an URL? Was there a final decision made whether to docu things locally in the repo (REDAME, doxygen, ...) vs. online www.mediawiki.org ?
The README file for pywikipedia IS A BLOODY JOKE!!!!
It provides ZERO information.
That point technically isn't true (it gives a sentence long description of what Pywikipediabot is), but I get what you're saying. It's not very helpful at all and we should change this. How, I'm not entirely sure, because as CONTENTS says, its a description for use with the nightlies, currently.
I've been building code on the net since it was the ARPAnet and the overall structure of pywikipedia simply flies in the face of conventions.
Were it not for the CONTENTS file, anyone downloading the code for the first time would be utterly lost.
The "docs" directory is a similar joke. If the instructions contained within it are so straight forward, why doesn't the "Installation script" do it automatically? ... oh, that's right, there IS NO installation script!!!
I first started trying to use pywikipedia on a private MediaWiki ... where "private" according to your definitions means NOT part of wikipedia.com. I downloaded it from the SVN at the end of June, just before you began converting to GIT.
Being used to things like Perl and MacPorts, getting pywikipedia working has been a painful experience.
As a Python and PHP person, I tend to say the same about anything that isn't Python or PHP!
I am a retired Unix SysAdmin, however, when I retired in 2003, Python was not yet a widely used language (and git had not yet been invented).
Your installation instructions still do not acknowledge the fact that "generate_family_file.py" is BROKEN.... that you CANNOT have a "user-config.py" file if you want to run it. "... Before you attempt to create one using the instructions below execute python generate_family_file.py. If it succeeds, ..." It CAN NOT succeed if you have followed the instructions up to that point, especially the step immediately before it where you are instructed to build "user-config.py" http://www.mediawiki.org/wiki/Manual:Pywikipediabot/Use_on_third-party_wikis
I've never had this problem, ever. I add new wikis to my Pywikipedia configuration all the time using generate_family_file.py without a problem.
The page: http://www.mediawiki.org/wiki/Manual:Pywikipediabot/Installation is simply an instruction on using GIT. I've already updated the page: http://www.mediawiki.org/wiki/Manual:Pywikipediabot/Mac to reflect "Mountain Lion," and to reflect this problem with the instructions.
I realize that I'm in a severe minority here, being both a Unix and OSX (i.e. Mac) person in a Windows based world, but I can live with the Windows distortions as I have been doing for the past 30 years. That is one reason why I have not had anything to say perviously... that plus the fact that I just barely have pywikipedia working for the MediaWiki I'm working on -- LOTRO-wiki.com.
A lot of F(L)OSS developers including Pywikipedia developers use *NIX as primary operating systems. There are bot runners that have success on both Windows and *NIX based systems.
But if you are going to package this software to look like a "correct" (i.e. conventional) Internet download, you need to fix things like the README file!
I realize also that from a developers point of view, maintaining a WIKI is a PAIN !!!! it is much easier to stay inside one's text editor (or whatever tool is being used to write and modify the code with) and to update things like the "CONTENTS" and "README" files. But that is a choice you have made... transferring your documentation from "standard" internet style to Wiki style. However, that choice implies much more work is necessary to keep the two in sync.
Maintaining any documentation is a pain, I actually find it easier to maintain a wiki however (I take no credit for the Pywikipedia documentation, it is something I tried to start some interest in cleaning up before, but nothing became of that), and it's rather fitting since this is a tool to be used with wikis.
One last point -- since you have these assorted "configure" scripts which MUST be run before one can begin using the product -- why not have an "install" script, which automatically runs them as needed. ... of course to do that, you will first need to fix "generate_family_file.py".
I wouldn't say that's really necessary, all you have to do is run generate_family_file.py and generate_user_files.py ... they are both pretty simple and quick. It's actually considerably easier than when I started using Pywikipedia, when you had to craft the family file by hand!
(I'll update the OSX instructions once I figure out how to use GIT. Fortunately, Apple includes the GIT client as part of OSX. )
T.T.F.N. William H. Magill # iMac11,3 Core i7 [2.93GHz - 8 GB 1067MHz] OS X 10.8.4 # MacBook Pro4.1 Core 2 Duo [2.5GHz - 4GB 667] OS X 10.6.8 # Macmini6,1 Intel Core i5 [2.5 Ghz - 4GB 1600MHz] OS X 10.8.3
magill@icloud.com magill@mac.com whmagill@gmail.com
Pywikipedia-l mailing list Pywikipedia-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
On Aug 20, 2013, at 11:49 AM, Lewis Cawte lewiscawte@googlemail.com wrote:
One last point -- since you have these assorted "configure" scripts which MUST be run before one can begin using the product -- why not have an "install" script, which automatically runs them as needed. ... of course to do that, you will first need to fix "generate_family_file.py".
I wouldn't say that's really necessary, all you have to do is run generate_family_file.py and generate_user_files.py ... they are both pretty simple and quick. It's actually considerably easier than when I started using Pywikipedia, when you had to craft the family file by hand!
Yes. if you do them in that order.... they work.
However, the Installation instructions tell you to run them in the reverse order; consequently they do not work -- because "generate_family_file.py" will fail if a "user-config.py" file exists. (Unless that has been changed in the past month.)
T.T.F.N. William H. Magill # iMac11,3 Core i7 [2.93GHz - 8 GB 1067MHz] OS X 10.8.4 # MacBook Pro4.1 Core 2 Duo [2.5GHz - 4GB 667] OS X 10.6.8 # Macmini6,1 Intel Core i5 [2.5 Ghz - 4GB 1600MHz] OS X 10.8.3
magill@icloud.com magill@mac.com whmagill@gmail.com
On 20/08/2013 19:19, William H. Magill wrote:
On Aug 20, 2013, at 11:49 AM, Lewis Cawte lewiscawte@googlemail.com wrote:
One last point -- since you have these assorted "configure" scripts which MUST be run before one can begin using the product -- why not have an "install" script, which automatically runs them as needed. ... of course to do that, you will first need to fix "generate_family_file.py".
I wouldn't say that's really necessary, all you have to do is run generate_family_file.py and generate_user_files.py ... they are both pretty simple and quick. It's actually considerably easier than when I started using Pywikipedia, when you had to craft the family file by hand!
Yes. if you do them in that order.... they work.
However, the Installation instructions tell you to run them in the reverse order; consequently they do not work -- because "generate_family_file.py" will fail if a "user-config.py" file exists. (Unless that has been changed in the past month.)
This is not true at all unless you've just found a very very weird bug. Generate_family_file.py works even if a user-config.py exists, and it has been that way since the script was created. I've been using it since it was first released, in all different versions.
T.T.F.N. William H. Magill # iMac11,3 Core i7 [2.93GHz - 8 GB 1067MHz] OS X 10.8.4 # MacBook Pro4.1 Core 2 Duo [2.5GHz - 4GB 667] OS X 10.6.8 # Macmini6,1 Intel Core i5 [2.5 Ghz - 4GB 1600MHz] OS X 10.8.3
magill@icloud.com magill@mac.com whmagill@gmail.com _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
On Aug 20, 2013, at 2:28 PM, Lewis Cawte lewiscawte@googlemail.com wrote:
On 20/08/2013 19:19, William H. Magill wrote:
On Aug 20, 2013, at 11:49 AM, Lewis Cawte lewiscawte@googlemail.com wrote:
One last point -- since you have these assorted "configure" scripts which MUST be run before one can begin using the product -- why not have an "install" script, which automatically runs them as needed. ... of course to do that, you will first need to fix "generate_family_file.py".
I wouldn't say that's really necessary, all you have to do is run generate_family_file.py and generate_user_files.py ... they are both pretty simple and quick. It's actually considerably easier than when I started using Pywikipedia, when you had to craft the family file by hand!
Yes. if you do them in that order.... they work.
However, the Installation instructions tell you to run them in the reverse order; consequently they do not work -- because "generate_family_file.py" will fail if a "user-config.py" file exists. (Unless that has been changed in the past month.)
This is not true at all unless you've just found a very very weird bug. Generate_family_file.py works even if a user-config.py exists, and it has been that way since the script was created. I've been using it since it was first released, in all different versions.
There is no version information in the distribution i.e. nothing in the README, or CONTENTS files. It was downloaded from the SVN in late June.
using the instructions found at: http://www.mediawiki.org/wiki/Manual:Pywikipediabot
======<cut here>======
python generate_family_file.py
Usage: generate_family_file.py <url> <short name> Example: generate_family_file.py http://www.mywiki.bogus/wiki/Main_Page mywiki This will create the file families/mywiki_family.py Please insert URL to wiki: http//www.lotro-wiki.com Please insert a short name (eg: freeciv): mywikitest Generating family file from http//www.lotro-wiki.com Traceback (most recent call last): File "generate_family_file.py", line 335, in <module> FamilyFileGenerator(*sys.argv[1:]).run() File "generate_family_file.py", line 69, in run w = Wiki(self.base_url) File "generate_family_file.py", line 266, in __init__ data = urlopen(fromurl).read() File "generate_family_file.py", line 29, in urlopen return urllib2.urlopen(req) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 386, in open protocol = req.get_type() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 248, in get_type raise ValueError, "unknown url type: %s" % self.__original ValueError: unknown url type: http//www.lotro-wiki.com ======<cut here>======
T.T.F.N. William H. Magill # iMac11,3 Core i7 [2.93GHz - 8 GB 1067MHz] OS X 10.8.4 # MacBook Pro4.1 Core 2 Duo [2.5GHz - 4GB 667] OS X 10.6.8 # Macmini6,1 Intel Core i5 [2.5 Ghz - 4GB 1600MHz] OS X 10.8.3
magill@icloud.com magill@mac.com whmagill@gmail.com
On Aug 20, 2013, at 4:18 PM, "William H. Magill" magill@icloud.com wrote:
On Aug 20, 2013, at 2:28 PM, Lewis Cawte lewiscawte@googlemail.com wrote:
On 20/08/2013 19:19, William H. Magill wrote:
On Aug 20, 2013, at 11:49 AM, Lewis Cawte lewiscawte@googlemail.com wrote:
One last point -- since you have these assorted "configure" scripts which MUST be run before one can begin using the product -- why not have an "install" script, which automatically runs them as needed. ... of course to do that, you will first need to fix "generate_family_file.py".
I wouldn't say that's really necessary, all you have to do is run generate_family_file.py and generate_user_files.py ... they are both pretty simple and quick. It's actually considerably easier than when I started using Pywikipedia, when you had to craft the family file by hand!
Yes. if you do them in that order.... they work.
However, the Installation instructions tell you to run them in the reverse order; consequently they do not work -- because "generate_family_file.py" will fail if a "user-config.py" file exists. (Unless that has been changed in the past month.)
This is not true at all unless you've just found a very very weird bug. Generate_family_file.py works even if a user-config.py exists, and it has been that way since the script was created. I've been using it since it was first released, in all different versions.
There is no version information in the distribution i.e. nothing in the README, or CONTENTS files. It was downloaded from the SVN in late June.
using the instructions found at: http://www.mediawiki.org/wiki/Manual:Pywikipediabot
======<cut here>======
python generate_family_file.py
Usage: generate_family_file.py <url> <short name> Example: generate_family_file.py http://www.mywiki.bogus/wiki/Main_Page mywiki This will create the file families/mywiki_family.py Please insert URL to wiki: http//www.lotro-wiki.com Please insert a short name (eg: freeciv): mywikitest Generating family file from http//www.lotro-wiki.com Traceback (most recent call last): File "generate_family_file.py", line 335, in <module> FamilyFileGenerator(*sys.argv[1:]).run() File "generate_family_file.py", line 69, in run w = Wiki(self.base_url) File "generate_family_file.py", line 266, in __init__ data = urlopen(fromurl).read() File "generate_family_file.py", line 29, in urlopen return urllib2.urlopen(req) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 386, in open protocol = req.get_type() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 248, in get_type raise ValueError, "unknown url type: %s" % self.__original ValueError: unknown url type: http//www.lotro-wiki.com ======<cut here>======
I should add: Also on page: http://www.mediawiki.org/wiki/Manual:Pywikipediabot/Quick_Start_Guide
8. Now you'll create a family file for your site. First see if it can be generated automatically. On the Shell prompt, type "python generate_family_file.py" and press enter. If you get an error and the file is not generated, then follow the example below.
T.T.F.N. William H. Magill # iMac11,3 Core i7 [2.93GHz - 8 GB 1067MHz] OS X 10.8.4 # MacBook Pro4.1 Core 2 Duo [2.5GHz - 4GB 667] OS X 10.6.8 # Macmini6,1 Intel Core i5 [2.5 Ghz - 4GB 1600MHz] OS X 10.8.3
magill@icloud.com magill@mac.com whmagill@gmail.com
On 20 August 2013 22:18, William H. Magill magill@icloud.com wrote:
Please insert URL to wiki: http//www.lotro-wiki.com
(...snip...)
ValueError: unknown url type: http//www.lotro-wiki.com
Try using http://www.lotro-wiki.com (note the colon) instead.
Merlijn
On Aug 20, 2013, at 4:43 PM, Merlijn van Deen valhallasw@arctus.nl wrote:
On 20 August 2013 22:18, William H. Magill magill@icloud.com wrote: Please insert URL to wiki: http//www.lotro-wiki.com
(...snip...)
ValueError: unknown url type: http//www.lotro-wiki.com
Try using http://www.lotro-wiki.com (note the colon) instead.
Duh... That is poor error processing! Definitely not "user-friendly." The traceback spew obscures the actual error message.
Not to mention "unknown url type" is not particularly descriptive of what is not a TYPE of URL, but the actual value being supplied.
I realize that is RFC correct, however is it even possible (i.,e. does it make sense) to use anything other than an "http" or "https" value? Clearly "gopher" or "news" won't work.
I'm guessing that is a standard Python library call, so there is not much this code can do with it. I.e. I don't know how or if you can eliminate the traceback.
On Aug 20, 2013, at 4:48 PM, Lewis Cawte lewiscawte@googlemail.com wrote:
I believe I've just spotted your problem:
"http//" isn't a valid protocol, its "http://". I've just tested your wiki in particular and it works for me, you could copy+paste in a URL to a page instead of hand typing it. :)
The instructions need to be more obvious that you can do that -- namely that the routine will "auto-truncate" the URL from a home page to the appropriate "parts".
Of course, that begs the question -- Any arbitrary page on the wiki, or only the home page?
Well, at least I can revise the Mac document so that the required information is more clear.... and what the "Spew" implies! I tended to wind up writing lots of documentation for Freshmen, who really did need to have everything explained to them :)
T.T.F.N. William H. Magill # iMac11,3 Core i7 [2.93GHz - 8 GB 1067MHz] OS X 10.8.4 # MacBook Pro4.1 Core 2 Duo [2.5GHz - 4GB 667] OS X 10.6.8 # Macmini6,1 Intel Core i5 [2.5 Ghz - 4GB 1600MHz] OS X 10.8.3
magill@icloud.com magill@mac.com whmagill@gmail.com
On 20/08/2013 21:18, William H. Magill wrote:
On Aug 20, 2013, at 2:28 PM, Lewis Cawte lewiscawte@googlemail.com wrote:
On 20/08/2013 19:19, William H. Magill wrote:
On Aug 20, 2013, at 11:49 AM, Lewis Cawte lewiscawte@googlemail.com wrote:
One last point -- since you have these assorted "configure" scripts which MUST be run before one can begin using the product -- why not have an "install" script, which automatically runs them as needed. ... of course to do that, you will first need to fix "generate_family_file.py".
I wouldn't say that's really necessary, all you have to do is run generate_family_file.py and generate_user_files.py ... they are both pretty simple and quick. It's actually considerably easier than when I started using Pywikipedia, when you had to craft the family file by hand!
Yes. if you do them in that order.... they work.
However, the Installation instructions tell you to run them in the reverse order; consequently they do not work -- because "generate_family_file.py" will fail if a "user-config.py" file exists. (Unless that has been changed in the past month.)
This is not true at all unless you've just found a very very weird bug. Generate_family_file.py works even if a user-config.py exists, and it has been that way since the script was created. I've been using it since it was first released, in all different versions.
There is no version information in the distribution i.e. nothing in the README, or CONTENTS files. It was downloaded from the SVN in late June.
using the instructions found at: http://www.mediawiki.org/wiki/Manual:Pywikipediabot
======<cut here>======
python generate_family_file.py
Usage: generate_family_file.py <url> <short name> Example: generate_family_file.py http://www.mywiki.bogus/wiki/Main_Page mywiki This will create the file families/mywiki_family.py Please insert URL to wiki: http//www.lotro-wiki.com Please insert a short name (eg: freeciv): mywikitest Generating family file from http//www.lotro-wiki.com Traceback (most recent call last): File "generate_family_file.py", line 335, in <module> FamilyFileGenerator(*sys.argv[1:]).run() File "generate_family_file.py", line 69, in run w = Wiki(self.base_url) File "generate_family_file.py", line 266, in __init__ data = urlopen(fromurl).read() File "generate_family_file.py", line 29, in urlopen return urllib2.urlopen(req) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 386, in open protocol = req.get_type() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 248, in get_type raise ValueError, "unknown url type: %s" % self.__original ValueError: unknown url type: http//www.lotro-wiki.com ======<cut here>======
I believe I've just spotted your problem: "http//" isn't a valid protocol, its "http://". I've just tested your wiki in particular and it works for me, you could copy+paste in a URL to a page instead of hand typing it. :)
T.T.F.N. William H. Magill # iMac11,3 Core i7 [2.93GHz - 8 GB 1067MHz] OS X 10.8.4 # MacBook Pro4.1 Core 2 Duo [2.5GHz - 4GB 667] OS X 10.6.8 # Macmini6,1 Intel Core i5 [2.5 Ghz - 4GB 1600MHz] OS X 10.8.3
magill@icloud.com magill@mac.com whmagill@gmail.com
Pywikipedia-l mailing list Pywikipedia-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 20.08.2013 22:18, William H. Magill wrote:
There is no version information in the distribution i.e. nothing in the README, or CONTENTS files. It was downloaded from the SVN in late June.
using the instructions found at: http://www.mediawiki.org/wiki/Manual:Pywikipediabot
To get version info, please use:
$ python version.py
On Aug 20, 2013, at 11:49 AM, Lewis Cawte lewiscawte@googlemail.com wrote:
I realize also that from a developers point of view, maintaining a WIKI is a PAIN !!!! it is much easier to stay inside one's text editor (or whatever tool is being used to write and modify the code with) and to update things like the "CONTENTS" and "README" files. But that is a choice you have made... transferring your documentation from "standard" internet style to Wiki style. However, that choice implies much more work is necessary to keep the two in sync.
Maintaining any documentation is a pain, I actually find it easier to maintain a wiki however (I take no credit for the Pywikipedia documentation, it is something I tried to start some interest in cleaning up before, but nothing became of that), and it's rather fitting since this is a tool to be used with wikis.
This is one point we absolutely agree on ... the only thing which has lower priority than documentation is "accounting" (both financial and tracking) -- They are both "after thoughts."
T.T.F.N. William H. Magill # iMac11,3 Core i7 [2.93GHz - 8 GB 1067MHz] OS X 10.8.4 # MacBook Pro4.1 Core 2 Duo [2.5GHz - 4GB 667] OS X 10.6.8 # Macmini6,1 Intel Core i5 [2.5 Ghz - 4GB 1600MHz] OS X 10.8.3
magill@icloud.com magill@mac.com whmagill@gmail.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Improving the docs and making them together with the code structure more compliant to usual conventions would be a good thing and any help (William?, Lewis? ;) is welcome... :)
Greetings DrTrigon
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 20.08.2013 03:40, Bináris wrote:
2013/8/19 Dr. Trigon <dr.trigon@surfeu.ch mailto:dr.trigon@surfeu.ch>
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
There is a very long message in before that now there will be some additional code installed. And then you get asked whether you want to do this or not. So I do not understand... (if there is a message missing this is a bug) ...what do you have in mind? Could you formulate a message for me please?
One more line: "See http://... for details."
Or is the actual message just unclear?
Obviously it is in the above mentioned questions.
Ok that sound like a good solution - in fact we should enhance the "Installation Manual" [1] and explain this there. The link should point to the manual, what do you think? Some time back you had something similar for transition to py2.7.? do you know what URL you had in use there?
[1] http://www.mediawiki.org/wiki/Manual:Pywikipediabot/Installation#Dependencie...
What about a README file instead of an URL? Was there a final decision made whether to docu things locally in the repo (REDAME, doxygen, ...) vs. online www.mediawiki.org ?
And the exe is also from an open source project, the source is freely available. Or does this not matter?!
But the user does not get the proper information about that. How could they know it?
Just trust me... ;)) ...no - honestly you should not... ;))
Proper information would exceed the limits of the screen, that's why I think a web page as an appropriate solution. There this can be written (what is the source of the exe, what it exactly does, where is the source etc.).
An exe is not necessarily of the devil, but a more detailed explanation is a friendly thing instead "you have to run this exe, yes or no?"
Exactly! Good thing... so we will manage to converge against a good solution sometime! Cool!
Greetings DrTrigon
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I completely forgot:
We had some discussions in the IRC channel about BS. There are so many problems about using BS in different systems. So many people don't want to run patch.exe and so many people are using servers for running bots and don't have permission to do that.
In fact in the most recent version patch.exe is not executed anymore for BS. For other modules yes but not for BS - I removed this recently.
And; no step is done before having asked the usere for explicit permission - in fact there were already complains about to many questions. May be we need a config variables or else...
Anyway we have to solve this disagreements... ;)
Greetings
pywikipedia-l@lists.wikimedia.org