I found the cause. I have several functions that are used by multiple scripts, so I have those in a custom module that each of my scripts imports. I recently added a subclass of catlib.Category to that module for use in a new script that will only run a couple times per month. To do that, I had to import pywikibot and catlib to that module, and it was the addition of those imports that broke things.

The reason is that that module also contained a subclass of StringIO that handles mixed str/unicode input better, and therefore I had to import it before I could redirect stdout and stderr to an instance of that class. In performing that import, the code in the module was run, which caused pywikibot to be imported, and pywikibot assigned the then-current stdout and stderr to pywikibot.ui. The subsequent redirection of stdout and stderr was then not reflected in pywikibot.ui, resulting in pywikibot.output() sending everything to the console instead of my StringIO subclass instance.

To fix it, I copied my StringIO subclass into one of my scripts proper, and made sure to redirect stdout and stderr before importing pywikibot or any other module that would import pywikibot. Subsequent tests successfully sent all output to my email.

Moral of the story: if you're going to redirect stdout and stderr by simply importing sys and reassigning sys.stdout and sys.stderr to something else, make sure to do it BEFORE you import pywikibot or any module that would import pywikibot. (Although it does kinda seem like a bug that pywikibot doesn't respond to future changes to sys.stdout and sys.stderr; maybe the output function in the interface class should check whether either of those has changed, and adjust its setup as needed?)

Jonathan Goble


On Tue, Jun 24, 2014 at 5:25 AM, Merlijn van Deen <valhallasw@arctus.nl> wrote:
Could you provide a minimal example that would allow us to reproduce the issue? With core, the following works for me (I don't have compat installed):


import sys, StringIO
oldstderr = sys.stderr
oldstdout = sys.stdout

sys.stderr = sys.stdout = StringIO.StringIO()
sys.stderr.fileno = lambda: 10

import pywikibot
oldstdout.write("Before output\n")
pywikibot.output('test')

oldstdout.write("After output; this is in our buffer:\n")
sys.stderr.seek(0)
oldstdout.write(repr(sys.stderr.read()))


$ PYWIKIBOT2_DIR=. python test.py
Before output
After output; this is in our buffer:
'WARNING: Running on Windows and transliteration_target is not set.\nPlease see
\n'



Nonetheless, I would suggest to use normal shell redirection techniques, instead of monkey-patching python's sys module. If you want to use python, the following should work (untested):

import subprocess
stdout, stderr = subprocess.Popen(["c:\python27\python", "pwb.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
# do something with stdout and stderr

I *think* it should also be possible to do this with the logging module, but I'm not sure how to do that in a quick-and-easy way.

Merlijn

On 24 June 2014 09:52, Jonathan Goble <jcgoble3@gmail.com> wrote:
When the error began occurring, I was using the Pywikibot nightly dump from February 25 of this year. It continues to occur with the nightly dump from yesterday (June 23). I just tried the fileno = lambda: 10 trick and it had no effect.

As for core, it's been four months since I last tried it so I don't recall the issues. It's past bedtime for me right now, but I'll look into giving it another try within the next day or two when I have some time.


On Tue, Jun 24, 2014 at 3:17 AM, Merlijn van Deen <valhallasw@arctus.nl> wrote:
On 24 June 2014 06:17, Jonathan Goble <jcgoble3@gmail.com> wrote:

I've given up trying to solve a bug that popped up in my scripts a couple days ago. I run a bot for Wookieepedia, over at Wikia, and run three simple scripts on a daily basis. They are set up to run automatically through Windows Task Scheduler. Since they run automatically, they run in the background through pythonw.exe, i.e. without a console, and therefore I need a means of getting the output. My solution for the past two months has been to redirect sys.stdout and sys.stderr to the same StringIO() instance, then at the end call getvalue() on that and email it to myself.

Could it be you haven't updated in a few years? It sounds like it's related to a feature that I added two or three years ago, that allowed Windows users to get full unicode out- and input. However, that does mean sys.stdin and sys.stdout are no longer being used. We /do/ check whether the user has redirected the output using normal shell redirection, but your method doesn't do that.

There are a two options I can think of.

  - trick the code into thinking you're doing 'regular' redirection by adding a fileno function to your streams:

    you'd need something like x = StringIO.StringIO(); x.fileno = lambda: 10
    and the same for stdout.

(By the way, the answer is NOT "switch to core". I have tried to get core to run on my system and failed miserably after two hours of repeated attempts without even getting it to talk to the wiki. Compat worked perfectly on the first try. Until such time as core can be installed by a beginner, it is not for me.)

I would appreciate it if you could clarify what the issues were you ran into.

Merlijn 


_______________________________________________
Pywikipedia-l mailing list
Pywikipedia-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l



_______________________________________________
Pywikipedia-l mailing list
Pywikipedia-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l



_______________________________________________
Pywikipedia-l mailing list
Pywikipedia-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l