I found the cause. I have several functions that
are used by multiple
scripts, so I have those in a custom module that each of my scripts
imports. I recently added a subclass of catlib.Category to that module for
use in a new script that will only run a couple times per month. To do
that, I had to import pywikibot and catlib to that module, and it was the
addition of those imports that broke things.
The reason is that that module also contained a subclass of StringIO that
handles mixed str/unicode input better, and therefore I had to import it
before I could redirect stdout and stderr to an instance of that class. In
performing that import, the code in the module was run, which caused
pywikibot to be imported, and pywikibot assigned the then-current stdout
and stderr to pywikibot.ui. The subsequent redirection of stdout and stderr
was then not reflected in pywikibot.ui, resulting in pywikibot.output()
sending everything to the console instead of my StringIO subclass instance.
To fix it, I copied my StringIO subclass into one of my scripts proper,
and made sure to redirect stdout and stderr before importing pywikibot or
any other module that would import pywikibot. Subsequent tests successfully
sent all output to my email.
Moral of the story: if you're going to redirect stdout and stderr by
simply importing sys and reassigning sys.stdout and sys.stderr to something
else, make sure to do it BEFORE you import pywikibot or any module that
would import pywikibot. (Although it does kinda seem like a bug that
pywikibot doesn't respond to future changes to sys.stdout and sys.stderr;
maybe the output function in the interface class should check whether
either of those has changed, and adjust its setup as needed?)
Jonathan Goble
On Tue, Jun 24, 2014 at 5:25 AM, Merlijn van Deen <valhallasw(a)arctus.nl>
wrote:
Could you provide a minimal example that would
allow us to reproduce the
issue? With core, the following works for me (I don't have compat
installed):
import sys, StringIO
oldstderr = sys.stderr
oldstdout = sys.stdout
sys.stderr = sys.stdout = StringIO.StringIO()
sys.stderr.fileno = lambda: 10
import pywikibot
oldstdout.write("Before output\n")
pywikibot.output('test')
oldstdout.write("After output; this is in our buffer:\n")
sys.stderr.seek(0)
oldstdout.write(repr(sys.stderr.read()))
$ PYWIKIBOT2_DIR=. python test.py
Before output
After output; this is in our buffer:
'WARNING: Running on Windows and transliteration_target is not
set.\nPlease see
https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Pywikibot/Windows\…
\n'
Nonetheless, I would suggest to use normal shell redirection techniques,
instead of monkey-patching python's sys module. If you want to use python,
the following should work (untested):
import subprocess
stdout, stderr = subprocess.Popen(["c:\python27\python", "pwb.py"],
stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
# do something with stdout and stderr
I *think* it should also be possible to do this with the logging module,
but I'm not sure how to do that in a quick-and-easy way.
Merlijn
On 24 June 2014 09:52, Jonathan Goble <jcgoble3(a)gmail.com> wrote:
When the error began occurring, I was using the
Pywikibot nightly dump
from February 25 of this year. It continues to occur with the nightly dump
from yesterday (June 23). I just tried the fileno = lambda: 10 trick and it
had no effect.
As for core, it's been four months since I last tried it so I don't
recall the issues. It's past bedtime for me right now, but I'll look into
giving it another try within the next day or two when I have some time.
On Tue, Jun 24, 2014 at 3:17 AM, Merlijn van Deen <valhallasw(a)arctus.nl
> wrote:
> On 24 June 2014 06:17, Jonathan Goble <jcgoble3(a)gmail.com> wrote:
>
> I've given up trying to solve a bug that popped up in my scripts a
>> couple days ago. I run a bot for Wookieepedia, over at Wikia, and run three
>> simple scripts on a daily basis. They are set up to run automatically
>> through Windows Task Scheduler. Since they run automatically, they run in
>> the background through pythonw.exe, i.e. without a console, and therefore I
>> need a means of getting the output. My solution for the past two months has
>> been to redirect sys.stdout and sys.stderr to the same StringIO() instance,
>> then at the end call getvalue() on that and email it to myself.
>>
>
> Could it be you haven't updated in a few years? It sounds like it's
> related to a feature that I added two or three years ago, that allowed
> Windows users to get full unicode out- and input. However, that does mean
> sys.stdin and sys.stdout are no longer being used. We /do/ check whether
> the user has redirected the output using normal shell redirection, but your
> method doesn't do that.
>
> There are a two options I can think of.
> - use normal shell redirection, e.g.
>
https://stackoverflow.com/questions/8662024/how-do-i-capture-the-output-of-…
>
> - trick the code into thinking you're doing 'regular' redirection by
> adding a fileno function to your streams:
> see
>
https://github.com/wikimedia/pywikibot-compat/blob/master/userinterfaces/wi…
>
> you'd need something like x = StringIO.StringIO(); x.fileno =
> lambda: 10
> and the same for stdout.
>
> (By the way, the answer is NOT "switch to core". I have tried to get
>> core to run on my system and failed miserably after two hours of repeated
>> attempts without even getting it to talk to the wiki. Compat worked
>> perfectly on the first try. Until such time as core can be installed by a
>> beginner, it is not for me.)
>>
>
> I would appreciate it if you could clarify what the issues were you
> ran into.
>
> Merlijn
>
>
> _______________________________________________
> Pywikipedia-l mailing list
> Pywikipedia-l(a)lists.wikimedia.org
>
https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
>
>
_______________________________________________
Pywikipedia-l mailing list
Pywikipedia-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
_______________________________________________
Pywikipedia-l mailing list
Pywikipedia-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
_______________________________________________
Pywikipedia-l mailing list
Pywikipedia-l(a)lists.wikimedia.org