Hi,
I run an own script on a remote Linux toolserver in the background. I suppressed my own screen outputs for that purpose. However, the program uses pagenerators.py, and my screen is infinitely full of lines lik these: Getting 60 pages from wikipedia:hu... Sleeping for 4.5 seconds, 2011-12-22 14:19:06 Getting 60 pages from wikipedia:hu... Sleeping for 4.6 seconds, 2011-12-22 14:19:12 Getting 60 pages from wikipedia:hu... Sleeping for 4.2 seconds, 2011-12-22 14:19:19 Getting 60 pages from wikipedia:hu... Sleeping for 4.1 seconds, 2011-12-22 14:19:25 Getting 60 pages from wikipedia:hu... Sleeping for 4.3 seconds, 2011-12-22 14:19:30 Getting 60 pages from wikipedia:hu... Sleeping for 4.2 seconds, 2011-12-22 14:19:36 Sleeping for 4.1 seconds, 2011-12-22 14:19:42 Getting 60 pages from wikipedia:hu... Getting 60 pages from wikipedia:hu...
This is not only ugly but prevents me of using the console. Is there any way to suppress these messages and run pagegenerators in silent mode?
Why not just use the screen command?
John
On Thu, Dec 22, 2011 at 8:24 AM, Bináris wikiposta@gmail.com wrote:
Hi,
I run an own script on a remote Linux toolserver in the background. I suppressed my own screen outputs for that purpose. However, the program uses pagenerators.py, and my screen is infinitely full of lines lik these: Getting 60 pages from wikipedia:hu... Sleeping for 4.5 seconds, 2011-12-22 14:19:06 Getting 60 pages from wikipedia:hu... Sleeping for 4.6 seconds, 2011-12-22 14:19:12 Getting 60 pages from wikipedia:hu... Sleeping for 4.2 seconds, 2011-12-22 14:19:19 Getting 60 pages from wikipedia:hu... Sleeping for 4.1 seconds, 2011-12-22 14:19:25 Getting 60 pages from wikipedia:hu... Sleeping for 4.3 seconds, 2011-12-22 14:19:30 Getting 60 pages from wikipedia:hu... Sleeping for 4.2 seconds, 2011-12-22 14:19:36 Sleeping for 4.1 seconds, 2011-12-22 14:19:42 Getting 60 pages from wikipedia:hu... Getting 60 pages from wikipedia:hu...
This is not only ugly but prevents me of using the console. Is there any way to suppress these messages and run pagegenerators in silent mode?
-- Bináris
Pywikipedia-l mailing list Pywikipedia-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
2011/12/22 John phoenixoverride@gmail.com
Why not just use the screen command?
Please help, how. I am not very familiar with Linux. After having a look at help and man, I just typed in "screen". The result is:
[Press Space or Return to end.] Getting 60 pages from wikipedia:hu... Sleeping for 4.0 seconds, 2011-12-22 14:34:45
Sleeping for 4.3 seconds, 2011-12-22 14:34:50 Getting 60 pages from wikipedia:hu... Getting 60 pages from wikipedia:hu... Sleeping for 4.4 seconds, 2011-12-22 14:34:57 Getting 60 pages from wikipedia:hu...
Sleeping for 4.5 seconds, 2011-12-22 14:35:03 Getting 60 pages from wikipedia:hu... Sleeping for 4.6 seconds, 2011-12-22 14:35:09 Getting 60 pages from wikipedia:hu... Sleeping for 4.3 seconds, 2011-12-22 14:35:15 So the messages still come, but at random cursor positions. :-) I quit with space, and now it looks like the same at the normal prompt, messages don't begin at the leftmost position any more.
Hit ctrl+a+d which will detach the screen session. To get it back type screen -ls and note the process id. Then screen -r (process id) to reattach.
Tom Morton
On 22 Dec 2011, at 13:38, "Bináris" wikiposta@gmail.com wrote:
2011/12/22 John phoenixoverride@gmail.com
Why not just use the screen command?
Please help, how. I am not very familiar with Linux. After having a look at help and man, I just typed in "screen". The result is:
[Press Space or Return to end.] Getting 60 pages from wikipedia:hu... Sleeping for 4.0 seconds, 2011-12-22 14:34:45
Sleeping for 4.3 seconds, 2011-12-22 14:34:50 Getting 60 pages from wikipedia:hu... Getting 60 pages from wikipedia:hu... Sleeping for 4.4 seconds, 2011-12-22 14:34:57 Getting 60 pages from wikipedia:hu...
Sleeping for 4.5 seconds, 2011-12-22 14:35:03 Getting 60 pages from wikipedia:hu... Sleeping for 4.6 seconds, 2011-12-22 14:35:09 Getting 60 pages from wikipedia:hu... Sleeping for 4.3 seconds, 2011-12-22 14:35:15 So the messages still come, but at random cursor positions. :-) I quit with space, and now it looks like the same at the normal prompt, messages don't begin at the leftmost position any more.
2011/12/22 Thomas Morton morton.thomas@googlemail.com
Hit ctrl+a+d which will detach the screen session. To get it back type screen -ls and note the process id. Then screen -r (process id) to reattach.
Thank you. But messages still come on both screens.
I see the problem: my script was already running when I used screen. Now I started another connection, and used screen, started the script and detached it, and this way it works. Thank you for the idea and the help.
However, it would be nice to be able to suppress unnecessary messages. Every script should have this feature. I often experience that a large amount of screen output makes programs run slower.
On 22 December 2011 14:58, Bináris wikiposta@gmail.com wrote:
However, it would be nice to be able to suppress unnecessary messages. Every script should have this feature. I often experience that a large amount of screen output makes programs run slower.
I sincerely doubt that the stdout/stderr output has an effect on the speed
of pywikipedia, as it is limited mainly by throttling. Even if you disable throttling, you will be limited by the network and not by your screen.
However, if you want, you can redirect the stdout/stderr output as normal using > and 2>.
Last but not least, feel free to rewrite all bots to use the python logging module, as it exists for long enough to be used (since python 2.3).
Best regards, Merlijn
Hi Merlijn,
Op 22-12-2011 21:48, Merlijn van Deen schreef:
Last but not least, feel free to rewrite all bots to use the python logging module, as it exists for long enough to be used (since python 2.3).
I think we can distinguish two kinds of output: 1. Output for user interaction (diff, followed by user interaction) 2. Various levels of information
If a bot is running in autonomous mode all messages fall in the second category. We should probably start using the python logging module and define an output level (see https://en.wikipedia.org/wiki/Syslog) for every message. Sounds like a nice programming project. Who is up to the challenge? ;-)
Maarten
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hello Merlijn Hello Maarten
I had a look into 'userinterfaces/terminal_interface.py'. As it looks to me we would have first to make a duplicate of it using 'logging' to replace 'targetStream.write' in 'printColorized'. (of course there will be some additional coding to adopt it well to the 'Colorized' scheme, since 'logging' does not support this - essentially we have to drop it here)
And voila we have a bot framework using the logging module, which whould have the features of printing messages to file AND console at the same time as well as other cool stuff... (!)
But since I do not have any commit access it is left to one of you to make this happen... ;)
Greetings DrTrigon
On 23.12.2011 19:17, Maarten Dammers wrote:
Hi Merlijn,
Op 22-12-2011 21:48, Merlijn van Deen schreef:
Last but not least, feel free to rewrite all bots to use the python logging module, as it exists for long enough to be used (since python 2.3).
I think we can distinguish two kinds of output: 1. Output for user interaction (diff, followed by user interaction) 2. Various levels of information
If a bot is running in autonomous mode all messages fall in the second category. We should probably start using the python logging module and define an output level (see https://en.wikipedia.org/wiki/Syslog) for every message. Sounds like a nice programming project. Who is up to the challenge? ;-)
Maarten
_______________________________________________ Pywikipedia-l mailing list Pywikipedia-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
Hello Dr. Trigon,
On 24 December 2011 22:21, Dr. Trigon dr.trigon@surfeu.ch wrote:
I had a look into 'userinterfaces/terminal_interface.py'. As it looks to me we would have first to make a duplicate of it using 'logging' to replace 'targetStream.write' in 'printColorized'.
I think this misses the point of switching to the Logging module. The benefit of the Logging module is the possibility of assigning 'logging levels' to a message, e.g. DEBUG, INFO or WARNING, and the user can then configure the verbosity.
Essentially, we would need to re-implement the user interface module to pass through the logging module, which allows for creating different output modules.
But as far as I know, the rewrite already does this - or at least partially ;-)
Merlijn
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I think this misses the point of switching to the Logging module. The benefit of the Logging module is the possibility of assigning 'logging levels' to a message, e.g. DEBUG, INFO or WARNING, and the user can then configure the verbosity.
This is absolutely correct but what I would have considered as a "second stage" - but that's up to you... ;)
But as far as I know, the rewrite already does this - or at least partially ;-)
So it will probably never make it into trunk...? ;)
Greetings and have Happy Holidays! DrTrigon
On 25 December 2011 10:57, Dr. Trigon dr.trigon@surfeu.ch wrote:
But as far as I know, the rewrite already does this - or at least partially ;-)
So it will probably never make it into trunk...? ;)
Not necessarily. My point is: it makes more sense to backport features
from rewrite than to re-develop them for trunk. In a perfect world, everybody should already be using rewrite and it should be 100x more awesome than trunk. In reality, we might want to consider backporting parts instead of trying to get people to switch.
Merlijn
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 27.12.2011 23:52, Merlijn van Deen wrote:
On 25 December 2011 10:57, Dr. Trigon <dr.trigon@surfeu.ch mailto:dr.trigon@surfeu.ch> wrote:
But as far as I know, the rewrite already does this - or at least partially ;-)
So it will probably never make it into trunk...? ;)
Not necessarily. My point is: it makes more sense to backport features from rewrite than to re-develop them for trunk. In a perfect world,
I fully agree! ;)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Anyway I have to thank this thread (!!) since I did not know the 'logging' module from python - know even my bot does... ;))
What about having a config.suppress parameter in config file (defaults to False), and teach pywikibot.output and some other methods to watch that and shut up when suppress is True?
Use of logging module proposed by Maarten and Merlijn may be a general solution, but is beyond my comprehension at the moment, so I cannot contribute to it. This would be much simpler.
So I propose:
1. Let's introduce a new config parameter suppress with default False. 2. pywikibot.output should return immediately upon detecting suppress=True. One central handle for majority of screen outputs. 3. Other methods and functions may use it as necessary. 4. After correctly implementing your grandious vision of logging levels, this may be deprecated.
Toolserver bots have an own config, so this seems to be comfortable.
Hmm?
On 22 February 2012 11:24, Bináris wikiposta@gmail.com wrote:
- Let's introduce a new config parameter suppress with default False.
- pywikibot.output should return immediately upon detecting
suppress=True. One central handle for majority of screen outputs.
How would this be different from
python bot.py 2>&1 > /dev/null
?
Furthermore, the logging module is *not* difficult and a lot of useful wrapper functions for pywikipedia already exist in the rewrite branch: http://svn.wikimedia.org/viewvc/pywikipedia/branches/rewrite/pywikibot/bot.p...
(around line 217)
Best, Merlijn
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Yeah Merlijin in right! The loggining module should be used!
I myself had a look at 'pywikibot.output' some time ago. To me it seems we would have to modify 'output' from modules at 'userinterfaces' directory (look at lines 7858-7860 in 'wikipedia.py'). Or even better create a new 'logging' user- interace module in order to use the 'logging' module there. Then finally we can suppress logging be setting the level or else there.
What do you think?
Greetings DrTrigon
ps.: It is not that simple and fast as it looks, that is the reason why it did not solve this already...
On 22.02.2012 11:38, Merlijn van Deen wrote:
On 22 February 2012 11:24, Bináris <wikiposta@gmail.com mailto:wikiposta@gmail.com> wrote:
- Let's introduce a new config parameter suppress with default
False. 2. pywikibot.output should return immediately upon detecting suppress=True. One central handle for majority of screen outputs.
How would this be different from
python bot.py 2>&1 > /dev/null
?
Furthermore, the logging module is *not* difficult and a lot of useful wrapper functions for pywikipedia already exist in the rewrite branch: http://svn.wikimedia.org/viewvc/pywikipedia/branches/rewrite/pywikibot/bot.p...
(around line 217)
Best, Merlijn
_______________________________________________ Pywikipedia-l mailing list Pywikipedia-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
Well, at the moment I cannot dig into that project. First I have to understand put_async as described in the other thread, and have a lot of other pending projects... So it remains for a while as it is.