Hi all, I'm a new and very ntimidited toolserver user :-) Presently I'm doing some "Hello world" tests only, as I told in a previuos message, but I'd like to run from toolserver my pywikipedia bot, Alebot, that is a rather busy one ( http://stats.wikimedia.org/wikisource/EN/BotActivityMatrix.htm).http://stats.wikimedia.org/wikisource/EN/BotActivityMatrix.htm
Alebot reads at intervals itwikisource RecentChanges, selects new edits by type, contributor and namespace, reads new or edited pages and "does things". Such things often are nothing, often imply ad edit of one wikisource page, sometimes are more complex, implying some more readings/editing of some different, related pages (max 3-4 readings/editing, reading/updating of local pickle files).
My question is: how many pywikipedia readings/editing per hour are a "heavy" toolserver job?
Alex
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Alex Brollo:
My question is: how many pywikipedia readings/editing per hour are a "heavy" toolserver job?
We don't usually measure resource use in "pywikipedia readings/editing per hour" ;-) The most common indicators of resource use are CPU time and memory.
There's also no clear point at which a job becomes "heavy" -- not least because "heavy" is a vague description and could mean different things to different people.
It would be easier to answer your question if you could provide some context...
- river.
2010/11/17 River Tarnell river.tarnell@wikimedia.de
Alex Brollo:
My question is: how many pywikipedia readings/editing per hour are a
"heavy"
toolserver job?
We don't usually measure resource use in "pywikipedia readings/editing per hour" ;-) The most common indicators of resource use are CPU time and memory.
My premise was "A newbie question" :-P
Ok. Have I some Unix tool to evaluate server resource used running a script? And - if such a tool exists and if it will give back some exoteric result to me - how can I evaluate it in terms of "heaviness"? I guess the best solution: to paste and copy here the result if any, and to ask you again. :-)
Feel free to send me your best link to a tutorial "Unix for dummies".
Alex
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Alex Brollo:
Have I some Unix tool to evaluate server resource used running a script?
Yes, but it depends on operating system. If you're using a Solaris server (willow), use prstat:
$ prstat -aU $LOGNAME
Or on Linux (nightshade), use top:
$ top -u $LOGNAME
Both of these will show a list of all your processes, and various statistics, including CPU and memory use. (prstat will also show a summary of your entire resource use.)
The interesting figures are CPU use and resident memory (called "RES" in top, and "RSS" in prstat). "CPU" is the amount of CPU (in percent) that your process is using. prstat shows this as total of all CPU cores, so 12.5% CPU means your process is using one entire CPU core (since the login servers have 8 cores, and 100/8=12.5). top shows this as total of a single core, so an entire core shows as 100% (and 200% would mean it was using two cores).
Generally you shouldn't use 100% CPU constantly, but it's okay for a program to use a lot of CPU sometimes and sleep (using none) at other times. What is acceptable CPU use really depends on what the program does; if it makes 1 edit to the wiki per hour and does nothing else, using 10% CPU is probably excessive. On the other hand, a script that does complicated processing might easily use 25%.
RES/RSS (resident memory) is the amount of RAM being used by the program. There is a hard limit of 1GB per user on each server, but most tools should use a *lot* less than that. Again, it really depends on what the tool does.
You shouldn't worry too much about resource use as long as it's not excessive. We (TS admins) will let you know if you seem to be using too many resources.
Feel free to send me your best link to a tutorial "Unix for dummies".
I usually recommend the book "Understanding UNIX" by Stan Kelly-Bootle, but unfortunately it's out of print, and it also doesn't answer this particular question. (But it does tell you just about everything else you'd want to know.)
- river.
2010/11/17 River Tarnell river.tarnell@wikimedia.de
You shouldn't worry too much about resource use as long as it's not excessive. We (TS admins) will let you know if you seem to be using too many resources.
This is exactly what I'd like to know. :-)
Feel free to send me your best link to a tutorial "Unix for dummies".
I usually recommend the book "Understanding UNIX" by Stan Kelly-Bootle, but unfortunately it's out of print, and it also doesn't answer this particular question. (But it does tell you just about everything else you'd want to know.)
Thanks. There are lots of tutorials online, presently I'm very proud to banal goals like changing PATH by python and running my first bash "Hello world" script. Just to let you know how much dummy I am.
Alex
toolserver-l@lists.wikimedia.org