Toolserver-l

toolserver-l@lists.wikimedia.org

1612 discussions

New job submission servers
by River Tarnell 12 Jan '11

12 Jan '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, We have set up a pair of redundant hosts to act as SGE job submission servers. These work by sharing each user's cronietab between both hosts, and executing jobs on whichever server is working. This avoids that problem where jobs run from cronie on one login server (such as willow) will fail to run if that host is down, even when other login servers are available. To use the new hosts, log into submit.toolserver.org and set up a cronietab (*not* a crontab) as normal. Note that these hosts are *only* for submitting SGE jobs, not for running tools on. For now this service could be considered experimental, but I don't foresee any problems with it. - river. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk0tewgACgkQIXd7fCuc5vJKpQCeOwDC6Y2/0rBkzGFj6A2WT1DS aaQAn0yhdkmZikrbXPdPL26EsQ5A+r+1 =PDMJ -----END PGP SIGNATURE-----

1 0

output buffer flushing broken
by Ryan Kaldari 11 Jan '11

11 Jan '11

Ever since the servers were moved to Solaris I haven't been able to get output buffer flushing in PHP to work at all. I've tried all the usual solutions: using implicit flushing, filling the buffer with 100000 chars before the flush, using ob_flush() and flush() together, setting output_buffering to 0, cleaning all the active buffers, making sure output compression is off, etc. but nothing will work. I had no problems with this a few months ago. Unfortunately, I need this to work for a toolserver tool that is going to be heavy used during the Wikipedia 10th Anniversary celebration this weekend. Here are a couple of basic test-case scripts that won't work for me on the toolserver: http://pastebin.com/sAwnvJrZ // manual flushing http://pastebin.com/TuKM14av // implicit flushing If anyone can get either of these test scripts working on the toolserver, please let me know ASAP. Thanks! Ryan Kaldari

1 1

Toolserver projects list
by Erik Moeller 11 Jan '11

11 Jan '11

Hi, is there still an up-to-date, canonical list of toolserver projects? Thanks, Erik -- Erik Möller Deputy Director, Wikimedia Foundation Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate

2 1

WM database dumps
by Mihajlo Andjelkovic 09 Jan '11

09 Jan '11

Hello, everyone Did I miss announcement when WM database dumps are to start being actualized again? M

2 1

cronsub
by River Tarnell 08 Jan '11

08 Jan '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I've made a couple of changes to cronsub in response to some issues reported by users. Specifically: * cronsub now requires that the script file be executable, and will raise an error if it's not. The previous behaviour was that non-executable scripts would be executed by /bin/sh. If this affects you, the fix is to make the script executable (chmod +x). * Previously, SGE would copy the job script to a shared directory and execute it there. For example, if you submitted $HOME/test.py, it would be copied to a file such as /sge62/default/spool/wolfsbane/job_scripts/117333 before starting. This was unfortunate for Python users who depended on Python's behaviour of treating the script's directory as part of the "import" search path. This behaviour has been changed, so that the script file will be executed in its original location. I appreciate that the first item may be a breaking change for some users. However, on balance this seems like the lesser evil. - river. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk0ny8sACgkQIXd7fCuc5vK1UwCfaIt9gOSWym8sDOKR8bekuUaK Qy4AoLLJIB1xaUjeKdpUHLZ6npg/0FiS =VV97 -----END PGP SIGNATURE-----

4 6

Maintenance
by River Tarnell 08 Jan '11

08 Jan '11

Hi, The previous announced maintenance has been completed successfully. In case of problems, open an issue in the TS project in JIRA. For problems relating to the conversion of nightshade to Solaris (e.g. missing software), open an issue in the TS project and include "[solaris]" at the start of the subject line. Due to the disruptive nature of this maintenance, such requests will be given priority. (However, I'll be away for at least the next 12 hours, so I won't be able to resolve them immediately.) - river.

4 8

Job scheduling a python bot script
by Dr. Trigon 07 Jan '11

07 Jan '11

Hello all I'm currently trying to get 'Job scheduling' working form my modified pywikipedia bot python script. According to https://wiki.toolserver.org/view/Job_scheduling (Converting an existing cron job to use the scheduler) I tried to convert ================= 0 2 * * * nice -n +15 python /home/drtrigon/pywikipedia/runbotrun.py -cron 0 0 1,16 * * nice -n +15 python /home/drtrigon/pywikipedia/runbotrun.py -skip_clean_user_sandbox -compress_history:[] -cron 0 6 * * * nice -n +15 python /home/drtrigon/pywikipedia/subster_beta.py 2>> /home/drtrigon/public_html/DrTrigonBot/subster.html ================= to ================= 0 2 * * * cronsub -s mainbot $HOME/pywikipedia/runbotrun.py -cron 0 0 1,16 * * cronsub -s compbot $HOME/pywikipedia/runbotrun.py -skip_clean_user_sandbox -compress_history:[] -cron 0 6 * * * cronsub -s substerbot $HOME/pywikipedia/subster_beta.py 2>> $HOME/public_html/DrTrigonBot/subster.html ================= AND I tried to run to first of those jobs with 'qsub', by: qsub $HOME/pywikipedia/runbotrun.py -cron BUT this leads to the follwing output in my $HOME: ================= /sge62/default/spool/wolfsbane/job_scripts/117222: ^JThis bot is the general DrTrigonBot caller. It runs all the different sub tasks,^Jthat DrTrigonBot does.^J^J...^J: not found import: unable to open X server `' @ error/import.c/ImportImageCommand/364. import: unable to open X server `' @ error/import.c/ImportImageCommand/364. import: unable to open X server `' @ error/import.c/ImportImageCommand/364. import: unable to open X server `' @ error/import.c/ImportImageCommand/364. import: unable to open X server `' @ error/import.c/ImportImageCommand/364. /sge62/default/spool/wolfsbane/job_scripts/117222: from: not found logname: extra operand `=' Try `logname --help' for more information. /sge62/default/spool/wolfsbane/job_scripts/117222: logger_tmsp: not found /sge62/default/spool/wolfsbane/job_scripts/117222: syntax error at line 65: `(' unexpected ================= ... and this is the point were I am getting lost... this message looks VERY strange to me, since I'm not using an X server (explicitely) and I have no clue what the 'error/import.c' means there..... Please can somebody give me a hint? Any ideas are welcome!! Thanks a lot and greetings Dr. Trigon ps: by the way; thanks for the good server maintenance and all that stuff!!

2 1

.environment file
by River Tarnell 07 Jan '11

07 Jan '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, It's now possible to set environment variables in your $HOME/.environment file, as a series of NAME=value pairs, for example: PYTHONPATH=$HOME/pylib This will affect interactive logins, cron jobs and SGE jobs, but currently not web server CGI scripts. - river. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk0mH1EACgkQIXd7fCuc5vJrWwCgt+nTHauHPBIJ2+jhw3J2tkPt COcAnAurVJt1Wncm6WDM8eX/h0RkTz7+ =KJ0E -----END PGP SIGNATURE-----

2 1

Submitting Python jobs to SGE from cron
by Russell Blau 06 Jan '11

06 Jan '11

Since a number of users seem to be struggling with the conversion to Solaris, I thought I would share the scripts I have been using to start my Python bots from Solaris cron using the job server. If you want to use these, be sure to substitute your username for "username" in the last line of listing 1, and your python path (if you have one defined) in the third line of listing 2. With these scripts in your home directly, you can then set up a cron job to run any Python script using the following syntax in your crontab: 30 9 * * * $HOME/pysub jobname $HOME/path/script.py arg1 arg2 "jobname" is an arbitrary name to be given to the job. Be sure to include $HOME/ in both places; replace /path/ with the path from your home directory to the Python script. I don't know as much about bash as I do about Python, so it is very possible someone else can improve on these scripts. :-) But feel free to use them if they are helpful. Russ Listing 1: pysub #! /bin/ksh # like cronsub, but runs a python script instead of a shell script . /sge62/default/common/settings.sh [[ $# -ge 1 ]] || { echo >&2 "usage: $0 <jobname> <script>" exit 1 } JOBNAME=$1 shift SCRIPT=$@ qstat -j $JOBNAME >/dev/null 2>&1 || qsub -N $JOBNAME /home/username/py.sh $SCRIPT >/dev/null Listing 2: py.sh #!/bin/sh # run an arbitrary Python script PYTHONPATH=/copy/of;/your/pythonpath export PYTHONPATH /usr/bin/python $@

2 3

cron
by River Tarnell 06 Jan '11

06 Jan '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Since the switch to Solaris a few people have complained about the Sun 'cron' not being very good. As an alternative to cron, we have now installed 'cronie', which is the RedHat Linux version of cron. This is based on Vixie cron and offers an almost identical feature set to the old cron on nightshade. To avoid a conflict with the system cron, the cronie crontab command is called 'cronie'. To convert your current crontab to a cronietab[0], you can use a command like this: % crontab -l >crontab # list existing crontab to file % cronie crontab # install same as a cronietab % crontab -r # remove old crontab You should probably use either crontab or cronie, but not both, since that could be confusing. - river. [0] Yes, "cronietab" is a fairly stupid name. Sorry. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk0lXhoACgkQIXd7fCuc5vKuEACdHkfvuZeTdT8MmMmNpgY1O/t6 7PwAn1tk+uJ9czNa+MBWQWP4vndzN+Et =FIKF -----END PGP SIGNATURE-----

2 1

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Toolserver-l