Hello all,
today I discovered (thanks to the mailing-list) that a few users have run several bot-instances in parallel on willow. I'm sure that these people did it by mistake, but it is annoying nevertheless and it is easy to fix: Use SGE.
The problem is that I wrote several of eMails about "use SGE!" already and somehow it did not work as good as it should (if you converted you stuff already: thank you and you can stop to read here ;-)). I understand that we all are busy with our lives and Wikipedia and that we all love to "do it right…later", but as you know that resources of the toolserver are limited. So I hereby declare the following new rule:
All bots have to run by SGE. A bot is every program or script that makes changes at a Wikimedia project. It does not matter if the bot runs periodically or continuous. The only exclusions are a.) interactive bots, b.) bots that can't run by SGE yet and c.) if you start a bot by hand for testing (no screen, no cron, no while). The rule will become active at Sunday, 10. February 2013. Exception b is almost NEVER the case, if it runs on a shell it is VERY likely that it can run by SGE.
Some time ago I wrote a simple SGE-how-to at [1]. Maybe you all can take a look and correct things and make things more clear. In very most cases the using of SGE IS easy.
Sincerely, DaB.
[1] https://wiki.toolserver.org/view/SGE_for_beginners
Hello all,
in the last 3 days I spent a few hours a day to enforce the new rule (starting on willow). I wrote many emails and commented-out even more cron-lines. I learned a few thing doing so (for example some users think that 1 cron-line for a bot is not enough, that some users still uses our old phoenix and newtask programs, and some users seems to do cron-task-sharing…). Until now nobody lost his account, but I killed all misbehaving bots. The load on willow is now appreciable lower than before (the rebooted helped there too of course). It's a more or less boring task and you would REALLY help me if you convert your stuff to SGE yourself, before I kill and disable your bot. If you find that your bot was disabled you are allowed to re-enable it IF you convert it to SGE FIRST! Don't make me find a bot I disabled before running without SGE – you and I would hate that.
To say something positive: I found also bots using SGE (few, but I found).
Sincerely, DaB.
toolserver-announce@lists.wikimedia.org