hi all
nightshade ran out of ram. I don't know why, because i couldn't even log in via serial to have a look. so i power-cycled it, should be back up shortly.
if you edited your scripts lately, please check if you made a fork bomb. or did something that would consume all the ram.
cheers, daniel
On Sat, Jan 31, 2009 at 2:00 PM, Daniel Kinzler daniel@brightbyte.de wrote:
hi all
nightshade ran out of ram. I don't know why, because i couldn't even log in via serial to have a look. so i power-cycled it, should be back up shortly.
if you edited your scripts lately, please check if you made a fork bomb. or did something that would consume all the ram.
cheers, daniel
Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
It won't allow connections again... Somebody has a fork bomb in his crontab?
Bryan
Il giorno 31/gen/09, alle ore 20:55, Bryan Tong Minh ha scritto:
On Sat, Jan 31, 2009 at 2:00 PM, Daniel Kinzler daniel@brightbyte.de wrote:
hi all
nightshade ran out of ram. I don't know why, because i couldn't even log in via serial to have a look. so i power-cycled it, should be back up shortly.
if you edited your scripts lately, please check if you made a fork bomb. or did something that would consume all the ram.
cheers, daniel
Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
It won't allow connections again... Somebody has a fork bomb in his crontab?
Bryan
Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Could you check my processes please? They are some python scripts. I had them killed by slayerd recently, but I fixed them next. They shouldn't be fork bombs, but who knows? Also, they should run in the night... the last time I did "top -u pietrodn" it was all OK.
Pietrodn powerpdn@gmail.com
It's a relief that nothing I did caused it.
On Sat, Jan 31, 2009 at 3:30 PM, DaB. WP@daniel.baur4.info wrote:
Hello, Am Saturday 31 January 2009 20:55:48 schrieb Bryan Tong Minh:
It won't allow connections again... Somebody has a fork bomb in his crontab?
powercyled it again and renamed the script that caused the probelm (AFAIS).
Bryan
Sincerly, DaB.
-- wp-blog.de
Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Il giorno 31/gen/09, alle ore 21:30, DaB. ha scritto:
Hello, Am Saturday 31 January 2009 20:55:48 schrieb Bryan Tong Minh:
It won't allow connections again... Somebody has a fork bomb in his crontab?
powercyled it again and renamed the script that caused the probelm (AFAIS).
That's not one of my scripts, luckily.
Pietrodn powerpdn@gmail.com
Ok, I'm really sorry guys. It would appear to be my script that screwed up (100 new messages from cron kinda was my notification). I'm going to try to fix that later today. I'll be keeping a close eye on it, so I hopefully won't kill the TS this time.
Soxred93
On Jan 31, 2009, at 3:45 PM [Jan 31, 2009 ], Pietrodn wrote:
Il giorno 31/gen/09, alle ore 21:30, DaB. ha scritto:
Hello, Am Saturday 31 January 2009 20:55:48 schrieb Bryan Tong Minh:
It won't allow connections again... Somebody has a fork bomb in his crontab?
powercyled it again and renamed the script that caused the probelm (AFAIS).
That's not one of my scripts, luckily.
Pietrodn powerpdn@gmail.com
Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Soxred93 wrote:
Ok, I'm really sorry guys. It would appear to be my script that screwed up (100 new messages from cron kinda was my notification). I'm going to try to fix that later today. I'll be keeping a close eye on it, so I hopefully won't kill the TS this time.
Come to think of it, shouldn't the toolserver have some resource use limits that would stop things like this from happening? A single non-root user running a fork bomb or anything similar really shouldn't be able to bring down the entire server.
On Sat, Jan 31, 2009 at 6:57 PM, Ilmari Karonen nospam@vyznev.net wrote:
Come to think of it, shouldn't the toolserver have some resource use limits that would stop things like this from happening? A single non-root user running a fork bomb or anything similar really shouldn't be able to bring down the entire server.
Maybe set ulimit -m to a gigabyte or something?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Sun, Feb 1, 2009 at 2:08 AM, Aryeh Gregor wrote:
On Sat, Jan 31, 2009 at 6:57 PM, Ilmari Karonen wrote:
Come to think of it, shouldn't the toolserver have some resource use limits that would stop things like this from happening? A single non-root user running a fork bomb or anything similar really shouldn't be able to bring down the entire server.
Maybe set ulimit -m to a gigabyte or something?
I guess this would effectively disable all those memory-eating interwiki bots ;)
Marco
On Sun, Feb 1, 2009 at 3:59 AM, Marco Schuster marco@harddisk.is-a-geek.org wrote:
I guess this would effectively disable all those memory-eating interwiki bots ;)
There's only 8G of RAM on nightshade. Make it a 2G limit, maybe, or 4G, but one non-root user using anything more than that seems unreasonable.
On Sun, Feb 1, 2009 at 6:42 AM, Bryan Tong Minh bryan.tongminh@gmail.com wrote:
And limit the number of processes a user can run. while (1) { fork(); } is gonna run you out of pids rather than memory.
ulimit -u is already set to 500 for me, so that shouldn't be possible.
0 15:30:12 ~$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) 2048000 scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 71680 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 500 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Aryeh Gregor:
ulimit -u is already set to 500 for me, so that shouldn't be possible.
i made this change after the problem occurred. hopefully, it will stop it happening again in the future.
regarding memory use: slayerd runs every 10 seconds to limit excessive memory use. since we already set a per-process data limit (2GB), it would be impossible for a user to use all ram+swap in under 10 seconds unless he was creating many child processes, which is what appears to have happened here. in this case, per-process memory limit would have no effect, since no individual process would exceed the limit. as far as i know, Linux has no per-user memory limit controls.
- river.
On Sun, Feb 1, 2009 at 2:08 AM, Aryeh Gregor Simetrical+wikilist@gmail.com wrote:
On Sat, Jan 31, 2009 at 6:57 PM, Ilmari Karonen nospam@vyznev.net wrote:
Come to think of it, shouldn't the toolserver have some resource use limits that would stop things like this from happening? A single non-root user running a fork bomb or anything similar really shouldn't be able to bring down the entire server.
Maybe set ulimit -m to a gigabyte or something?
And limit the number of processes a user can run. while (1) { fork(); } is gonna run you out of pids rather than memory.
Bryan
hi all
many of our tools uses messages from betawiki because they are translated in many languages. But this messages must be synchronized and I think it's not very economically if every user/tool do that self. Because of that I propose to create one database for all users which will be sync periodically.
What do you think? Are any other users interested for such a db?
Greetings, Luxo
Il giorno 01/feb/09, alle ore 15:35, Luxo ha scritto:
hi all
many of our tools uses messages from betawiki because they are translated in many languages. But this messages must be synchronized and I think it's not very economically if every user/tool do that self. Because of that I propose to create one database for all users which will be sync periodically.
A database with one table per user? Sounds good.
What do you think? Are any other users interested for such a db?
I would be interested.
Greetings, Luxo
Pietrodn powerpdn@gmail.com
2009/2/1 Luxo luxo.de@gmail.com:
hi all
many of our tools uses messages from betawiki because they are translated in many languages. But this messages must be synchronized and I think it's not very economically if every user/tool do that self. Because of that I propose to create one database for all users which will be sync periodically.
What do you think? Are any other users interested for such a db?
I second that! this would be great :)
toolserver-l@lists.wikimedia.org