Skip to content

Stopping the run away process

July 23, 2010

We had a user who had managed to write a fork bomb and so had consumed all the resources allocated to them and could no longer login. The odd thing, something I have still not quite got to the bottom of, was that when I killed all the processes they each produced a child which when killed also produced a child making them unkillable.

So I wrote this one-liner to stop any newly created processes for that user before they could do anything:

/usr/sbin/dtrace -w -n 'proc:::create / uid == 12345 / { printf("%d",  pid); stop() }'

This simply stops the process just at the point of creation, before it can go on to do anything. Once stopped they could be killed.  I regret that I did not have time to investigate the child producing a child symptom and so far I’ve not been able to reproduce it.

I’m certain it will happen again though so I’ll investigate it then.

Advertisements

From → bsc, Solaris

6 Comments
  1. Mike Gerdts permalink

    You can do this without dtrace as well

    pkill -STOP -u $baduser
    pkill -STOP -u $baduser
    pkill -STOP -u $baduser

    Repeat until ps -ltu $baduser shows all processes with state T. Normally two or three iterations is all that is needed. Then

    pkill -9 -u $baduser

    This does assume that you have tuned your system a bit such that there are some process table slots left for the sysadmin to log in and fix things. I normally tune systems such that maxuprc is in the 5000 – 10000 range, thus requiring more than two $badusers to force me to crash the system.

    • chrisgerhard permalink

      The is indeed what I had tried to do to get them to stop. I suspect having a large MP system was not helping me but I had tried to kill all the processes many times and still the process count matched the max-lwps limit for the project.

      I have a theory how this can happen that I will test when I can get hold of a suitably large system to try it out on.

  2. Mike Gerdts permalink

    If all of the processes are stopped before you do kill -9, you should have no problems. If any are not stopped, as soon as you kill -9 one of the stopped processes it will be replaced by one of the processes that is not stopped. This approach has worked quite reliably for me for over a decade. I suspect that you did not have them all kill -STOP’d or you used kill -15, which will allow (a non-default) signal handler to break free from your kill attempt.

    • chrisgerhard permalink

      That must be it. Not stopped. It only takes one to recreate the havoc.

      I still like the dtrace solution though.

      Thanks

      • chrisgerhard permalink

        Ito looks like the reason I need the dtrace to stop the fork bomb is due to bug 6898322.

        With out that bug Mike’s solution would be just fine.

  3. alecm permalink

    yr fixed width blog theme truncates code. 😦

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: