Dev hunts the CPU killer

Dev's monitoring fired at 11am Tuesday. CPU 98%. App response time 45 seconds. Users complaining on Twitter. His manager said: You have 10 minutes.

Step 1: Confirm the problem.

uptime
# load average: 15.2, 12.1, 8.3

Load average 15.2 on a 4-core server means 11 processes are queued waiting for CPU. The server is drowning.

Step 2: Find the culprit.

ps aux --sort=-%cpu | head -10

Output:

USER    PID   %CPU %MEM  COMMAND
app    4521   97.3  2.1  java -jar analytics.jar
tomcat 1234    0.8 12.3  java -jar payment.jar

analytics.jar was eating 97.3% CPU. Someone triggered a large report that went into an infinite loop.

Every program in Linux is a process with a PID (unique ID), %CPU, %MEM, and COMMAND.

ps aux              # all processes, all users
ps aux | grep java  # find all java processes
top                 # live dashboard (press P for CPU, M for memory, q to quit)

Dev noted the PID 4521. Before killing anything he confirmed it with: cat /proc/4521/cmdline

Then he killed it correctly:

kill 4521         # polite - gives process a chance to clean up
sleep 10
ps aux | grep 4521   # is it gone?
# if still running:
kill -9 4521      # force kill - last resort only

CPU dropped from 97% to 8% in 3 seconds. Response time went from 45s to 200ms.

Never jump to kill -9 immediately. Forced kills on databases can corrupt data.

Inside top keyboard shortcuts: P sorts by CPU, M sorts by memory, 1 shows each core individually, k kills a process by PID, q quits.

The OOM Killer: when memory is completely full, Linux kills the process using the most memory to protect itself. Your app disappears with nothing in its own log.

grep -i "killed process" /var/log/kern.log | tail -5
# kernel: Kill process 1234 (java) score 890