Linux Process Management: ps, top, kill, systemd and cgroups
See what runs on Linux with ps and top, stop processes with signals, change priorities, read load and memory, and manage services with systemd.
Every program that runs on Linux is a process: your shell, the web server, the SSH daemon, the ls you just typed. When a server is slow, a port is taken, or memory runs out, the answer is in the process list. This part shows how to find processes, read what they are doing, stop them cleanly, change their priority, and manage long-running services with systemd.
Prerequisites
- The command line basics from Part 3 and users from Part 6.
- Python 3, which Ubuntu, Debian and Fedora install by default. It runs a tiny web server as the example process.
- A fresh
~/labfrom Part 3.
The output comes from Ubuntu 24.04.5 LTS, run inside a small lab container. A container has its own process list, so it shows only a handful of processes with low process IDs (PIDs); a real server lists hundreds. The systemctl output comes from a Debian 12 machine and says so.
What a Process Is
Each process has:
- a PID (process ID) and a PPID, the PID of the parent that started it;
- an owner (the user whose permissions it has);
- a state: running, sleeping, stopped, or zombie;
- its own memory, open files, environment variables and current directory.
New processes are created by existing ones, so all processes form a tree. At the root is PID 1: on almost every distribution that is systemd, which starts services at boot and adopts orphaned processes. On a typical Ubuntu server, a branch of pstree looks like this (example output):
systemd─┬─cron
├─sshd───sshd───sshd───bash───python3
├─systemd-journal
├─systemd-logind
└─systemd-resolve
Inside a container, PID 1 is usually the application itself or a small init program, not systemd. When that process exits, the container stops, which is why some containers exit right away.
See What Is Running
Start a process to look at: a small web server that serves ~/lab/site in the background. & puts it in the background and gives you the prompt back; $! holds its PID.
The two classic ways to list everything:
ps aux(BSD style): every process withUSER,PID,%CPU,%MEM,VSZ(virtual memory),RSS(memory actually in RAM, in KiB),TTY,STAT,START,TIME(CPU time used) andCOMMAND.ps -ef(System V style):UID,PID,PPID,C,STIME,TTY,TIMEandCMD. Use it when you need the parent PID.
Add --sort=-%cpu or --sort=-%mem to sort, and --forest (with -ef) to draw the tree. To find a process by name, prefer pgrep -a name over ps aux | grep name, which also finds the grep itself.
The STAT column shows the state:
| Code | State | Meaning |
|---|---|---|
R |
Running | On a CPU, or ready to run |
S |
Sleeping | Waiting for something, such as network input; most processes are here |
D |
Uninterruptible sleep | Waiting for I/O, usually disk or NFS. Many D processes point to a storage problem, and they cannot be killed until the I/O finishes |
T |
Stopped | Paused, for example with Ctrl+Z |
Z |
Zombie | Finished, but the parent has not collected its exit code yet |
Extra letters: s session leader, l multi-threaded, + in the foreground, < high priority, N low priority.
Look Inside a Process: /proc
Everything ps shows comes from /proc/<PID>/. You can read it directly:
File descriptors 0, 1 and 2 are standard input, output and error: input comes from /dev/null and both outputs go to server.log, as the redirection asked. Descriptor 3 is the listening network socket.
Which Process Uses a Port or a File
ss lists sockets and lsof lists open files. Both answer "who is using this?":
ss -ltnp means: listening sockets, tCP, numeric addresses and ports, with the process. You see processes of other users only with sudo. The usual "Address already in use" error when a service starts means another process already holds the port, and these commands tell you which one. See how to kill a process running on a port.
Signals: Stopping and Reloading Processes
You control a running process by sending it a signal. The most important ones:
| Signal | Number | Default effect | Used for |
|---|---|---|---|
SIGTERM |
15 | Terminate | Polite stop request. The process can clean up first. The default of kill |
SIGINT |
2 | Terminate | What Ctrl+C sends |
SIGHUP |
1 | Terminate | Many daemons (nginx, sshd) reload their configuration on it instead |
SIGKILL |
9 | Terminate, cannot be caught | Last resort: no cleanup, files may be left half-written |
SIGSTOP / SIGCONT |
19 / 18 | Pause / resume | What job control uses (Ctrl+Z sends the similar SIGTSTP) |
Always try SIGTERM first and give the process a few seconds. Use kill -9 only when it does not react. A process in state D ignores even SIGKILL until its I/O returns, and a zombie is already dead: to get rid of zombies, fix or restart the parent, which must collect them.
pkill pattern matches against process names; pkill -f pattern matches the whole command line, which is powerful and easy to get wrong. Check what will match with pgrep -a (or pgrep -af) first. killall name (from psmisc) kills by exact name.
Kill Processes by Part of Their Name
Preview with pgrep, then send the signal with pkill and the same pattern. -e makes pkill print what it killed:
pgrep -af report-worker # what would match, with full command lines
pkill -e -f report-worker # stop them (SIGTERM), and list each one
Two traps decide which flag you need:
- Without
-f, the pattern matches the process name, which the kernel cuts to 15 characters. A longer pattern never matches.pgrepwarns about it,pkillgives the same warning, and both exit with 1. - With
-f, the pattern matches every command line that contains it, including the shell or script that runspkill.pkillnever kills itself, but it kills its parent if that has the pattern in its arguments. That ends abash -cline before its next command runs, and it killed our test session's own shell the first time we tried. Anchor the pattern, or use-xto match the whole command line exactly.
report-worker-n in the pkill output is the 15-character process name. Other filters narrow a match: -u deploy (only that user's processes), -n or -o (only the newest or oldest match) and -P 1234 (only children of process 1234).
Foreground, Background and Jobs
In an interactive shell:
- Ctrl+C stops the foreground command.
- Ctrl+Z pauses it.
bglets it continue in the background,fgbrings it back. command &starts it in the background directly.jobs -llists the background jobs of this shell with their PIDs, andkill %1signals job number 1.
Background jobs belong to your shell session. When you close the terminal or the SSH connection drops, the shell sends them SIGHUP and they usually die. For work that must keep running:
tmuxorscreen: a terminal session that survives disconnects. Run long jobs (upgrades, data migrations) inside one, and reattach withtmux attach.nohup command &: ignoresSIGHUP; output goes tonohup.out.- A systemd service (below and in Part 10): the right home for anything that should run permanently, restart on failure and start at boot.
Priorities: nice and renice
The nice value tells the scheduler how generously to treat a process: from -20 (highest priority) to 19 (lowest), with 0 as the default. Regular users can only make their processes nicer (lower priority); only root can raise the priority.
Use nice -n 19 for backups, compression and other batch jobs on a busy server. For disk-heavy jobs, ionice -c 3 command gives the process disk time only when nobody else needs it.
CPU, Load and Memory
Load Average
The three load averages are the average number of processes that were running or waiting to run (including D processes waiting for disk) over the last 1, 5 and 15 minutes. As a rule of thumb, compare them with the number of CPUs from nproc: on this 4-CPU machine, a sustained load of 4 from CPU-bound work keeps every CPU busy, and 8 means processes are queuing. Because processes waiting for disk count too, check the CPU usage in top before you blame the CPUs. A rising 1-minute value with a low 15-minute value means the load just started.
Find and Calm a CPU Hog
yes prints y forever, which makes it a perfect runaway process:
top shows the same thing live. On a server, top -b -n 1 -o %CPU | head -n 8 prints the summary and the busiest process (example output):
top - 09:14:02 up 16 days, 3:05, 1 user, load average: 1.12, 0.61, 0.38
Tasks: 187 total, 2 running, 185 sleeping, 0 stopped, 0 zombie
%Cpu(s): 25.4 us, 0.8 sy, 0.0 ni, 73.6 id, 0.0 wa, 0.0 hi, 0.2 si, 0.0 st
MiB Mem : 3796.8 total, 1767.6 free, 1026.5 used, 1080.1 buff/cache
MiB Swap: 2048.0 total, 2048.0 free, 0.0 used. 2770.3 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4131 dev 20 0 2700 1500 1396 R 100.0 0.0 0:03.18 yes
In the header, the load averages repeat uptime, and on the %Cpu(s) line us is time spent in user programs, sy in the kernel, wa waiting for disk, and st (steal) time a virtual machine waited for the hypervisor. A high wa points to storage, not CPU. Inside interactive top, press P to sort by CPU, M by memory, 1 to show each CPU, k to kill a process and q to quit. htop and btop are friendlier alternatives in the package repositories.
Memory
Do not read the free column as "memory left". Linux uses idle RAM as a disk cache (buff/cache) and gives it back as soon as programs need it. The number that matters is available: how much memory programs can get without the system starting to swap. Per process, look at RSS in ps or RES in top.
When memory runs out completely, the kernel's OOM killer ends the process it judges most expendable, often the biggest one. You find the evidence in the kernel log:
journalctl -k | grep -i -E 'out of memory|oom-kill'
On kernels with pressure stall information (PSI), /proc/pressure/cpu, /proc/pressure/memory and /proc/pressure/io show how much of the time processes were stalled waiting for each resource, which is a clearer warning sign than raw usage. For example, some avg10=12.50 in /proc/pressure/memory means at least one process waited for memory 12.5% of the last 10 seconds. Part 10 covers vmstat and the sysstat tools (iostat, pidstat, sar) for longer investigations.
How Do I Measure the Actual Memory Usage of a Process?
For one process, read RSS. For a program that runs as several processes, add up PSS from /proc/<pid>/smaps_rollup, because RSS counts shared memory once for every process that shares it. The columns in ps and top answer different questions:
| Number | Where you find it | What it counts |
|---|---|---|
VSZ, VIRT |
ps, top |
All address space the process has mapped, also memory it never touched. Not a measure of RAM |
RSS, RES |
ps, top, VmRSS in /proc/<pid>/status |
Pages of the process that are in RAM now, shared pages included in full |
| PSS | /proc/<pid>/smaps_rollup |
Like RSS, but each shared page is divided by the number of processes that share it. PSS values add up |
| USS | Private_Clean + Private_Dirty in smaps_rollup |
Memory that only this process uses: about what you get back when it exits |
The difference is large for servers that fork workers, such as PostgreSQL, nginx, gunicorn and PHP-FPM. In this test, a Python script fills a 200 MiB buffer and then forks a child, which shares the buffer with its parent:
# mem.py
import os, time
data = bytearray(200 * 1024 * 1024) # 200 MiB, written once by the zero fill
os.fork() # the child shares these pages until one side writes to them
time.sleep(30)
ps shows about 208 MiB for each process, 414 MiB together. They share 203 MiB (Shared_Dirty), so together they use about 206 MiB of RAM, which is the sum of the two Pss values. To add up the PSS of every process of one program:
for pid in $(pgrep -x postgres); do
sudo awk '/^Pss:/ {print $2}' "/proc/$pid/smaps_rollup"
done | awk '{ kb += $1 } END { printf "%.0f MiB\n", kb / 1024 }'
Reading smaps_rollup of another user's process needs root. The smem package prints PSS and USS for each process in one table (smem -k -P postgres).
Three more questions come up often:
- What was the peak?
VmHWMin/proc/<pid>/statusis the highest RSS so far. For a command that you start yourself,/usr/bin/time -v ./job.sh(the GNUtimepackage, not the shell keyword) printsMaximum resident set sizewhen it ends. - How much does a service or a container use? Look at its cgroup.
systemctl status app.serviceshows aMemory:line, and/sys/fs/cgroup/system.slice/app.service/memory.currenthas the value in bytes. It includes the page cache that the service's file reads created, which RSS does not, and it is the number that the cgroup'sMemoryMax=limit applies to.docker statsreads the same cgroup files but subtracts inactive cache. cgroup v2 memory.high vs memory.max, measured shows what happens near the limit. - Is it a leak? Watch the same number over hours, not seconds:
pidstat -r -p <pid> 60(fromsysstat) prints RSS every minute. Memory that grows after start-up and then stays level is usually a cache or a heap that the runtime keeps. Memory that keeps growing under steady traffic can be a leak, but caches and allocator fragmentation look the same from outside, so confirm it with the heap tools of the runtime. Runtimes report their own heap:process.memoryUsage()in Node.js,tracemallocin Python, and heap metrics in the JVM. A JVM heap limit (-Xmx) is not the RSS: the JVM also uses memory outside the heap.
Services With systemd
Long-running programs (web servers, databases, agents) run as services managed by systemd. systemd starts them at boot in the right order, restarts them when they crash, collects their output in the journal and limits their resources. You control them with systemctl. Output from a Debian 12 machine:
Read the status from the top: Loaded shows the unit file and whether the service starts at boot (enabled); Active shows whether it runs now and since when; Main PID and CGroup list its processes. On Ubuntu 24.04 and Debian 13, unit files that come with packages live in /usr/lib/systemd/system/; Debian 12 still uses /lib/systemd/system/, as above. Your own units go in /etc/systemd/system/.
The commands you will use most:
sudo systemctl start nginx # start now
sudo systemctl stop nginx # stop now
sudo systemctl restart nginx # stop, then start
sudo systemctl reload nginx # re-read the configuration without stopping (if supported)
sudo systemctl enable --now nginx # start at boot, and start now
sudo systemctl disable nginx # do not start at boot
systemctl status nginx # state, recent log lines, processes
journalctl -u nginx -f # follow the service's log
systemctl list-units --type=service --state=running
Part 10 writes a complete unit file for the lab web server and schedules jobs with timers.
cgroups: Resource Limits for Groups of Processes
systemd puts every service in its own control group (cgroup), and the kernel accounts CPU, memory, I/O and process counts per cgroup. All current distributions use the unified cgroup v2 hierarchy; systemd 258 and later no longer support the old v1 layout at all. Check yours:
You set limits in the unit file, and systemd writes them to the cgroup:
[Service]
# hard memory limit; the OOM killer acts inside the service only
MemoryMax=512M
# at most half of one CPU
CPUQuota=50%
# at most 100 processes and threads, which stops fork bombs
TasksMax=100
# a smaller share of disk time than other services (default 100)
IOWeight=50
systemd only treats a line as a comment when # or ; is its first character, so keep comments on their own lines: MemoryMax=512M # limit is an invalid value.
systemd-cgtop shows live usage per cgroup, and systemd-cgls the tree. Old tutorials use MemoryLimit= and write to paths such as /sys/fs/cgroup/memory/. Those come from cgroup v1: the paths do not exist on a cgroup v2 system, and MemoryLimit= is a deprecated name that newer systemd versions only translate for compatibility, so write MemoryMax=. To try a limit on a single command: systemd-run --user --scope -p MemoryMax=200M -- ./big-job.sh.
Namespaces: How Containers Isolate Processes
cgroups limit what a group of processes may use. Namespaces limit what it can see: its own process list, network stack, mount table and hostname. Docker, Podman and Kubernetes combine both. You can try a PID namespace with unshare:
Inside the new namespace, ps is PID 1 and sees nothing else. On Ubuntu 23.10 and later, AppArmor blocks unprivileged user namespaces for most programs, so this command fails with write failed /proc/self/uid_map: Operation not permitted; run sudo unshare --pid --fork --mount-proc ps -ef there instead. The Docker under the hood simulator walks through how a container runtime uses these features.
Scheduled Jobs
Processes that should run at set times (backups, reports, cleanup) are started by cron or by systemd timers. Part 10 covers both, with a timer for the backup script from Part 9.
Common Mistakes
- Reaching for
kill -9first. SendSIGTERM, wait, and only thenSIGKILL. - Killing zombies. They are already dead; restart or fix the parent process.
- Reading
freeinstead ofavailableand concluding the server is out of memory. - Comparing load average with 1 on a multi-core machine. Compare it with
nproc. - Running long jobs in a plain SSH session. Use
tmux, or make it a service. pkill -fwith a short pattern that also matches other users' processes or your own shell. Check withpgrep -affirst.- Following cgroup v1 tutorials (
MemoryLimit=,/sys/fs/cgroup/memory/...); useMemoryMax=and the cgroup v2 files.
Summary
- Processes have a PID, a parent, an owner and a state; PID 1 is systemd on most systems.
ps,pgrep -a,pstree,topand/proc/<PID>/show what runs;ss -ltnpandlsofshow who holds a port or a file.killsendsSIGTERMby default;SIGKILLis the last resort;pkillandkillallwork by name.niceandrenicechange CPU priority; only root can raise it.- Load average is compared with the CPU count; for memory, read
available. - systemd manages services with
systemctl, and cgroup v2 limits their resources.
Next: Networking in Linux, where you configure interfaces, follow routes and troubleshoot connections.
Keep Learning
- cgroup v2 memory.high vs memory.max, measured: what a service does near its memory limit
- Disk full but du disagrees: deleted files that a running process still holds
- Exercise: Linux performance tuning and troubleshooting
- Interview questions: Linux process debugging
- See what happens when processes multiply: fork bomb simulator
Tags
Found an issue?