Skip to content

Linux & Networking for DevOps

What you’ll learn. How Linux models processes, files, and permissions; how systemd supervises services; how signals work (and why they matter for containers); how a request travels over TCP/IP, DNS, and TLS; and a practical debugging method with ss, curl, dig, strace, and tcpdump. Everything in Docker and Kubernetes is built from these primitives.

A process is a running program: an address space, open file descriptors, a PID, a parent, credentials, and one or more threads. New processes are made by fork() (clone the caller) followed by execve() (replace the image with a new program). PID 1 is the first process (init/systemd) and adopts orphans.

Terminal window
ps -eo pid,ppid,user,stat,%cpu,%mem,cmd --sort=-%cpu | head
pstree -p $$ # who is my parent chain
cat /proc/$$/status # per-process kernel view
ls -l /proc/$$/fd # open file descriptors

Everything is a file: /proc exposes kernel and process state, /sys exposes devices and cgroups. Learning to read /proc makes you independent of tools.

Process states in ps STAT: R running, S sleeping (interruptible), D uninterruptible sleep (usually blocked on I/O; cannot be killed), Z zombie (exited, parent has not called wait()), T stopped.

uptime shows load averages: the average number of tasks that are runnable or in uninterruptible sleep over 1, 5, and 15 minutes. Compare with core count (nproc): load 8 on 8 cores is saturated; load 8 on 64 cores is mostly idle. High load with low CPU usually means I/O wait (D state). vmstat 1, iostat -x 1, and top/htop separate the cases.

The Linux filesystem is a single tree rooted at /, assembled by mounting devices. Useful directories: /etc (config), /var (variable data, logs), /usr (programs), /tmp, /proc, /sys, /dev.

A file is an inode (metadata and block pointers) plus one or more directory entries (names). That is why hard links work and why deleting a file that a process holds open does not free space until the process closes it:

Terminal window
df -h # blocks used per filesystem
df -i # inodes used (can run out with millions of tiny files)
du -sh /var/* | sort -h
lsof +L1 # deleted-but-open files (the "disk full but du says no" mystery)

Each file has an owner, a group, and mode bits for user/group/other, each rwx (read 4, write 2, execute 1):

Terminal window
ls -l deploy.sh
# -rwxr-x--- 1 alice ops 812 Nov 17 10:02 deploy.sh
chmod 750 deploy.sh # numeric
chmod g-w,o-rwx deploy.sh # symbolic
chown alice:ops deploy.sh

For directories, x means “may traverse/enter”, r means “may list names”, w means “may create/delete entries” (subject to the sticky bit on /tmp). Special bits: setuid/setgid (run with the file’s owner/group privileges; a classic privilege escalation vector), sticky (only the owner can delete in a shared directory). umask sets the default mask; umask 022 gives files 644.

Beyond classic mode bits: POSIX ACLs (setfacl), capabilities (split root into fine-grained powers such as CAP_NET_BIND_SERVICE), and mandatory access control (SELinux, AppArmor). Containers rely heavily on capability dropping; see DevSecOps.

Terminal window
sudo setcap 'cap_net_bind_service=+ep' ./myserver # bind port 80 without root
getcap ./myserver

Never run services as root if avoidable. Create a system user with no login shell:

Terminal window
sudo useradd --system --no-create-home --shell /usr/sbin/nologin myapp
id myapp

Edit sudoers only through visudo; prefer drop-ins under /etc/sudoers.d/ and narrow command lists to blanket ALL.

Signals are asynchronous notifications to a process:

Signal Number Default Typical use
SIGHUP 1 terminate terminal closed; daemons often reload config
SIGINT 2 terminate Ctrl-C
SIGQUIT 3 core dump Ctrl-\
SIGKILL 9 terminate cannot be caught or ignored
SIGTERM 15 terminate polite “please shut down”
SIGSTOP 19 stop cannot be caught
SIGCHLD 17 ignore child exited

Graceful shutdown matters in production: orchestrators send SIGTERM, wait a grace period (Kubernetes default 30s), then SIGKILL. Your program must catch SIGTERM, stop accepting new work, finish in-flight requests, flush, and exit.

import signal, sys, threading
stop = threading.Event()
signal.signal(signal.SIGTERM, lambda *_: stop.set())
signal.signal(signal.SIGINT, lambda *_: stop.set())
while not stop.is_set():
... # do work
sys.exit(0)

A classic container bug: a shell-form CMD ./start.sh runs under /bin/sh -c, which does not forward SIGTERM to the child, so the app is killed hard after the grace period. Use exec form (CMD ["./app"]) or exec ./app in scripts.

systemd is PID 1 on nearly all mainstream distros. It manages units: services, timers, sockets, mounts, targets. A service unit:

/etc/systemd/system/myapp.service
[Unit]
Description=My application
After=network-online.target
Wants=network-online.target
[Service]
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/myapp/env
ExecStart=/opt/myapp/bin/myapp --port 8080
Restart=on-failure
RestartSec=3
# hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/myapp
# resource limits via cgroups
MemoryMax=512M
CPUQuota=100%
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
Terminal window
sudo systemctl daemon-reload
sudo systemctl enable --now myapp
systemctl status myapp
journalctl -u myapp -f # follow logs
journalctl -u myapp --since "1 hour ago" -p err
systemd-analyze security myapp # score the sandboxing

Note Type=: simple (default; process in foreground), forking, notify (service signals readiness via sd_notify), oneshot. Restart=on-failure plus RestartSec gives you a crude supervisor; StartLimitBurst prevents crash loops from hammering.

Timers replace cron; they are covered in Shell Scripting.

Containers are not a kernel feature; they are processes with restricted views. Namespaces limit what a process can see (PID, network, mount, UTS, IPC, user, cgroup), cgroups limit what it can use (CPU, memory, I/O, PIDs). You can feel this with unshare:

Terminal window
sudo unshare --pid --fork --mount-proc bash
ps aux # only shows processes in this namespace; bash is PID 1

The full picture is in Docker & Containers.

The practical stack (TCP/IP, four layers):

Application HTTP, DNS, TLS, SSH, gRPC (your code, libraries)
Transport TCP, UDP, QUIC (ports, reliability)
Internet IP, ICMP (addresses, routing)
Link Ethernet, Wi-Fi, ARP (local delivery)

An IP address identifies an interface; a port (0-65535) identifies a socket on a host. A connection is identified by the 5-tuple (protocol, src IP, src port, dst IP, dst port).

10.0.1.0/24 means the first 24 bits are the network; 8 host bits give 28=2562^8 = 256 addresses (254 usable in classic IPv4 LAN, and cloud providers reserve a few more). A /16 has 216=655362^{16} = 65536 addresses. Private ranges: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16. Overlapping CIDRs across VPCs or clusters prevent peering later, so plan address space early (Cloud Fundamentals).

Terminal window
ip -br addr # interfaces and addresses
ip route # routing table; "default via" is the gateway
ip route get 8.8.8.8 # which route/interface would be used

TCP provides a reliable ordered byte stream. Connection setup is a three-way handshake:

Client Server
| ---- SYN ---------> |
| <--- SYN+ACK ------ |
| ---- ACK ---------> | (connection established; 1 RTT spent)

Teardown uses FIN/ACK in each direction; after closing, the side that closes first sits in TIME_WAIT (typically 60s) so stray packets from the old connection do not corrupt a new one. Thousands of TIME_WAIT sockets are normal for a busy client; connection pooling and keep-alive avoid the cost. TCP also provides flow control (receive window) and congestion control (slow start, CUBIC/BBR). Latency-sensitive systems pay for handshakes: TCP (1 RTT) plus TLS 1.3 (1 more RTT, 0 with resumption) before the first byte. HTTP/3 over QUIC (UDP) merges these.

UDP is connectionless, unordered, and unreliable; DNS, QUIC, and many real-time protocols use it.

DNS maps names to records. A lookup goes: application -> stub resolver (/etc/resolv.conf, systemd-resolved) -> recursive resolver (ISP, 1.1.1.1, 8.8.8.8, a cloud resolver) -> root -> TLD (.com) -> authoritative server for the zone. Results are cached for the record’s TTL.

Record types you meet constantly: A/AAAA (IPv4/IPv6 address), CNAME (alias, cannot coexist with other records at the same name, so not at the zone apex), MX, TXT (SPF, domain verification), NS, SRV, CAA (which CAs may issue certs).

Terminal window
dig example.com A +short
dig @1.1.1.1 example.com AAAA
dig +trace example.com # walk the delegation from the root
dig -x 93.184.216.34 # reverse lookup
getent hosts example.com # what libc (and thus most programs) resolves

“It’s always DNS” is a joke with substance: stale caches after changing a record (lower the TTL before a migration, not after), split-horizon setups, and Kubernetes’ ndots:5 default (which makes short names do several failing lookups) all cause real outages.

HTTP/1.1 is text-based request/response over TCP; HTTP/2 multiplexes streams over one TCP connection (binary framing, header compression); HTTP/3 uses QUIC. The status code classes: 2xx success, 3xx redirect, 4xx client error, 5xx server error. Know the usual suspects: 301/302/307/308, 401 vs 403, 429 (rate limited), 502 (bad gateway: proxy got garbage from upstream), 503 (unavailable), 504 (upstream timeout).

TLS provides confidentiality, integrity, and server authentication. Simplified TLS 1.3 handshake: client sends supported ciphers and a key share; server returns its key share and certificate chain; both derive session keys (ephemeral Diffie-Hellman gives forward secrecy); client validates the chain against trusted roots, checks the hostname (SAN) and validity dates. Certificates come from CAs; Let’s Encrypt automates this with ACME and 90-day certs, which you should renew automatically (certbot, cert-manager).

Terminal window
curl -vI https://example.com 2>&1 | grep -E 'HTTP/|TLS|subject|expire|issuer'
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates

Expired certificates are among the most common preventable outages. Monitor expiry (see Observability).

Ports below 1024 are privileged (need root or CAP_NET_BIND_SERVICE). A firewall filters packets. On Linux the kernel facility is netfilter, programmed via nftables (modern) or iptables (legacy), with friendlier front-ends ufw and firewalld.

Terminal window
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status verbose

NAT rewrites addresses so many private hosts share one public IP; Kubernetes service proxying and Docker bridge networking are both NAT/iptables under the hood. Cloud security groups are stateful allow-lists attached to instances; network ACLs are stateless and subnet-level. Default to deny, open only what is needed.

A method beats memorizing tools: go up the stack, and at each layer ask “does this work?”

  1. Is the thing running? systemctl status, ps.
  2. Is it listening? ss -ltnp shows listening TCP sockets with owning process. Listening on 127.0.0.1 rather than 0.0.0.0 is a top cause of “works locally, unreachable remotely”.
  3. Can I reach it locally? curl -v localhost:8080/health.
  4. Can I reach it over the network? nc -zv host 8080, curl from another host. Failure types differ: connection refused (nothing listening, or RST by firewall), timeout (packets dropped by firewall/route), no route to host.
  5. Does DNS resolve correctly? dig.
  6. Is TLS healthy? openssl s_client.
  7. What does the process do? strace, logs.
  8. What is on the wire? tcpdump.
Terminal window
# sockets
ss -ltnp # listening TCP + process
ss -tan state established '( dport = :443 )' | head
ss -s # summary counts per state
# HTTP with timing breakdown
curl -s -o /dev/null -w \
'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' \
https://example.com
# force a name to an IP without touching DNS (test before cutover)
curl --resolve example.com:443:203.0.113.10 https://example.com/
# trace syscalls: what files/sockets does it touch, why does it fail?
strace -f -e trace=network,openat -p "$(pgrep -f myapp)"
strace -c -f ./myprog # syscall time summary
# packets
sudo tcpdump -i any -nn port 53 # DNS traffic
sudo tcpdump -i eth0 -nn 'tcp port 8080 and host 10.0.1.5' -w /tmp/cap.pcap
# open the pcap in Wireshark for dissection
# path and loss
mtr -rwc 20 example.com # traceroute+ping combined

The timing breakdown from curl -w is especially valuable: it tells you instantly whether latency is DNS, TCP connect, TLS, or the server’s think time.

Symptom: the app returns 502 from the load balancer intermittently.

  • ss -ltnp on the backend shows the app listening. Good.
  • ss -s shows many connections in CLOSE_WAIT: the remote closed, but our app has not called close(). A file-descriptor leak candidate.
  • ls /proc/$(pgrep myapp)/fd | wc -l returns 65,000 against a LimitNOFILE of 65536 (see the systemd unit above). When the limit is hit, accept() fails, LB health checks time out, 502s follow.
  • Fix: close connections properly (a code fix), and meanwhile alert on fd usage.

The pattern (hypothesis, minimal tool, evidence) is the entire skill.

Defaults are fine for most workloads. When you do tune, use sysctl and persist in /etc/sysctl.d/:

Terminal window
sysctl net.core.somaxconn net.ipv4.ip_local_port_range
echo 'net.core.somaxconn=4096' | sudo tee /etc/sysctl.d/90-tuning.conf
sudo sysctl --system

Change one thing at a time and measure before and after. Cargo-cult sysctl lists from blog posts are a common source of obscure problems.

  • A process is created by fork+exec, is supervised by its parent (or systemd), and is controlled by signals; implement graceful SIGTERM handling and use exec-form commands in containers.
  • Permissions are owner/group/other bits plus capabilities and MAC; run services as unprivileged users and sandbox them with systemd directives.
  • Containers are namespaces (visibility) plus cgroups (resource limits); this explains most container behavior.
  • Debug networking layer by layer: listening, local reach, remote reach, DNS, TLS, syscalls, packets. curl -w timing splits latency by phase.
  • Distinguish “refused” (active reject) from “timeout” (dropped) from “no route”.
  • Plan CIDR ranges early; lower DNS TTLs before migrations; monitor certificate expiry.
  1. Hands-on. Write a systemd unit for a simple Python HTTP server (python3 -m http.server 8080) running as a dedicated user with Restart=on-failure. Kill the process with kill -9 and observe the restart in journalctl.
  2. Conceptual. Why can a CNAME not be placed at a zone apex (e.g., example.com itself)? How do DNS providers work around it?
  3. Hands-on. Using only curl -w, determine how much of the latency to a site is DNS, TCP connect, TLS, and server time. Repeat with --http2 and --http3 if your curl supports it.
  4. Diagnosis. df -h shows a disk 100% full but du -sh /* totals far less. List two causes and the command that detects each.
  5. Calculation. How many usable host addresses are in a /22? How many /24 subnets fit in a /16?
Solutions (2, 4, 5)
  1. A CNAME means “this name is an alias for another name; use that name’s records”. The apex must hold SOA and NS records, and a CNAME cannot coexist with other record types at the same name. Providers offer proprietary ALIAS/ANAME/flattening records that resolve the target server-side and return A/AAAA records.
  2. (a) Deleted files still held open by a process: lsof +L1; restart or truncate via /proc/PID/fd/N. (b) A different filesystem mounted over a directory that already contained data (hidden files under the mount point): unmount or bind-mount / elsewhere and du it. (Also inode exhaustion: df -i.)
  3. A /22 has 210=10242^{10} = 1024 addresses, so 1022 usable in classic terms (cloud providers reserve a few more). A /16 contains 28=2562^{8} = 256 /24 subnets.
  • Brendan Gregg, Systems Performance (2nd ed., 2020).
  • W. Richard Stevens, TCP/IP Illustrated, Volume 1; Kerrisk, The Linux Programming Interface (2010).
  • Julia Evans’ zines on networking and Linux debugging tools.
  • The man pages: ss(8), tcpdump(8), systemd.service(5), capabilities(7), signal(7).
  • Ilya Grigorik, High Performance Browser Networking (2013), free online.