Linux & Networking for DevOps
What you’ll learn. How Linux models processes, files, and permissions; how systemd supervises services; how signals work (and why they matter for containers); how a request travels over TCP/IP, DNS, and TLS; and a practical debugging method with ss, curl, dig, strace, and tcpdump. Everything in Docker and Kubernetes is built from these primitives.
Processes
Section titled “Processes”A process is a running program: an address space, open file descriptors, a PID, a parent, credentials, and one or more threads. New processes are made by fork() (clone the caller) followed by execve() (replace the image with a new program). PID 1 is the first process (init/systemd) and adopts orphans.
ps -eo pid,ppid,user,stat,%cpu,%mem,cmd --sort=-%cpu | headpstree -p $$ # who is my parent chaincat /proc/$$/status # per-process kernel viewls -l /proc/$$/fd # open file descriptorsEverything is a file: /proc exposes kernel and process state, /sys exposes devices and cgroups. Learning to read /proc makes you independent of tools.
Process states in ps STAT: R running, S sleeping (interruptible), D uninterruptible sleep (usually blocked on I/O; cannot be killed), Z zombie (exited, parent has not called wait()), T stopped.
Load average and CPU
Section titled “Load average and CPU”uptime shows load averages: the average number of tasks that are runnable or in uninterruptible sleep over 1, 5, and 15 minutes. Compare with core count (nproc): load 8 on 8 cores is saturated; load 8 on 64 cores is mostly idle. High load with low CPU usually means I/O wait (D state). vmstat 1, iostat -x 1, and top/htop separate the cases.
Filesystems and permissions
Section titled “Filesystems and permissions”The Linux filesystem is a single tree rooted at /, assembled by mounting devices. Useful directories: /etc (config), /var (variable data, logs), /usr (programs), /tmp, /proc, /sys, /dev.
A file is an inode (metadata and block pointers) plus one or more directory entries (names). That is why hard links work and why deleting a file that a process holds open does not free space until the process closes it:
df -h # blocks used per filesystemdf -i # inodes used (can run out with millions of tiny files)du -sh /var/* | sort -hlsof +L1 # deleted-but-open files (the "disk full but du says no" mystery)Permissions
Section titled “Permissions”Each file has an owner, a group, and mode bits for user/group/other, each rwx (read 4, write 2, execute 1):
ls -l deploy.sh# -rwxr-x--- 1 alice ops 812 Nov 17 10:02 deploy.shchmod 750 deploy.sh # numericchmod g-w,o-rwx deploy.sh # symbolicchown alice:ops deploy.shFor directories, x means “may traverse/enter”, r means “may list names”, w means “may create/delete entries” (subject to the sticky bit on /tmp). Special bits: setuid/setgid (run with the file’s owner/group privileges; a classic privilege escalation vector), sticky (only the owner can delete in a shared directory). umask sets the default mask; umask 022 gives files 644.
Beyond classic mode bits: POSIX ACLs (setfacl), capabilities (split root into fine-grained powers such as CAP_NET_BIND_SERVICE), and mandatory access control (SELinux, AppArmor). Containers rely heavily on capability dropping; see DevSecOps.
sudo setcap 'cap_net_bind_service=+ep' ./myserver # bind port 80 without rootgetcap ./myserverUsers and sudo
Section titled “Users and sudo”Never run services as root if avoidable. Create a system user with no login shell:
sudo useradd --system --no-create-home --shell /usr/sbin/nologin myappid myappEdit sudoers only through visudo; prefer drop-ins under /etc/sudoers.d/ and narrow command lists to blanket ALL.
Signals
Section titled “Signals”Signals are asynchronous notifications to a process:
| Signal | Number | Default | Typical use |
|---|---|---|---|
SIGHUP |
1 | terminate | terminal closed; daemons often reload config |
SIGINT |
2 | terminate | Ctrl-C |
SIGQUIT |
3 | core dump | Ctrl-\ |
SIGKILL |
9 | terminate | cannot be caught or ignored |
SIGTERM |
15 | terminate | polite “please shut down” |
SIGSTOP |
19 | stop | cannot be caught |
SIGCHLD |
17 | ignore | child exited |
Graceful shutdown matters in production: orchestrators send SIGTERM, wait a grace period (Kubernetes default 30s), then SIGKILL. Your program must catch SIGTERM, stop accepting new work, finish in-flight requests, flush, and exit.
import signal, sys, threadingstop = threading.Event()signal.signal(signal.SIGTERM, lambda *_: stop.set())signal.signal(signal.SIGINT, lambda *_: stop.set())while not stop.is_set(): ... # do worksys.exit(0)A classic container bug: a shell-form CMD ./start.sh runs under /bin/sh -c, which does not forward SIGTERM to the child, so the app is killed hard after the grace period. Use exec form (CMD ["./app"]) or exec ./app in scripts.
systemd
Section titled “systemd”systemd is PID 1 on nearly all mainstream distros. It manages units: services, timers, sockets, mounts, targets. A service unit:
[Unit]Description=My applicationAfter=network-online.targetWants=network-online.target
[Service]User=myappGroup=myappWorkingDirectory=/opt/myappEnvironmentFile=/etc/myapp/envExecStart=/opt/myapp/bin/myapp --port 8080Restart=on-failureRestartSec=3# hardeningNoNewPrivileges=trueProtectSystem=strictProtectHome=truePrivateTmp=trueReadWritePaths=/var/lib/myapp# resource limits via cgroupsMemoryMax=512MCPUQuota=100%LimitNOFILE=65536
[Install]WantedBy=multi-user.targetsudo systemctl daemon-reloadsudo systemctl enable --now myappsystemctl status myappjournalctl -u myapp -f # follow logsjournalctl -u myapp --since "1 hour ago" -p errsystemd-analyze security myapp # score the sandboxingNote Type=: simple (default; process in foreground), forking, notify (service signals readiness via sd_notify), oneshot. Restart=on-failure plus RestartSec gives you a crude supervisor; StartLimitBurst prevents crash loops from hammering.
Timers replace cron; they are covered in Shell Scripting.
Namespaces and cgroups, briefly
Section titled “Namespaces and cgroups, briefly”Containers are not a kernel feature; they are processes with restricted views. Namespaces limit what a process can see (PID, network, mount, UTS, IPC, user, cgroup), cgroups limit what it can use (CPU, memory, I/O, PIDs). You can feel this with unshare:
sudo unshare --pid --fork --mount-proc bashps aux # only shows processes in this namespace; bash is PID 1The full picture is in Docker & Containers.
Networking fundamentals
Section titled “Networking fundamentals”The model
Section titled “The model”The practical stack (TCP/IP, four layers):
Application HTTP, DNS, TLS, SSH, gRPC (your code, libraries)Transport TCP, UDP, QUIC (ports, reliability)Internet IP, ICMP (addresses, routing)Link Ethernet, Wi-Fi, ARP (local delivery)An IP address identifies an interface; a port (0-65535) identifies a socket on a host. A connection is identified by the 5-tuple (protocol, src IP, src port, dst IP, dst port).
CIDR and subnets
Section titled “CIDR and subnets”10.0.1.0/24 means the first 24 bits are the network; 8 host bits give addresses (254 usable in classic IPv4 LAN, and cloud providers reserve a few more). A /16 has addresses. Private ranges: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16. Overlapping CIDRs across VPCs or clusters prevent peering later, so plan address space early (Cloud Fundamentals).
ip -br addr # interfaces and addressesip route # routing table; "default via" is the gatewayip route get 8.8.8.8 # which route/interface would be usedTCP provides a reliable ordered byte stream. Connection setup is a three-way handshake:
Client Server | ---- SYN ---------> | | <--- SYN+ACK ------ | | ---- ACK ---------> | (connection established; 1 RTT spent)Teardown uses FIN/ACK in each direction; after closing, the side that closes first sits in TIME_WAIT (typically 60s) so stray packets from the old connection do not corrupt a new one. Thousands of TIME_WAIT sockets are normal for a busy client; connection pooling and keep-alive avoid the cost. TCP also provides flow control (receive window) and congestion control (slow start, CUBIC/BBR). Latency-sensitive systems pay for handshakes: TCP (1 RTT) plus TLS 1.3 (1 more RTT, 0 with resumption) before the first byte. HTTP/3 over QUIC (UDP) merges these.
UDP is connectionless, unordered, and unreliable; DNS, QUIC, and many real-time protocols use it.
DNS maps names to records. A lookup goes: application -> stub resolver (/etc/resolv.conf, systemd-resolved) -> recursive resolver (ISP, 1.1.1.1, 8.8.8.8, a cloud resolver) -> root -> TLD (.com) -> authoritative server for the zone. Results are cached for the record’s TTL.
Record types you meet constantly: A/AAAA (IPv4/IPv6 address), CNAME (alias, cannot coexist with other records at the same name, so not at the zone apex), MX, TXT (SPF, domain verification), NS, SRV, CAA (which CAs may issue certs).
dig example.com A +shortdig @1.1.1.1 example.com AAAAdig +trace example.com # walk the delegation from the rootdig -x 93.184.216.34 # reverse lookupgetent hosts example.com # what libc (and thus most programs) resolves“It’s always DNS” is a joke with substance: stale caches after changing a record (lower the TTL before a migration, not after), split-horizon setups, and Kubernetes’ ndots:5 default (which makes short names do several failing lookups) all cause real outages.
HTTP and TLS
Section titled “HTTP and TLS”HTTP/1.1 is text-based request/response over TCP; HTTP/2 multiplexes streams over one TCP connection (binary framing, header compression); HTTP/3 uses QUIC. The status code classes: 2xx success, 3xx redirect, 4xx client error, 5xx server error. Know the usual suspects: 301/302/307/308, 401 vs 403, 429 (rate limited), 502 (bad gateway: proxy got garbage from upstream), 503 (unavailable), 504 (upstream timeout).
TLS provides confidentiality, integrity, and server authentication. Simplified TLS 1.3 handshake: client sends supported ciphers and a key share; server returns its key share and certificate chain; both derive session keys (ephemeral Diffie-Hellman gives forward secrecy); client validates the chain against trusted roots, checks the hostname (SAN) and validity dates. Certificates come from CAs; Let’s Encrypt automates this with ACME and 90-day certs, which you should renew automatically (certbot, cert-manager).
curl -vI https://example.com 2>&1 | grep -E 'HTTP/|TLS|subject|expire|issuer'openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \ | openssl x509 -noout -subject -issuer -datesExpired certificates are among the most common preventable outages. Monitor expiry (see Observability).
Ports, firewalls, NAT
Section titled “Ports, firewalls, NAT”Ports below 1024 are privileged (need root or CAP_NET_BIND_SERVICE). A firewall filters packets. On Linux the kernel facility is netfilter, programmed via nftables (modern) or iptables (legacy), with friendlier front-ends ufw and firewalld.
sudo ufw default deny incomingsudo ufw default allow outgoingsudo ufw allow 22/tcpsudo ufw allow 443/tcpsudo ufw enablesudo ufw status verboseNAT rewrites addresses so many private hosts share one public IP; Kubernetes service proxying and Docker bridge networking are both NAT/iptables under the hood. Cloud security groups are stateful allow-lists attached to instances; network ACLs are stateless and subnet-level. Default to deny, open only what is needed.
The debugging toolbox
Section titled “The debugging toolbox”A method beats memorizing tools: go up the stack, and at each layer ask “does this work?”
- Is the thing running?
systemctl status,ps. - Is it listening?
ss -ltnpshows listening TCP sockets with owning process. Listening on127.0.0.1rather than0.0.0.0is a top cause of “works locally, unreachable remotely”. - Can I reach it locally?
curl -v localhost:8080/health. - Can I reach it over the network?
nc -zv host 8080,curlfrom another host. Failure types differ: connection refused (nothing listening, or RST by firewall), timeout (packets dropped by firewall/route), no route to host. - Does DNS resolve correctly?
dig. - Is TLS healthy?
openssl s_client. - What does the process do?
strace, logs. - What is on the wire?
tcpdump.
# socketsss -ltnp # listening TCP + processss -tan state established '( dport = :443 )' | headss -s # summary counts per state
# HTTP with timing breakdowncurl -s -o /dev/null -w \ 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' \ https://example.com
# force a name to an IP without touching DNS (test before cutover)curl --resolve example.com:443:203.0.113.10 https://example.com/
# trace syscalls: what files/sockets does it touch, why does it fail?strace -f -e trace=network,openat -p "$(pgrep -f myapp)"strace -c -f ./myprog # syscall time summary
# packetssudo tcpdump -i any -nn port 53 # DNS trafficsudo tcpdump -i eth0 -nn 'tcp port 8080 and host 10.0.1.5' -w /tmp/cap.pcap# open the pcap in Wireshark for dissection
# path and lossmtr -rwc 20 example.com # traceroute+ping combinedThe timing breakdown from curl -w is especially valuable: it tells you instantly whether latency is DNS, TCP connect, TLS, or the server’s think time.
A worked diagnosis
Section titled “A worked diagnosis”Symptom: the app returns 502 from the load balancer intermittently.
ss -ltnpon the backend shows the app listening. Good.ss -sshows many connections inCLOSE_WAIT: the remote closed, but our app has not calledclose(). A file-descriptor leak candidate.ls /proc/$(pgrep myapp)/fd | wc -lreturns 65,000 against aLimitNOFILEof 65536 (see the systemd unit above). When the limit is hit,accept()fails, LB health checks time out, 502s follow.- Fix: close connections properly (a code fix), and meanwhile alert on fd usage.
The pattern (hypothesis, minimal tool, evidence) is the entire skill.
Kernel tuning, sparingly
Section titled “Kernel tuning, sparingly”Defaults are fine for most workloads. When you do tune, use sysctl and persist in /etc/sysctl.d/:
sysctl net.core.somaxconn net.ipv4.ip_local_port_rangeecho 'net.core.somaxconn=4096' | sudo tee /etc/sysctl.d/90-tuning.confsudo sysctl --systemChange one thing at a time and measure before and after. Cargo-cult sysctl lists from blog posts are a common source of obscure problems.
Key takeaways
Section titled “Key takeaways”- A process is created by fork+exec, is supervised by its parent (or systemd), and is controlled by signals; implement graceful SIGTERM handling and use exec-form commands in containers.
- Permissions are owner/group/other bits plus capabilities and MAC; run services as unprivileged users and sandbox them with systemd directives.
- Containers are namespaces (visibility) plus cgroups (resource limits); this explains most container behavior.
- Debug networking layer by layer: listening, local reach, remote reach, DNS, TLS, syscalls, packets.
curl -wtiming splits latency by phase. - Distinguish “refused” (active reject) from “timeout” (dropped) from “no route”.
- Plan CIDR ranges early; lower DNS TTLs before migrations; monitor certificate expiry.
Exercises
Section titled “Exercises”- Hands-on. Write a systemd unit for a simple Python HTTP server (
python3 -m http.server 8080) running as a dedicated user withRestart=on-failure. Kill the process withkill -9and observe the restart injournalctl. - Conceptual. Why can a
CNAMEnot be placed at a zone apex (e.g.,example.comitself)? How do DNS providers work around it? - Hands-on. Using only
curl -w, determine how much of the latency to a site is DNS, TCP connect, TLS, and server time. Repeat with--http2and--http3if your curl supports it. - Diagnosis.
df -hshows a disk 100% full butdu -sh /*totals far less. List two causes and the command that detects each. - Calculation. How many usable host addresses are in a
/22? How many/24subnets fit in a/16?
Solutions (2, 4, 5)
- A CNAME means “this name is an alias for another name; use that name’s records”. The apex must hold SOA and NS records, and a CNAME cannot coexist with other record types at the same name. Providers offer proprietary ALIAS/ANAME/flattening records that resolve the target server-side and return A/AAAA records.
- (a) Deleted files still held open by a process:
lsof +L1; restart or truncate via/proc/PID/fd/N. (b) A different filesystem mounted over a directory that already contained data (hidden files under the mount point): unmount or bind-mount/elsewhere andduit. (Also inode exhaustion:df -i.) - A
/22has addresses, so 1022 usable in classic terms (cloud providers reserve a few more). A/16contains/24subnets.
Further reading
Section titled “Further reading”- Brendan Gregg, Systems Performance (2nd ed., 2020).
- W. Richard Stevens, TCP/IP Illustrated, Volume 1; Kerrisk, The Linux Programming Interface (2010).
- Julia Evans’ zines on networking and Linux debugging tools.
- The
manpages:ss(8),tcpdump(8),systemd.service(5),capabilities(7),signal(7). - Ilya Grigorik, High Performance Browser Networking (2013), free online.