Shell Scripting & Automation
What you’ll learn. The parts of bash that earn their keep in automation: quoting, exit codes, pipes and redirection, strict mode and its limits, parameter expansion, trap cleanup, jq for JSON, scheduling (cron, systemd timers), Makefiles as a task runner, and a decision rule for switching to Python. It builds on Linux & Networking.
Why shell still matters
Section titled “Why shell still matters”The shell is the glue language of operations: every CI step (CI/CD), every Dockerfile RUN, every incident-time one-liner is shell. It is excellent at composing programs and poor at data structures, error handling, and testing. Learn it well enough to write 30-line scripts safely and to recognize when a script has outgrown the tool.
Core mechanics
Section titled “Core mechanics”Exit codes and control flow
Section titled “Exit codes and control flow”Every command returns an exit status: 0 is success, non-zero is failure. if, &&, || branch on it, not on output.
if grep -q "ERROR" app.log; then echo "found errors"fi
mkdir -p /tmp/work && cd /tmp/work || exit 1command -v jq >/dev/null || { echo "jq required" >&2; exit 1; }$? holds the last status. [[ ... ]] is bash’s extended test (prefer it over [ ... ]): [[ -f file ]], [[ $x == foo* ]], [[ -z $var ]].
Quoting: the number-one bug source
Section titled “Quoting: the number-one bug source”Unquoted variable expansions undergo word splitting and globbing. Always quote:
file="my report.txt"rm $file # BAD: tries to remove "my" and "report.txt"rm -- "$file" # good; -- ends option parsing so a file named -rf cannot be an optionUse "$@" to forward arguments faithfully, arrays for lists:
args=(--verbose --out "$dir/out dir")mytool "${args[@]}"Single quotes are literal; double quotes allow $var, $(cmd), and backslash escapes.
Pipes and redirection
Section titled “Pipes and redirection”A pipe connects stdout of one process to stdin of the next; each stage runs concurrently.
# top 5 client IPs hitting an nginx logawk '{print $1}' access.log | sort | uniq -c | sort -rn | head -5
# redirectionscmd > out.txt # stdout to file (truncate)cmd >> out.txt # appendcmd 2> err.txt # stderrcmd > all.txt 2>&1 # both to the same file (order matters!)cmd &> all.txt # bash shorthandcmd < input.txtcmd <<'EOF' # here-document, quoted delimiter => no expansionliteral $text hereEOFdiff <(sort a) <(sort b) # process substitutionFile descriptors: 0 stdin, 1 stdout, 2 stderr. Send diagnostics to stderr (echo "msg" >&2) so stdout stays machine-parseable.
The text toolkit
Section titled “The text toolkit”grep, sed, awk, cut, sort, uniq, tr, xargs, find, tee, head/tail. A few high-value forms:
find . -name '*.log' -mtime +7 -print0 | xargs -0 rm -f # null-delimited: safe for odd namessed -i.bak 's/old/new/g' config.yaml # in-place with backupawk -F, '$3 > 100 {sum += $3} END {print sum}' sales.csvgrep -rn --include='*.py' 'TODO' src/xargs -P4 -n1 curl -s -o /dev/null -w '%{http_code}\n' < urls.txt # 4 parallelSafe scripting
Section titled “Safe scripting”The strict-mode header
Section titled “The strict-mode header”#!/usr/bin/env bashset -euo pipefailIFS=$'\n\t'-e(errexit): exit when a command fails.-u(nounset): referencing an unset variable is an error (catchesrm -rf "$DIR/"whenDIRis empty).-o pipefail: a pipeline’s status is that of the last non-zero stage, not just the last command.
Know the limits; set -e is famously quirky. It is ignored inside conditions (if cmd, cmd || ..., cmd && ...), in functions called from such contexts, and does not trigger for failures inside $(...) assigned within local x=$(cmd) (declare then assign separately). Handle important failures explicitly:
local outout=$(some_command) || { echo "some_command failed" >&2; return 1; }Also pipefail can bite: grep -q closes the pipe early and the upstream gets SIGPIPE (exit 141). Example: cmd | grep -q pattern may be reported as failed. Capture to a variable first or tolerate 141.
Cleanup with trap
Section titled “Cleanup with trap”tmp=$(mktemp -d)cleanup() { rm -rf "$tmp"; }trap cleanup EXIT # runs on normal exit, error exit, and (usually) signalstrap 'echo "failed at line $LINENO" >&2' ERRParameter expansion
Section titled “Parameter expansion”Bash can do a lot without spawning processes:
: "${ENVIRONMENT:?ENVIRONMENT must be set}" # fail if unset/emptyname="${1:-world}" # default valuefile=/var/log/app/server.logecho "${file##*/}" # server.log (basename)echo "${file%/*}" # /var/log/app (dirname)echo "${file%.log}.txt" # /var/log/app/server.txtecho "${name^^}" # uppercase (bash 4+)Argument parsing
Section titled “Argument parsing”usage() { echo "usage: $0 [-n namespace] [-v] <deployment>" >&2; exit 2; }ns=default verbose=0while getopts ":n:vh" opt; do case $opt in n) ns=$OPTARG ;; v) verbose=1 ;; h) usage ;; *) usage ;; esacdoneshift $((OPTIND - 1))[[ $# -eq 1 ]] || usagedeploy=$1Lint and test
Section titled “Lint and test”- ShellCheck (
shellcheck script.sh) catches quoting bugs, bad tests, and portability issues; run it in CI. It is the single best investment. - shfmt for formatting.
- bats-core for tests when a script matters enough.
A complete example
Section titled “A complete example”#!/usr/bin/env bash# rollout-wait.sh: wait for a deployment, roll back on failure.set -euo pipefail
usage() { echo "usage: $0 [-n namespace] <deployment>" >&2; exit 2; }ns=defaultwhile getopts ":n:" o; do case $o in n) ns=$OPTARG ;; *) usage ;; esac; doneshift $((OPTIND - 1))[[ $# -eq 1 ]] || usagedeploy=$1
log() { printf '%s %s\n' "$(date -u +%FT%TZ)" "$*" >&2; }
if kubectl -n "$ns" rollout status "deploy/$deploy" --timeout=180s; then log "rollout of $deploy succeeded"else log "rollout of $deploy failed; rolling back" kubectl -n "$ns" rollout undo "deploy/$deploy" exit 1fiIdempotency and automation principles
Section titled “Idempotency and automation principles”A script that automates infrastructure should be idempotent: running it twice leaves the same end state as running it once. Prefer mkdir -p, ln -sfn, useradd guarded by id user ||, and “ensure” semantics over “do” semantics. When state management gets complex, that is the signal to use a declarative tool (Infrastructure as Code) rather than more bash.
Other habits: log with timestamps, print what you are about to do (a --dry-run flag is cheap), never embed secrets (read from env or a secret manager; see DevSecOps), and avoid curl | bash for anything unpinned and unverified.
jq: JSON on the command line
Section titled “jq: JSON on the command line”Cloud CLIs, kubectl, and APIs speak JSON. jq is a small functional language over JSON.
# sample: kubectl get pods -o jsonkubectl get pods -o json | jq -r '.items[] | [.metadata.name, .status.phase] | @tsv'
# filter and selectkubectl get pods -o json | jq -r ' .items[] | select(.status.containerStatuses[]?.restartCount > 3) | .metadata.name'
# build JSON safely (never string-concatenate JSON in shell)jq -n --arg env "$ENVIRONMENT" --argjson replicas 3 \ '{environment: $env, replicas: $replicas}'
# aggregatecurl -s https://api.example.com/orders | \ jq '[.[] | .total] | add / length' # averageKey forms: .field, .[] (iterate), | (pipe), select(cond), map(f), group_by(.k), -r (raw strings, no quotes), --arg/--argjson (safe variable injection), ? (suppress errors on missing). yq does the same for YAML; both kubectl and the AWS CLI also have built-in --query (JMESPath) and -o jsonpath options.
Scheduling
Section titled “Scheduling”# m h dom mon dow command */5 * * * * /opt/scripts/healthcheck.sh 15 3 * * 1-5 /opt/scripts/backup.sh >> /var/log/backup.log 2>&1Edit with crontab -e. Pitfalls: cron runs with a minimal PATH and environment (use absolute paths, set PATH= at the top); output is emailed or discarded unless redirected; no built-in protection against overlapping runs (wrap in flock -n /var/lock/backup.lock cmd); time zone is the system’s; a missed run (host down) is simply skipped; and there is no central view of failures.
systemd timers
Section titled “systemd timers”Timers add logging via the journal, dependency handling, resource limits, randomized delay, and catch-up of missed runs.
[Unit]Description=Nightly backup[Service]Type=oneshotUser=backupExecStart=/opt/scripts/backup.sh[Unit]Description=Run backup nightly[Timer]OnCalendar=*-*-* 03:15:00RandomizedDelaySec=300Persistent=true # run at boot if a run was missed[Install]WantedBy=timers.targetsudo systemctl enable --now backup.timersystemctl list-timersjournalctl -u backup.servicesystemd-analyze calendar '*-*-* 03:15:00' # validate a scheduleIn Kubernetes use a CronJob; in CI use scheduled workflows. For “must not silently fail” jobs, add a dead-man’s-switch (a monitoring service that alerts when a heartbeat stops arriving); see Observability.
Makefiles as a task runner
Section titled “Makefiles as a task runner”make is installed everywhere and gives a uniform entry point (make test, make deploy) so humans and CI use identical commands. Used as a task runner, targets are mostly .PHONY.
.DEFAULT_GOAL := help.PHONY: help build test lint run clean
IMAGE ?= myappTAG ?= $(shell git rev-parse --short HEAD)
help: ## Show this help @grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) | \ awk 'BEGIN{FS=":.*?## "}{printf " %-12s %s\n", $$1, $$2}'
build: ## Build the container image docker build -t $(IMAGE):$(TAG) .
test: ## Run unit tests pytest -q
lint: ## Static checks shellcheck scripts/*.sh ruff check .
run: build ## Run locally docker run --rm -p 8080:8080 $(IMAGE):$(TAG)
clean: rm -rf dist .pytest_cacheRules: recipe lines must start with a tab; $$ escapes $ for the shell; each line runs in its own shell unless you use .ONESHELL:; ?= allows override from the environment (make build TAG=v1). Real build-graph features (file targets with dependencies, so unchanged outputs are skipped) are what make make more than a script wrapper. Alternatives: just (a cleaner command runner without the build-system baggage), task (Taskfile), and language-native runners (npm run, poe).
When to switch to Python (or Go)
Section titled “When to switch to Python (or Go)”Bash excels at orchestrating processes and short pipelines. Switch when you see these signals:
| Signal | Why bash hurts |
|---|---|
| More than ~100 lines, or you need functions with real return values | No data structures beyond strings and flat arrays; everything is text |
Nested JSON/YAML processing beyond a few jq calls |
Unreadable, untestable |
| Needing HTTP with retries, auth refresh, pagination | You are rebuilding a client library |
| Parallelism with result collection and error handling | xargs -P and wait are crude |
| Cross-platform (macOS vs Linux flag differences, Windows) | Tool flags vary (GNU vs BSD sed, date) |
| Unit tests required | Possible with bats but awkward |
| Complex error handling and rollback | set -e semantics are subtle |
A comparable task in Python, which gains structure and testability:
#!/usr/bin/env python3import json, subprocess, sys
def sh(*cmd: str) -> str: return subprocess.run(cmd, check=True, text=True, capture_output=True).stdout
pods = json.loads(sh("kubectl", "get", "pods", "-o", "json"))["items"]flappy = [ p["metadata"]["name"] for p in pods if any(c.get("restartCount", 0) > 3 for c in p["status"].get("containerStatuses", []))]print("\n".join(flappy))sys.exit(1 if flappy else 0)Use subprocess.run([...], check=True) with a list (never shell=True with interpolated input), argparse or typer for flags, pathlib for paths, and uv/venv for dependency management. For a static distributable binary, Go. A pragmatic rule: start in bash for glue; port to Python at the first moment you wish you had a dictionary or a unit test.
Common pitfalls
Section titled “Common pitfalls”- Unquoted variables and
for f in $(ls); use globs (for f in *.txt) orfind -print0. - Parsing
lsoutput; usefind, globs, orstat. cd dir; rm -rf *withoutcd dir || exit. Useset -e, or better, absolute paths.- Relying on
set -ealone; check critical commands explicitly. - Cron jobs that fail silently.
- Secrets in command-line arguments (visible in
psand shell history). #!/bin/bashversus#!/usr/bin/env bash: the latter is more portable (macOS ships bash 3.2; associative arrays need bash 4+). Declare#!/bin/shonly if you truly stay POSIX (Alpine’sash, Debian’sdash).
Key takeaways
Section titled “Key takeaways”- Quote every expansion, use
"$@", arrays,--, and null-delimited pipelines for safety. - Start scripts with
set -euo pipefail, but understand its exceptions; addtrap ... EXITfor cleanup. - Use
jqto build and parse JSON instead of string-gluing it; lint with ShellCheck in CI. - Make automation idempotent and dry-runnable; prefer declarative tools when state grows.
- systemd timers beat cron for logging, catch-up, and dependency handling; use
flockagainst overlap. - A Makefile gives humans and CI the same verbs. Switch to Python once data structures, tests, or HTTP logic appear.
Exercises
Section titled “Exercises”- Hands-on. Write a script with strict mode that takes a URL list file, fetches each URL’s HTTP status with
curl, prints failures to stderr, and exits non-zero if any failed. Run ShellCheck on it. - Conceptual. Explain why
cmd > out.txt 2>&1andcmd 2>&1 > out.txtbehave differently. - Hands-on. Convert a cron entry running a nightly job into a systemd service + timer with
Persistent=true. Test it withsystemd-analyze calendarandsystemctl start. - Debugging. What does
set -e; local x=$(false); echo reachedprint inside a function, and why? - Design. Take a 150-line bash script from your own work. List which of the “switch to Python” signals apply and sketch the module layout.
Solutions (2, 4)
- Redirections are processed left to right. In the first, stdout goes to the file, then stderr is duplicated onto what stdout currently points to (the file): both end up in the file. In the second, stderr is duplicated onto the original stdout (the terminal), then stdout goes to the file: only stdout is in the file.
- It prints
reached. The exit status oflocal x=$(false)is that oflocal(0), masking the failure. Declare then assign:local x; x=$(false)triggers errexit.
Further reading
Section titled “Further reading”- Bash Reference Manual (GNU), and the
bash(1)man page. - Greg’s Wiki “BashGuide”, “BashFAQ”, and “BashPitfalls”.
- Ken Thompson, Brian Kernighan et al., The UNIX Programming Environment (Kernighan, Pike, 1984), for the pipe philosophy.
- The ShellCheck wiki (explains each warning).
- Stephen Dolan’s
jqmanual, and Al Sweigart’s Automate the Boring Stuff with Python for the Python side.