Skip to content

Shell Scripting & Automation

What you’ll learn. The parts of bash that earn their keep in automation: quoting, exit codes, pipes and redirection, strict mode and its limits, parameter expansion, trap cleanup, jq for JSON, scheduling (cron, systemd timers), Makefiles as a task runner, and a decision rule for switching to Python. It builds on Linux & Networking.

The shell is the glue language of operations: every CI step (CI/CD), every Dockerfile RUN, every incident-time one-liner is shell. It is excellent at composing programs and poor at data structures, error handling, and testing. Learn it well enough to write 30-line scripts safely and to recognize when a script has outgrown the tool.

Every command returns an exit status: 0 is success, non-zero is failure. if, &&, || branch on it, not on output.

Terminal window
if grep -q "ERROR" app.log; then
echo "found errors"
fi
mkdir -p /tmp/work && cd /tmp/work || exit 1
command -v jq >/dev/null || { echo "jq required" >&2; exit 1; }

$? holds the last status. [[ ... ]] is bash’s extended test (prefer it over [ ... ]): [[ -f file ]], [[ $x == foo* ]], [[ -z $var ]].

Unquoted variable expansions undergo word splitting and globbing. Always quote:

Terminal window
file="my report.txt"
rm $file # BAD: tries to remove "my" and "report.txt"
rm -- "$file" # good; -- ends option parsing so a file named -rf cannot be an option

Use "$@" to forward arguments faithfully, arrays for lists:

Terminal window
args=(--verbose --out "$dir/out dir")
mytool "${args[@]}"

Single quotes are literal; double quotes allow $var, $(cmd), and backslash escapes.

A pipe connects stdout of one process to stdin of the next; each stage runs concurrently.

Terminal window
# top 5 client IPs hitting an nginx log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -5
# redirections
cmd > out.txt # stdout to file (truncate)
cmd >> out.txt # append
cmd 2> err.txt # stderr
cmd > all.txt 2>&1 # both to the same file (order matters!)
cmd &> all.txt # bash shorthand
cmd < input.txt
cmd <<'EOF' # here-document, quoted delimiter => no expansion
literal $text here
EOF
diff <(sort a) <(sort b) # process substitution

File descriptors: 0 stdin, 1 stdout, 2 stderr. Send diagnostics to stderr (echo "msg" >&2) so stdout stays machine-parseable.

grep, sed, awk, cut, sort, uniq, tr, xargs, find, tee, head/tail. A few high-value forms:

Terminal window
find . -name '*.log' -mtime +7 -print0 | xargs -0 rm -f # null-delimited: safe for odd names
sed -i.bak 's/old/new/g' config.yaml # in-place with backup
awk -F, '$3 > 100 {sum += $3} END {print sum}' sales.csv
grep -rn --include='*.py' 'TODO' src/
xargs -P4 -n1 curl -s -o /dev/null -w '%{http_code}\n' < urls.txt # 4 parallel
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
  • -e (errexit): exit when a command fails.
  • -u (nounset): referencing an unset variable is an error (catches rm -rf "$DIR/" when DIR is empty).
  • -o pipefail: a pipeline’s status is that of the last non-zero stage, not just the last command.

Know the limits; set -e is famously quirky. It is ignored inside conditions (if cmd, cmd || ..., cmd && ...), in functions called from such contexts, and does not trigger for failures inside $(...) assigned within local x=$(cmd) (declare then assign separately). Handle important failures explicitly:

Terminal window
local out
out=$(some_command) || { echo "some_command failed" >&2; return 1; }

Also pipefail can bite: grep -q closes the pipe early and the upstream gets SIGPIPE (exit 141). Example: cmd | grep -q pattern may be reported as failed. Capture to a variable first or tolerate 141.

Terminal window
tmp=$(mktemp -d)
cleanup() { rm -rf "$tmp"; }
trap cleanup EXIT # runs on normal exit, error exit, and (usually) signals
trap 'echo "failed at line $LINENO" >&2' ERR

Bash can do a lot without spawning processes:

Terminal window
: "${ENVIRONMENT:?ENVIRONMENT must be set}" # fail if unset/empty
name="${1:-world}" # default value
file=/var/log/app/server.log
echo "${file##*/}" # server.log (basename)
echo "${file%/*}" # /var/log/app (dirname)
echo "${file%.log}.txt" # /var/log/app/server.txt
echo "${name^^}" # uppercase (bash 4+)
Terminal window
usage() { echo "usage: $0 [-n namespace] [-v] <deployment>" >&2; exit 2; }
ns=default verbose=0
while getopts ":n:vh" opt; do
case $opt in
n) ns=$OPTARG ;;
v) verbose=1 ;;
h) usage ;;
*) usage ;;
esac
done
shift $((OPTIND - 1))
[[ $# -eq 1 ]] || usage
deploy=$1
  • ShellCheck (shellcheck script.sh) catches quoting bugs, bad tests, and portability issues; run it in CI. It is the single best investment.
  • shfmt for formatting.
  • bats-core for tests when a script matters enough.
#!/usr/bin/env bash
# rollout-wait.sh: wait for a deployment, roll back on failure.
set -euo pipefail
usage() { echo "usage: $0 [-n namespace] <deployment>" >&2; exit 2; }
ns=default
while getopts ":n:" o; do case $o in n) ns=$OPTARG ;; *) usage ;; esac; done
shift $((OPTIND - 1))
[[ $# -eq 1 ]] || usage
deploy=$1
log() { printf '%s %s\n' "$(date -u +%FT%TZ)" "$*" >&2; }
if kubectl -n "$ns" rollout status "deploy/$deploy" --timeout=180s; then
log "rollout of $deploy succeeded"
else
log "rollout of $deploy failed; rolling back"
kubectl -n "$ns" rollout undo "deploy/$deploy"
exit 1
fi

A script that automates infrastructure should be idempotent: running it twice leaves the same end state as running it once. Prefer mkdir -p, ln -sfn, useradd guarded by id user ||, and “ensure” semantics over “do” semantics. When state management gets complex, that is the signal to use a declarative tool (Infrastructure as Code) rather than more bash.

Other habits: log with timestamps, print what you are about to do (a --dry-run flag is cheap), never embed secrets (read from env or a secret manager; see DevSecOps), and avoid curl | bash for anything unpinned and unverified.

Cloud CLIs, kubectl, and APIs speak JSON. jq is a small functional language over JSON.

Terminal window
# sample: kubectl get pods -o json
kubectl get pods -o json | jq -r '.items[] | [.metadata.name, .status.phase] | @tsv'
# filter and select
kubectl get pods -o json | jq -r '
.items[]
| select(.status.containerStatuses[]?.restartCount > 3)
| .metadata.name'
# build JSON safely (never string-concatenate JSON in shell)
jq -n --arg env "$ENVIRONMENT" --argjson replicas 3 \
'{environment: $env, replicas: $replicas}'
# aggregate
curl -s https://api.example.com/orders | \
jq '[.[] | .total] | add / length' # average

Key forms: .field, .[] (iterate), | (pipe), select(cond), map(f), group_by(.k), -r (raw strings, no quotes), --arg/--argjson (safe variable injection), ? (suppress errors on missing). yq does the same for YAML; both kubectl and the AWS CLI also have built-in --query (JMESPath) and -o jsonpath options.

# m h dom mon dow command
*/5 * * * * /opt/scripts/healthcheck.sh
15 3 * * 1-5 /opt/scripts/backup.sh >> /var/log/backup.log 2>&1

Edit with crontab -e. Pitfalls: cron runs with a minimal PATH and environment (use absolute paths, set PATH= at the top); output is emailed or discarded unless redirected; no built-in protection against overlapping runs (wrap in flock -n /var/lock/backup.lock cmd); time zone is the system’s; a missed run (host down) is simply skipped; and there is no central view of failures.

Timers add logging via the journal, dependency handling, resource limits, randomized delay, and catch-up of missed runs.

/etc/systemd/system/backup.service
[Unit]
Description=Nightly backup
[Service]
Type=oneshot
User=backup
ExecStart=/opt/scripts/backup.sh
/etc/systemd/system/backup.timer
[Unit]
Description=Run backup nightly
[Timer]
OnCalendar=*-*-* 03:15:00
RandomizedDelaySec=300
Persistent=true # run at boot if a run was missed
[Install]
WantedBy=timers.target
Terminal window
sudo systemctl enable --now backup.timer
systemctl list-timers
journalctl -u backup.service
systemd-analyze calendar '*-*-* 03:15:00' # validate a schedule

In Kubernetes use a CronJob; in CI use scheduled workflows. For “must not silently fail” jobs, add a dead-man’s-switch (a monitoring service that alerts when a heartbeat stops arriving); see Observability.

make is installed everywhere and gives a uniform entry point (make test, make deploy) so humans and CI use identical commands. Used as a task runner, targets are mostly .PHONY.

.DEFAULT_GOAL := help
.PHONY: help build test lint run clean
IMAGE ?= myapp
TAG ?= $(shell git rev-parse --short HEAD)
help: ## Show this help
@grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) | \
awk 'BEGIN{FS=":.*?## "}{printf " %-12s %s\n", $$1, $$2}'
build: ## Build the container image
docker build -t $(IMAGE):$(TAG) .
test: ## Run unit tests
pytest -q
lint: ## Static checks
shellcheck scripts/*.sh
ruff check .
run: build ## Run locally
docker run --rm -p 8080:8080 $(IMAGE):$(TAG)
clean:
rm -rf dist .pytest_cache

Rules: recipe lines must start with a tab; $$ escapes $ for the shell; each line runs in its own shell unless you use .ONESHELL:; ?= allows override from the environment (make build TAG=v1). Real build-graph features (file targets with dependencies, so unchanged outputs are skipped) are what make make more than a script wrapper. Alternatives: just (a cleaner command runner without the build-system baggage), task (Taskfile), and language-native runners (npm run, poe).

Bash excels at orchestrating processes and short pipelines. Switch when you see these signals:

Signal Why bash hurts
More than ~100 lines, or you need functions with real return values No data structures beyond strings and flat arrays; everything is text
Nested JSON/YAML processing beyond a few jq calls Unreadable, untestable
Needing HTTP with retries, auth refresh, pagination You are rebuilding a client library
Parallelism with result collection and error handling xargs -P and wait are crude
Cross-platform (macOS vs Linux flag differences, Windows) Tool flags vary (GNU vs BSD sed, date)
Unit tests required Possible with bats but awkward
Complex error handling and rollback set -e semantics are subtle

A comparable task in Python, which gains structure and testability:

#!/usr/bin/env python3
import json, subprocess, sys
def sh(*cmd: str) -> str:
return subprocess.run(cmd, check=True, text=True, capture_output=True).stdout
pods = json.loads(sh("kubectl", "get", "pods", "-o", "json"))["items"]
flappy = [
p["metadata"]["name"]
for p in pods
if any(c.get("restartCount", 0) > 3 for c in p["status"].get("containerStatuses", []))
]
print("\n".join(flappy))
sys.exit(1 if flappy else 0)

Use subprocess.run([...], check=True) with a list (never shell=True with interpolated input), argparse or typer for flags, pathlib for paths, and uv/venv for dependency management. For a static distributable binary, Go. A pragmatic rule: start in bash for glue; port to Python at the first moment you wish you had a dictionary or a unit test.

  • Unquoted variables and for f in $(ls); use globs (for f in *.txt) or find -print0.
  • Parsing ls output; use find, globs, or stat.
  • cd dir; rm -rf * without cd dir || exit. Use set -e, or better, absolute paths.
  • Relying on set -e alone; check critical commands explicitly.
  • Cron jobs that fail silently.
  • Secrets in command-line arguments (visible in ps and shell history).
  • #!/bin/bash versus #!/usr/bin/env bash: the latter is more portable (macOS ships bash 3.2; associative arrays need bash 4+). Declare #!/bin/sh only if you truly stay POSIX (Alpine’s ash, Debian’s dash).
  • Quote every expansion, use "$@", arrays, --, and null-delimited pipelines for safety.
  • Start scripts with set -euo pipefail, but understand its exceptions; add trap ... EXIT for cleanup.
  • Use jq to build and parse JSON instead of string-gluing it; lint with ShellCheck in CI.
  • Make automation idempotent and dry-runnable; prefer declarative tools when state grows.
  • systemd timers beat cron for logging, catch-up, and dependency handling; use flock against overlap.
  • A Makefile gives humans and CI the same verbs. Switch to Python once data structures, tests, or HTTP logic appear.
  1. Hands-on. Write a script with strict mode that takes a URL list file, fetches each URL’s HTTP status with curl, prints failures to stderr, and exits non-zero if any failed. Run ShellCheck on it.
  2. Conceptual. Explain why cmd > out.txt 2>&1 and cmd 2>&1 > out.txt behave differently.
  3. Hands-on. Convert a cron entry running a nightly job into a systemd service + timer with Persistent=true. Test it with systemd-analyze calendar and systemctl start.
  4. Debugging. What does set -e; local x=$(false); echo reached print inside a function, and why?
  5. Design. Take a 150-line bash script from your own work. List which of the “switch to Python” signals apply and sketch the module layout.
Solutions (2, 4)
  1. Redirections are processed left to right. In the first, stdout goes to the file, then stderr is duplicated onto what stdout currently points to (the file): both end up in the file. In the second, stderr is duplicated onto the original stdout (the terminal), then stdout goes to the file: only stdout is in the file.
  2. It prints reached. The exit status of local x=$(false) is that of local (0), masking the failure. Declare then assign: local x; x=$(false) triggers errexit.
  • Bash Reference Manual (GNU), and the bash(1) man page.
  • Greg’s Wiki “BashGuide”, “BashFAQ”, and “BashPitfalls”.
  • Ken Thompson, Brian Kernighan et al., The UNIX Programming Environment (Kernighan, Pike, 1984), for the pipe philosophy.
  • The ShellCheck wiki (explains each warning).
  • Stephen Dolan’s jq manual, and Al Sweigart’s Automate the Boring Stuff with Python for the Python side.