Incident Response & Reliability Engineering
What you’ll learn. How to run on-call sustainably; the structure of a well-run incident (roles, severity, communication); how to write runbooks that work at 3 a.m.; blameless postmortems that produce real fixes; chaos engineering as a way to find weaknesses before users do; capacity planning with queueing intuition; and backup and disaster-recovery design around RPO and RTO. Builds on Observability (SLOs, alerts) and Cloud Fundamentals (multi-region).
Reliability as a feature
Section titled “Reliability as a feature”No system is 100% reliable, and chasing it is ruinously expensive. Reliability is a product requirement with a target (an SLO) and an error budget to spend (Observability). This lecture covers what happens when the budget is spent faster than planned: incidents, how to respond, learn, and prevent, and how to prepare.
Reliability work has four loops:
Prevent <--- postmortem action items --- Respond | ^ design, test, chaos, capacity on-call, incident command, runbooks v | Detect --------- alerts, SLO burn --------->+On-call
Section titled “On-call”On-call means a rotation of engineers who are reachable and able to respond to production alerts. Done badly, it burns people out and drives attrition; done well, it keeps ownership close to the code (“you build it, you run it”) and is a powerful source of feedback.
Structure
Section titled “Structure”- Rotation: typically one week per person; primary and secondary (backup/escalation). Minimum viable rotation size is around 5-8 people for a 24/7 follow-the-sun-less rotation so that each person is on-call at most about one week in 6-8. Smaller rotations lead to fatigue.
- Follow-the-sun (handoff across time zones) avoids night pages when you have teams in multiple regions.
- Tooling: PagerDuty, Opsgenie (Atlassian’s end-of-sale is announced; check status), Grafana OnCall, incident.io, Rootly, FireHydrant. Escalation policies, schedules, overrides, and acknowledgement timeouts.
- Compensation and norms: pay or time-off in lieu; no-meetings day after a rough night; the right to say “I can’t take this shift.”
- Handoffs: a written summary of ongoing issues, recent changes, and flaky alerts at each rotation change.
Sustainable load
Section titled “Sustainable load”Google’s SRE guidance suggests on-call engineers handle at most about two incidents per 12-hour shift so there is time to respond properly, root-cause, and write up. Measure: pages per shift, percentage of non-actionable pages, time to acknowledge, time spent on toil. If pages exceed what a person can handle, that is a priority-one reliability problem (fix alerts, fix the system, or add people), not something to tolerate. Review all pages weekly; each should end in a fix, a better alert, a runbook, or deletion.
Responding to a page: the first minutes
Section titled “Responding to a page: the first minutes”- Acknowledge so the system knows a human has it.
- Assess impact (is this user-facing? how many users? which SLO?). Check the service dashboard and recent changes (deploys, flag flips, config, dependency incidents).
- Mitigate first, diagnose later. The first goal is restoring service, not finding root cause. The usual mitigations: roll back the last deploy, flip a feature flag, fail over, scale out, shed load, disable a failing dependency path. Rolling back is the best default if a deploy correlates with the start of the problem (CI/CD).
- Escalate early if you are not making progress in about 15 minutes or the impact is large. Asking for help is a competence, not a weakness.
- Communicate status as you go.
Incident management
Section titled “Incident management”For anything beyond a simple page, use a lightweight, explicit process (derived from the Incident Command System used by emergency services and adapted by Google, PagerDuty, and others).
| Role | Responsibility |
|---|---|
| Incident Commander (IC) | Owns the response and decisions; coordinates; does not debug. Delegates and keeps the big picture |
| Operations/Technical lead(s) | Hands-on investigation and mitigation |
| Communications lead | Updates stakeholders and status page at regular intervals |
| Scribe | Records a timeline (timestamps, actions, hypotheses) in the incident channel/doc |
| Subject-matter experts | Pulled in as needed |
One person per role matters. When the IC is also debugging, nobody is watching the whole incident. In a small team one person may hold two roles, but the IC should hand off if they dive into the technical work.
Severity
Section titled “Severity”Define severity levels in advance so people do not argue mid-incident. Example:
| Sev | Meaning | Response |
|---|---|---|
| SEV1 | Major user-visible outage or data loss, core flow down | Page immediately, all hands, exec comms, postmortem required |
| SEV2 | Significant degradation, partial outage, SLO at risk | Page on-call, IC assigned, postmortem required |
| SEV3 | Minor issue, workaround exists | Ticket, handled in business hours |
Declare incidents liberally. A false alarm costs little; a late declaration costs a lot. Anyone should be able to declare one.
Running the incident
Section titled “Running the incident”- Open a dedicated channel (
#inc-2026-11-17-checkout) and a status document. Use a bot/slash command (/incident declare) to automate the setup. - IC states: “I am IC. Impact: … Current hypothesis: … Next update in 15 minutes.”
- Time-box hypotheses and work parallel threads; keep a visible list of “what we know, what we tried, what we think.”
- Change control during an incident: every change announced and recorded; avoid unrelated changes (“while we’re here”).
- Communicate on a cadence (e.g., every 30 minutes for SEV1) even if the update is “no change”; silence breeds anxiety. External updates are plain-language, state user impact, and avoid speculation about cause.
- Resolve when the SLI has recovered and is stable, not when the fix is merged. Then schedule the postmortem while memories are fresh (within about a week).
- Hand off with explicit transfer if the incident exceeds a shift; fatigued responders make mistakes.
[10:02] IC (alice): Declared SEV2. Checkout error rate 14%. Started 09:48, correlates with deploy 1.4.2 at 09:46.[10:04] ops (bob): Rolling back to 1.4.1. ETA 3 min.[10:07] comms (carol): Status page updated: "Some customers see errors at checkout. We are mitigating."[10:08] scribe (dan): Rollback complete 10:07. Error rate 14% -> 1.2% -> 0.2%.[10:15] IC: Impact mitigated. Monitoring 30 min. Postmortem owner: bob.Runbooks
Section titled “Runbooks”A runbook is a documented procedure for diagnosing and mitigating a specific alert or task. Good runbooks make the 3 a.m. page survivable for someone who did not write the service.
Properties of a useful runbook:
- Linked directly from the alert (annotation
runbook_url). - Starts with impact and urgency: what does this alert mean for users?
- Copy-pasteable commands with expected outputs, not prose essays.
- Decision tree: if X then do Y; includes how to confirm each hypothesis.
- Safe mitigations first and explicit warnings for dangerous steps (data deletion).
- Escalation contacts and when to use them.
- Owned and tested: updated after every incident; reviewed quarterly; validated in game days.
# Runbook: CheckoutHighErrorRateImpact: customers cannot complete purchases. Page severity.
## 1. Confirm- Dashboard: https://grafana.example.com/d/checkout (look at 5xx by route and version)- `kubectl -n prod get pods -l app=checkout` : any CrashLoopBackOff?
## 2. Recent change?- `kubectl -n prod rollout history deploy/checkout` ; check flags changed in last hour.- If errors began within 10 min of a deploy: `kubectl -n prod rollout undo deploy/checkout`
## 3. Dependencies- Payment provider status; `inventory` p99 latency panel; DB connections panel.- If DB pool saturated: scale `checkout` down by 25% (reduces connections), page DB on-call.
## 4. Escalate- No improvement in 15 min: page @payments-secondary and the IC.The best runbook is a script or an automated remediation. Convert repeated runbook steps into code (self-healing, auto-rollback in the pipeline), leaving humans for judgment (Shell Scripting).
Blameless postmortems
Section titled “Blameless postmortems”A postmortem (or incident review) documents what happened, why, and what to change. Blameless does not mean “no accountability”; it means assuming that people acted reasonably given the information and incentives they had, and that human error is a symptom of system design, not a root cause. Punishing individuals teaches people to hide mistakes, which destroys the information you need. Sidney Dekker’s “Field Guide to Understanding Human Error” and the “Just Culture” literature underpin this approach.
Template
Section titled “Template”- Summary: one paragraph. Impact and duration.
- Impact: users affected, SLO budget consumed, revenue/support effect.
- Timeline: detection, response, mitigation, with timestamps (UTC).
- Contributing factors / root causes: usually several. Go beyond the trigger.
- What went well / what went poorly / where we got lucky.
- Action items: concrete, owned, prioritized, with deadlines and tracked in the normal backlog.
- Lessons learned and glossary for readers outside the team.
Analysis techniques
Section titled “Analysis techniques”- Five whys: iterate “why” to move from proximate to systemic causes, but beware the linear chain that stops at a person. “Why did the engineer push a bad config? Because the config had no validation. Why? Because … ” Aim for latent conditions.
- Contributing factors over single root cause: complex systems fail through combinations (a latent bug, a missing alert, a confusing runbook, an unusual load).
- Detection, mitigation, and prevention as three separate questions: how could we have found it sooner, fixed it faster, or made it impossible?
- Distinguish triggers from causes: a deploy triggered the outage; the missing canary and lack of validation allowed it.
Bad versus good action items
Section titled “Bad versus good action items”- Weak: “Be more careful with config changes.” (Not a system change.)
- Strong: “Add schema validation for the config to CI, owner: bob, due Dec 1.” “Add canary stage that blocks on error rate regression.” “Add alert on DB pool saturation with runbook.”
Track completion; a postmortem with unfinished action items is a failure. Share postmortems widely (internally at least); several companies publish them, and reading public postmortems is excellent education.
Chaos engineering
Section titled “Chaos engineering”Chaos engineering is “the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production” (Principles of Chaos Engineering). It is empirical testing of resilience assumptions, not random destruction.
The experiment loop
Section titled “The experiment loop”- Define steady state with a measurable metric (e.g., checkout success rate, p99 latency).
- Form a hypothesis: “If we terminate one of three API pods, the steady state holds.”
- Introduce a real-world fault with a small blast radius: kill instances, add latency, drop packets, fail a dependency, exhaust disk, fail over a database, take down an AZ.
- Observe the metric; look for deviation.
- Fix weaknesses found; automate the experiment as a regression test; gradually expand scope.
Always have an abort condition (“stop if error rate exceeds 2%”) and run during working hours with the team present until you are confident.
Tools: Chaos Mesh and LitmusChaos (Kubernetes), Gremlin (commercial), AWS Fault Injection Service, Azure Chaos Studio, Chaos Monkey (Netflix), Toxiproxy (network faults in tests), and tc netem for latency:
# add 200ms +/- 50ms latency and 1% loss on eth0 for 60 seconds (test environment!)sudo tc qdisc add dev eth0 root netem delay 200ms 50ms loss 1%sleep 60sudo tc qdisc del dev eth0 root netem# Chaos Mesh: kill one random pod of the api deploymentapiVersion: chaos-mesh.org/v1alpha1kind: PodChaosmetadata: {name: kill-one-api-pod, namespace: chaos-testing}spec: action: pod-kill mode: one selector: namespaces: [staging] labelSelectors: {app: api}Typical discoveries: missing timeouts (a hung dependency cascades), retry storms (retries amplify an outage; use exponential backoff with jitter and retry budgets), unbounded queues, health checks that do not reflect reality, single points of failure, failover that has never been exercised. Game days run these as team exercises, practicing incident response too (Kubernetes Fundamentals PodDisruptionBudgets and topology spread are what you are testing).
Resilience patterns the experiments verify: timeouts everywhere, bounded retries with backoff and jitter, circuit breakers, bulkheads (isolate resource pools), load shedding, graceful degradation (serve stale data, disable non-essential features), idempotency so retries are safe, and backpressure.
Capacity planning
Section titled “Capacity planning”Capacity planning ensures resources meet demand with headroom, at acceptable cost (Cloud Fundamentals for FinOps).
Intuition from queueing theory
Section titled “Intuition from queueing theory”Latency does not grow linearly with load. For a single server with random arrivals and service times (the M/M/1 model), the mean time in system is
where is the arrival rate, the service rate, and the utilization. At 50% utilization, (twice the bare service time); at 90%, ; at 95%, ; as , latency explodes. This is why you do not run services at 90% CPU: tail latency collapses long before 100%. Plan for peak utilization around 50-70% for latency-sensitive services, and use autoscaling plus headroom for spikes.
Little’s Law relates concurrency to throughput: . If a service handles 200 requests per second with 50 ms mean latency, average concurrency is . Size connection pools, thread pools, and worker counts from this (DevOps Culture uses the same law for WIP).
Practice
Section titled “Practice”- Know your demand: requests per second, seasonality (daily, weekly, events), growth rate. Use historical metrics.
- Know your per-unit capacity with load tests (k6, Locust, Gatling, vegeta): find the saturation point, the knee of the latency curve, and the bottleneck resource (CPU, memory, connections, disk IOPS, a downstream dependency).
- Forecast: demand times growth, plus headroom for failure: with N+1 or N+2 redundancy, ensure that losing an AZ or node still leaves enough capacity. With 3 AZs, each running at utilization , losing one pushes the others to , so keep or so if you must survive an AZ loss at 90% (since ).
- Lead time: quotas, reservations, hardware (GPUs; see MLOps) may take weeks. Request quota increases early.
- Review regularly (quarterly) and before known events (launches, sales). Alert on saturation and on time-to-exhaustion (
predict_linear) for disks and quotas. - Load test in CI or nightly to catch performance regressions before capacity surprises.
# quick load test with k6cat > load.js <<'EOF'import http from 'k6/http';import { check } from 'k6';export const options = { stages: [{duration: '2m', target: 100}, {duration: '5m', target: 100}, {duration: '1m', target: 0}], thresholds: {http_req_failed: ['rate<0.01'], http_req_duration: ['p(95)<300']},};export default function () { const r = http.get('https://staging.example.com/health'); check(r, {'status 200': (res) => res.status === 200});}EOFk6 run load.jsBackups and disaster recovery
Section titled “Backups and disaster recovery”RPO and RTO
Section titled “RPO and RTO”Two numbers define your recovery requirements:
- RPO (Recovery Point Objective): the maximum tolerable data loss, measured in time. RPO of 15 minutes means you can lose up to the last 15 minutes of writes. It sets backup/replication frequency.
- RTO (Recovery Time Objective): the maximum tolerable downtime until service is restored. It sets how much automation and standby capacity you need.
last good copy disaster service restored |---- RPO ----->| | |-------- RTO ---------->|Both are business decisions with a cost curve: smaller RPO/RTO needs synchronous replication, hot standby, and rehearsals. Tier your systems (payments: RPO near 0, RTO minutes; internal wiki: RPO 24 h, RTO 1 day) rather than giving everything gold-plated treatment.
Backup principles
Section titled “Backup principles”- 3-2-1 rule: at least 3 copies, on 2 different media/systems, 1 offsite (different account/region). Modern variant: add immutable/air-gapped copies (object lock, WORM) to survive ransomware and a compromised admin account, and keep backups in a separate account.
- Replication is not backup. Replicas faithfully copy
DROP TABLEand corruption. You need point-in-time recovery and versioned backups. - A backup that has not been restored does not exist. Schedule automated restore tests and measure real RTO. This is the most commonly skipped step and the most often fatal.
- Encrypt backups; manage keys separately and ensure you can still get them during a disaster.
- Cover everything that makes the system: databases, object storage, secrets, configuration, IaC state, DNS, certificates, CI/CD, and the knowledge of how to rebuild. Infrastructure as code makes rebuilds feasible (IaC); Kubernetes needs etcd or cluster-state backup and persistent volume snapshots (Velero) (Kubernetes Fundamentals).
# PostgreSQL logical backup + upload, with integrity check and retention via lifecycle rulesset -euo pipefailts=$(date -u +%Y%m%dT%H%M%SZ)pg_dump --format=custom --file="/tmp/app-$ts.dump" "$DATABASE_URL"pg_restore --list "/tmp/app-$ts.dump" >/dev/null # sanity-check the archive is readableaws s3 cp "/tmp/app-$ts.dump" "s3://myco-backups/app/$ts.dump" --sse aws:kmsrm "/tmp/app-$ts.dump"(For larger databases prefer continuous WAL archiving / managed point-in-time recovery; logical dumps are slow to restore.)
DR drills
Section titled “DR drills”A DR plan is a tested procedure: declare disaster, decide failover, execute steps (promote replica, switch DNS/traffic, scale up, verify), communicate, and later fail back. Run tabletop exercises (talk through scenarios) and live failover drills regularly. Record measured RTO/RPO against targets; mismatches become action items. Also plan for failures of your tooling (what if the CI/CD, the identity provider, or the observability stack is the thing that is down?), and keep emergency “break-glass” access and out-of-band communication channels.
Putting it together: a reliability maturity path
Section titled “Putting it together: a reliability maturity path”- Define SLOs, alerts on symptoms, and a basic on-call with runbooks.
- Adopt incident roles, severity, and blameless postmortems with tracked actions.
- Automate mitigations (auto-rollback, autoscaling, self-healing).
- Practice: game days, chaos experiments, restore tests.
- Plan capacity from load tests and forecasts; design for AZ loss.
- Define tiered RPO/RTO and rehearse DR.
Key takeaways
Section titled “Key takeaways”- Treat reliability as a target (SLO and error budget), not a goal of perfection; on-call must be sustainable and every non-actionable page is a bug.
- In an incident: mitigate first (rollback, flag, failover), assign an IC who does not debug, communicate on a cadence, and keep a timeline.
- Runbooks are linked from alerts, copy-pasteable, owned, and increasingly replaced by automation.
- Blameless postmortems treat human error as a symptom of system design and produce owned, tracked, systemic action items.
- Chaos engineering tests hypotheses about steady state with controlled blast radius; it finds missing timeouts, retry storms, and untested failover.
- Latency explodes as utilization approaches 100% (); plan headroom for peak and failure domains; use load tests, not guesses.
- RPO bounds data loss and RTO bounds downtime; replication is not backup; untested restores are not backups.
Exercises
Section titled “Exercises”- Hands-on. Write a runbook for one alert in a service you run, including at least three copy-pasteable diagnostic commands and a mitigation. Have a colleague follow it cold and note where they get stuck.
- Writing. Take a public postmortem (or an outage you remember) and rewrite its action items so each is a specific system change with an owner and a measure.
- Calculation. A service handles 120 requests/s at 80 ms mean latency. What is average concurrency (Little’s Law)? If you size the worker pool at 8, what happens? Using M/M/1, how much does mean latency grow when utilization goes from 60% to 90%?
- Hands-on. In a staging Kubernetes cluster, kill one pod at random every minute for 10 minutes while running a load test. Does the success rate stay within your SLO? Which setting would you change?
- Design. For a 3-AZ deployment, what per-AZ utilization lets you survive losing one AZ without exceeding 80% utilization on the remaining two?
Solutions (3, 5)
- Concurrency on average. A pool of 8 workers cannot serve average load: requests queue, latency grows without bound (utilization above 100%). Latency scales as : at 60% it is the service time; at 90% it is ; so growth is .
- Losing one AZ leaves 2/3 of capacity, so utilization becomes . For at most 0.8: , so about 53% per-AZ utilization before the failure.
Further reading
Section titled “Further reading”- Betsy Beyer et al., Site Reliability Engineering (2016): chapters on being on-call, managing incidents, postmortem culture; and The Site Reliability Workbook (2018).
- Sidney Dekker, The Field Guide to Understanding ‘Human Error’ (3rd ed., 2014).
- Casey Rosenthal and Nora Jones, Chaos Engineering (2020); the “Principles of Chaos Engineering” site.
- Michael Nygard, Release It! (2nd ed., 2018), on stability patterns.
- PagerDuty’s open-source Incident Response documentation.
- Neil Gunther, Guerrilla Capacity Planning (2007), for queueing models in practice.