Skip to content

Observability & Monitoring

What you’ll learn. The difference between monitoring and observability; the three signals (metrics, logs, traces) and when to use each; Prometheus data model, PromQL, and Grafana dashboards that actually help; OpenTelemetry instrumentation and collectors; alerting on symptoms rather than causes; SLIs, SLOs, error budgets, and burn-rate alerts; and log pipeline design. Builds on Kubernetes Fundamentals; feeds Incident Response.

Monitoring checks known failure modes: “is CPU above 90%? is the disk full?” You decide the questions in advance. Observability is the property of a system that lets you ask new questions about its internal state from its external outputs, without shipping new code. In distributed systems the failures that hurt you are novel (“only checkout requests from iOS clients hitting shard 7 after the 14:02 deploy”), so you need high-cardinality, explorable data rather than only pre-built graphs.

Practical consequence: collect rich, structured telemetry, and be able to slice it by arbitrary dimensions (service, version, region, customer tier).

Signal What it is Strength Weakness
Metrics Numeric time series, aggregated Cheap, fast, great for alerting and trends Low cardinality; hides individual requests
Logs Discrete timestamped events Rich detail and context Volume/cost; hard to aggregate; free-text is unqueryable
Traces A request’s path across services as a tree of spans Shows where latency/errors originate in a call chain Needs propagation instrumentation; sampling loses data

A worked flow: an alert fires from a metric (error rate up on the checkout service). You open a dashboard to see it began after a deploy. A trace of a failing request shows that checkout waits 5 s on inventory. Logs from inventory for that trace ID show a database connection pool exhausted. Each signal answers a different question, and you move between them by shared identifiers (trace ID, service, labels). Some add profiles (continuous profiling, e.g., Pyroscope, Parca) as a fourth signal and events (deploys, config changes) as annotations.

Prometheus is the de facto open-source metrics system, CNCF-graduated. Architecture:

targets (/metrics endpoints, HTTP) <---scrape (pull)--- Prometheus server --> TSDB
| \
| +--> Alertmanager --> Slack/PagerDuty
+--> Grafana (queries via PromQL)

It pulls (scrapes) text-format metrics from targets at intervals (commonly 15-30 s), discovers targets via service discovery (Kubernetes, EC2, Consul), stores them in a local time-series database, evaluates rules, and sends alerts to Alertmanager. For short-lived jobs use the Pushgateway sparingly; for long-term storage and global views use remote-write to Thanos, Mimir, Cortex, VictoriaMetrics, or a managed service (Grafana Cloud, Amazon Managed Prometheus, Google Managed Prometheus).

A time series is identified by a metric name and a set of labels: http_requests_total{service="api", method="GET", status="200"}, with a float value per timestamp.

  • Counter: monotonically increasing (requests, errors); use rate(). Resets on restart are handled.
  • Gauge: goes up and down (memory in use, queue depth).
  • Histogram: counts observations in buckets (_bucket{le="0.5"}), plus _sum and _count; lets you compute quantiles server-side and aggregate across instances. Native (exponential) histograms reduce the bucket-design burden.
  • Summary: client-side quantiles; cannot be aggregated across instances, so prefer histograms.
from prometheus_client import Counter, Histogram, start_http_server
import time
REQUESTS = Counter("http_requests_total", "Requests", ["method", "route", "status"])
LATENCY = Histogram(
"http_request_duration_seconds", "Request latency", ["route"],
buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5),
)
def handle(route, method="GET"):
start = time.perf_counter()
status = "200"
try:
... # real work
except Exception:
status = "500"
raise
finally:
LATENCY.labels(route).observe(time.perf_counter() - start)
REQUESTS.labels(method, route, status).inc()
start_http_server(9100) # exposes /metrics

Use route templates (/users/{id}), never raw paths, as labels.

# requests per second, by route, over 5 minutes
sum by (route) (rate(http_requests_total[5m]))
# error ratio (SLI-style)
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
# p99 latency from a histogram, aggregated across instances
histogram_quantile(0.99,
sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
# saturation: memory usage vs limit per pod (cAdvisor + kube-state-metrics)
max by (pod) (container_memory_working_set_bytes{container!=""})
/ on (pod) group_left
max by (pod) (kube_pod_container_resource_limits{resource="memory"})
# predict disk full within 4 hours
predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0

Notes: rate() works on counters and gives per-second average increase over the window; use a window at least 4x the scrape interval. Always aggregate before histogram_quantile, and never average quantiles. Quantile accuracy is limited by bucket boundaries.

  • RED (for request-driven services): Rate, Errors, Duration.
  • USE (for resources like CPU, disks, pools; Brendan Gregg): Utilization, Saturation, Errors.
  • Four Golden Signals (Google SRE book): latency, traffic, errors, saturation. Distinguish latency of successful versus failed requests.

Instrument these uniformly across services (via a shared library or the mesh/OpenTelemetry) and you get consistent dashboards for free.

Grafana visualizes data from Prometheus, Loki, Tempo, Elasticsearch, cloud monitors, SQL databases, and more. Dashboards rot into wallpaper unless designed. Principles:

  1. Start from questions and users. A dashboard for on-call triage differs from one for capacity planning or executives. Name it for its audience.
  2. Top-down hierarchy: a service overview (the golden signals and SLO status) at the top, with drill-downs to dependencies, then infrastructure. A triage should go: overview, which service, which instance/dependency, then logs and traces.
  3. Show SLO status and the user-visible symptoms first, causes second.
  4. Consistent layout across services (same rows: rate, errors, latency, saturation), so reading one teaches you all.
  5. Right visualization: time series for trends, heatmaps for latency distributions (an average hides bimodality), stat panels for current state. Label units and axes; fix y-axis minimums when comparing.
  6. Annotate deploys and incidents so changes correlate with behavior.
  7. Limit panel count (a dozen well-chosen panels beats sixty), avoid pie charts for time series, and avoid “vanity” panels nobody acts on.
  8. Dashboards as code: provision from Git (Grafana provisioning, Terraform provider, Jsonnet/Grafonnet, Grafana Foundation SDK) so they are reviewable and reproducible. Use template variables ($service, $env) instead of copy-pasted dashboards.
  9. Link everything: panel to logs filtered by the same labels, to traces via exemplars, to the runbook (Incident & Reliability).

Logs are the most flexible and the most abused signal.

Structured logging: emit JSON (or logfmt) with consistent fields so logs are queryable rather than grep-able prose.

{"ts":"2026-11-17T10:02:11.482Z","level":"error","service":"checkout","version":"1.4.2",
"trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","user_tier":"pro",
"msg":"payment provider timeout","provider":"stripe","duration_ms":5003}

Guidelines: log to stdout/stderr (the platform collects it; see Docker); use levels meaningfully (error means someone might need to act); include trace IDs for correlation; log events with context rather than fragments; never log secrets or unnecessary personal data (PII, tokens); sample or rate-limit noisy logs; and prefer metrics for counting things (counting log lines to compute rates is expensive).

apps -> stdout -> node agent (Fluent Bit / Vector / OTel Collector / Promtail)
| parse, enrich (k8s metadata), filter, redact, route
v
buffer/queue (Kafka, optional) -> storage/index -> query UI
(Loki | Elasticsearch/OpenSearch | ClickHouse | cloud logging)
  • Collect with a DaemonSet agent reading container log files (Fluent Bit is light; Vector and the OpenTelemetry Collector are flexible).
  • Enrich with Kubernetes metadata (namespace, pod, labels) and parse JSON.
  • Redact and filter early (drop debug noise, mask PII) to control both cost and compliance risk.
  • Store: Loki indexes only labels and stores compressed chunks (cheap, good with Grafana, query with LogQL); Elasticsearch/OpenSearch full-text indexes everything (powerful, costly); ClickHouse-based systems offer fast columnar analytics on logs; managed cloud logging is convenient but watch ingestion pricing.
  • Retention tiers: hot (days), warm, cold/archive in object storage (compliance, with lifecycle rules).
  • Backpressure and loss: agents need buffering; decide whether to drop or block under overload (usually drop with a metric counting drops).
{namespace="prod", app="checkout"} | json | level="error" | line_format "{{.msg}} ({{.provider}})"
sum by (provider) (rate({app="checkout"} |= "payment provider timeout" [5m]))

Observability data volume grows faster than traffic; it is often one of the largest cloud line items (Cloud Fundamentals). Budget it, sample it, and set retention deliberately.

A trace is the end-to-end journey of one request; it consists of spans (an operation with start time, duration, attributes, status), arranged as a tree via parent IDs. Context propagation passes the trace and span IDs between services in headers (W3C traceparent) so spans from different processes join into one trace.

trace 4bf9... GET /checkout [=============================] 820 ms
├─ auth.verify [==] 40 ms
├─ inventory.reserve [===========] 310 ms
│ └─ SELECT stock (postgres) [=========] 290 ms <- slow
└─ payment.charge [==========] 420 ms

OpenTelemetry (OTel) is the CNCF standard for generating and exporting telemetry: vendor-neutral APIs and SDKs (many languages), a wire protocol (OTLP), semantic conventions (standard attribute names such as http.request.method), and the Collector. Traces and metrics are stable; logs support is stable in the specification and maturing across SDKs; check the status per language. Instrument once, send to any backend (Jaeger, Tempo, Prometheus, Honeycomb, Datadog, Grafana Cloud, etc.), which reduces vendor lock-in.

Auto-instrumentation covers common frameworks and clients; add manual spans for business operations.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
provider = TracerProvider(resource=Resource.create({"service.name": "checkout", "service.version": "1.4.2"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317")))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)
def charge(order):
with tracer.start_as_current_span("payment.charge") as span:
span.set_attribute("order.value_cents", order.total)
span.set_attribute("payment.provider", "stripe")
return provider_client.charge(order)

A standalone pipeline between apps and backends: receivers (OTLP, Prometheus, filelog), processors (batch, memory_limiter, attributes, tail_sampling, k8sattributes), exporters.

otel-collector.yaml
receivers:
otlp:
protocols: {grpc: {}, http: {}}
processors:
memory_limiter: {check_interval: 1s, limit_mib: 512}
batch: {}
tail_sampling:
decision_wait: 10s
policies:
- {name: errors, type: status_code, status_code: {status_codes: [ERROR]}}
- {name: slow, type: latency, latency: {threshold_ms: 1000}}
- {name: baseline, type: probabilistic, probabilistic: {sampling_percentage: 5}}
exporters:
otlp/tempo: {endpoint: tempo:4317, tls: {insecure: true}}
prometheusremotewrite: {endpoint: http://mimir:9009/api/v1/push}
service:
pipelines:
traces: {receivers: [otlp], processors: [memory_limiter, tail_sampling, batch], exporters: [otlp/tempo]}
metrics: {receivers: [otlp], processors: [memory_limiter, batch], exporters: [prometheusremotewrite]}

Sampling: tracing every request is costly. Head sampling decides at the start (cheap, but may miss rare errors). Tail sampling decides after the trace completes (keep all errors and slow traces plus a baseline; needs the collector to buffer whole traces, so route by trace ID). Exemplars link a metric data point to a representative trace, bridging signals.

Alerts are interrupts to humans. Each costs attention and sleep, so the bar is high. Principles (from the Google SRE book and Rob Ewaschuk’s “My Philosophy on Alerting”):

  1. Page on symptoms, not causes. Alert on user-visible pain (error rate, latency, availability) rather than “CPU > 80%”. Cause-based alerts (disk filling, queue backing up) belong as tickets or dashboards unless they imply imminent user impact.
  2. Every page must be urgent, actionable, and require a human. If it can wait until morning, it is a ticket. If the response is “wait and see,” delete the alert. If the response is always the same automated fix, automate the fix.
  3. Link a runbook and relevant dashboards in the alert.
  4. Alert fatigue kills reliability: track pages per shift; every non-actionable page is a bug to fix in the alert or the system. Review alerts in a regular cadence.
  5. Use severity tiers: page (wake someone), ticket (business hours), log/dashboard only.
  6. Group and route (Alertmanager): deduplicate, group related alerts, inhibit downstream alerts when an upstream one is firing, silence during maintenance, route by team and severity.
  7. Dead man’s switch: an always-firing alert whose absence pages you, so a broken monitoring pipeline is itself detected.
# prometheus rule: symptom alert with a runbook
groups:
- name: checkout
rules:
- alert: CheckoutHighErrorRate
expr: |
sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m]))
/ sum(rate(http_requests_total{service="checkout"}[5m])) > 0.02
for: 10m
labels: {severity: page, team: payments}
annotations:
summary: "Checkout 5xx ratio above 2% for 10m"
runbook_url: "https://runbooks.example.com/checkout-errors"

The for: clause prevents flapping on transient blips.

  • SLI (Service Level Indicator): a measured proportion of good events, e.g. SLI=good eventsvalid events\text{SLI} = \frac{\text{good events}}{\text{valid events}}: requests answered successfully within 300 ms divided by all valid requests.
  • SLO (Service Level Objective): the target for the SLI over a window: “99.9% of requests are good over 30 days.”
  • SLA: a contractual promise with consequences (credits), typically looser than the internal SLO.
  • Error budget: 1−SLO1 - \text{SLO}. The allowed unreliability.

Budget arithmetic for a 30-day window (43,20043{,}200 minutes):

SLO Error budget Allowed downtime per 30 days
99% 1% about 7.2 hours
99.9% 0.1% about 43 minutes
99.95% 0.05% about 21.6 minutes
99.99% 0.01% about 4.3 minutes

Each extra nine costs roughly 10x more effort; pick the SLO that your users actually need, not the highest conceivable. Dependencies cap you: serial dependencies multiply availability, so three services at 99.9% each give about 0.9993≈99.7%0.999^3 \approx 99.7\%.

The error budget is a negotiation tool: while budget remains, ship features; when exhausted, prioritize reliability work. This resolves the dev-vs-ops tension numerically (DevOps Culture).

SLIs should reflect the user’s experience at a good measurement point: load balancer or client-side metrics rather than server internals. Typical SLI types: availability, latency (via histogram thresholds), correctness/freshness (data pipelines), throughput.

Alerting on “error rate above SLO” is either too slow or too noisy. A burn rate is how fast you consume budget relative to the steady rate that would exactly exhaust it at the window’s end: burn rate 1 means exhausting it in exactly 30 days, and a burn rate of BB exhausts it in 30/B30/B days (so 14.4 means about 2 days). The multi-window, multi-burn-rate approach (SRE Workbook) pages on fast burns and tickets on slow ones, requiring both a long and a short window to avoid stale alerts:

# 99.9% SLO => budget 0.001. Burn rate 14.4 over 1h (long) AND 5m (short) => page
- alert: CheckoutSLOFastBurn
expr: |
(
sum(rate(http_requests_total{service="checkout",status=~"5.."}[1h]))
/ sum(rate(http_requests_total{service="checkout"}[1h])) > (14.4 * 0.001)
)
and
(
sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m]))
/ sum(rate(http_requests_total{service="checkout"}[5m])) > (14.4 * 0.001)
)
labels: {severity: page}

At burn rate 14.4, 1 hour consumes about 2% of a 30-day budget (14.4/72014.4 / 720 hours). Tools like Sloth, Pyrra, and OpenSLO generate these rules from a declarative SLO spec.

  • Instrument early and consistently; retrofitting is harder.
  • Correlate: trace IDs in logs, exemplars on metrics, deploy markers on graphs, consistent resource attributes (service.name, env, version).
  • Monitor the monitoring: Prometheus down, scrape failures, ingestion lag, dropped logs.
  • Synthetic probes (blackbox exporter, checkly) measure from the user’s side, including DNS, TLS expiry, and external dependencies (Linux & Networking).
  • Control cost: drop unused metrics, limit label cardinality, sample traces, tier log retention.
  • Don’t build it all yourself unless you must: managed backends trade money for operational burden. The self-hosted Prometheus/Grafana/Loki/Tempo (“LGTM”) stack is mature but needs care at scale.
  • Metrics are cheap and good for alerting, logs give detail, traces show cross-service causality; connect them via shared IDs and labels.
  • Keep Prometheus label cardinality bounded; use counters with rate(), histograms for latency, and aggregate before computing quantiles.
  • Use RED for services, USE for resources, or the four golden signals; build dashboards top-down, consistently, and as code.
  • OpenTelemetry gives vendor-neutral instrumentation; use the Collector for processing and tail sampling.
  • Page on user-visible symptoms; every page must be actionable; fight alert fatigue and add a dead man’s switch.
  • SLOs and error budgets turn reliability into a quantified trade-off; multi-window burn-rate alerts are the practical way to alert on them.
  1. Hands-on. Run Prometheus and Grafana with Docker Compose, instrument a small app with a counter and histogram, and build a RED dashboard.
  2. Calculation. For a 99.95% SLO over 28 days, compute the error budget in minutes. A deploy causes a 100% outage for 6 minutes. What fraction of the budget is consumed?
  3. PromQL. Write queries for (a) the per-route 95th-percentile latency and (b) the fraction of requests faster than 300 ms (using the le="0.25" and le="0.5" buckets and its limitation).
  4. Conceptual. Why is “CPU above 90% for 5 minutes” usually a bad paging alert? Under what conditions would it be justified?
  5. Design. Define two SLIs and SLOs for a batch data pipeline (hint: freshness and correctness) and the alerts you would derive.
Solutions (2, 4)
  1. 28 days = 40,320 minutes; budget = 0.0005 × 40,320 = 20.16 minutes. A 6-minute full outage consumes 6 / 20.16 ≈ 30% of the budget.
  2. High CPU is a cause or resource indicator, not necessarily user impact: a batch node or an efficient service may legitimately run hot. Page only if saturation demonstrably causes user-visible errors or latency, or imminent failure with no autoscaling; otherwise use a ticket or dashboard. Better alert on latency and error SLOs.
  • Betsy Beyer et al., Site Reliability Engineering (2016), chapters on monitoring and alerting; The Site Reliability Workbook (2018), chapter “Alerting on SLOs”.
  • Charity Majors, Liz Fong-Jones, George Miranda, Observability Engineering (2022).
  • Brian Brazil, Prometheus: Up & Running (2nd ed., 2023).
  • Rob Ewaschuk, “My Philosophy on Alerting” (Google SRE notes).
  • OpenTelemetry documentation and semantic conventions; Grafana Labs documentation for Loki, Tempo, and Mimir.
  • Brendan Gregg, “The USE Method”; Tom Wilkie, “The RED Method”.