Production, Observability and Cost Engineering
What you’ll learn. What changes when an agent leaves the notebook: tracing and observability, regression evaluation, a cost and latency model with concrete optimization levers (prompt caching, model routing, parallelism, context control), durable execution, deployment and rollout practices, and an operational checklist.
Production differs from demos
Section titled “Production differs from demos”Demos run happy paths once. Production means thousands of runs, adversarial and messy inputs, drifting model versions, rate limits, partial outages, and users who care about cost and latency. The failure distribution has a long tail: a few percent of runs loop, time out, or do something odd. You cannot see this tail without instrumentation, and you cannot fix it without an evaluation loop.
Observability
Section titled “Observability”What to record
Section titled “What to record”Record every run as a trace: a tree of spans.
run (task_id, user, model, prompt_version, total_cost, outcome) |-- step 1: llm_call (input_tokens, output_tokens, latency, cached_tokens, stop_reason) | `-- tool_call: search (args, result_size, latency, error?) |-- step 2: llm_call ... | `-- tool_call: read_file ... `-- final answerPer span: inputs and outputs (with redaction), timing, token counts, model and parameters, versions of prompts and tools, errors, and a run id for correlating logs. Include human feedback and downstream outcomes (ticket reopened, change reverted) as later annotations on the trace. The OpenTelemetry GenAI semantic conventions and several commercial and open-source LLM observability tools define span schemas; adopting a standard keeps you portable.
A minimal tracer:
import json, time, contextlib, uuid
class Tracer: def __init__(self, path, run_id=None): self.path, self.run_id = path, run_id or uuid.uuid4().hex
@contextlib.contextmanager def span(self, name, **attrs): start, span_id = time.time(), uuid.uuid4().hex[:8] rec = {"run": self.run_id, "span": span_id, "name": name, **attrs} try: yield rec # caller can add fields: rec["tokens"] = ... rec["status"] = "ok" except Exception as e: rec["status"] = "error"; rec["error"] = repr(e) raise finally: rec["duration_s"] = round(time.time() - start, 4) with open(self.path, "a") as f: f.write(json.dumps(rec, default=str) + "\n")What to watch
Section titled “What to watch”- Outcome metrics: success proxies (task completion, user acceptance, escalation rate), not just “no exception.”
- Cost and latency distributions: p50, p95, p99 per task type; the tail matters.
- Step counts and loop rates: runs hitting the step limit.
- Tool error rates per tool; a spike indicates a broken integration or a tool-description regression.
- Token mix: input versus output versus cached.
- Drift: changing input distribution, new failure categories, silent model updates on the provider side.
- Safety signals: denied actions, approval rates, injection detections (Safety).
Privacy
Section titled “Privacy”Traces contain user data and often secrets. Redact PII and credentials before storage, set retention limits, restrict access, and check provider data-handling terms. The ability to delete a user’s traces is a legal requirement in many jurisdictions.
Evaluation in the loop
Section titled “Evaluation in the loop”- Golden set in CI. A fixed suite of representative tasks (Evaluation) runs on every change to prompts, tools, models or harness; block regressions.
- Mine production for new cases. Failed or low-rated traces become new eval tasks; this keeps the suite aligned with reality.
- Shadow and canary: run a new configuration on a copy of traffic (shadow, no user impact) or a small fraction (canary), compare outcome and cost distributions before full rollout.
- Pin model versions where possible and re-evaluate on upgrades; provider “aliases” can change behavior without notice.
- Online LLM judges can score samples of production traces for quality drift, calibrated against periodic human review.
Cost and latency model
Section titled “Cost and latency model”For a loop of steps, with input context at step of tokens and output tokens, with per-token prices :
Since grows roughly linearly with , input cost grows quadratically in unless cached or trimmed. Typical agent runs are input-dominated. Latency per step is roughly time-to-first-token (grows with context length) plus output tokens times per-token decode time plus tool time; end-to-end latency is the sum over sequential steps, which is why step count is the first-order driver.
Worked example. 20 steps, context growing from 4k to 40k tokens (average 22k), output 300 tokens per step. Input tokens processed k, output k. Whatever the prices, input tokens outnumber output about 70:1, and since output tokens are usually priced several times higher per token, input still dominates cost unless caching applies.
Levers, roughly by impact
Section titled “Levers, roughly by impact”1. Prompt caching. Providers cache the processed prefix of a prompt: repeated prefixes are billed at a steep discount (typically a large fraction off input price for cache reads) and are faster. To exploit it: keep system prompt, tool definitions and early history byte-identical across steps; append only; place volatile content (timestamps, user-specific data) at the end; avoid reordering tools; consider explicit cache breakpoints where the API offers them. Cache lifetimes are short (minutes), so bursty traffic benefits most. Check your provider’s current pricing and TTL rules.
2. Fewer steps. Better tools (one search returning what two did), parallel tool calls, clear stopping criteria, planning that avoids wandering. Each removed step removes its latency and re-read of the context.
3. Context control. Elide old tool outputs, compact, delegate noisy work to subagents (Memory). Smaller context lowers both cost and time-to-first-token and often improves accuracy.
4. Model routing and cascades. Send easy work to a small fast model and escalate on difficulty or low confidence. Patterns: a router classifier on the task; a cascade that tries the cheap model, verifies, and escalates only on failure; using a large model to plan and a small one to execute steps. Measure on your eval: the cheap path must preserve quality. Reasoning-effort settings are a similar knob within one model (Test-time compute).
def cascade(task, small, large, verify): out = small(task) return out if verify(task, out) else large(task)5. Parallelism. Independent tool calls and independent subagents run concurrently; cost the same, latency lower.
6. Output control. Constrain output length and format; avoid asking for long reasoning where unnecessary; use structured outputs to avoid retries from malformed JSON.
7. Caching results. Memoize deterministic, expensive tool calls (search, fetch, embeddings) keyed by arguments, with TTL. Semantic caching of whole answers is riskier; use only where staleness and near-duplicate matching errors are acceptable.
8. Batch APIs for offline workloads (evaluation runs, bulk classification) usually offer a substantial discount in exchange for delayed completion.
9. Self-hosting and smaller fine-tuned models for high-volume narrow tasks, after you have data (RL and distillation); weigh against operational burden.
Latency UX
Section titled “Latency UX”Streaming partial output and status (“Searching… reading 3 files…”) makes waits tolerable. For long runs, run asynchronously, notify on completion, and allow cancellation and steering. Set expectations: a user will wait seconds for a chat answer but minutes for a research report if told so.
Reliability in operation
Section titled “Reliability in operation”- Durable execution: long-running agents should survive process restarts and provider errors. Persist the transcript and step state; resume from the last completed step, with idempotent tools so that replays do not duplicate side effects. Workflow engines (Temporal-style) provide this; a simple version is a database row per run plus per-step checkpoints.
- Rate limits and backoff: retry with exponential backoff and jitter; respect provider headers; queue and shed load rather than failing everything.
- Fallbacks: a secondary provider or model when the primary is down; degrade to a simpler workflow or human handoff.
- Timeouts at every layer: model call, tool call, whole run.
- Budgets and quotas per user and per tenant, enforced server-side; a runaway loop should hit a hard USD cap.
- Kill switch: the ability to stop all runs and disable a tool or model quickly.
- Concurrency control: avoid two agents editing the same resource; use locks or optimistic concurrency.
Deployment patterns
Section titled “Deployment patterns”- Stateless request-response for short agents behind an API.
- Queue-backed workers for long runs, with run records and status polling or webhooks.
- Sandboxed per-run environments (container or microVM per task) for code and computer-use agents, pooled and pre-warmed to hide start-up time (Code agents).
- Versioning: treat the prompt, tool schemas and model name as a versioned configuration with changelogs; record the version on every trace so regressions can be bisected.
- Feature flags for rollout, kill, and A/B testing.
Human feedback loop
Section titled “Human feedback loop”Provide users a way to correct or flag behavior; route low-confidence or high-risk cases to humans; review a random sample of successes (silent errors hide there); close the loop by converting reviewed cases into eval tasks, memory entries or training data. Mature teams run a weekly “read the traces” ritual; it is cheap and finds the most important problems.
An operational checklist
Section titled “An operational checklist”- Every run has a trace with run id, versions, tokens, cost, outcome.
- Hard limits: steps, time, tokens, USD per run; per-user daily caps.
- Golden eval set in CI; shadow or canary for changes.
- Pinned model versions and re-eval on upgrade.
- Redaction, retention and access control on traces.
- Prefix-stable prompts to maximize cache hits; cache hit rate dashboarded.
- Alerts on success-rate drop, cost spike, loop rate, tool error rate.
- Idempotent tools; durable state for long runs; kill switch.
- Security controls from Safety in place and tested.
- A written incident runbook, including how to revoke agent credentials.
When costs do not add up
Section titled “When costs do not add up”Compute value per task versus cost per task. If a resolved ticket is worth USD 5 and the agent costs USD 0.40 per attempt at 70% success with human cleanup on failures, the economics may be fine; if the agent costs USD 8 per attempt, no amount of prompt tuning rescues it, and you should reconsider scope, routing or whether a workflow suffices (What is an agent?). Quote cost per successful task:
Key takeaways
Section titled “Key takeaways”- Trace every run with versions, tokens, cost and outcome; redact and secure the data.
- Evaluate in CI, canary changes, pin model versions and turn production failures into eval tasks.
- Agent cost is input-dominated and grows quadratically with steps without caching; latency scales with sequential steps.
- Biggest levers: prompt-cache-friendly prefixes, fewer steps, context trimming, model routing, parallelism.
- Production needs hard budgets, durable and idempotent execution, fallbacks and a kill switch.
Exercises
Section titled “Exercises”- Compute. An agent runs 15 steps with a 3k-token fixed prefix and adds 1.5k tokens per step. Compute total input tokens processed without caching, and tokens billed at the full rate with perfect prefix caching of everything previously seen.
- Compute. A cascade uses a small model (cost 1 unit, 60% pass on the verifier) and escalates to a large model (cost 10 units). The verifier costs 0.2 units. Expected cost per task versus always using the large model?
- Hands-on. Add the
Tracerto the agent from the project lecture and write a script that reads the JSONL to report p50 and p95 latency per tool. - Design. Specify the alerts and thresholds you would set for a customer-support agent, and what action each triggers.
Solutions (1, 2)
- Context at step is tokens. Sum over 15 steps: . With perfect caching the new (uncached) tokens are the first prefix (3,000) plus 1,500 per later step: at the full rate; everything else is billed at the cheaper cache-read rate (ignoring cache-write surcharges).
- Cascade: units versus 10 for always-large, a 48% saving, provided the verifier is accurate and quality is preserved.
Further reading
Section titled “Further reading”- Anthropic, “Building effective agents” (2024), sections on evaluation and when to add complexity.
- Chen et al., “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance” (2023).
- Kapoor et al., “AI Agents That Matter” (2024), on cost-controlled evaluation.
- OpenTelemetry, GenAI semantic conventions (opentelemetry.io).
- Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (vLLM, 2023), for serving-side context.
- Provider documentation on prompt caching and batch APIs (check current terms).