Skip to content

Building LLM Applications

What you’ll learn. How to turn the previous lectures into a production application: typical architectures and when to use which; concrete API usage patterns (messages, system prompts, tool use, structured output, streaming) with the Anthropic and OpenAI-style SDKs; prompt caching and response caching; retries, rate limits and fallbacks; guardrails; observability and evaluation in production; cost and latency engineering; and safety, privacy and compliance. Code is illustrative: SDK method names and model IDs change, so verify against current documentation.

client -> API gateway/auth -> app backend
|-> input guardrails (PII, injection/abuse checks, rate limit)
|-> context assembly (system prompt, history, retrieval, memory, tool defs)
|-> LLM call(s) (router -> model) <-> tools / retrieval / sandbox
|-> output guardrails (schema validation, moderation, grounding checks)
|-> response (streamed) -> client
observability: traces, logs, metrics, eval sampling, feedback

Common shapes, from simplest:

  1. Single call: a prompt template, one model call, validation. Many production features (summarize, classify, extract, rewrite) are this. Start here.
  2. Workflow / chain: fixed steps (classify, retrieve, generate, verify), possibly in parallel or with routing. Predictable and testable.
  3. RAG app: workflow with retrieval (RAG).
  4. Agent: model-directed loop with tools (Agents & Tool Use). Highest flexibility, cost and risk.

Principles: keep the LLM behind your own thin interface (so you can swap providers and models; the best model of today won’t be next quarter’s); keep business logic and authorization in code, not in prompts; treat model output as untrusted input to the rest of your system; and log everything needed to reproduce a call.

Model selection. Match model to step: small/fast models for routing, classification, extraction and simple rewriting; large or reasoning models for planning, hard reasoning and synthesis (Reasoning Models). A cascade tries a cheap model first and escalates when confidence is low or validation fails. Decide with your eval set (Evaluating LLMs), not reputation. Hosted API versus self-hosted open-weight (Inference & Serving): hosted is the fastest start and best quality per effort; self-hosting offers data control, fixed cost at high volume, and customization (Fine-Tuning in Practice), at the price of operations work.

Chat-style APIs take a list of messages (roles: user, assistant, plus a separate or special system prompt), generation parameters, and return content blocks plus usage (input/output token counts) and a stop reason.

import anthropic
client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY from the environment
MODEL = "claude-sonnet-4-5" # placeholder: use a current model ID from the docs
resp = client.messages.create(
model=MODEL,
max_tokens=512,
system="You are a concise assistant for an internal HR knowledge base.",
messages=[{"role": "user", "content": "Summarize our parental leave policy in 3 bullets."}],
temperature=0.2,
)
print(resp.content[0].text)
print(resp.usage.input_tokens, resp.usage.output_tokens, resp.stop_reason)

Note: max_tokens is required, and always check stop_reason (end_turn, max_tokens, tool_use…) to detect truncation. Some newer models restrict or ignore certain sampling parameters; check the model’s documentation.

from openai import OpenAI
client = OpenAI() # also works against OpenAI-compatible servers via base_url
resp = client.chat.completions.create(
model="gpt-4.1-mini", # placeholder model ID
messages=[
{"role": "system", "content": "You are a concise assistant for an internal HR knowledge base."},
{"role": "user", "content": "Summarize our parental leave policy in 3 bullets."},
],
temperature=0.2,
)
print(resp.choices[0].message.content, resp.usage)

The shapes differ in details (system prompt placement, field names, newer “responses”-style endpoints) but the concepts are identical. Many open-source servers (vLLM, llama.cpp, Ollama) expose OpenAI-compatible endpoints, which makes the OpenAI-style interface a practical lingua franca; wrap whichever you use in a small adapter.

tools = [{
"name": "get_order_status",
"description": "Look up an order by ID. Use when the user asks where their order is.",
"input_schema": {"type": "object",
"properties": {"order_id": {"type": "string", "description": "e.g. A-10293"}},
"required": ["order_id"]},
}]
messages = [{"role": "user", "content": "Where is order A-10293?"}]
while True:
resp = client.messages.create(model=MODEL, max_tokens=1024, tools=tools, messages=messages)
if resp.stop_reason != "tool_use":
break
messages.append({"role": "assistant", "content": resp.content})
results = []
for block in resp.content:
if block.type == "tool_use":
out = lookup_order(**block.input) # your code; validate args, enforce authz
results.append({"type": "tool_result", "tool_use_id": block.id, "content": str(out)})
messages.append({"role": "user", "content": results})
print(resp.content[-1].text)

Production additions: step cap, per-tool timeouts, argument validation and authorization checks using the end user’s identity (never trust the model to enforce permissions), error results returned to the model, and logging. See Agents & Tool Use.

Use the provider’s schema-constrained “structured output” feature if available, or define a tool whose input schema is your output schema and force its use; then validate with Pydantic and retry with the error message on failure (Prompting).

APIs are stateless (unless you use a provider’s stored-conversation feature): you resend the history each turn. Manage it: truncate, summarize old turns, drop stale tool results. Count tokens before sending to avoid context-limit errors, using the provider’s token-counting facility or the model’s tokenizer (Tokenization).

LLMs generate token by token; waiting for a full response makes the app feel slow. Streaming returns partial output over server-sent events (SSE). It improves perceived latency: users see output at time-to-first-token (TTFT), not at completion.

with client.messages.stream(model=MODEL, max_tokens=1024,
messages=[{"role": "user", "content": "Explain RAG briefly."}]) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
final = stream.get_final_message() # includes usage and stop reason

Backend to browser, pass SSE through (or use WebSockets). Considerations:

  • Structured output and tools while streaming: partial JSON isn’t valid; use incremental parsers, or stream only the human-readable part and parse structured results at the end.
  • Moderation of streamed output: either buffer small windows before display, or accept post-hoc retraction.
  • Cancellation: stop generation when the client disconnects, to avoid paying for unused tokens.
  • Error handling mid-stream: connection drops or overload errors can occur after partial output; design the UI for retry.
  • Intermediate events (tool-call starts, reasoning summaries) can be shown as progress for agents.
  1. Provider prompt caching. Providers cache the processed KV state of a prompt prefix (Inference & Serving) and charge a fraction for cached input tokens plus faster TTFT; some require explicit cache markers, others are automatic. Structure prompts as [stable: system prompt, tool definitions, reference documents] then [variable: history, user query]. Any change early in the prefix invalidates everything after it, so don’t put timestamps or per-user data at the top. Cache entries expire after minutes of inactivity, so benefits depend on traffic patterns.
resp = client.messages.create(
model=MODEL, max_tokens=512,
system=[{"type": "text", "text": LONG_POLICY_DOCUMENT,
"cache_control": {"type": "ephemeral"}}], # mark the end of the stable prefix
messages=[{"role": "user", "content": question}],
)
print(resp.usage) # inspect cache read/creation token counts to verify hits
  1. Response (exact) cache. Hash (model, prompt, parameters) to a stored answer. Only valid for deterministic, non-personalized requests; invalidate on prompt/model changes.
  2. Semantic cache. Embed the query, and return a stored answer if a past query is nearly identical. Cheap wins for FAQ-like traffic but risks serving wrong answers for subtly different questions (negation, entities, numbers); set a high threshold, keep it to low-stakes use cases, and measure the false-hit rate.
  3. Embedding and tool-result caches for repeated computations; cache retrieval results with short TTLs.

Reliability: retries, limits and fallbacks

Section titled “Reliability: retries, limits and fallbacks”
  • Timeouts on every call (including streaming idle timeouts).
  • Retries with exponential backoff and jitter for transient errors (HTTP 429 rate limits, 5xx, overload); do not retry client errors (400, 401). Honor Retry-After headers. The official SDKs implement basic retries; configure the limits.
  • Rate limits: respect requests-per-minute and tokens-per-minute; use queues, concurrency limits and priority classes (interactive over batch).
  • Fallbacks: secondary model or provider on persistent failure, degraded mode (cached answer, simpler path, message to try later). Test fallbacks: prompts and tool formats often need adaptation across models.
  • Idempotency for side-effecting tool calls so retries don’t duplicate actions.
  • Batch APIs for offline work (evals, bulk classification, embeddings) typically give substantial discounts for asynchronous completion.
  • Pin model versions and re-run your eval suite before upgrading: behavior shifts between versions, and aliases may move.

Layered checks around the model, each catching different failures:

  • Input: authentication and quotas; PII detection/redaction; abuse and jailbreak/injection classifiers (a mitigation, not a guarantee); topic scoping; size limits.
  • Prompt-level: clear system instructions and scope; delimiting untrusted content (Prompting).
  • Tool-level: least privilege, allowlists, approval gates for side effects, sandboxed execution, per-user authorization in code (Agents).
  • Output: schema validation; moderation classifiers; grounding/citation checks for RAG (is each claim supported by retrieved text?); regex/secret scanners (API keys, PII leakage); safe rendering (sanitize HTML/markdown; block untrusted image URLs that could exfiltrate data); never execute output as code on trusted hosts.
  • Policy: a fallback response for blocked content; escalate to a human.

Each guardrail adds latency, cost and false positives: measure precision/recall for each on labelled data. Run independent checks in parallel with generation where possible (stream, but cancel if the input check fails). Guardrails reduce risk; they don’t remove the need for architecture that limits blast radius.

LLM apps fail in ways logs of HTTP status codes won’t show. Capture, per request, a trace with spans for each step (retrieval, rerank, each LLM call, each tool call):

  • full prompts (after assembly) and completions, model and version, parameters, stop reason;
  • token counts, cost, latency (TTFT and total), cache hit/miss;
  • retrieved document IDs and scores; tool inputs/outputs and errors;
  • user/session/tenant IDs (pseudonymized as appropriate), prompt/template version, feature flags;
  • user feedback (thumbs, edits, retries) and downstream outcomes.

Tooling: OpenTelemetry (with its emerging generative-AI semantic conventions) plus LLM-specific platforms (Langfuse, LangSmith, Arize Phoenix, Braintrust, Helicone, and others) or your own tracing store. Handle privacy: prompts contain user data; apply retention limits, access controls and redaction.

Closing the loop:

  1. Sample production traces for review; cluster failures; add them to the eval set (Evaluating LLMs).
  2. Run online judges or heuristics for drift (format-failure rate, refusal rate, hallucination checks, length, latency).
  3. Alert on cost per request, error rates, p95 latency, and quality proxies.
  4. Replay traces against new prompts/models offline before rollout; canary and A/B changes.

Cost model. cost=nin cin+nout cout\text{cost} = n_{in}\,c_{in} + n_{out}\,c_{out} (with cached input at a discount; reasoning tokens billed as output). Output tokens typically cost several times input tokens. Multiply by calls per task for pipelines and agents (context regrows each iteration).

Levers, roughly by impact:

  1. Route to the cheapest adequate model (classifier/router, cascades); evaluate quality loss.
  2. Cut tokens: shorter system prompts, fewer/relevant retrieved chunks, summarize history, concise tool outputs, cap max_tokens, ask for terse formats (JSON fields rather than prose).
  3. Prompt caching for long stable prefixes.
  4. Batch API for non-interactive workloads.
  5. Avoid unnecessary reasoning: set reasoning effort low for easy tasks.
  6. Cache responses where safe.
  7. Fine-tune or distill a small model for a high-volume narrow task (Fine-Tuning in Practice); self-host when volume justifies.
  8. Eliminate calls: deterministic code for what code can do; early exits.

Latency levers: stream; reduce input size (prefill time) and output length (decode is serial, so latency is dominated by output tokens: roughly TTFT+nout⋅TPOT\text{TTFT}+n_{out}\cdot\text{TPOT}); parallelize independent calls and tool calls; use faster/smaller models for the critical path; prefetch/speculatively run likely next steps; cache; co-locate with the provider region; split work so the user sees progress early.

Budget and enforce: per-request token caps, per-user and per-tenant spend limits, circuit breakers on runaway agents.

  • Data handling: know what you send to providers: retention, training-use policies, regional residency, zero-data-retention options, enterprise terms. Minimize and redact sensitive data (PII, secrets, health and financial data) before it leaves your boundary; avoid logging secrets.
  • Access control: enforce permissions in retrieval and tools using the end user’s identity; multi-tenant isolation in indexes, caches and memory (a shared semantic cache can leak across tenants).
  • Prompt injection and exfiltration: assume untrusted content can steer the model (Prompting); remove the “lethal trifecta” legs where possible; sanitize rendered output.
  • Secrets: API keys in a secret manager, server-side only, never in client code or prompts; rotate; per-environment keys; spending limits.
  • Transparency and consent: tell users they’re interacting with AI where required or expected; provide feedback and report channels; human escalation for consequential decisions.
  • Regulation and risk: privacy law (GDPR-style rights of access and deletion apply to stored prompts, memories and embeddings: design deletion paths), sector rules, and AI-specific regulation (e.g. the EU AI Act’s risk-based obligations). Consult legal counsel for your domain; this lecture is engineering guidance, not legal advice.
  • Content risks: copyright and licensing of outputs and training data, defamation risk from hallucinated claims about real people, bias and fairness testing across user groups, and clear UX for uncertainty.
  • Abuse prevention: authentication, rate limits, anomaly detection, usage policies; red-team before launch and continuously (Evaluating LLMs).
  • Human oversight: for high-stakes outputs (medical, legal, financial, safety-critical), keep a qualified human in the loop and design the UI to support verification (citations, highlighted sources).
  1. Task defined; success metrics and an eval set (including adversarial and out-of-scope cases) exist.
  2. Simplest architecture that meets the metric; model chosen by evals; provider abstracted.
  3. Prompts versioned in source control; model versions pinned.
  4. Structured outputs validated; retries with backoff; timeouts; fallbacks tested.
  5. Guardrails at input, tools and output; permissions enforced in code.
  6. Streaming and caching configured; cost/latency budgets with alerts.
  7. Tracing, feedback capture and a pipeline from failures to eval cases.
  8. Privacy review, data retention, deletion path, incident response plan.
  9. Red-team completed; staged rollout with canary and kill switch.
  • Start with the simplest architecture (single call, then workflow, then agent) and keep the LLM behind a swappable interface; keep authorization and business logic in code.
  • Chat APIs are stateless: you manage history, tool loops, structured-output validation and stop reasons.
  • Stream for perceived latency; use prompt caching with a stable-prefix-first layout; be careful with semantic caches.
  • Engineer for failure: timeouts, backoff retries, rate-limit handling, fallbacks, pinned versions and idempotent tools.
  • Layer guardrails (input, tool, output) and measure their error rates; they reduce, not eliminate, risk.
  • Observability (full traces, cost, feedback) feeds the eval set; evaluate changes before rollout.
  • Control cost via routing, token reduction, caching, batching and distillation; protect privacy through minimization, tenant isolation and retention policies.
  1. Hands-on. Implement a thin llm.complete(messages, tools=None, schema=None) interface with two provider adapters (Anthropic-style and OpenAI-style) and run the same eval suite on both.
  2. Hands-on. Add retries with exponential backoff and jitter, a timeout, and a fallback model; simulate failures with a mock server and confirm behavior.
  3. Conceptual. Your agent resends a 20K-token prefix each step for 15 steps. Estimate the input tokens billed with and without caching (assume cached reads cost 10% of normal), ignoring growth of the context.
  4. Hands-on. Instrument an LLM app with OpenTelemetry spans (retrieval, LLM call, tool call) recording token counts and cost; build a dashboard of cost per request and p95 latency.
  5. Conceptual. List the places a multi-tenant RAG chatbot might leak one tenant’s data to another, and a control for each.
Solutions (sketch)
  1. Without caching: 15×20K=300K15\times20\text{K}=300\text{K} billed input tokens. With caching: the first step pays full (maybe a write premium), the other 14 pay 10%: 20K+14×2K=48K20\text{K}+14\times2\text{K}=48\text{K} token-equivalents, about an 84% reduction.
  2. Vector index without tenant filters (enforce filter/namespace per tenant), shared semantic or response caches (key by tenant), shared conversation memory (partition), logs and traces visible across tenants (RBAC), fine-tuning on pooled data (separate adapters or consent), prompt templates with data baked in (parameterize), tool credentials shared across tenants (per-tenant credentials).
  • Anthropic and OpenAI API documentation and cookbooks (messages, tool use, streaming, prompt caching, batch APIs, structured outputs); the Anthropic “Building Effective Agents” essay.
  • Chip Huyen, AI Engineering: Building Applications with Foundation Models (2025).
  • Huyen, “Building A Generative AI Platform” (blog, 2024).
  • OWASP, “Top 10 for LLM Applications” (security risk catalogue; versions updated yearly).
  • NIST, “AI Risk Management Framework” (2023).
  • Hamel Husain and Shreya Shankar’s writing and course material on evals and application debugging.
  • OpenTelemetry documentation on generative-AI semantic conventions; documentation for Langfuse, Arize Phoenix and similar tracing tools.