Building LLM Applications
What you’ll learn. How to turn the previous lectures into a production application: typical architectures and when to use which; concrete API usage patterns (messages, system prompts, tool use, structured output, streaming) with the Anthropic and OpenAI-style SDKs; prompt caching and response caching; retries, rate limits and fallbacks; guardrails; observability and evaluation in production; cost and latency engineering; and safety, privacy and compliance. Code is illustrative: SDK method names and model IDs change, so verify against current documentation.
Architecture
Section titled “Architecture”client -> API gateway/auth -> app backend |-> input guardrails (PII, injection/abuse checks, rate limit) |-> context assembly (system prompt, history, retrieval, memory, tool defs) |-> LLM call(s) (router -> model) <-> tools / retrieval / sandbox |-> output guardrails (schema validation, moderation, grounding checks) |-> response (streamed) -> client observability: traces, logs, metrics, eval sampling, feedbackCommon shapes, from simplest:
- Single call: a prompt template, one model call, validation. Many production features (summarize, classify, extract, rewrite) are this. Start here.
- Workflow / chain: fixed steps (classify, retrieve, generate, verify), possibly in parallel or with routing. Predictable and testable.
- RAG app: workflow with retrieval (RAG).
- Agent: model-directed loop with tools (Agents & Tool Use). Highest flexibility, cost and risk.
Principles: keep the LLM behind your own thin interface (so you can swap providers and models; the best model of today won’t be next quarter’s); keep business logic and authorization in code, not in prompts; treat model output as untrusted input to the rest of your system; and log everything needed to reproduce a call.
Model selection. Match model to step: small/fast models for routing, classification, extraction and simple rewriting; large or reasoning models for planning, hard reasoning and synthesis (Reasoning Models). A cascade tries a cheap model first and escalates when confidence is low or validation fails. Decide with your eval set (Evaluating LLMs), not reputation. Hosted API versus self-hosted open-weight (Inference & Serving): hosted is the fastest start and best quality per effort; self-hosting offers data control, fixed cost at high volume, and customization (Fine-Tuning in Practice), at the price of operations work.
API usage
Section titled “API usage”Chat-style APIs take a list of messages (roles: user, assistant, plus a separate or special system prompt), generation parameters, and return content blocks plus usage (input/output token counts) and a stop reason.
Basic call with the Anthropic Python SDK
Section titled “Basic call with the Anthropic Python SDK”import anthropic
client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY from the environmentMODEL = "claude-sonnet-4-5" # placeholder: use a current model ID from the docs
resp = client.messages.create( model=MODEL, max_tokens=512, system="You are a concise assistant for an internal HR knowledge base.", messages=[{"role": "user", "content": "Summarize our parental leave policy in 3 bullets."}], temperature=0.2,)print(resp.content[0].text)print(resp.usage.input_tokens, resp.usage.output_tokens, resp.stop_reason)Note: max_tokens is required, and always check stop_reason (end_turn, max_tokens, tool_use…) to detect truncation. Some newer models restrict or ignore certain sampling parameters; check the model’s documentation.
The same with an OpenAI-style API
Section titled “The same with an OpenAI-style API”from openai import OpenAIclient = OpenAI() # also works against OpenAI-compatible servers via base_urlresp = client.chat.completions.create( model="gpt-4.1-mini", # placeholder model ID messages=[ {"role": "system", "content": "You are a concise assistant for an internal HR knowledge base."}, {"role": "user", "content": "Summarize our parental leave policy in 3 bullets."}, ], temperature=0.2,)print(resp.choices[0].message.content, resp.usage)The shapes differ in details (system prompt placement, field names, newer “responses”-style endpoints) but the concepts are identical. Many open-source servers (vLLM, llama.cpp, Ollama) expose OpenAI-compatible endpoints, which makes the OpenAI-style interface a practical lingua franca; wrap whichever you use in a small adapter.
Tool use (function calling)
Section titled “Tool use (function calling)”tools = [{ "name": "get_order_status", "description": "Look up an order by ID. Use when the user asks where their order is.", "input_schema": {"type": "object", "properties": {"order_id": {"type": "string", "description": "e.g. A-10293"}}, "required": ["order_id"]},}]
messages = [{"role": "user", "content": "Where is order A-10293?"}]while True: resp = client.messages.create(model=MODEL, max_tokens=1024, tools=tools, messages=messages) if resp.stop_reason != "tool_use": break messages.append({"role": "assistant", "content": resp.content}) results = [] for block in resp.content: if block.type == "tool_use": out = lookup_order(**block.input) # your code; validate args, enforce authz results.append({"type": "tool_result", "tool_use_id": block.id, "content": str(out)}) messages.append({"role": "user", "content": results})print(resp.content[-1].text)Production additions: step cap, per-tool timeouts, argument validation and authorization checks using the end user’s identity (never trust the model to enforce permissions), error results returned to the model, and logging. See Agents & Tool Use.
Structured output
Section titled “Structured output”Use the provider’s schema-constrained “structured output” feature if available, or define a tool whose input schema is your output schema and force its use; then validate with Pydantic and retry with the error message on failure (Prompting).
Conversation state
Section titled “Conversation state”APIs are stateless (unless you use a provider’s stored-conversation feature): you resend the history each turn. Manage it: truncate, summarize old turns, drop stale tool results. Count tokens before sending to avoid context-limit errors, using the provider’s token-counting facility or the model’s tokenizer (Tokenization).
Streaming
Section titled “Streaming”LLMs generate token by token; waiting for a full response makes the app feel slow. Streaming returns partial output over server-sent events (SSE). It improves perceived latency: users see output at time-to-first-token (TTFT), not at completion.
with client.messages.stream(model=MODEL, max_tokens=1024, messages=[{"role": "user", "content": "Explain RAG briefly."}]) as stream: for text in stream.text_stream: print(text, end="", flush=True) final = stream.get_final_message() # includes usage and stop reasonBackend to browser, pass SSE through (or use WebSockets). Considerations:
- Structured output and tools while streaming: partial JSON isn’t valid; use incremental parsers, or stream only the human-readable part and parse structured results at the end.
- Moderation of streamed output: either buffer small windows before display, or accept post-hoc retraction.
- Cancellation: stop generation when the client disconnects, to avoid paying for unused tokens.
- Error handling mid-stream: connection drops or overload errors can occur after partial output; design the UI for retry.
- Intermediate events (tool-call starts, reasoning summaries) can be shown as progress for agents.
Caching
Section titled “Caching”- Provider prompt caching. Providers cache the processed KV state of a prompt prefix (Inference & Serving) and charge a fraction for cached input tokens plus faster TTFT; some require explicit cache markers, others are automatic. Structure prompts as [stable: system prompt, tool definitions, reference documents] then [variable: history, user query]. Any change early in the prefix invalidates everything after it, so don’t put timestamps or per-user data at the top. Cache entries expire after minutes of inactivity, so benefits depend on traffic patterns.
resp = client.messages.create( model=MODEL, max_tokens=512, system=[{"type": "text", "text": LONG_POLICY_DOCUMENT, "cache_control": {"type": "ephemeral"}}], # mark the end of the stable prefix messages=[{"role": "user", "content": question}],)print(resp.usage) # inspect cache read/creation token counts to verify hits- Response (exact) cache. Hash (model, prompt, parameters) to a stored answer. Only valid for deterministic, non-personalized requests; invalidate on prompt/model changes.
- Semantic cache. Embed the query, and return a stored answer if a past query is nearly identical. Cheap wins for FAQ-like traffic but risks serving wrong answers for subtly different questions (negation, entities, numbers); set a high threshold, keep it to low-stakes use cases, and measure the false-hit rate.
- Embedding and tool-result caches for repeated computations; cache retrieval results with short TTLs.
Reliability: retries, limits and fallbacks
Section titled “Reliability: retries, limits and fallbacks”- Timeouts on every call (including streaming idle timeouts).
- Retries with exponential backoff and jitter for transient errors (HTTP 429 rate limits, 5xx, overload); do not retry client errors (400, 401). Honor
Retry-Afterheaders. The official SDKs implement basic retries; configure the limits. - Rate limits: respect requests-per-minute and tokens-per-minute; use queues, concurrency limits and priority classes (interactive over batch).
- Fallbacks: secondary model or provider on persistent failure, degraded mode (cached answer, simpler path, message to try later). Test fallbacks: prompts and tool formats often need adaptation across models.
- Idempotency for side-effecting tool calls so retries don’t duplicate actions.
- Batch APIs for offline work (evals, bulk classification, embeddings) typically give substantial discounts for asynchronous completion.
- Pin model versions and re-run your eval suite before upgrading: behavior shifts between versions, and aliases may move.
Guardrails
Section titled “Guardrails”Layered checks around the model, each catching different failures:
- Input: authentication and quotas; PII detection/redaction; abuse and jailbreak/injection classifiers (a mitigation, not a guarantee); topic scoping; size limits.
- Prompt-level: clear system instructions and scope; delimiting untrusted content (Prompting).
- Tool-level: least privilege, allowlists, approval gates for side effects, sandboxed execution, per-user authorization in code (Agents).
- Output: schema validation; moderation classifiers; grounding/citation checks for RAG (is each claim supported by retrieved text?); regex/secret scanners (API keys, PII leakage); safe rendering (sanitize HTML/markdown; block untrusted image URLs that could exfiltrate data); never execute output as code on trusted hosts.
- Policy: a fallback response for blocked content; escalate to a human.
Each guardrail adds latency, cost and false positives: measure precision/recall for each on labelled data. Run independent checks in parallel with generation where possible (stream, but cancel if the input check fails). Guardrails reduce risk; they don’t remove the need for architecture that limits blast radius.
Observability
Section titled “Observability”LLM apps fail in ways logs of HTTP status codes won’t show. Capture, per request, a trace with spans for each step (retrieval, rerank, each LLM call, each tool call):
- full prompts (after assembly) and completions, model and version, parameters, stop reason;
- token counts, cost, latency (TTFT and total), cache hit/miss;
- retrieved document IDs and scores; tool inputs/outputs and errors;
- user/session/tenant IDs (pseudonymized as appropriate), prompt/template version, feature flags;
- user feedback (thumbs, edits, retries) and downstream outcomes.
Tooling: OpenTelemetry (with its emerging generative-AI semantic conventions) plus LLM-specific platforms (Langfuse, LangSmith, Arize Phoenix, Braintrust, Helicone, and others) or your own tracing store. Handle privacy: prompts contain user data; apply retention limits, access controls and redaction.
Closing the loop:
- Sample production traces for review; cluster failures; add them to the eval set (Evaluating LLMs).
- Run online judges or heuristics for drift (format-failure rate, refusal rate, hallucination checks, length, latency).
- Alert on cost per request, error rates, p95 latency, and quality proxies.
- Replay traces against new prompts/models offline before rollout; canary and A/B changes.
Cost and latency engineering
Section titled “Cost and latency engineering”Cost model. (with cached input at a discount; reasoning tokens billed as output). Output tokens typically cost several times input tokens. Multiply by calls per task for pipelines and agents (context regrows each iteration).
Levers, roughly by impact:
- Route to the cheapest adequate model (classifier/router, cascades); evaluate quality loss.
- Cut tokens: shorter system prompts, fewer/relevant retrieved chunks, summarize history, concise tool outputs, cap
max_tokens, ask for terse formats (JSON fields rather than prose). - Prompt caching for long stable prefixes.
- Batch API for non-interactive workloads.
- Avoid unnecessary reasoning: set reasoning effort low for easy tasks.
- Cache responses where safe.
- Fine-tune or distill a small model for a high-volume narrow task (Fine-Tuning in Practice); self-host when volume justifies.
- Eliminate calls: deterministic code for what code can do; early exits.
Latency levers: stream; reduce input size (prefill time) and output length (decode is serial, so latency is dominated by output tokens: roughly ); parallelize independent calls and tool calls; use faster/smaller models for the critical path; prefetch/speculatively run likely next steps; cache; co-locate with the provider region; split work so the user sees progress early.
Budget and enforce: per-request token caps, per-user and per-tenant spend limits, circuit breakers on runaway agents.
Safety, privacy and compliance
Section titled “Safety, privacy and compliance”- Data handling: know what you send to providers: retention, training-use policies, regional residency, zero-data-retention options, enterprise terms. Minimize and redact sensitive data (PII, secrets, health and financial data) before it leaves your boundary; avoid logging secrets.
- Access control: enforce permissions in retrieval and tools using the end user’s identity; multi-tenant isolation in indexes, caches and memory (a shared semantic cache can leak across tenants).
- Prompt injection and exfiltration: assume untrusted content can steer the model (Prompting); remove the “lethal trifecta” legs where possible; sanitize rendered output.
- Secrets: API keys in a secret manager, server-side only, never in client code or prompts; rotate; per-environment keys; spending limits.
- Transparency and consent: tell users they’re interacting with AI where required or expected; provide feedback and report channels; human escalation for consequential decisions.
- Regulation and risk: privacy law (GDPR-style rights of access and deletion apply to stored prompts, memories and embeddings: design deletion paths), sector rules, and AI-specific regulation (e.g. the EU AI Act’s risk-based obligations). Consult legal counsel for your domain; this lecture is engineering guidance, not legal advice.
- Content risks: copyright and licensing of outputs and training data, defamation risk from hallucinated claims about real people, bias and fairness testing across user groups, and clear UX for uncertainty.
- Abuse prevention: authentication, rate limits, anomaly detection, usage policies; red-team before launch and continuously (Evaluating LLMs).
- Human oversight: for high-stakes outputs (medical, legal, financial, safety-critical), keep a qualified human in the loop and design the UI to support verification (citations, highlighted sources).
A launch checklist
Section titled “A launch checklist”- Task defined; success metrics and an eval set (including adversarial and out-of-scope cases) exist.
- Simplest architecture that meets the metric; model chosen by evals; provider abstracted.
- Prompts versioned in source control; model versions pinned.
- Structured outputs validated; retries with backoff; timeouts; fallbacks tested.
- Guardrails at input, tools and output; permissions enforced in code.
- Streaming and caching configured; cost/latency budgets with alerts.
- Tracing, feedback capture and a pipeline from failures to eval cases.
- Privacy review, data retention, deletion path, incident response plan.
- Red-team completed; staged rollout with canary and kill switch.
Key takeaways
Section titled “Key takeaways”- Start with the simplest architecture (single call, then workflow, then agent) and keep the LLM behind a swappable interface; keep authorization and business logic in code.
- Chat APIs are stateless: you manage history, tool loops, structured-output validation and stop reasons.
- Stream for perceived latency; use prompt caching with a stable-prefix-first layout; be careful with semantic caches.
- Engineer for failure: timeouts, backoff retries, rate-limit handling, fallbacks, pinned versions and idempotent tools.
- Layer guardrails (input, tool, output) and measure their error rates; they reduce, not eliminate, risk.
- Observability (full traces, cost, feedback) feeds the eval set; evaluate changes before rollout.
- Control cost via routing, token reduction, caching, batching and distillation; protect privacy through minimization, tenant isolation and retention policies.
Exercises
Section titled “Exercises”- Hands-on. Implement a thin
llm.complete(messages, tools=None, schema=None)interface with two provider adapters (Anthropic-style and OpenAI-style) and run the same eval suite on both. - Hands-on. Add retries with exponential backoff and jitter, a timeout, and a fallback model; simulate failures with a mock server and confirm behavior.
- Conceptual. Your agent resends a 20K-token prefix each step for 15 steps. Estimate the input tokens billed with and without caching (assume cached reads cost 10% of normal), ignoring growth of the context.
- Hands-on. Instrument an LLM app with OpenTelemetry spans (retrieval, LLM call, tool call) recording token counts and cost; build a dashboard of cost per request and p95 latency.
- Conceptual. List the places a multi-tenant RAG chatbot might leak one tenant’s data to another, and a control for each.
Solutions (sketch)
- Without caching: billed input tokens. With caching: the first step pays full (maybe a write premium), the other 14 pay 10%: token-equivalents, about an 84% reduction.
- Vector index without tenant filters (enforce filter/namespace per tenant), shared semantic or response caches (key by tenant), shared conversation memory (partition), logs and traces visible across tenants (RBAC), fine-tuning on pooled data (separate adapters or consent), prompt templates with data baked in (parameterize), tool credentials shared across tenants (per-tenant credentials).
Further reading
Section titled “Further reading”- Anthropic and OpenAI API documentation and cookbooks (messages, tool use, streaming, prompt caching, batch APIs, structured outputs); the Anthropic “Building Effective Agents” essay.
- Chip Huyen, AI Engineering: Building Applications with Foundation Models (2025).
- Huyen, “Building A Generative AI Platform” (blog, 2024).
- OWASP, “Top 10 for LLM Applications” (security risk catalogue; versions updated yearly).
- NIST, “AI Risk Management Framework” (2023).
- Hamel Husain and Shreya Shankar’s writing and course material on evals and application debugging.
- OpenTelemetry documentation on generative-AI semantic conventions; documentation for Langfuse, Arize Phoenix and similar tracing tools.