Skip to content

Memory and Context Engineering

What you’ll learn. Why the context window is the agent’s scarce resource; a taxonomy of agent memory; techniques (truncation, summarization/compaction, scratchpads, retrieval, sub-agent isolation, caching); and the failure modes of long contexts.

An LLM has no memory between calls except what you put in the prompt (and the weights, frozen at inference). At each step an agent’s behavior is a function of its context: system prompt, tool definitions, conversation so far, tool results, retrieved documents. “Memory” in agents is therefore a policy about what to put into the context.

Context engineering is the practice of designing that policy: deciding at each step which information the model sees, in what form and order. Prompt engineering (wording instructions) is one subset of it.

A million-token window does not mean you should fill it.

  • Cost and latency: attention cost grows with context; every agent step re-sends the transcript. With TT steps each adding kk tokens, total tokens processed are about kT2/2k T^2 / 2, quadratic in steps unless prefix caching absorbs it (Production and cost).
  • Attention dilution: accuracy tends to degrade as irrelevant tokens accumulate, and models often use information in the middle of long contexts less reliably than at the edges (“lost in the middle”, Liu et al., 2023). Practitioners call the gradual degradation context rot.
  • Distraction and poisoning: stale facts, earlier wrong conclusions or injected text stay in context and keep influencing decisions.
  • Hard limit: eventually the window is exceeded and the run fails.

The guiding principle: the smallest set of high-signal tokens that makes the correct next action likely.

Kind Analogy Where it lives Example
Working RAM The context window Current transcript, plan
Episodic Diary External store of past trajectories “Last time deploy X failed because …”
Semantic Encyclopedia Vector/keyword/graph store Docs, user profile facts
Procedural Skills Prompts, tools, instruction files “How we run tests in this repo”

Procedural memory in practice is often a plain file the agent reads at start (a project instructions file, playbooks, skills directories). It is cheap, human-editable and versionable, and works better than it sounds.

The cheapest intervention: drop old, low-value content. The best candidates are old tool results (a 20 KB file read from 40 steps ago). Keep the call, replace the bulk with a stub: [output elided; re-run read_file to see again]. Because tools are re-callable, elision is nearly lossless.

When the transcript nears a threshold, ask the model to summarize it, then restart a fresh context containing the system prompt plus the summary plus a few recent turns.

def maybe_compact(llm, messages, count_tokens, limit=100_000, keep_last=6):
if count_tokens(messages) < 0.8 * limit:
return messages
head, tail = messages[:-keep_last], messages[-keep_last:]
summary = llm(
"Summarize this agent session for your future self. Preserve: the goal, "
"decisions made and why, files touched, unresolved errors, next steps. "
"Omit raw tool output.\n\n" + render(head)
)
return [messages[0], {"role": "user", "content": f"[Session summary]\n{summary}"}] + tail

Compaction quality is a real research and engineering problem. Summaries that lose a constraint (“never modify the migrations folder”) cause subtle regressions. Tune the summarization prompt on failures, and keep critical constraints in the system prompt where they survive compaction.

Let the agent write notes to a file or memory tool outside the window and read them back when needed: a todo list, a decisions log, a PROGRESS.md. This lets long tasks survive many context resets, because the state lives outside the window. It is the agentic analogue of a lab notebook.

Store large corpora externally and fetch relevant pieces on demand. Classic retrieval-augmented generation is covered in RAG. Agent-specific wrinkles:

  • Just-in-time retrieval: instead of pre-loading everything, give the agent search tools (grep, glob, semantic search) and let it pull what it needs. Pre-loaded embeddings are fast but can be stale; exploration by tool is slower but more precise, and coding agents mostly use the latter, often combined with pre-loaded high-level files.
  • Progressive disclosure: show an index (file names, titles), let the agent open details.
  • Hybrid search: combine lexical (BM25) and dense retrieval; exact identifiers are poorly handled by embeddings alone.

Delegate a noisy subtask (reading 30 files to answer one question) to a sub-agent with a fresh context; it returns a short distilled answer. The parent’s context stays clean. This is the principal context-management motivation for multi-agent systems.

For cross-session memory (user preferences, past outcomes) decide:

  • What to store: facts, preferences, resolved lessons, not raw transcripts.
  • When to write: at session end, or when the agent decides via a remember(fact) tool.
  • How to retrieve: by similarity to the current task, recency, and importance score (Generative Agents, Park et al. 2023, combined recency, importance and relevance).
  • How to forget: deduplicate, expire, and let users inspect and delete. Memory that cannot be audited is a privacy and correctness hazard.
import time, numpy as np
class Memory:
def __init__(self, embed):
self.embed, self.items = embed, [] # (vec, text, t)
def add(self, text):
self.items.append((self.embed(text), text, time.time()))
def search(self, query, k=3, half_life=7*86400):
q = self.embed(query)
def score(item):
v, _, t = item
sim = float(q @ v / (np.linalg.norm(q) * np.linalg.norm(v) + 1e-9))
recency = 0.5 ** ((time.time() - t) / half_life)
return 0.8 * sim + 0.2 * recency
return [x[1] for x in sorted(self.items, key=score, reverse=True)[:k]]

Ordering matters for both quality and cost.

[ system prompt: role, rules, constraints ] <- stable, cacheable prefix
[ tool definitions ] <- stable
[ long-term memory / project instructions ] <- stable per session
[ retrieved documents for this task ] <- varies
[ conversation + tool results ] <- grows
[ latest user message / current goal ] <- recency position

Keep the prefix byte-stable so provider prompt caching applies: editing the system prompt mid-run, or inserting a timestamp near the top, invalidates the cache and multiplies cost. Append rather than rewrite earlier history.

A recurring trick is recitation: have the agent restate the todo list at the end of the context so the goal sits in the most-attended region.

  • Silent information loss in summaries; mitigate with structured summary templates and checks.
  • Memory poisoning: untrusted content (a web page) written to long-term memory becomes a persistent prompt injection. Never auto-store tool output without scrutiny (Safety).
  • Retrieval noise: irrelevant chunks hurt more than they help; measure retrieval precision.
  • Stale memory: facts that were true last month. Store timestamps and verify before acting on volatile facts.
  • Over-remembering: personalizing on a single stray remark.
  • Agent memory is a policy about what enters the context window; context engineering is the discipline of designing it.
  • Long contexts cost quadratic compute across steps without caching and degrade attention quality.
  • Core tools: elide old tool outputs, compact with structured summaries, keep external notes, retrieve just in time, isolate noisy work in sub-agents.
  • Keep a stable prefix for caching; append, do not rewrite.
  • Long-term memory needs write, retrieve and forget policies and is an attack surface.
  1. Compute. An agent adds 2,000 tokens per step for 40 steps with no caching and no trimming. How many total input tokens are processed (ignore the initial prompt)? With a perfect prefix cache, how many uncached tokens?
  2. Hands-on. Implement tool-result elision: after each step, replace any tool message older than 5 turns and longer than 500 characters with a stub that preserves the tool name and call arguments.
  3. Conceptual. Give two kinds of information that should live in the system prompt rather than conversation, because compaction might drop them.
  4. Design. Design a long-term memory for a coding agent used across a team. What do you store, and who may edit it?
Solutions (1)

Step ii sends 2000i2000 i tokens, so total =2000⋅∑i=140i=2000⋅820=1,640,000= 2000 \cdot \sum_{i=1}^{40} i = 2000 \cdot 820 = 1{,}640{,}000. With a perfect cache only the new 2,000 tokens per step are uncached: 80,00080{,}000.

  • Liu et al., “Lost in the Middle: How Language Models Use Long Contexts” (2023).
  • Park et al., “Generative Agents: Interactive Simulacra of Human Behavior” (2023).
  • Packer et al., “MemGPT: Towards LLMs as Operating Systems” (2023).
  • Anthropic, “Effective context engineering for AI agents” (2025).
  • Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020); see also RAG.