Short-Term vs. Long-Term Memory

Human memory is often described in terms of two broad systems: working memory, which holds a small amount of information actively in mind right now, and long-term memory, which stores a vast archive of knowledge, experiences, and skills that can be retrieved when needed. AI agents face a structurally identical problem, and the solution architecture maps onto the same two-tier structure. Short-term memory for an agent is the context window — everything the model can see in the current call. Long-term memory is any persistent store outside the model — a database, a vector store, a file system — that survives between calls and can be selectively retrieved.

Short-Term Memory: The Context Window as Working Memory

Working memory in a cognitive science sense holds roughly seven items simultaneously, give or take two. The context window is a far more generous working memory — 128,000 tokens can hold an entire book — but it shares the essential characteristic of working memory: it is finite and it resets. When the call ends, whatever was in the context is gone unless the application code saves it somewhere. For an agent, short-term memory includes everything explicitly placed in the current prompt: the system prompt, conversation history replayed so far, any tool outputs retrieved this step, the current task description, and any intermediate reasoning the model has produced in chain-of-thought format. The key property is that the model can directly attend to all of this — it requires no lookup, no retrieval step, no translation. It is simply there, immediately and uniformly accessible.

The costs of relying exclusively on short-term memory are predictable. First, cost: long prompts consume more tokens and cost more per call. Second, capacity: even a 200,000-token window fills up — a month of agent interactions will far exceed it. Third, noise: a context packed with mostly irrelevant history degrades model performance. The model has to attend over everything, including material that is no longer relevant to the current step.

Long-Term Memory: Persistent Stores Outside the Model

Long-term memory for an agent is any external system that persists information beyond a single model call. In practice, agents use several distinct types of external store depending on what they need to remember. Relational databases store structured records: user profiles, task logs, completed steps, configuration. They support precise queries — find all tasks completed since yesterday, get the user's preferred language — but struggle with fuzzy conceptual similarity searches. Vector databases store dense numerical embeddings of text. They are optimized for semantic similarity search: given a query like 'papers about climate adaptation in agriculture,' retrieve the ten stored documents whose meaning is closest to that query. This is the engine behind retrieval-augmented generation, covered in depth in the next lesson. Key-value stores like Redis give agents a fast scratchpad for structured state: the current task's status, intermediate counters, session flags. They are simple and extremely fast but unstructured — the agent must know exactly what key to look up. File systems store raw text, code, or binary data. An agent that processes large documents often stores raw content on disk and reads it back only when needed, rather than keeping it in the context.

The Write-Read Cycle

Long-term memory is only useful if the agent both writes to it and reads from it at the right moments. The write step happens after the model produces output worth keeping: a completed task, a useful piece of retrieved information, a decision the agent made that future steps must respect. The read step happens before a model call that needs that information: retrieve relevant records from the store and inject them into the prompt. This write-read cycle is explicit and engineered — it does not happen automatically. An agent that never writes to long-term memory cannot benefit from it. An agent that writes but never retrieves is just filling a store that does nothing. Getting both steps right, at the right moments in the agent loop, is the core skill of memory system design.