Anatomy of a Real Agent
Abstract architecture diagrams clarify structure; concrete worked examples clarify behavior. This lesson traces the execution of a single realistic task through every layer of a fully specified agent — an AI research assistant tasked with producing a comparative summary of recent AI safety research. We will examine each component's role, the decisions it makes, and the failures it must handle. By the end you will be able to read any agent's architecture documentation and mentally simulate its execution.
The Agent: An AI Research Assistant
Our example agent is ResearchBot, built on Claude 3.5 Sonnet as its LLM core. Its tool layer exposes four tools: search_arxiv(query, max_results) retrieves paper metadata and abstracts from the arXiv preprint server; fetch_paper(arxiv_id) downloads and extracts the full text of a specific paper; write_file(filename, content) writes a text file to a local directory; and summarize_chunk(text, focus) calls the model recursively to summarize a long text with a given focus (used for papers too long to fit in context). Its memory system uses a vector store populated with the current session's retrieved paper chunks, plus in-context memory for the current conversation. The orchestration loop is ReAct-style with a 30-step maximum and a human confirmation checkpoint before any write_file call.
It may seem odd to have a tool that calls the LLM recursively. This pattern — using the model to process content that is too long for the main context window — is called map-reduce summarization. The tool's implementation splits the long paper into chunks, calls the LLM on each chunk to produce a partial summary, then calls the LLM again to synthesize the partial summaries. Wrapping this in a tool keeps the main agent loop clean and allows the pattern to be reused across different agent tasks.
The task arrives: 'Compare the main approaches to AI alignment taken by three different research groups over the past year. Write a 500-word summary and save it to alignment_summary.txt.' Step 1 — Planning: The LLM core reasons through the task. Thought: 'I need to find three research groups working on AI alignment, find their recent papers, extract the main approach each takes, compare them, write a synthesis, and save the file. I will start by searching for recent alignment papers to identify prominent groups.' Step 2 — Search: Action: search_arxiv('AI alignment approaches 2024', max_results=20). Observation: 20 paper abstracts returned, including work from Anthropic's alignment team, DeepMind's safety team, and the Center for Human-Compatible AI (CHAI). Step 3 — Retrieval: The agent identifies two key papers from each group and calls fetch_paper for each. Three of the six papers are within the context window; the other three trigger summarize_chunk calls to compress them. Step 4 — Memory: The six paper summaries are written to the vector store. The agent formulates a synthesis query and retrieves the most relevant chunks. Step 5 — Synthesis: With the retrieved content in context, the LLM core drafts the 500-word comparison. Thought: 'I have material from all three groups. Anthropic focuses on constitutional AI and RLHF refinements; DeepMind emphasizes reward modeling under distributional shift; CHAI focuses on value learning and cooperative inverse reinforcement learning.' Step 6 — Write: Action: write_file('alignment_summary.txt', [drafted content]). The orchestration loop detects this is a write_file call and pauses for human confirmation. The user approves. The file is written. Step 7 — Terminate: The model produces a final message confirming the file has been saved. The loop ends after 11 steps, well within the 30-step budget.
Match each ResearchBot component to the specific role it played in the worked example above.
Terms
Definitions
Drag terms onto their definitions, or click a term then click a definition to match.
Failure Modes and Mitigations
Even well-designed agents encounter failures. Three failure modes are worth examining in the ResearchBot context. First, retrieval gaps: if arXiv returns no recent papers from a prominent group (perhaps because their work was published in a venue not indexed by arXiv), the agent's synthesis will be skewed. A robust design would add a fallback tool — search_semantic_scholar or search_web — and the planner should detect under-coverage and trigger the fallback. Second, context overflow in synthesis: if the six summaries plus conversation history approach the context limit, the synthesis step will either be truncated or the model will drop earlier context. The solution is aggressive chunking and retrieval from the vector store rather than placing all six summaries directly in context. Third, hallucinated citations: LLMs sometimes generate plausible-sounding but non-existent paper titles or authors. A verification step — running fetch_paper on each cited work to confirm it exists — would catch this, though it adds steps and cost.
Every failure mode in the worked example traces to a specific architectural decision or gap: no fallback retrieval tool, no context overflow detection, no citation verification step. This is the value of the systems view. Failures are not random — they are predictable consequences of architectural choices, which means they can be anticipated, designed around, and monitored.
In the ResearchBot example, the summarize_chunk tool calls the LLM recursively. Which architectural problem does this tool specifically address?
The ResearchBot's orchestration loop pauses for human confirmation before write_file. If this checkpoint were removed, which category of failure would become possible that was previously prevented?
Architect a Variant Agent
- ResearchBot is designed for academic literature. You will redesign it for a different domain.
- Choose one of these domains: (a) competitive market intelligence — monitor competitor product announcements and summarize weekly, (b) legal research — find and summarize case law relevant to a given legal question, (c) software dependency auditing — check a codebase's dependencies for known security vulnerabilities.
- Step 1: Specify the LLM core (any current model).
- Step 2: List the tools your variant agent needs (3-5 tools, with name and one-sentence description each).
- Step 3: Describe the memory architecture (what gets stored, in what type of memory, and when it is retrieved).
- Step 4: Describe any human-in-the-loop checkpoints and justify why they are or are not needed.
- Step 5: Anticipate two failure modes specific to your domain and describe how your architecture addresses each.
- Goal: apply the full five-component framework to a novel domain from scratch.