Inference, Context, and Compute

Training a large language model is the most expensive step, but once trained, the model must be run millions or billions of times to serve users. This is called inference: the process of running a trained model on new input to produce output. Understanding inference is essential for understanding the economics, limitations, and design trade-offs of deployed AI systems. Inference is not simply the reverse of training. Training adjusts parameters; inference holds parameters fixed and propagates input through the model to produce an output. But in language models, inference has a specific structure, autoregressive generation, that creates important constraints and opportunities.

Autoregressive Generation

A language model does not generate an entire response in one forward pass. It generates one token at a time, appending each generated token to the context and then running another forward pass to generate the next token. This is called autoregressive generation, and it is an important architectural consequence of next-token prediction pretraining. Here is the process for generating a response to the prompt 'Explain photosynthesis in one sentence.' Step 1: The model runs a forward pass on the full prompt. Its output is a probability distribution over the vocabulary: maybe 'Photosynthesis' has probability 0.45, 'Plants' has probability 0.22, and so on. The model samples or selects the highest-probability token, say 'Photosynthesis.' Step 2: 'Photosynthesis' is appended to the context. The model runs another forward pass on 'Explain photosynthesis in one sentence. Photosynthesis' and produces a new distribution. The next token is selected, perhaps 'is.' Step 3: This repeats until the model generates an end-of-sequence token or reaches a length limit. The key consequence: generating a response of length L tokens requires L sequential forward passes through the entire model. These passes cannot be fully parallelized because each depends on the previous token. This is fundamentally different from training, where many token predictions can be parallelized within a forward pass. Inference latency, the time from query to complete response, scales with response length and model size.

The context window is the total amount of text the model can process in one inference call: the prompt plus all tokens generated so far. Every token in the context window participates in the self-attention computation for each new token being generated. If the context window is 128,000 tokens and the model has generated 60,000 tokens of a response, the next token's generation requires computing attention over all 128,000 tokens currently in context. This is why long contexts are computationally expensive: attention cost scales quadratically with context length. A response generated in a 128,000-token context is not just more expensive than one in a 4,000-token context because of the extra tokens generated. It is also more expensive per token, because each step of generation must attend over a longer context. KV caching is the primary technique for managing this cost. During a forward pass, the model computes intermediate representations called keys (K) and values (V) for each token in the context. These are reused for every subsequent forward pass on the same context. Rather than recomputing K and V from scratch for all previous tokens at every step, the model stores them in a cache and only computes them fresh for the new token being appended. KV caching dramatically reduces the computation per generated token but requires significant memory: a 128,000-token context for a large model may require tens to hundreds of gigabytes of KV cache. Batch size is another inference variable. Rather than serving one user at a time, inference systems typically process requests from multiple users simultaneously (batching). Larger batches improve hardware utilization but increase latency for each individual user. Inference optimization is an entire engineering discipline balancing throughput, latency, cost, and model accuracy.

Test-Time Compute and Reasoning Models

A significant research direction in 2024-2026 has been scaling inference-time compute rather than training-time compute. The traditional paradigm is: train a bigger model (more training compute) to get a better model. The emerging alternative: use more compute at inference time on a given model to get better answers. The most prominent implementation of this idea is chain-of-thought reasoning, where the model is prompted or trained to reason step by step before producing a final answer. More compute at inference time means more tokens generated in the reasoning chain, which on many tasks (mathematics, formal logic, coding) produces substantially better final answers. OpenAI's o1 and o3 models, released in 2024-2025, were explicitly designed around this principle: they generate extended internal reasoning traces before producing their final answer, using considerably more inference compute per query than a standard model, in exchange for significantly better performance on complex reasoning tasks. Anthropic's extended thinking in Claude 3.7 Sonnet follows the same paradigm. This creates a new design variable for deployers: how much inference compute should be spent per query? For simple questions, a short response from a small model is optimal. For complex mathematical proofs or multi-step planning tasks, spending 10x or 100x more inference compute on a larger reasoning model may be worth the cost. The economics of AI deployment are increasingly shaped by this trade-off between per-query compute cost and answer quality.