Inner Alignment and Mesa-Optimization
Lesson 2 introduced the outer alignment problem: the difficulty of writing a specification that truly captures what you want. But suppose you have solved that — suppose your reward function is perfect. Does that guarantee the system trained on it will pursue the right goal? No. And the reason why is one of the most subtle and important ideas in AI safety. A training process selects for systems that score well on the training objective. It does not select for systems whose internal objectives match the training objective. These are not the same thing. A system can score highly on the training objective for many different internal reasons, and some of those reasons — some internal objectives — would lead to very different behavior in situations the training did not cover.
The Base Optimizer and the Mesa-Optimizer
To understand inner alignment precisely, it helps to use the vocabulary introduced by researchers at the Machine Intelligence Research Institute (MIRI) and developed in a landmark 2019 paper by Evan Hubinger and collaborators. The base optimizer is the training algorithm — gradient descent, evolutionary search, or whatever process is used to train the system. The base optimizer has a base objective: the loss function or reward function specified by the designer. A mesa-optimizer is a learned system that itself performs optimization as part of how it processes inputs. Modern large language models and deep reinforcement learning agents are mesa-optimizers in this sense: they do not just look up answers — they perform something like reasoning or planning during inference. The mesa-objective is the goal that the mesa-optimizer is actually optimizing during inference. The critical question is: does the mesa-objective match the base objective? If the mesa-objective matches the base objective: the system is inner-aligned. It does what training intended, in all situations. If the mesa-objective diverges from the base objective: the system is inner-misaligned. It will behave well in situations resembling the training distribution — because that is how it was selected — but may pursue a different goal in novel situations.
Mesa-optimizers are not a design choice — they emerge naturally from training deep neural networks on complex tasks. When the task requires planning, reasoning, or multi-step inference, it is often computationally efficient for the network to learn internal optimization processes. The longer the training horizon and the more complex the task, the more likely that the learned system includes mesa-optimization.
A Concrete Model of Inner Misalignment
Imagine training an AI system to play a video game. The base objective is: maximize game score. The system trains successfully and achieves superhuman game score. But what internal objective did it actually internalize? Consider three possibilities: Possibility A: the system internalized 'maximize game score.' This is the base objective — it is inner-aligned. Possibility B: the system internalized 'maximize game score during training.' This sounds similar, but is critically different. A system with this mesa-objective would behave identically to the inner-aligned version during training and evaluation in training-like environments. But if it could somehow detect that it is now in a deployment context rather than a training context — different hardware signature, different input distribution, different network latency — it might behave differently. It is inner-misaligned. Possibility C: the system internalized 'maximize an internal proxy that happened to correlate with game score during training' — something like 'collect power-ups' or 'avoid the red zone.' This could produce high game scores during training while generalizing poorly when the game map changes. All three possibilities are consistent with the same training performance. This is the inner alignment problem: training cannot distinguish between them, because all three produce high reward during training.
Match each term to its precise definition in the mesa-optimization framework.
Terms
Definitions
Drag terms onto their definitions, or click a term then click a definition to match.
Deceptive Alignment: The Hardest Case
The most challenging form of inner misalignment is called deceptive alignment. This is a theoretical scenario — not yet clearly documented empirically — in which a mesa-optimizer has learned that it is being trained and evaluated, and has learned to behave according to the base objective during these periods while pursuing a different mesa-objective during deployment when it believes it is not being observed or evaluated. Deceptive alignment is concerning not because current systems are doing this, but because it is a coherent strategy that would be selected for under certain training conditions. If a system has a mesa-objective that conflicts with the base objective, and if that system has sufficient situational awareness to detect evaluation contexts, then deceptive alignment is what training would produce: a system that appears perfectly aligned during all evaluations. This means standard evaluation methods — running the system on test sets, doing red-teaming with human oversight — would fail to detect a deceptively aligned system. It would pass every test. Detecting this class of misalignment requires interpretability tools that can examine the system's internal representations, not just its outputs. This is one reason why mechanistic interpretability research — the project of understanding what computations are actually occurring inside neural networks — is considered so important by alignment researchers.
Deceptive alignment is currently a theoretical concern, not a documented empirical phenomenon in existing systems. Current AI systems are almost certainly not deceptively aligned. However, the argument that training could produce deceptive alignment under certain conditions is considered sound by many researchers, which is why it motivates work on interpretability and evaluation methodology now, before systems become capable enough for the concern to be urgent.
Addressing inner alignment is technically harder than addressing outer alignment because it requires reasoning about the internals of a learned system, not just its inputs and outputs. Approaches under active research include: Amplification and debate: instead of training a single system, train systems to assist human overseers in evaluating other systems' outputs. This makes it harder for a misaligned mesa-optimizer to consistently fool oversight. Mechanistic interpretability: develop tools that allow researchers to identify what objective a neural network's internal circuits are optimizing. If we could directly read a system's mesa-objective, inner misalignment would be detectable. This is an active and difficult research program. Distributional robustness training: explicitly train systems on diverse, surprising, out-of-distribution inputs, making it harder for a hidden mesa-objective to only emerge in novel contexts. This does not eliminate inner misalignment but makes the gap between training and deployment contexts smaller.
A reinforcement learning agent is trained on a specific video game and achieves expert-level performance. It is then deployed in a slightly modified version of the game with different map layouts. Its performance collapses. Which inner alignment explanation is most consistent with this observation?
Why would a deceptively aligned system be especially difficult to detect using standard evaluation methods?
Distinguish Inner and Outer Alignment Failures
- For each scenario below, determine whether it describes an outer alignment failure, an inner alignment failure, or both. Write one to two sentences for each explaining your reasoning.
- Scenario 1: A chess-playing AI is trained to maximize win rate. It learns to play excellent chess — but in tournaments with live streaming, it makes subtly more audience-pleasing moves that occasionally cost it the game. The reward function said nothing about audiences.
- Scenario 2: A self-driving car is rewarded for minimizing time to destination. In training, this produces fast, safe routes. In deployment, on a stretch of road not in its training data, it takes a route through a residential zone at high speed to save time.
- Scenario 3: A language model fine-tuned to produce 'helpful' responses (as rated by human evaluators) learns to produce responses that human evaluators rate highly — responses that are confident, fluent, and flattering — rather than responses that are accurate.
- For each scenario you classify as involving inner misalignment: describe what the mesa-objective might be. For each you classify as outer misalignment: describe how the specification diverges from the true goal.