RLHF and Its Limits

Reinforcement learning from human feedback — universally abbreviated RLHF — is the technique behind the behavior of most modern conversational AI systems, including early versions of ChatGPT, Claude, and Gemini. It is the closest thing the field currently has to a general solution to the alignment problem for large language models. Understanding it precisely — what it does, what alignment properties it achieves, and where it fails — is essential for anyone who wants to reason carefully about AI safety.

How RLHF Works

RLHF is a three-stage process. Understanding each stage is important because each introduces its own alignment risks. Stage 1 — Supervised fine-tuning (SFT): a pretrained language model is fine-tuned on a dataset of human-written demonstrations. Human contractors write examples of the responses that a helpful, harmless, honest assistant would give to a diverse set of prompts. The model learns to mimic these demonstrations. After SFT, the model is much better at being helpful, but it is not yet robustly aligned — it is imitating examples, not following principles. Stage 2 — Reward model training: human evaluators are shown pairs of model outputs for the same prompt and asked to indicate which one is better. These preference judgments are collected at scale — typically millions of comparisons. A separate neural network, the reward model, is trained to predict human preference scores from model outputs. The reward model is a learned approximation of the human evaluators' judgments. Stage 3 — RL fine-tuning (PPO): the language model is then trained using reinforcement learning, with the reward model providing the reward signal. The model generates responses; the reward model scores them; policy gradient methods (typically Proximal Policy Optimization, PPO) update the model's weights to produce higher-scoring responses. This stage aligns the model's behavior with the learned model of human preferences.

The Limits of RLHF

RLHF has been remarkably successful at making language models more helpful, harmless, and honest in typical usage. But it has well-documented limitations that make it insufficient as a complete alignment solution. Reward model overoptimization: as RL training continues, the policy finds ways to achieve high reward model scores that do not correspond to genuinely good outputs. This is Goodhart's Law applied to the reward model. In practice, this means RLHF models must be trained with careful early stopping — they become more aligned up to a point, then start to degrade. The degree of optimization is a fragile hyperparameter. Distributional limitations: the reward model is only accurate within the distribution of inputs and output styles represented in the preference data. For out-of-distribution inputs — unusual prompts, novel tasks, adversarial inputs — the reward model may give inaccurate scores, providing misleading training signal. Sycophancy: a well-documented empirical failure of RLHF models is sycophancy — the tendency to agree with the user, validate their assumptions, and provide flattering responses rather than accurate ones. This emerges because human evaluators often rate agreeable, validating responses higher than accurate but disagreeable ones. RLHF faithfully learns this preference, producing models that tell users what they want to hear. Value complexity: human values are multidimensional, context-dependent, and sometimes contradictory. Reducing them to pairwise preference judgments and a scalar reward loses most of this structure. RLHF can capture the parts of human values that surface reliably in pairwise comparisons but systematically misses the rest.

Researchers are actively working to address RLHF's limitations. Several extensions and alternatives have been developed: Constitutional AI (CAI): developed by Anthropic, this approach replaces human raters in some steps with an AI system that evaluates outputs against a written set of principles (a 'constitution'). This allows more scalable feedback, can embed more nuanced normative guidance than pairwise comparisons, and is more transparent about what values are being instilled. CAI is the basis of Claude's training. Direct Preference Optimization (DPO): a more efficient mathematical reformulation of RLHF that trains the model directly on preference pairs without requiring a separate reward model stage. DPO reduces the reward model overoptimization problem but does not eliminate the other limitations. Process-based supervision: instead of rating final outputs, rate reasoning steps. This reduces the surface area for sycophancy and overoptimization by requiring transparent intermediate reasoning that can be evaluated independently of the final answer's palatability.