Capability Assessment

AI capability claims arrive constantly. A company announces that its new model 'surpasses human experts' in a domain. A research paper claims a system has 'achieved superhuman performance' on a benchmark. A news headline declares that AI can now 'do X' — where X is something previously thought to require human intelligence. These claims range from carefully evidenced and honestly scoped to misleading, premature, or simply wrong. Being able to assess them rigorously — to distinguish genuine capability from hype, to understand what the evidence actually shows, and to identify the gap between benchmark performance and real-world reliability — is one of the most practically valuable skills you can develop as an AI-literate person.

A Framework for Assessing Capability Claims

When you encounter a claim that an AI system can do something, apply this five-part framework. One: What exactly is the claim? Strip it of vague language. 'AI surpasses doctors' becomes: which AI system, on which specific task, with which patient population, compared to which doctors, under which conditions? Precision exposes what the claim is actually asserting. Two: What is the evidence? Was this demonstrated on a benchmark, a real-world deployment, a controlled experiment, or an anecdote? Benchmark evidence is more standardized but subject to contamination and saturation. Real-world deployment evidence is more ecologically valid but harder to interpret. Anecdotal evidence (cherry-picked impressive outputs) is the weakest form. Three: What is the evaluation's scope and setting? Does the benchmark match the real task? A model that reads radiology images accurately in a controlled study may perform differently on a hospital's actual scanner images, with their real noise, compression artifacts, and unusual presentations. The distributional match between evaluation and deployment matters enormously. Four: What are the failure modes? Even systems that perform well on average fail on specific inputs. What kinds of inputs does the system struggle with? Are those inputs common in real deployment? A system that fails 5% of the time on average but fails 40% of the time on rare high-stakes cases is not safe for high-stakes deployment. Five: What is the reliability gap? Capability (can it ever do this?) and reliability (does it always do this correctly and safely?) are different. A system that generates a correct answer 80% of the time is capable but not reliable enough for applications where errors are costly.

Common Patterns of Capability Misrepresentation

Understanding how capability claims go wrong helps you spot problems faster. Several patterns recur. Benchmark mismatch: the cited benchmark does not measure what the claim says it measures. A model that achieves high accuracy on a reading comprehension benchmark written by NLP researchers may struggle on reading comprehension tasks as medical professionals or lawyers actually encounter them. Unfair baseline comparison: 'better than human' claims often compare to a narrow or unrepresentative set of humans — novice doctors, time-pressured nurses, students. The same model may underperform specialists with adequate time, access to context, and the ability to ask follow-up questions. Capability without context: demonstrating that a system can generate a correct answer in isolation does not mean it will generate correct answers when integrated into a real workflow with real users, real noise, and real edge cases. The system exists in a context, and context changes performance. Equivocation on 'human-level': 'human-level' means many different things in different papers. Sometimes it means 'within the range of individual human variability,' which is a low bar. Sometimes it means 'matching the best human expert under controlled conditions,' which is a high bar. The phrase itself is almost meaningless without specification.