Neural Nets Aren't Magic

Neural networks have beaten world champions at chess and Go, diagnosed diseases from medical images, and learned to write convincingly in dozens of languages. It is easy to walk away from those headlines thinking that these systems are mysterious, perhaps even intelligent in a deep sense. They are not. They are math — extraordinarily effective math, but math nonetheless. Being clear-eyed about their limitations is not pessimism. It is the foundation of using AI responsibly and building systems that do not fail in dangerous ways. This lesson is about those limitations, and they are real.

They Need Enormous Amounts of Data

A human child can learn to recognize a dog from a handful of examples. A neural network typically needs thousands or millions of labeled examples to reach comparable accuracy — and that is for a narrow, well-defined task like 'is this a dog?' For rarer categories, performance drops sharply. A skin cancer classifier trained on mostly light-skin training photos will perform worse on dark-skin images — not because the network is biased in intention, but because its weights encoded what it actually saw, and if that data was not representative, neither are the weights. This is the data dependency problem: a neural network can only be as good as the data it was trained on, and real-world data is almost always incomplete, skewed, or imbalanced in some way. Collecting, cleaning, and labeling training data is often the most expensive and time-consuming part of building an AI system. A network trained on data from one hospital may fail at a different hospital where scanners are calibrated differently. A translation model trained on formal writing may struggle with slang. This brittleness to data distribution shifts is a known and active research problem.

Neural networks also fail in ways that are surprising and revealing. Consider adversarial examples: carefully crafted inputs that fool a network while looking completely normal to humans. Researchers have shown that adding invisible noise to an image — changes too small for a human eye to notice — can cause a confident image classifier to misidentify a panda as a gibbon with 99% confidence. These attacks exploit the fact that networks learn statistical patterns in pixel space that do not always match human perception. Similarly, networks can be confidently wrong. A network might classify a photo with 98% confidence as 'boat' when the image is pure static — because static does not look like any class the network knows, and the softmax output will still sum to 1, so some class gets a high number even if the input is meaningless. This is called out-of-distribution failure: the network was not designed to say 'I don't know,' so it does not.

They Cannot Explain Their Reasoning

When a neural network classifies your X-ray as showing a tumor, it cannot tell you why. It cannot point to a specific feature and say 'because this region has these characteristics and they correlate with malignancy in ways I can articulate.' It can only produce a confidence score. This lack of interpretability is a genuine problem in high-stakes applications. Researchers work in the field of explainable AI (XAI) to develop tools that reveal which parts of an input most influenced a network's output — for example, highlighting the pixels in a medical image that drove a classification decision. These tools help, but they are approximations. The weights of a large neural network are fundamentally not a human-readable explanation of anything. In fields like medicine, criminal justice, loan approvals, and autonomous vehicles, the inability to explain decisions is not just intellectually unsatisfying — it is a safety and fairness issue. If you cannot understand why a system made a decision, you cannot reliably predict when it will fail.