When Learning Goes Wrong

Machine learning is not a magic button that transforms data into good predictions automatically. There are many ways the process can fail — and understanding those failure modes is just as important as knowing how training works when everything goes right. In this lesson, you will study the most common ways learning goes wrong, why they happen, and how to recognize the warning signs.

Overfitting: Too Much of a Good Thing

Overfitting happens when a model learns the training data too well — memorizing specific examples, including their noise and quirks, instead of learning the general pattern. An overfit model performs brilliantly on training data and poorly on new data. You already saw the test-set gap that reveals it in Lesson 7. Imagine studying for a history exam by memorizing every exact question from last year's practice test, word for word. If the teacher uses a few different questions this year, you are lost. You memorized the surface, not the underlying history. An overfit machine learning model has done the same thing. It 'memorized' the training examples — including random noise in those specific examples — rather than learning the true underlying pattern. Signs of overfitting: very high training accuracy, much lower test accuracy; the model struggles with examples slightly different from the training set.

Underfitting is the opposite failure. An underfit model is too simple to capture the real pattern in the data — like trying to summarize an intricate symphony with a single note. It performs poorly on both the training data and new data, because it has not learned much at all. Think of drawing a straight line through data points that actually follow a curved pattern. The straight line is the underfit model: it is wrong everywhere, not because it memorized the training data, but because it lacked the complexity to represent the true relationship. Common causes of underfitting: a model architecture that is too simple (not enough layers, neurons, or parameters), training for too few iterations, or using features that do not capture enough information about the problem. The goal is the middle ground: a model complex enough to capture the real pattern, but not so complex that it memorizes noise. Finding this balance is one of the core challenges of machine learning practice.

Beyond the model's complexity, the data itself can cause learning to go wrong in serious ways. Too little data: the training loop needs enough examples to see the full range of variation in a problem. A model trained to recognize all handwritten digits from 50 total examples will struggle — it simply has not seen enough variety to generalize. More data usually helps, up to a point. Biased or unrepresentative data: if the training data does not reflect the real-world distribution the model will encounter, the model will fail in systematic and potentially harmful ways. A hiring algorithm trained on a company's historical hire data will encode whatever biases shaped those past decisions. A medical model trained entirely on data from adult men may perform poorly on women and children. The model reflects its data, not the world. Misleading features: as you learned in Lesson 3, features that are correlated with the outcome by accident — not by any real causal link — can cause the model to learn spurious patterns that do not hold up on new data. The model finds the pattern because it is there in the training set; but the pattern is a mirage.

How Practitioners Fight These Problems

Overfitting is fought with a toolkit called regularization — techniques that penalize the model for becoming too complex, forcing it to prefer simpler, more general solutions. Another powerful remedy is more data: a model with ten million training examples is harder to overfit than one with a thousand, because there is simply too much variety to memorize. A technique called dropout (used in neural networks) randomly ignores some of the model's neurons during training, forcing different parts of the model to learn the pattern independently — like making sure no single person on a team becomes a single point of failure. Underfitting is usually fixed by using a more complex model, training for more iterations, or engineering better features. Biased data is the hardest problem of all. It requires careful thought about where the data came from, who or what it represents, and whose perspectives or experiences might be missing. There is no algorithm that fixes biased data — it requires human judgment, diverse teams, and honest audits of model behavior across different groups.