Iterations and Improvement

One round of 'make a prediction, measure error, adjust weights' improves a model a tiny amount. A model trained for a single round would still be nearly useless. The secret ingredient is repetition — running that loop thousands or even millions of times. This lesson is about how those repetitions are organized, what they look like on a chart, and how to tell whether training is actually working.

Epochs and Batches

Training data is usually too large to process all at once, so it is split into smaller chunks called batches. A batch is a small group of training examples (often 32, 64, or 256 at a time). The model processes one batch, computes the loss, runs gradient descent, and updates its weights. Then it processes the next batch. An epoch is one complete pass through the entire training dataset — all batches processed once. After one epoch, every training example has been seen exactly once. Training usually runs for many epochs. A small image-recognition model might train for 20 epochs. A large language model might train for 1–3 epochs over a dataset so vast that even one pass takes weeks. Why batches instead of processing all data at once? Two reasons: (1) the full dataset often does not fit in memory, and (2) smaller batches provide noisier but more frequent weight updates, which can help escape stuck points in the loss landscape.

Here is a worked example to make the numbers concrete. Suppose you have 5,000 training photos labeled cat or dog. You set batch size = 50. That means 5000 ÷ 50 = 100 iterations per epoch. If you train for 30 epochs, total iterations = 3,000. Each iteration: the model sees 50 photos, computes loss, and updates all weights once. After 3,000 updates the model has seen real signal from every photo 30 times.

Reading the Loss Curve

As training runs, practitioners plot the training loss after each epoch. This chart is called the loss curve (or learning curve). A healthy loss curve looks like a downward slope that gradually flattens — the model improves quickly at first, then squeezes out smaller gains. If the curve is flat from the start, the learning rate may be too small. If it bounces wildly up and down, the learning rate may be too large. If it drops steeply and then rises back up, the model has started overfitting (more on that in Lesson 6). Most training frameworks (like TensorFlow or PyTorch) print the loss after each epoch automatically. Reading loss curves is one of the most practical skills a machine learning practitioner has.