The Transformer, Conceptually

From the outside, a large language model looks like a black box: text goes in, text comes out. From the inside, it is an architecture called the Transformer, introduced in a 2017 paper titled 'Attention Is All You Need.' The Transformer is the backbone of virtually every major language model built since: GPT, BERT, LLaMA, Claude, Gemini. You do not need to implement one to understand it — but you do need a clear conceptual model, because the architecture directly shapes what these systems can and cannot do.

The Core Problem: Relating Tokens to Each Other

Earlier neural architectures for language — recurrent neural networks (RNNs) — processed tokens one at a time in sequence, passing a hidden state forward like a baton. This meant information from early tokens had to survive many steps to influence predictions far later. In practice, RNNs forgot long-range context. The Transformer's solution was radical: abandon sequential processing entirely. Instead, every token attends to every other token simultaneously. The mechanism that makes this possible is called self-attention. Here is the idea. For each token, the model asks: which other tokens in this sequence are most relevant to understanding what this token means here? It computes a relevance score between every pair of tokens — a number that says how much token i should 'pay attention to' token j when building its representation. These scores are used to produce a weighted combination of all other tokens' information, resulting in a richer representation of each token that has been informed by context from the entire sequence. Concrete example: In the sentence 'The animal didn't cross the street because it was too tired,' what does 'it' refer to? A human reader immediately knows: 'animal.' An attention mechanism can learn to assign a high relevance score between 'it' and 'animal' in this context, allowing the model to resolve the reference correctly even though the words are far apart.

How are relevance scores computed? Each token's representation is projected into three vectors, called Query (Q), Key (K), and Value (V) — names borrowed from database retrieval metaphors. The relevance score between token i and token j is the dot product of token i's Query vector and token j's Key vector, scaled and passed through a softmax function to produce a probability distribution. The output for token i is then a weighted sum of all tokens' Value vectors, where the weights are those probabilities. In practice, this happens in parallel across multiple 'heads' — called multi-head attention. Each head can learn to attend to different kinds of relationships simultaneously: one head might track syntactic subject-verb agreement; another might track semantic similarity; a third might track coreference (which pronoun refers to which noun). The outputs of all heads are concatenated and projected to produce the next layer's input. One important addition: the Transformer adds positional encodings to each token's initial embedding. Because self-attention treats all tokens in parallel (no inherent order), it needs explicit information about each token's position in the sequence. Positional encodings inject this information numerically.

Layers, Feed-Forward Networks, and Scale

A full Transformer is not one attention layer but many — stacked on top of each other. In each layer, every token's representation is updated by attending to all other tokens (the attention step) and then passed through a position-wise feed-forward network — a small fully connected network applied independently to each token. Then the process repeats in the next layer. Why stack layers? Early layers tend to capture low-level patterns — local syntax, common phrases. Deeper layers build more abstract, contextually rich representations. Empirically, stacking more layers (and using larger hidden dimensions) consistently improves model quality, which is part of why modern models have dozens or even hundreds of layers. Scale: GPT-3 has 96 layers and 175 billion parameters. The 'parameters' are the numbers — in the weight matrices of the attention and feed-forward components — that are learned during training. Each forward pass through the model (generating one token) involves billions of floating-point multiplications. This is why LLMs require specialized hardware (GPUs or TPUs) and significant energy.