The Transformer Architecture

In June 2017, a team at Google Brain published a paper with a deceptively modest title: Attention Is All You Need. The paper introduced the transformer, an architecture for processing sequences that discarded the recurrent structure that dominated previous models and replaced it with a mechanism called self-attention. Within two years, the transformer had become the dominant architecture not just for language, but for vision, audio, protein structure prediction, and reinforcement learning. Understanding it is essential for anyone who wants to reason about modern AI.

The Problem with Recurrence

Before transformers, sequence models were dominated by recurrent neural networks and their variant, long short-term memory networks. These models processed sequences token by token, maintaining a hidden state that summarized everything seen so far. To understand the word 'bank' in a sentence about finance, the RNN had to thread through every preceding word in order. This created two serious problems. First, limited parallelism: because each step depends on the previous hidden state, you cannot process multiple tokens simultaneously. Training on long sequences was slow. Second, long-range dependencies: relevant information must survive through many intermediate hidden state updates. In practice, RNNs struggled to maintain relevant information across gaps of more than a few dozen tokens, a severe constraint when understanding a paragraph or document. LSTMs improved the second problem with gated memory cells, but not enough for the sentence lengths and document structures researchers wanted to handle. Something fundamentally different was needed. The transformer's answer was self-attention: every token attends directly to every other token in the sequence simultaneously, computing a weighted mixture of all positions. There is no sequential bottleneck. Long-range dependencies are as cheap to learn as local ones. And the entire operation can be parallelized across modern GPU hardware. A transformer is organized into layers called transformer blocks, stacked on top of each other. Each block contains two main components: a multi-head self-attention sublayer and a feedforward sublayer. Residual connections and layer normalization sit around each sublayer to stabilize training. The flow of information: input tokens are converted to dense vector embeddings, positional encodings are added to indicate token order, the embeddings pass through all stacked transformer blocks, and the output embeddings are mapped to the model's output distribution.

Why Transformers Generalize Across Domains

One of the most striking facts about the transformer is its domain-agnosticism. The same core architecture that processes language also processes images, treating 16x16 pixel patches as tokens in Vision Transformers. It processes audio by treating spectrogram frames as tokens, protein sequences by treating each amino acid as a token, and even game states in reinforcement learning. Why does one architecture work across such different domains? Because the transformer makes very few assumptions about the structure of its input. It does not assume spatial locality, as convolutional neural networks do. It does not assume sequential order is the primary source of information, as RNNs do. It simply asks: given a set of tokens with associated positions, which tokens should inform the representation of each other? This abstraction is general enough to accommodate any structured sequence. The trade-off for this generality is computational cost. Self-attention scales quadratically with sequence length: doubling the number of tokens quadruples the computation. This is why context windows were historically limited to a few thousand tokens, and why extending them to hundreds of thousands required algorithmic innovations like flash attention and sparse attention. Parameter count scales with the number of layers and the width of each layer. A small transformer might have 100 million parameters. GPT-3 has 175 billion. Estimates for GPT-4 suggest roughly 1.8 trillion, potentially as a mixture-of-experts model where only a fraction of parameters are active for any given input.