Pretraining and Self-Supervision

A language model trained from scratch on a narrow task, such as answering customer service questions, would face a brutal problem: it must simultaneously learn grammar, world knowledge, reasoning patterns, and the specific task, all from a small number of task-specific examples. This is asking a model to build a skyscraper without a foundation. Pretraining solves this by separating foundation-building from task-specific specialization. First, train the model on an enormous, general corpus using a self-supervised objective. The model learns rich representations of language and the world. Then, adapt this pretrained model to specific tasks with far smaller labeled datasets. This two-phase approach is one of the central design decisions of the current AI paradigm.

Self-Supervised Learning: Supervision Without Labels

Traditional supervised learning requires labeled examples: someone must annotate each data point with the correct answer. For a small dataset of thousands of examples, this is feasible. For hundreds of billions of tokens from the internet, it is not: the cost would be astronomical and the coverage impossibly narrow. Self-supervised learning sidesteps this by generating the training signal from the data itself. No human annotation is needed. The most successful self-supervised objective for language is next-token prediction, also called causal language modeling. The model receives a sequence of tokens and must predict the next one. Consider the sentence: The neural network learned to recognize patterns in the training data. A language model in training might receive the tokens up to 'the' and must predict 'training.' The loss is computed by comparing the model's probability distribution over the vocabulary to the true next token. The model updates its parameters to make 'training' more probable given that prefix. This repeats for every consecutive subsequence in the training corpus. Across hundreds of billions or trillions of such predictions, the model is forced to develop internal representations that capture grammar, syntax, semantics, factual knowledge, logical structure, and stylistic variation. Predicting the next word accurately requires understanding all of these things.

What data is used for pretraining? Modern frontier language models train on a curated mixture of sources. Common Crawl, a public archive of web pages scraped at regular intervals, contributes the majority of tokens by volume. Books provide dense, high-quality prose with coherent long-form reasoning. Wikipedia contributes factual encyclopedic content with structured citations. Code repositories such as GitHub contribute a domain that rewards precise logical structure. Interestingly, training on code improves reasoning ability on tasks unrelated to programming. Academic papers and filtered web content round out the corpus. Data curation matters enormously. Raw web data is noisy: spam, duplicate content, and low-quality text are common. The major labs invest significant engineering in filtering, deduplication, and quality scoring their training corpora. The exact composition, how much code, how much multilingual content, how much scientific text, significantly affects what abilities emerge and where the model is weak. The scale of pretraining is staggering. GPT-3 was trained on roughly 300 billion tokens. Models like GPT-4, Claude, and Gemini are estimated to have trained on one to ten trillion tokens or more. A single training run at frontier scale can cost tens to hundreds of millions of dollars in compute.

What Pretraining Produces: Representations and World Knowledge

At the end of pretraining, a model has learned billions of parameters whose values encode patterns extracted from the training corpus. What exactly is stored in those parameters? The model has learned token embeddings: dense vector representations where semantically related words cluster together in high-dimensional space. The words 'king' and 'queen' have nearby embedding vectors. The word 'bank' as a financial institution and 'bank' as a river shore end up as context-dependent mixtures of distinct clusters. The model has learned attention patterns: which tokens tend to be relevant to which other tokens in various syntactic and semantic contexts. A verb attends strongly to its subject and object. A pronoun attends to its antecedent. Critically, the model also encodes factual knowledge in its parameters. Probing experiments, where researchers analyze model representations without any fine-tuning, show that large pretrained models encode factual associations like 'Paris is the capital of France,' causal relationships like 'fire causes heat,' and commonsense associations like 'knives are sharp.' This knowledge is implicit, stored in weights as patterns that make certain completions more probable, rather than explicit as a database of facts. This is what enables transfer learning: the pretrained model's representations are useful starting points for fine-tuning on specific tasks, because the model already knows a great deal about language and the world. A fine-tuned model only needs to learn the specific format and focus of the new task, not build from scratch.