Tokens and Tokenization

You have just learned that a language model predicts the next token given preceding tokens. But what exactly is a token? It is not simply a word. Modern language models break text into subword units — pieces that are sometimes full words, sometimes word fragments, and sometimes single characters. This design choice, called tokenization, is not arbitrary. It solves a real engineering problem, and understanding it will help you reason clearly about both the capabilities and the odd failures of LLMs.

The Problem Tokenization Solves

Imagine you wanted to build a language model that worked at the word level: each word in the vocabulary is one unit. English has hundreds of thousands of words, and any real corpus also contains names, technical terms, misspellings, URLs, and code. A purely word-level model faces a choice: include only the most common words (leaving most text unrepresentable) or build an enormous vocabulary (making the model slow and memory-hungry). Worse, word-level models cannot handle words they have never seen — a new medication name or a newly coined slang term simply does not exist in the vocabulary. At the other extreme, a character-level model that treats each letter as a token can represent any text, but it must predict thousands of characters to generate one sentence, and each character carries very little information about meaning. Subword tokenization is the compromise: split text into fragments that are common enough to be reused efficiently, but rare enough that unusual words can be built from familiar pieces. The word 'unhelpfulness,' for example, might be tokenized as 'un,' 'help,' 'ful,' 'ness' — four pieces, each of which appears in thousands of other words.

Walk through a concrete example using Byte-Pair Encoding (BPE), the algorithm used by GPT-family models. BPE starts with a vocabulary of individual characters (or bytes). It then repeatedly finds the most frequent pair of adjacent symbols in the training corpus and merges them into a new symbol. After enough merges — typically thousands — the vocabulary contains common words as single units, common word pieces, and the original characters as a fallback. Example tokenization (approximate GPT-4 tokenization): 'Hello, world!' -> ['Hello', ',', ' world', '!'] 'tokenization' -> ['token', 'ization'] 'ChatGPT' -> ['Chat', 'G', 'PT'] 'antidisestablishmentarianism' -> ['anti', 'dis', 'est', 'ablish', 'ment', 'arian', 'ism'] Notice: spaces are often attached to the following word as a leading space character (' world' rather than 'world'). Punctuation is usually its own token. Capitalization matters — 'hello' and 'Hello' may be different tokens. Each token is then mapped to an integer ID. 'Hello' might be token 9906; ',' might be token 11; ' world' might be token 1917. The model never processes the raw characters — it processes these integer IDs, and its first layer converts them into vectors (numerical lists) before any computation begins.

Why Tokenization Shapes Model Behavior

Tokenization is not merely an implementation detail — it has visible consequences for how models behave. Arithmetic and spelling: Models often struggle with character-level tasks (counting letters, spelling unusual words) because they do not see characters directly — they see tokens. Asking a GPT-family model 'How many r's are in strawberry?' is tricky because 'strawberry' may be tokenized as 'straw' and 'berry,' hiding the individual letters. Language coverage: Languages with large character sets (Chinese, Japanese, Arabic) or highly inflected morphology (Turkish, Finnish) tend to tokenize into more tokens per word than English. This means the same amount of text takes more tokens, making these languages more expensive to process and sometimes reducing model quality compared to English. Context window: The context window — the maximum amount of text a model can consider at once — is measured in tokens, not words or characters. A model with a 128,000-token context window can consider roughly 90,000-100,000 words of English, but fewer words of a morphologically rich language. Prompt cost: APIs charge per token. Verbose prompts and outputs cost more than concise ones. Understanding tokenization helps you write efficient prompts.