Tokenization
What you’ll learn. Why LLMs operate on subword tokens rather than characters or words; how Byte-Pair Encoding, WordPiece and the Unigram model are trained and applied; why byte-level BPE became the default; how vocabulary size trades off against sequence length, embedding cost and multilingual fairness; the strange failure modes tokenization causes (arithmetic, spelling, glitch tokens, whitespace); and how to inspect tokenizers with tiktoken and Hugging Face tokenizers.
Why tokenize at all
Section titled “Why tokenize at all”A model needs a finite vocabulary of discrete symbols mapped to embedding rows. Three extremes:
- Words: short sequences, but the vocabulary is open-ended (names, typos, morphology, code identifiers). Any unseen word becomes
<unk>, losing information. - Characters / bytes: no unknown symbols and a tiny vocabulary, but sequences become 4 to 5 times longer. Attention cost grows quadratically in length, and the model must spend capacity composing characters into meaningful units.
- Subwords: frequent strings get their own token, rare ones decompose into smaller pieces. This is the compromise used by essentially every production LLM.
Tokenization is a fixed preprocessing step trained once on a corpus, separate from the neural network. The model never sees characters; it sees IDs. Every quirk of the tokenizer becomes a quirk of the model.
The pipeline
Section titled “The pipeline”raw string -> normalization (optional: Unicode NFC/NFKC, lowercasing in some models) -> pre-tokenization (split into "words": whitespace, punctuation, digits, regex) -> model (BPE merges / WordPiece / Unigram) -> subword pieces -> post-processing (add special tokens like <bos>, chat markers) -> integer IDsThe pre-tokenizer matters a lot: it decides that merges cannot cross certain boundaries (for example, a token won’t span a space and a following punctuation mark, or won’t merge digits across a number boundary). GPT-style tokenizers use a regular expression for this; SentencePiece instead treats the raw stream with spaces replaced by a visible marker (U+2581).
Byte-Pair Encoding (BPE)
Section titled “Byte-Pair Encoding (BPE)”BPE was a data-compression algorithm (Gage, 1994) adapted for NMT by Sennrich et al. (2016).
Training:
- Start with a base vocabulary of individual symbols (characters, or in byte-level BPE the 256 byte values).
- Count all adjacent symbol pairs in the corpus (weighted by word frequency).
- Merge the most frequent pair into a new symbol; record the merge.
- Repeat until the vocabulary reaches the target size .
Encoding a new word: split into base symbols and apply the learned merges in the order they were learned (priority = merge rank), until none apply.
Toy implementation
Section titled “Toy implementation”from collections import Counter
def get_stats(words): pairs = Counter() for symbols, freq in words.items(): for a, b in zip(symbols, symbols[1:]): pairs[(a, b)] += freq return pairs
def merge(words, pair): out = {} for symbols, freq in words.items(): new, i = [], 0 while i < len(symbols): if i < len(symbols) - 1 and (symbols[i], symbols[i + 1]) == pair: new.append(symbols[i] + symbols[i + 1]); i += 2 else: new.append(symbols[i]); i += 1 out[tuple(new)] = freq return out
corpus = "low low low low low lower lower newest newest newest widest widest".split()words = {tuple(w): c for w, c in Counter(corpus).items()}merges = []for _ in range(10): stats = get_stats(words) if not stats: break best = max(stats, key=stats.get) words = merge(words, best) merges.append(best)print(merges) # [('w','e'), ('l','o'), ...] - frequent substrings emerge firstReal implementations are the same idea with far more efficient data structures (priority queues over pair counts, incremental updates).
Byte-level BPE
Section titled “Byte-level BPE”Character-level BPE still has an unknown-character problem for the enormous Unicode space. Byte-level BPE (GPT-2 and later) runs BPE over the UTF-8 bytes of the text. The base vocabulary is exactly 256 symbols, so any string, in any script, emoji or binary garbage, is representable. A character such as an accented letter or a CJK ideograph is 2 to 4 bytes, and common ones get merged into single tokens during training. This guarantees zero <unk>. The cost: rare scripts split into many byte tokens.
WordPiece
Section titled “WordPiece”Used by BERT. Training resembles BPE but selects merges by a likelihood criterion rather than raw frequency: pick the pair maximizing
the pair whose joint occurrence most exceeds chance (like pointwise mutual information). Encoding is greedy longest-match-first against the vocabulary; continuation pieces are marked with ## (“playing” becomes play, ##ing). If no piece matches, it emits [UNK].
Unigram language model (SentencePiece)
Section titled “Unigram language model (SentencePiece)”Kudo (2018) takes a different, probabilistic approach:
- Start with a large over-complete candidate vocabulary (all frequent substrings).
- Model the text as a bag of independent pieces: .
- Fit piece probabilities with EM; for each piece compute how much total corpus likelihood would drop if it were removed; prune the lowest-impact fraction (e.g. 20%). Repeat until reaching the target size.
At inference, the best segmentation of a string is found by Viterbi over the lattice of possible splits, . Because the model is probabilistic, it also supports subword regularization: sampling alternative segmentations during training as data augmentation, which helps robustness, especially in low-resource and multilingual settings. T5, Llama 1/2 and many multilingual models use SentencePiece (Llama 3 moved to a tiktoken-style BPE).
Special tokens and chat templates
Section titled “Special tokens and chat templates”Beyond text pieces, vocabularies contain special tokens: beginning/end of sequence, padding, and, for chat models, role markers (such as start-of-turn and end-of-turn). The chat template is the exact serialization of a conversation into tokens that the model saw during post-training. Using the wrong template is a common cause of silently degraded quality when running open-weight models. Always use the model’s own template (tokenizer.apply_chat_template in Hugging Face) rather than hand-concatenating strings.
Vocabulary size trade-offs
Section titled “Vocabulary size trade-offs”Let be the vocabulary size and the model width.
- Embedding and output parameters: input embeddings plus an unembedding matrix cost each (they can be tied). For and that is 524M parameters per matrix, a large fraction of a small model but minor for a 70B one.
- Softmax cost: the final logits layer is per token, and the logits tensor in training is , a real memory hog that fused cross-entropy kernels address.
- Sequence length: a bigger vocabulary compresses text into fewer tokens (typical English: roughly 4 characters or 0.75 words per token for large modern vocabularies), reducing attention cost, KV-cache size and API price per document, and letting the same context window hold more text.
- Token under-training: rare tokens get few gradient updates; embeddings of tokens that almost never appear are poorly trained, and tokens that exist in the vocabulary but nearly vanished from the model’s training data become “glitch tokens” (below).
- Multilingual fairness: vocabularies trained mostly on English fragment other languages. The same sentence may cost 2 to 5 times more tokens in Thai, Burmese or some Indic languages than in English, which means higher price, less effective context and often weaker quality. Modern large-vocabulary models (100K to 250K) mitigate but don’t eliminate this.
Trend: vocabulary sizes have grown from 32K (original Llama, BERT about 30K) to 100K-250K for recent multilingual models. There is no universal optimum; it depends on language mix and model size, and larger models can afford larger vocabularies.
Quirks and failure modes
Section titled “Quirks and failure modes”Tokenization explains many “LLMs are dumb at X” results:
- Spelling and character counting. The word “strawberry” may be 2 or 3 tokens; the model doesn’t see letters. Counting letters requires memorized facts about token spellings or step-by-step reasoning.
- Arithmetic. Numbers are split arbitrarily by greedy merges (“12345” might be
123+45, and “12346” might be1234+6), so digit alignment varies. Many recent tokenizers split digits individually or into fixed 1-3 digit groups, which noticeably helps arithmetic. Right-to-left grouping (as in thousands separators) is another remedy. - Leading whitespace. In GPT-style BPE, ” hello” (with a space) and “hello” are different tokens. Prompts ending in a trailing space put the model off-distribution because it expects the space to be attached to the next word. This matters for few-shot templates and for completion prompts.
- Case and Unicode variants. “Hello”, “hello”, “HELLO” can be unrelated token sequences. Visually identical strings with different Unicode normalization forms tokenize differently.
- Glitch / under-trained tokens. Tokens that exist in the vocabulary (e.g. from a scraped username or forum artifact) but almost never appear in model training can trigger bizarre behavior when prompted. Evidence for this was published for several early GPT models.
- Code and structured data. Indentation, repeated whitespace and long identifiers need good whitespace-run tokens. Tokenizers trained with code in the mix handle runs of spaces as single tokens; otherwise Python files explode in length.
- Token healing / boundary issues. If you split a prompt at an arbitrary character position and continue generation, you may have forced a tokenization the model would never choose. “Token healing” backs up one token and constrains the continuation to start with it.
- Different tokenizers per model. Token counts, and therefore cost and context limits, aren’t portable across providers. Always count with the target model’s tokenizer.
Practice with tiktoken
Section titled “Practice with tiktoken”import tiktoken
enc = tiktoken.get_encoding("cl100k_base") # a GPT-4-era BPE; others exist (o200k_base)for s in ["hello", " hello", "Hello", "strawberry", "12345", " indented"]: ids = enc.encode(s) print(repr(s), ids, [enc.decode([i]) for i in ids])
# token counting for budgetingtext = "Tokenization is where many LLM bugs begin."print(len(enc.encode(text)), "tokens for", len(text), "characters")
# safe handling of untrusted text containing special-token stringsprint(enc.encode("<|endoftext|>", disallowed_special=())) # treated as plain textEncoding names change between model generations; check the library docs rather than assuming a name.
Training your own with Hugging Face tokenizers
Section titled “Training your own with Hugging Face tokenizers”from tokenizers import Tokenizer, models, trainers, pre_tokenizers, decoders
tok = Tokenizer(models.BPE(unk_token=None))tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)tok.decoder = decoders.ByteLevel()
trainer = trainers.BpeTrainer( vocab_size=32_000, special_tokens=["<|bos|>", "<|eos|>", "<|pad|>"], initial_alphabet=pre_tokenizers.ByteLevel.alphabet(), # all 256 bytes)tok.train(files=["corpus.txt"], trainer=trainer)tok.save("my_tokenizer.json")
enc = tok.encode("Tokenizers are fast.")print(enc.tokens, enc.ids)print(tok.decode(enc.ids)) # round-trips exactlyA good habit when training: measure fertility (tokens per word, or bytes per token) on held-out text in each language/domain you care about, and check round-trip fidelity (decode(encode(x)) == x).
Tokenizer-free and alternative directions
Section titled “Tokenizer-free and alternative directions”Because tokenization is a lossy, hand-designed preprocessing step, researchers keep trying to remove it. Byte-level models (ByT5, MegaByte-style hierarchical models, and newer approaches that dynamically group bytes into patches inside the network) trade longer sequences for robustness to spelling, scripts and noise. As of 2026 subword tokenizers remain the production default because of efficiency, but this is an active research area and worth watching.
Also relevant: extending a vocabulary when adapting a model to a new language. Adding tokens requires initializing new embedding rows (commonly the mean of the sub-token embeddings that previously spelled the new token) and then continued pretraining so they are learned; otherwise they are noise. See Fine-Tuning in Practice.
Key takeaways
Section titled “Key takeaways”- Subword tokenization balances open vocabulary against sequence length; byte-level BPE guarantees any input is encodable.
- BPE merges by frequency, WordPiece by a likelihood-like score with greedy longest-match encoding, Unigram prunes a probabilistic vocabulary and supports sampling segmentations.
- Larger vocabularies shorten sequences and cut KV-cache and API cost but enlarge embedding/softmax layers and risk under-trained tokens.
- Many LLM weaknesses (spelling, arithmetic, trailing spaces, non-English cost) trace to tokenization.
- Token counts are tokenizer-specific; always use the target model’s tokenizer and chat template, and don’t let untrusted text parse into special tokens.
Exercises
Section titled “Exercises”- Conceptual. Why does byte-level BPE never need an
<unk>token, and what does it cost for scripts like Thai? - Hands-on. Run the toy BPE above on a larger text file with 500 merges. Print the 30 longest learned tokens. What do they tell you about the corpus?
- Hands-on. Using
tiktoken, compute tokens per character for English, Spanish, Hindi and Python source code of comparable meaning/size. Explain the ordering. - Conceptual. A model fails at reversing the string “lollipop”. Give a tokenization-based explanation and two mitigations.
- Hands-on. Train two byte-level BPE tokenizers with vocabulary 8K and 64K on the same corpus. Plot compression (bytes per token) against vocabulary size; where do returns diminish?
Solutions (sketch)
- Every string is a sequence of bytes and all 256 byte values are in the base vocabulary. Non-Latin scripts take 3 bytes per character in UTF-8; unless merges were learned for them, each character costs up to 3 tokens.
- The model sees a few multi-character tokens, not letters, so reversal requires knowing each token’s spelling and recomposing. Mitigations: ask it to spell out the letters first (chain of thought), or give it a code tool.
Further reading
Section titled “Further reading”- Sennrich, Haddow, Birch, “Neural Machine Translation of Rare Words with Subword Units” (2016).
- Kudo, “Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates” (2018); Kudo and Richardson, “SentencePiece” (2018).
- Schuster and Nakajima, “Japanese and Korean Voice Search” (2012), origin of WordPiece; Wu et al., “Google’s Neural Machine Translation System” (2016).
- Radford et al., “Language Models are Unsupervised Multitask Learners” (GPT-2, 2019), for byte-level BPE.
- Xue et al., “ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models” (2021).
- Andrej Karpathy, “Let’s build the GPT Tokenizer” (video lecture), and the Hugging Face
tokenizersandtiktokendocumentation.