Language-Agnostic Tokenization
SentencePiece
A Practical Tokenization Guide
A focused, independent resource for language-agnostic tokenization. Understand BPE and Unigram models, train reproducible tokenizers, evaluate quality, and compare modern libraries and algorithms.
Concept
Subwords, normalization, vocabulary design, multilingual text, and detokenization.
Math
Tokenizer quality, sequence length, fertility, and vocabulary trade-offs.
Code
Train, evaluate, and integrate SentencePiece and other tokenizer libraries.
Compare
SentencePiece, BPE, WordPiece, Unigram, tiktoken, and modern alternatives.
Papers
Annotated tokenization research, benchmarks, fairness, and emerging methods.
Latest entries
BPE Dropout · Making Subword Segmentation More Robust for Unseen Words
What BPE-dropout does inside a fixed BPE framework — why deterministic segmentation hurts, how SentencePiece samples multiple segmentations on the fly, and how it relates to Unigram subword regularization.
SentencePiece Unigram vs BPE · model_type, Character Coverage and Split Rules
A practical comparison of the SentencePiece training parameters that actually decide segmentation output — model_type, character_coverage, split_by_whitespace, byte_fallback — what each one changes, and how to compare Unigram and BPE fairly on the same corpus.
Margin Notes · Building a Tokenization-Aware Application
Practical guidance for developers building tokenization-aware applications — token counting, prompt budget management, and cost optimisation with tokenizer APIs.
NLP Tokenization Benchmarks · Fertility, BLEU, and Downstream Performance
How tokenizer choices affect downstream NLP task performance — empirical benchmarks comparing BPE, Unigram, WordPiece, and character-level tokenization.
The Future of Tokenization · Beyond BPE and Unigram
Emerging tokenization research — learned tokenization, differentiable tokenizers, vision-inspired approaches, and whether tokenizers will survive the next generation of models.
Tokenization and Model Fairness · Bias in Subword Segmentation
How tokenization choices create bias in language models — the differential treatment of languages, gendered tokenization, and emerging fairness research.