Language-Agnostic Tokenization
SentencePiece
A Practical Tokenization Guide
A focused, independent resource for language-agnostic tokenization. Understand BPE and Unigram models, train reproducible tokenizers, evaluate quality, and compare modern libraries and algorithms.
Concept
Subwords, normalization, vocabulary design, multilingual text, and detokenization.
Math
Tokenizer quality, sequence length, fertility, and vocabulary trade-offs.
Code
Train, evaluate, and integrate SentencePiece and other tokenizer libraries.
Compare
SentencePiece, BPE, WordPiece, Unigram, tiktoken, and modern alternatives.
Papers
Annotated tokenization research, benchmarks, fairness, and emerging methods.
Latest entries
Margin Notes · Building a Tokenization-Aware Application
Practical guidance for developers building tokenization-aware applications — token counting, prompt budget management, and cost optimisation with tokenizer APIs.
NLP Tokenization Benchmarks · Fertility, BLEU, and Downstream Performance
How tokenizer choices affect downstream NLP task performance — empirical benchmarks comparing BPE, Unigram, WordPiece, and character-level tokenization.
The Future of Tokenization · Beyond BPE and Unigram
Emerging tokenization research — learned tokenization, differentiable tokenizers, vision-inspired approaches, and whether tokenizers will survive the next generation of models.
Tokenization and Model Fairness · Bias in Subword Segmentation
How tokenization choices create bias in language models — the differential treatment of languages, gendered tokenization, and emerging fairness research.
OpenAI Tokenizer · tiktoken and the GPT Token Ecosystem
How OpenAI's tokenizer works — the tiktoken library, cl100k_base encoding, counting tokens for API pricing, and the tokenizer's impact on GPT-4 costs.
Tokenization in Information Retrieval · Search, TF-IDF, BM25
How tokenization works in information retrieval systems — the differences from NLP tokenization, stemming, stopword removal, and why subword tokenizers aren't always better for search.