Concept
11 articles
Character-Aware Tokenization · Can We Skip Subwords?
Exploring character-level and character-aware NLP models — CANINE, ByT5, CharFormer, and whether character-level models can replace subword tokenization entirely.
Normalization in Tokenization · NFKC, Lowercasing, and Unicode
How text normalisation affects tokenization — Unicode normalisation forms, the case folding debate, and when normalisation helps vs hurts model performance.
Pre-tokenization · Why You Shouldn't Split on Whitespace First
Why whitespace-based pre-tokenization is harmful — how it breaks CJK and other non-segmented languages, and why modern tokenizers (SentencePiece) avoid it.
Byte-Level BPE · How GPT Tokenization Handles Every Language
How byte-level BPE works — starting from bytes instead of characters, handling any Unicode input, and the tradeoffs compared to character-level BPE.
Detokenization · Reconstructing Text from Tokens Losslessly
How to reverse the tokenization process — the challenges of detokenization, why SentencePiece is lossless, and common detokenization errors.
Tokenization for Multilingual NLP · Challenges and Solutions
How tokenization handles multiple languages — script-specific challenges, the CJK problem, vocabulary allocation across writing systems, and the byte-fallback approach.
Transformer Tokenization · How Tokens Become Embeddings
The path from text to model input — tokenization, token IDs, embedding lookup, positional encoding, and attention masking in the transformer pipeline.
WordPiece Tokenization · Google's Subword Algorithm
How WordPiece tokenization works — the likelihood-based merge criterion, the ## prefix for continuation tokens, its use in BERT, and comparison with BPE.
Subword Tokenization · Why BPE, WordPiece and Unigram Work
The theory behind subword tokenization — why character-level is too long, word-level fails on OOV, and subword units achieve an optimal balance of vocabulary size and sequence length.
Vocabulary Size in Tokenization · How Many Tokens is Optimal?
How vocabulary size affects tokenizer performance — the tradeoff between vocabulary size, sequence length, model embedding parameters, and multilingual coverage.
SentencePiece · A Lossless, Language-Agnostic Tokenizer
A deep dive into SentencePiece — how it works, the difference between BPE and Unigram models, why it outperforms word-level tokenizers, and its role in modern LLM pipelines.