Concept
13 articles
BPE Dropout · Making Subword Segmentation More Robust for Unseen Words
What BPE-dropout does inside a fixed BPE framework — why deterministic segmentation hurts, how SentencePiece samples multiple segmentations on the fly, and how it relates to Unigram subword regularization.
SentencePiece Unigram vs BPE · model_type, Character Coverage and Split Rules
A practical comparison of the SentencePiece training parameters that actually decide segmentation output — model_type, character_coverage, split_by_whitespace, byte_fallback — what each one changes, and how to compare Unigram and BPE fairly on the same corpus.
Character-Aware Tokenization · Can We Skip Subwords?
Exploring character-level and character-aware NLP models — CANINE, ByT5, CharFormer, and whether character-level models can replace subword tokenization entirely.
Normalization in Tokenization · NFKC, Lowercasing, and Unicode
How text normalisation affects tokenization — Unicode normalisation forms, the case folding debate, and when normalisation helps vs hurts model performance.
Pre-tokenization · Why You Shouldn't Split on Whitespace First
Why whitespace-based pre-tokenization is harmful — how it breaks CJK and other non-segmented languages, and why modern tokenizers (SentencePiece) avoid it.
Byte-Level BPE · How GPT Tokenization Handles Every Language
How byte-level BPE works — starting from bytes instead of characters, handling any Unicode input, and the tradeoffs compared to character-level BPE.
Detokenization · Reconstructing Text from Tokens Losslessly
How to reverse the tokenization process — the challenges of detokenization, why SentencePiece is lossless, and common detokenization errors.
Tokenization for Multilingual NLP · Challenges and Solutions
How tokenization handles multiple languages — script-specific challenges, the CJK problem, vocabulary allocation across writing systems, and the byte-fallback approach.
Transformer Tokenization · How Tokens Become Embeddings
The path from text to model input — tokenization, token IDs, embedding lookup, positional encoding, and attention masking in the transformer pipeline.
WordPiece Tokenization · Google's Subword Algorithm
How WordPiece tokenization works — the likelihood-based merge criterion, the ## prefix for continuation tokens, its use in BERT, and comparison with BPE.
Subword Tokenization · Why BPE, WordPiece and Unigram Work
The theory behind subword tokenization — why character-level is too long, word-level fails on OOV, and subword units achieve an optimal balance of vocabulary size and sequence length.
Vocabulary Size in Tokenization · How Many Tokens is Optimal?
How vocabulary size affects tokenizer performance — the tradeoff between vocabulary size, sequence length, model embedding parameters, and multilingual coverage.
SentencePiece · A Lossless, Language-Agnostic Tokenizer
A deep dive into SentencePiece — how it works, the difference between BPE and Unigram models, why it outperforms word-level tokenizers, and its role in modern LLM pipelines.