Compare

4 articles

Tokenization in Information Retrieval · Search, TF-IDF, BM25

How tokenization works in information retrieval systems — the differences from NLP tokenization, stemming, stopword removal, and why subword tokenizers aren't always better for search.

Word-Level Tokenization · Why It's Obsolete in Modern NLP

Why word-level tokenization has been replaced by subword methods — the OOV problem, vocabulary explosion, and comparison with modern subword approaches.

Tokenization in Large Language Models · LLaMA, GPT, and BERT

How major LLMs handle tokenization — LLaMA's SentencePiece BPE, GPT-4's tokenizer, BERT's WordPiece, and why tokenization choices affect model performance and behaviour.

BPE vs WordPiece vs Unigram vs SentencePiece · Tokenization Algorithms Compared

A systematic comparison of four subword tokenization algorithms — how they differ, which models use each one, and the practical trade-offs in vocabulary quality and training speed.