Code

6 articles

Margin Notes · Building a Tokenization-Aware Application

Practical guidance for developers building tokenization-aware applications — token counting, prompt budget management, and cost optimisation with tokenizer APIs.

OpenAI Tokenizer · tiktoken and the GPT Token Ecosystem

How OpenAI's tokenizer works — the tiktoken library, cl100k_base encoding, counting tokens for API pricing, and the tokenizer's impact on GPT-4 costs.

Custom Tokenizer for Domain-Specific Text · Medical, Legal, Code

How to train a tokenizer for specialised domains — medical text, legal documents, source code, and the importance of domain-specific training data.

Tokenizer Libraries · SentencePiece, HuggingFace Tokenizers, tiktoken

A comparison of the major tokenizer libraries — SentencePiece, HuggingFace tokenizers, tiktoken (OpenAI), and when to use each.

Special Tokens · [CLS], [SEP], [PAD], [MASK] and Beyond

How special tokens work in NLP tokenization — the role of control tokens in BERT, GPT, and T5 models, and how to configure them correctly.

Training a SentencePiece Model · From Corpus to Tokenizer

How to train a SentencePiece tokenizer from scratch — data preparation, parameter selection, training a BPE or Unigram model, and evaluating tokenizer quality.