1 articles
How to evaluate tokenizer quality — intrinsic metrics (fertility, coverage, token/word ratio) and extrinsic evaluation through downstream task performance.