npm / gpt-tokenizer
gpt-tokenizer
What changed in gpt-tokenizer 3 to 4, from its official release notes, upgrade guides and advisories. Sources checked 2026-11-03.
4 breaking7 new APIs and options2 performance2 fixes and other
See which of these apply to your code, ranked, in your agent:
npx @unstale/cli initgpt-tokenizer 49 changes
Breaking changes 2
UMD bundles are removed.
The unpkg global entry point is removed.
New APIs and options 3
Adds a tokenizer-integrated incremental token counter.
GPT-2 tokenizer support was added.
Price data was added to gpt-5-nano models.
Performance 2
Optimized BPE hot paths while preserving Unicode and streaming correctness.
Byte pair merge is now O(n log n) on long pieces.
Fixes and other changes 2
Allowed special tokens are now recognised anywhere in the text.
Splitting and encoding now match tiktoken for U+0085 and U+FEFF.
gpt-tokenizer 36 changes
Breaking changes 2
Corrects the split regexp for o200, affecting how its text is tokenized.
Changes the default encoding to o200k_base, which the notes say most modern models use.
New APIs and options 4
Adds token counting for function calling.
Adds support for the harmony format.
Adds the ability to estimate cost.
Adds support for new models.