# What changed in gpt-tokenizer

> 15 changes in gpt-tokenizer 3 to 4 from its official release notes, upgrade guides and advisories: 4 breaking, 7 new APIs and options. Each one cited. Sources checked 2026-11-03. HTML: https://unstale.sh/npm/gpt-tokenizer/

To see which of these apply to a repository, run `npx @unstale/cli init` in it.

## gpt-tokenizer 4

### Breaking changes

- UMD bundles are removed. (4.0.0) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/4.0.0)
- The unpkg global entry point is removed. (4.0.0) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/4.0.0)

### New APIs and options

- Adds a tokenizer-integrated incremental token counter. (4.1.0) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/4.1.0)
- GPT-2 tokenizer support was added. (4.0.0) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/4.0.0)
- Price data was added to gpt-5-nano models. (4.0.0) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/4.0.0)

### Performance

- Optimized BPE hot paths while preserving Unicode and streaming correctness. (4.0.4) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/4.0.4)
- Byte pair merge is now O(n log n) on long pieces. (4.0.1) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/4.0.1)

### Fixes and other changes

- Allowed special tokens are now recognised anywhere in the text. (4.0.3) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/4.0.3)
- Splitting and encoding now match tiktoken for U+0085 and U+FEFF. (4.0.2) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/4.0.2)

## gpt-tokenizer 3

### Breaking changes

- Corrects the split regexp for o200, affecting how its text is tokenized. (3.3.0) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/3.3.0)
- Changes the default encoding to o200k_base, which the notes say most modern models use. (3.0.0) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/3.0.0)

### New APIs and options

- Adds token counting for function calling. (3.4.0) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/3.4.0)
- Adds support for the harmony format. (3.3.0) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/3.3.0)
- Adds the ability to estimate cost. (3.0.0) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/3.0.0)
- Adds support for new models. (3.0.0) — [release notes](https://github.com/niieani/gpt-tokenizer/releases/tag/3.0.0)