# What changed in tokenizers

> 18 changes in tokenizers 0.23 from its official release notes, upgrade guides and advisories: 2 breaking, 4 new APIs and options. Each one cited. Sources checked 2026-10-26. HTML: https://unstale.sh/pypi/tokenizers/

To see which of these apply to a repository, run `npx @unstale/cli init` in it.

## tokenizers 0.23

### Breaking changes

- Token content is normalized at insertion, so re-saved tokenizer.json files may differ in the added_tokens block. (0.23.1) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1)
- Python 3.9 is no longer supported; the requirement is now Python >=3.10. (0.23.1) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1)

### New APIs and options

- Adds ParityBpeTrainer as part of implementing Parity-aware BPE. (0.23.2) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.2)
- Unigram exposes alpha for subword regularization, providing parity with Google's implementation. (0.23.1) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1)
- Unigram exposes nbest_size for subword regularization, providing parity with Google's implementation. (0.23.1) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1)
- Tokenizer now supports weak references, useful for long-lived caches that should not keep tokenizers alive. (0.23.1) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1)

### Performance

- Avoids cloning the full vocabulary when getting the vocabulary size. (0.23.2) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.2)
- A per-thread BPE cache improves performance on highly parallel workloads. (0.23.1) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1)
- Right-direction truncation exits without pre-tokenizing past max_length; left-truncation is unchanged. (0.23.1) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1)

### Behavior and default changes

- Requirements are loosened to allow huggingface_hub v2, unblocking transformers installs. (0.23.3) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.3)
- Dedicated free-threaded Python 3.14 wheels are available and do not force the GIL back on when tokenizers is imported. (0.23.1) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1)

### Tooling

- The npm package now ships binaries for 13 platforms, including Apple Silicon, Linux ARM, Alpine, Windows, and Android targets. (0.23.1) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1)
- Every Python binding class now has precise stubs and signatures; stricter typing may expose errors previously hidden as Any. (0.23.1) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1)

### Fixes and other changes

- Fixes a breaking change from 0.23.1 regarding truncation and overflowing. (0.23.3) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.3)
- Fixes type hints that were not resolving for pyright. (0.23.3) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.3)
- DecodeStream now implements __copy__. (0.23.1) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1)
- DecodeStream now implements __deepcopy__. (0.23.1) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1)
- An unclosed annotation span is fixed and HTML escaping is applied to output. (0.23.1) — [release notes](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1)