pypi / tokenizers
tokenizers
What changed in tokenizers 0.23, from its official release notes, upgrade guides and advisories. Sources checked 2026-10-26.
2 breaking4 new APIs and options3 performance2 behavior and defaults2 tooling5 fixes and other
See which of these apply to your code, ranked, in your agent:
npx @unstale/cli inittokenizers 0.2318 changes
Breaking changes 2
Token content is normalized at insertion, so re-saved tokenizer.json files may differ in the added_tokens block.
Python 3.9 is no longer supported; the requirement is now Python >=3.10.
New APIs and options 4
Adds ParityBpeTrainer as part of implementing Parity-aware BPE.
Unigram exposes alpha for subword regularization, providing parity with Google's implementation.
Unigram exposes nbest_size for subword regularization, providing parity with Google's implementation.
Tokenizer now supports weak references, useful for long-lived caches that should not keep tokenizers alive.
Performance 3
Avoids cloning the full vocabulary when getting the vocabulary size.
A per-thread BPE cache improves performance on highly parallel workloads.
Right-direction truncation exits without pre-tokenizing past max_length; left-truncation is unchanged.
Behavior and default changes 2
Requirements are loosened to allow huggingface_hub v2, unblocking transformers installs.
Dedicated free-threaded Python 3.14 wheels are available and do not force the GIL back on when tokenizers is imported.
Tooling 2
The npm package now ships binaries for 13 platforms, including Apple Silicon, Linux ARM, Alpine, Windows, and Android targets.
Every Python binding class now has precise stubs and signatures; stricter typing may expose errors previously hidden as Any.
Fixes and other changes 5
Fixes a breaking change from 0.23.1 regarding truncation and overflowing.
Fixes type hints that were not resolving for pyright.
DecodeStream now implements __copy__.
DecodeStream now implements __deepcopy__.
An unclosed annotation span is fixed and HTML escaping is applied to output.