# What changed in sentencepiece

> 19 changes in sentencepiece 0.2 from its official release notes, upgrade guides and advisories: 3 breaking, 2 security, 6 new APIs and options. Each one cited. Sources checked 2026-10-27. HTML: https://unstale.sh/pypi/sentencepiece/

To see which of these apply to a repository, run `npx @unstale/cli init` in it.

## sentencepiece 0.2

### Breaking changes

- BOS/EOS adding now follows C++ constraints and raises ValueError instead of silently failing or crashing. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)
- Vocabulary pieces containing null characters are explicitly disallowed. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)
- Support and distribution of 32-bit Windows wheels stopped. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)

### Security

- Replaced internal CHECK macros with DCHECK or returned errors to prevent host application crashes on invalid inputs. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)
- Model loading verifies darts-clone trie offsets to prevent segfaults on corrupted or malicious model files. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)

### New APIs and options

- Processors and normalizers can now be loaded directly from in-memory Proto instances. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)
- C++ now supports loading a normalizer from raw maps. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)
- Added direct encoding and decoding to and from NumPy objects. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)
- Added offset mapping for Unicode character offsets and raw byte offsets. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)
- Introduced parallel_encode to tokenize single large documents across multiple CPU cores. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)
- Introduced ThreadPool for parallel Python tokenization of single large documents. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)

### Performance

- Ported an O(log n) lazy priority queue update optimization to the BPE trainer to speed up training. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)
- Integer/string input encoding and decoding skip SentencePieceText conversion, resulting in about a 2x speedup. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)
- C++ execution releases the GIL to improve multi-threaded Python performance. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)

### Behavior and default changes

- Improved unigram pruning using token-normalized frequency calculations. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)

### Tooling

- Added inline type stubs and a PEP 561 marker for IDE auto-completion and static type safety. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)

### Fixes and other changes

- Validated value offsets in DecompileCharsMap. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)
- Implemented a global timeout to prevent NBest A* search from hanging on pathological inputs. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)
- Added bounds checks to piece accessors. (0.2.2) — [release notes](https://github.com/google/sentencepiece/releases/tag/v0.2.2)