unstale

pypi / sentencepiece

sentencepiece

What changed in sentencepiece 0.2, from its official release notes, upgrade guides and advisories. Sources checked 2026-10-27.

3 breaking2 security6 new APIs and options3 performance1 behavior and defaults1 tooling3 fixes and other

See which of these apply to your code, ranked, in your agent:

npx @unstale/cli init

@unstale/cli on npm

sentencepiece 0.219 changes

Breaking changes 3

  • BOS/EOS adding now follows C++ constraints and raises ValueError instead of silently failing or crashing.

    0.2.2release notes

  • Vocabulary pieces containing null characters are explicitly disallowed.

    0.2.2release notes

  • Support and distribution of 32-bit Windows wheels stopped.

    0.2.2release notes

Security 2

  • Replaced internal CHECK macros with DCHECK or returned errors to prevent host application crashes on invalid inputs.

    0.2.2release notes

  • Model loading verifies darts-clone trie offsets to prevent segfaults on corrupted or malicious model files.

    0.2.2release notes

New APIs and options 6

  • Processors and normalizers can now be loaded directly from in-memory Proto instances.

    0.2.2release notes

  • C++ now supports loading a normalizer from raw maps.

    0.2.2release notes

  • Added direct encoding and decoding to and from NumPy objects.

    0.2.2release notes

  • Added offset mapping for Unicode character offsets and raw byte offsets.

    0.2.2release notes

  • Introduced parallel_encode to tokenize single large documents across multiple CPU cores.

    0.2.2release notes

  • Introduced ThreadPool for parallel Python tokenization of single large documents.

    0.2.2release notes

Performance 3

  • Ported an O(log n) lazy priority queue update optimization to the BPE trainer to speed up training.

    0.2.2release notes

  • Integer/string input encoding and decoding skip SentencePieceText conversion, resulting in about a 2x speedup.

    0.2.2release notes

  • C++ execution releases the GIL to improve multi-threaded Python performance.

    0.2.2release notes

Behavior and default changes 1

  • Improved unigram pruning using token-normalized frequency calculations.

    0.2.2release notes

Tooling 1

  • Added inline type stubs and a PEP 561 marker for IDE auto-completion and static type safety.

    0.2.2release notes

Fixes and other changes 3

  • Validated value offsets in DecompileCharsMap.

    0.2.2release notes

  • Implemented a global timeout to prevent NBest A* search from hanging on pathological inputs.

    0.2.2release notes

  • Added bounds checks to piece accessors.

    0.2.2release notes