npm / tokenx
tokenx
What changed in tokenx 2, from its official release notes, upgrade guides and advisories. Sources checked 2026-10-28.
4 breaking1 new APIs and options1 performance1 behavior and defaults9 fixes and other
See which of these apply to your code, ranked, in your agent:
npx @unstale/cli inittokenx 216 changes
Breaking changes 4
CJK text is split at ideographic and fullwidth punctuation, and punctuation is attached to the clause before it.
CJK punctuation is attached to the preceding clause when splitting text.
CJK segments are priced by script rates without trying built-in language configs.
The CLI rejects numeric option values that are too large to represent.
New APIs and options 1
Adds a CLI for counting, slicing and splitting text.
Performance 1
The function stops once the token limit is exceeded.
Behavior and default changes 1
The CLI accepts decimal values for --chars-per-token.
Fixes and other changes 9
A line break is merged into preceding CJK punctuation, as with ASCII punctuation.
Custom language configs are checked before built-in configs instead of replacing them.
Built-in language configs retain the ASCII shortcut when custom configs are set.
Token-slicing boundaries are kept consistent and safe for surrogate pairs.
Overlap no longer repeats a segment that fills a chunk.
The CLI explains directory inputs instead of failing with EISDIR.
The CLI expands glob patterns when the shell does not.
Short option values are split correctly.
Usage output is kept off stdout, so it no longer interferes with command output.