# What changed in tokenx

> 16 changes in tokenx 2 from its official release notes, upgrade guides and advisories: 4 breaking, 1 new APIs and options. Each one cited. Sources checked 2026-10-28. HTML: https://unstale.sh/npm/tokenx/

To see which of these apply to a repository, run `npx @unstale/cli init` in it.

## tokenx 2

### Breaking changes

- CJK text is split at ideographic and fullwidth punctuation, and punctuation is attached to the clause before it. (2.2.0) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.2.0)
- CJK punctuation is attached to the preceding clause when splitting text. (2.2.0) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.2.0)
- CJK segments are priced by script rates without trying built-in language configs. (2.2.0) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.2.0)
- The CLI rejects numeric option values that are too large to represent. (2.2.0) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.2.0)

### New APIs and options

- Adds a CLI for counting, slicing and splitting text. (2.1.0) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.1.0)

### Performance

- The function stops once the token limit is exceeded. (2.2.0) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.2.0)

### Behavior and default changes

- The CLI accepts decimal values for --chars-per-token. (2.2.0) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.2.0)

### Fixes and other changes

- A line break is merged into preceding CJK punctuation, as with ASCII punctuation. (2.2.0) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.2.0)
- Custom language configs are checked before built-in configs instead of replacing them. (2.2.0) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.2.0)
- Built-in language configs retain the ASCII shortcut when custom configs are set. (2.2.0) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.2.0)
- Token-slicing boundaries are kept consistent and safe for surrogate pairs. (2.2.0) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.2.0)
- Overlap no longer repeats a segment that fills a chunk. (2.2.0) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.2.0)
- The CLI explains directory inputs instead of failing with EISDIR. (2.1.1) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.1.1)
- The CLI expands glob patterns when the shell does not. (2.1.1) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.1.1)
- Short option values are split correctly. (2.1.1) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.1.1)
- Usage output is kept off stdout, so it no longer interferes with command output. (2.1.1) — [release notes](https://github.com/johannschopplich/tokenx/releases/tag/v2.1.1)