# What changed in datasets

> 56 changes in datasets 5 from its official release notes, upgrade guides and advisories: 5 breaking, 3 security, 20 new APIs and options. Each one cited. Sources checked 2026-10-20. HTML: https://unstale.sh/pypi/datasets/

To see which of these apply to a repository, run `npx @unstale/cli init` in it.

## datasets 5

### Breaking changes

- Rejects a non-callable generator supplied in GeneratorConfig. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- External links in HDF5 files are now disallowed. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Batched mapping raises an error when output lengths do not match. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- The default shuffling mechanism now uses multiple input shards; pass max_buffer_input_shards=1 to restore the old mechanism. (5.0.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.0)
- None values in Json columns remain real nulls instead of being converted to the string "null". (5.0.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.0)

### Security

- Fixes archive path checking that allowed paths to escape into sibling directories. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Fixes arbitrary file writes caused by following symlinks during archive extraction. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Hugging Face Datasets folder-based builders allow path traversal through file_name metadata (5.0.1) — [security advisory](https://github.com/advisories/GHSA-379C-QX7V-6H59), [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)

### New APIs and options

- Adds BioSequence as a decodable feature for biological sequence data. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Adds BioStructure as a decodable feature for biological structures. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Adds lightweight support for the FASTA file format. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Adds lightweight support for the FASTQ file format. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Adds support for GenBank files containing biological sequence data. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Adds support for mmCIF files containing macromolecular structures. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Adds lightweight support for PDB files. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Adds a strategy argument for splitting iterable datasets by node. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Datasets can now be backed by the Vortex file format. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Supports batched=True when converting a Dataset to a dictionary. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Adds support for Droid agent traces. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Adds support for Hermes traces. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Apache Iceberg format is now supported. (5.0.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.0)
- Streaming datasets now support composed splits. (5.0.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.0)
- A dataset format loader now supports CoNLL-2003, CoNLL-2000, and CoNLL-U files. (5.0.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.0)
- Batch rows by the values of a specified column, enabling batches such as grouped robotics episodes. (5.0.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.0)
- Dataset.to_sql now accepts a num_proc argument. (5.0.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.0)
- Agent traces can now be loaded and parsed to messages for training, using the optional teich dependency. (5.0.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.0)
- 3D mesh support and the MeshFolder builder are now available. (5.0.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.0)
- A packaged builder now supports TsFile (Apache IoTDB) in per-device wide format. (5.0.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.0)

### Performance

- JSON export of indexed datasets is faster. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)

### Behavior and default changes

- Importing datasets no longer imports aiohttp. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- num_shards can now exceed dataset.num_shards when pushing an iterable dataset to the Hub. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)

### Fixes and other changes

- BioSequence and BioStructure records are decoded with universal newlines. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Advances a user-supplied generator even when shuffle uses a cache hit. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Fixes handling of fixed-size binary data. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Fixes inverted spans for minus-strand multi-exon GenBank locations. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Shuffling iterable datasets now preserves source shards. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Fixes filtering iterable datasets using arrow, pandas, and polars formats. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Fixes pushing iterable datasets when their features are unknown. (5.1.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.1.0)
- Fixed-shape ArrayXD columns with nulls keep integers on the Python read path. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Validates Arrow IPC record batches. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Preserves on_bad_lines and encoding_errors on pandas 2.0–2.2. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Resuming twice no longer resets the dataloader. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Dataset fingerprints no longer depend on Arrow chunking. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Calling batch(by_column=...) no longer crashes after shard, shuffle, or split operations. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Fixes column dropping in the Arrow path for axis=1 concatenation. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Decodes Json() columns when converting a Dataset to pandas. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Removed splits are no longer left in the dataset card after pushing to the Hub. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Rebatches the Arrow source before formatting to prevent data loss when resuming. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Schema inference now works for JSON files that start with a UTF-8 BOM. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Fixes authentication for Lance. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Fixes casting a nullable LargeList to a different inner type. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Flat numeric columns with nulls remain numeric in numpy format. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Preserves nullable integer columns when exporting to JSON, CSV, or SQL. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Fixes streaming for traces. (5.0.1) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.1)
- Fixes skipping over full Arrow blocks in iterable datasets. (5.0.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.0)
- A fix addresses Parquet streaming hangs at the end of a script. (5.0.0) — [release notes](https://github.com/huggingface/datasets/releases/tag/5.0.0)