pypi / datasets
datasets
What changed in datasets 5, from its official release notes, upgrade guides and advisories. Sources checked 2026-10-20.
5 breaking3 security20 new APIs and options1 performance2 behavior and defaults25 fixes and other
See which of these apply to your code, ranked, in your agent:
npx @unstale/cli initdatasets 556 changes
Breaking changes 5
Rejects a non-callable generator supplied in GeneratorConfig.
External links in HDF5 files are now disallowed.
Batched mapping raises an error when output lengths do not match.
The default shuffling mechanism now uses multiple input shards; pass max_buffer_input_shards=1 to restore the old mechanism.
None values in Json columns remain real nulls instead of being converted to the string "null".
Security 3
Fixes archive path checking that allowed paths to escape into sibling directories.
Fixes arbitrary file writes caused by following symlinks during archive extraction.
Hugging Face Datasets folder-based builders allow path traversal through file_name metadata
New APIs and options 20
Adds BioSequence as a decodable feature for biological sequence data.
Adds BioStructure as a decodable feature for biological structures.
Adds lightweight support for the FASTA file format.
Adds lightweight support for the FASTQ file format.
Adds support for GenBank files containing biological sequence data.
Adds support for mmCIF files containing macromolecular structures.
Adds lightweight support for PDB files.
Adds a strategy argument for splitting iterable datasets by node.
Datasets can now be backed by the Vortex file format.
Supports batched=True when converting a Dataset to a dictionary.
Adds support for Droid agent traces.
Adds support for Hermes traces.
Apache Iceberg format is now supported.
Streaming datasets now support composed splits.
A dataset format loader now supports CoNLL-2003, CoNLL-2000, and CoNLL-U files.
Batch rows by the values of a specified column, enabling batches such as grouped robotics episodes.
Dataset.to_sql now accepts a num_proc argument.
Agent traces can now be loaded and parsed to messages for training, using the optional teich dependency.
3D mesh support and the MeshFolder builder are now available.
A packaged builder now supports TsFile (Apache IoTDB) in per-device wide format.
Performance 1
JSON export of indexed datasets is faster.
Behavior and default changes 2
Importing datasets no longer imports aiohttp.
num_shards can now exceed dataset.num_shards when pushing an iterable dataset to the Hub.
Fixes and other changes 25
BioSequence and BioStructure records are decoded with universal newlines.
Advances a user-supplied generator even when shuffle uses a cache hit.
Fixes handling of fixed-size binary data.
Fixes inverted spans for minus-strand multi-exon GenBank locations.
Shuffling iterable datasets now preserves source shards.
Fixes filtering iterable datasets using arrow, pandas, and polars formats.
Fixes pushing iterable datasets when their features are unknown.
Fixed-shape ArrayXD columns with nulls keep integers on the Python read path.
Validates Arrow IPC record batches.
Preserves on_bad_lines and encoding_errors on pandas 2.0–2.2.
Resuming twice no longer resets the dataloader.
Dataset fingerprints no longer depend on Arrow chunking.
Calling batch(by_column=...) no longer crashes after shard, shuffle, or split operations.
Fixes column dropping in the Arrow path for axis=1 concatenation.
Decodes Json() columns when converting a Dataset to pandas.
Removed splits are no longer left in the dataset card after pushing to the Hub.
Rebatches the Arrow source before formatting to prevent data loss when resuming.
Schema inference now works for JSON files that start with a UTF-8 BOM.
Fixes authentication for Lance.
Fixes casting a nullable LargeList to a different inner type.
Flat numeric columns with nulls remain numeric in numpy format.
Preserves nullable integer columns when exporting to JSON, CSV, or SQL.
Fixes streaming for traces.
Fixes skipping over full Arrow blocks in iterable datasets.
A fix addresses Parquet streaming hangs at the end of a script.