datasets Documentation

File: README.md Copy Chapter

🤗 Datasets is a lightweight library providing two main features:

• one-line dataloaders for many public datasets: one-liners to download and pre-process any of the major public datasets (image datasets, audio datasets, text datasets in 467 languages and dialects, 3D medical images, video datasets, agent traces, etc.) provided on the HuggingFace Datasets Hub. With a simple command like squad_dataset = load_dataset("rajpurkar/squad"), get any of these datasets ready to use in a dataloader for training/evaluating a ML model (Numpy/Pandas/PyTorch/TensorFlow/JAX/Polars),
• efficient data pre-processing: simple, fast and reproducible data pre-processing for the public datasets as well as your own local datasets in CSV, JSON, JSONL, Parquet, HDF5, XML, text, PNG, JPEG, WAV, MP3, PDF, NIfTI, and more. With simple commands like processed_dataset = dataset.map(process_example), efficiently prepare the dataset for inspection and ML model evaluation and training.

🎓 Documentation 🔎 Find a dataset in the Hub 🌟 Share a dataset on the Hub

[](https://hf.co/course)

🚀 Key Features

🤗 Datasets is designed to let the community easily add and share new datasets, and provides powerful capabilities for data manipulation:

Installation

With pip Copy Chapter

🤗 Datasets can be installed from PyPi and should be installed in a virtual environment (venv or conda for instance):

pip install datasets

For the latest development version:

pip install "datasets @ git+https://github.com/huggingface/datasets.git"

With conda Copy Chapter

conda install -c huggingface -c conda-forge datasets

Optional dependencies Copy Chapter

🤗 Datasets supports various optional features via extras:

# For audio (torchcodec)
pip install datasets[audio]

# For image/video (Pillow, torchcodec)
pip install datasets[vision]

# For PDFs/NIfTI (pdfplumber, nibabel)
pip install datasets[pdfs,nibabel]

# For PyTorch/TensorFlow/JAX integration
pip install datasets[torch,tensorflow,jax]

For more details on installation, check the installation page.

Quick Start

🤗 Datasets is made to be very simple to use — the API is centered around a single function, datasets.load_dataset(dataset_name, **kwargs), that instantiates a dataset.

Here is a quick example:

from datasets import load_dataset

# Load a dataset and print the first example in the training set
squad_dataset = load_dataset('rajpurkar/squad')
print(squad_dataset['train'][0])

# Process the dataset - add a column with the length of the context texts
dataset_with_length = squad_dataset.map(lambda x: {"length": len(x["context"])})

# Tokenize the context texts (using a tokenizer from the 🤗 Transformers library)
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')

tokenized_dataset = squad_dataset.map(lambda x: tokenizer(x['context']), batched=True)

# Tokenize chat conversations with a chat template (using a model that supports chat templates)
# This is useful for fine-tuning instruction/chat models

# Load a popular chat dataset (ultrachat_200k contains ~200k AI assistant conversations)
chat_dataset = load_dataset('HuggingFaceH4/ultrachat_200k', split='train_sft')

chat_tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-7B-Instruct')

def tokenize_chat(examples):
    # Apply the chat template and tokenize in one step
    return chat_tokenizer.apply_chat_template(examples["messages"])

tokenized_chat_dataset = chat_dataset.map(tokenize_chat, batched=True)

Streaming mode Copy Chapter

If your dataset is bigger than your disk or if you don't want to wait to download the data, you can use streaming:

# Stream the dataset without downloading anything
image_dataset = load_dataset('timm/imagenet-1k-wds', streaming=True)
for example in image_dataset["train"]:
    print(example["image"])
    break

Multi-modal data Copy Chapter

🤗 Datasets supports a wide variety of data types out of the box:

# Audio dataset
dataset = load_dataset("openslr/librispeech_asr", "clean")

# Image dataset
dataset = load_dataset("ILSVRC/imagenet-1k")

# Video dataset
dataset = load_dataset("Shofo/shofo-tiktok-general-small")

# PDF documents
dataset = load_dataset("pixparse/pdfa-eng-wds")

# NIfTI (3D medical imaging)
dataset = load_dataset("dartbrains/localizer", "betas")

From local files Copy Chapter

# Load from local CSV
dataset = load_dataset('csv', data_files='my_data.csv')

# Load from local Parquet
dataset = load_dataset('parquet', data_files='data/*.parquet')

# Load from a local directory (auto-detect format)
dataset = load_dataset('./path/to/data')

From Python objects Copy Chapter

from datasets import Dataset

# From a dictionary
dataset = Dataset.from_dict({"text": ["Hello world", "How are you?"]})

# From a list
dataset = Dataset.from_list([{"text": "Hello world"}, {"text": "How are you?"}])

# From Pandas
import pandas as pd
df = pd.DataFrame({"col1": [1, 2, 3], "col2": ["a", "b", "c"]})
dataset = Dataset.from_pandas(df)

# From a generator
def gen():
    for i in range(10):
        yield {"value": i}
dataset = Dataset.from_generator(gen)

For more details on using the library, check the quick start guide and the specific pages on:

• Loading a dataset
• What's in a Dataset
• Processing data with 🤗 Datasets
  • Processing audio data
  • Processing image data
  • Processing text data
  • Processing PDF data
  • Processing video data
• Streaming a dataset

Core Classes

The library provides two main dataset classes:

Both are wrapped in DatasetDict / IterableDatasetDict for multi-split datasets (e.g., train/test/val).

Add a new dataset to the Hub

We have a very detailed step-by-step guide to add a new dataset to the datasets already provided on the HuggingFace Datasets Hub.

You can find:

• how to upload a dataset to the Hub using your web browser or Python and also
• how to upload it using Git.

Disclaimers

You can use 🤗 Datasets to load datasets based on versioned git repositories maintained by the dataset authors. For reproducibility reasons, we ask users to pin the revision of the repositories they use.

If you're a dataset owner and wish to update any part of it (description, citation, license, etc.), or do not want your dataset to be included in the Hugging Face Hub, please get in touch by opening a discussion or a pull request in the Community tab of the dataset page. Thanks for your contribution to the ML community!

Contributing

We welcome contributions! Please see our Contributing Guide for details on:

• How to submit issues and pull requests
• Code style guidelines (we use Ruff)
• Testing requirements
• Documentation standards

BibTeX

If you want to cite our 🤗 Datasets library, you can use our paper:

/* Detailed source-code truncated for AI context efficiency. */

If you need to cite a specific version of our 🤗 Datasets library for reproducibility, you can use the corresponding version Zenodo DOI from this list.

File: docs/source/package_reference/builder_classes.mdx Copy Chapter

Builder classes

Builders Copy Chapter

🤗 Datasets relies on two main classes during the dataset building process: [DatasetBuilder] and [BuilderConfig].

[[autodoc]] datasets.DatasetBuilder

[[autodoc]] datasets.GeneratorBasedBuilder

[[autodoc]] datasets.ArrowBasedBuilder

[[autodoc]] datasets.BuilderConfig

Download Copy Chapter

[[autodoc]] datasets.DownloadManager

[[autodoc]] datasets.StreamingDownloadManager

[[autodoc]] datasets.DownloadConfig

[[autodoc]] datasets.DownloadMode

Verification Copy Chapter

[[autodoc]] datasets.VerificationMode

Splits Copy Chapter

[[autodoc]] datasets.SplitGenerator

[[autodoc]] datasets.Split

[[autodoc]] datasets.NamedSplit

[[autodoc]] datasets.NamedSplitAll

[[autodoc]] datasets.ReadInstruction

Version Copy Chapter

[[autodoc]] datasets.utils.Version

File: docs/source/package_reference/loading_methods.mdx Copy Chapter

Loading methods

Methods for listing and loading datasets:

Datasets Copy Chapter

[[autodoc]] datasets.load_dataset

[[autodoc]] datasets.load_from_disk

[[autodoc]] datasets.load_dataset_builder

[[autodoc]] datasets.get_dataset_config_names

[[autodoc]] datasets.get_dataset_infos

[[autodoc]] datasets.get_dataset_split_names

From files Copy Chapter

Configurations used to load data files.
They are used when loading local files or a dataset repository:

• local files: load_dataset("parquet", data_dir="path/to/data/dir")
• dataset repository: load_dataset("allenai/c4")

You can pass arguments to load_dataset to configure data loading.
For example you can specify the sep parameter to define the [~datasets.packaged_modules.csv.CsvConfig] that is used to load the data:

load_dataset("csv", data_dir="path/to/data/dir", sep="\t")

Text Copy Section

[[autodoc]] datasets.packaged_modules.text.TextConfig

[[autodoc]] datasets.packaged_modules.text.Text

CSV Copy Section

[[autodoc]] datasets.packaged_modules.csv.CsvConfig

[[autodoc]] datasets.packaged_modules.csv.Csv

JSON Copy Section

[[autodoc]] datasets.packaged_modules.json.JsonConfig

[[autodoc]] datasets.packaged_modules.json.Json

XML Copy Section

[[autodoc]] datasets.packaged_modules.xml.XmlConfig

[[autodoc]] datasets.packaged_modules.xml.Xml

Parquet Copy Section

[[autodoc]] datasets.packaged_modules.parquet.ParquetConfig

[[autodoc]] datasets.packaged_modules.parquet.Parquet

Arrow Copy Section

[[autodoc]] datasets.packaged_modules.arrow.ArrowConfig

[[autodoc]] datasets.packaged_modules.arrow.Arrow

SQL Copy Section

[[autodoc]] datasets.packaged_modules.sql.SqlConfig

[[autodoc]] datasets.packaged_modules.sql.Sql

Images Copy Section

[[autodoc]] datasets.packaged_modules.imagefolder.ImageFolderConfig

[[autodoc]] datasets.packaged_modules.imagefolder.ImageFolder

Audio Copy Section

[[autodoc]] datasets.packaged_modules.audiofolder.AudioFolderConfig

[[autodoc]] datasets.packaged_modules.audiofolder.AudioFolder

Videos Copy Section

[[autodoc]] datasets.packaged_modules.videofolder.VideoFolderConfig

[[autodoc]] datasets.packaged_modules.videofolder.VideoFolder

HDF5 Copy Section

[[autodoc]] datasets.packaged_modules.hdf5.HDF5Config

[[autodoc]] datasets.packaged_modules.hdf5.HDF5

TsFile Copy Section

[[autodoc]] datasets.packaged_modules.tsfile.TsFileConfig

[[autodoc]] datasets.packaged_modules.tsfile.TsFile

Pdf Copy Section

[[autodoc]] datasets.packaged_modules.pdffolder.PdfFolderConfig

[[autodoc]] datasets.packaged_modules.pdffolder.PdfFolder

Nifti Copy Section

[[autodoc]] datasets.packaged_modules.niftifolder.NiftiFolderConfig

[[autodoc]] datasets.packaged_modules.niftifolder.NiftiFolder

WebDataset Copy Section

[[autodoc]] datasets.packaged_modules.webdataset.WebDataset

File: docs/source/package_reference/main_classes.mdx Copy Chapter

Main classes

DatasetInfo Copy Chapter

[[autodoc]] datasets.DatasetInfo

Dataset Copy Chapter

The base class [Dataset] implements a Dataset backed by an Apache Arrow table.

[[autodoc]] datasets.Dataset
- add_column
- add_item
- from_file
- from_buffer
- from_pandas
- from_dict
- from_list
- from_generator
- data
- cache_files
- num_columns
- num_rows
- column_names
- shape
- unique
- flatten
- cast
- cast_column
- remove_columns
- rename_column
- rename_columns
- select_columns
- class_encode_column
- len
- iter
- iter
- formatted_as
- set_format
- set_transform
- reset_format
- with_format
- with_transform
- getitem
- cleanup_cache_files
- map
- filter
- select
- sort
- shuffle
- skip
- take
- train_test_split
- shard
- repeat
- to_tf_dataset
- push_to_hub
- save_to_disk
- load_from_disk
- flatten_indices
- to_csv
- to_pandas
- to_dict
- to_json
- to_parquet
- to_sql
- to_iterable_dataset
- add_faiss_index
- add_faiss_index_from_external_arrays
- save_faiss_index
- load_faiss_index
- add_elasticsearch_index
- load_elasticsearch_index
- list_indexes
- get_index
- drop_index
- search
- search_batch
- get_nearest_examples
- get_nearest_examples_batch
- info
- split
- builder_name
- citation
- config_name
- dataset_size
- description
- download_checksums
- download_size
- features
- homepage
- license
- size_in_bytes
- supervised_keys
- version
- from_csv
- from_json
- from_parquet
- from_text
- from_sql
- align_labels_with_mapping

[[autodoc]] datasets.concatenate_datasets

[[autodoc]] datasets.interleave_datasets

[[autodoc]] datasets.distributed.split_dataset_by_node

[[autodoc]] datasets.enable_caching

[[autodoc]] datasets.disable_caching

[[autodoc]] datasets.is_caching_enabled

[[autodoc]] datasets.Column

DatasetDict Copy Chapter

Dictionary with split names as keys ('train', 'test' for example), and Dataset objects as values.
It also has dataset transform methods like map or filter, to process all the splits at once.

[[autodoc]] datasets.DatasetDict
- data
- cache_files
- num_columns
- num_rows
- column_names
- shape
- unique
- cleanup_cache_files
- map
- filter
- sort
- shuffle
- set_format
- reset_format
- formatted_as
- with_format
- with_transform
- flatten
- cast
- cast_column
- remove_columns
- rename_column
- rename_columns
- select_columns
- class_encode_column
- push_to_hub
- save_to_disk
- load_from_disk
- from_csv
- from_json
- from_parquet
- from_text

IterableDataset Copy Chapter

The base class [IterableDataset] implements an iterable Dataset backed by python generators.

[[autodoc]] datasets.IterableDataset
- from_file
- from_pandas
- from_dict
- from_list
- from_generator
- remove_columns
- select_columns
- cast_column
- cast
- decode
- iter
- iter
- map
- rename_column
- filter
- shuffle
- batch
- skip
- take
- shard
- reshard
- repeat
- to_csv
- to_pandas
- to_dict
- to_json
- to_parquet
- to_sql
- push_to_hub
- load_state_dict
- state_dict
- info
- split
- builder_name
- citation
- config_name
- dataset_size
- description
- download_checksums
- download_size
- features
- homepage
- license
- size_in_bytes
- supervised_keys
- version
- from_csv
- from_json
- from_parquet
- from_text

[[autodoc]] datasets.IterableColumn

IterableDatasetDict Copy Chapter

Dictionary with split names as keys ('train', 'test' for example), and IterableDataset objects as values.

[[autodoc]] datasets.IterableDatasetDict
- map
- filter
- shuffle
- with_format
- cast
- cast_column
- remove_columns
- rename_column
- rename_columns
- select_columns
- push_to_hub

Features Copy Chapter

[[autodoc]] datasets.Features

Scalar Copy Section

[[autodoc]] datasets.Value

[[autodoc]] datasets.ClassLabel

Composite Copy Section

[[autodoc]] datasets.LargeList

[[autodoc]] datasets.List

[[autodoc]] datasets.Sequence

Translation Copy Section

[[autodoc]] datasets.Translation

[[autodoc]] datasets.TranslationVariableLanguages

Arrays Copy Section

[[autodoc]] datasets.Array2D

[[autodoc]] datasets.Array3D

[[autodoc]] datasets.Array4D

[[autodoc]] datasets.Array5D

Audio Copy Section

[[autodoc]] datasets.Audio

Image Copy Section

[[autodoc]] datasets.Image

Video Copy Section

[[autodoc]] datasets.Video

Mesh Copy Section

[[autodoc]] datasets.Mesh

Json Copy Section

[[autodoc]] datasets.Json

Pdf Copy Section

[[autodoc]] datasets.Pdf

Nifti Copy Section

[[autodoc]] datasets.Nifti

Filesystems Copy Chapter

[[autodoc]] datasets.filesystems.is_remote_filesystem

Fingerprint Copy Chapter

[[autodoc]] datasets.fingerprint.Hasher

File: docs/source/package_reference/table_classes.mdx Copy Chapter

Table Classes

Each Dataset object is backed by a PyArrow Table.
A Table can be loaded from either the disk (memory mapped) or in memory.
Several Table types are available, and they all inherit from [table.Table].

Table Copy Chapter

[[autodoc]] datasets.table.Table
- validate
- equals
- to_batches
- to_pydict
- to_pandas
- to_string
- field
- column
- itercolumns
- schema
- columns
- num_columns
- num_rows
- shape
- nbytes

InMemoryTable Copy Chapter

[[autodoc]] datasets.table.InMemoryTable
- validate
- equals
- to_batches
- to_pydict
- to_pandas
- to_string
- field
- column
- itercolumns
- schema
- columns
- num_columns
- num_rows
- shape
- nbytes
- column_names
- slice
- filter
- flatten
- combine_chunks
- cast
- replace_schema_metadata
- add_column
- append_column
- remove_column
- set_column
- rename_columns
- select
- drop
- from_file
- from_buffer
- from_pandas
- from_arrays
- from_pydict
- from_batches

MemoryMappedTable Copy Chapter

[[autodoc]] datasets.table.MemoryMappedTable
- validate
- equals
- to_batches
- to_pydict
- to_pandas
- to_string
- field
- column
- itercolumns
- schema
- columns
- num_columns
- num_rows
- shape
- nbytes
- column_names
- slice
- filter
- flatten
- combine_chunks
- cast
- replace_schema_metadata
- add_column
- append_column
- remove_column
- set_column
- rename_columns
- select
- drop
- from_file

ConcatenationTable Copy Chapter

[[autodoc]] datasets.table.ConcatenationTable
- validate
- equals
- to_batches
- to_pydict
- to_pandas
- to_string
- field
- column
- itercolumns
- schema
- columns
- num_columns
- num_rows
- shape
- nbytes
- column_names
- slice
- filter
- flatten
- combine_chunks
- cast
- replace_schema_metadata
- add_column
- append_column
- remove_column
- set_column
- rename_columns
- select
- drop
- from_blocks
- from_tables

Utils Copy Chapter

[[autodoc]] datasets.table.concat_tables

[[autodoc]] datasets.table.list_table_cache_files

File: docs/source/package_reference/utilities.mdx Copy Chapter

Utilities

Configure logging Copy Chapter

🤗 Datasets strives to be transparent and explicit about how it works, but this can be quite verbose at times. We have included a series of logging methods which allow you to easily adjust the level of verbosity of the entire library. Currently the default verbosity of the library is set to WARNING.

To change the level of verbosity, use one of the direct setters. For instance, here is how to change the verbosity to the INFO level:

import datasets
datasets.logging.set_verbosity_info()

You can also use the environment variable DATASETS_VERBOSITY to override the default verbosity, and set it to one of the following: debug, info, warning, error, critical:

DATASETS_VERBOSITY=error ./myprogram.py

All the methods of this logging module are documented below. The main ones are:

• [logging.get_verbosity] to get the current level of verbosity in the logger
• [logging.set_verbosity] to set the verbosity to the level of your choice

In order from the least to the most verbose (with their corresponding int values):

1. logging.CRITICAL or logging.FATAL (int value, 50): only report the most critical errors.
2. logging.ERROR (int value, 40): only report errors.
3. logging.WARNING or logging.WARN (int value, 30): only reports error and warnings. This the default level used by the library.
4. logging.INFO (int value, 20): reports error, warnings and basic information.
5. logging.DEBUG (int value, 10): report all information.

[[autodoc]] datasets.logging.get_verbosity

[[autodoc]] datasets.logging.set_verbosity

[[autodoc]] datasets.logging.set_verbosity_info

[[autodoc]] datasets.logging.set_verbosity_warning

[[autodoc]] datasets.logging.set_verbosity_debug

[[autodoc]] datasets.logging.set_verbosity_error

[[autodoc]] datasets.logging.disable_propagation

[[autodoc]] datasets.logging.enable_propagation

Configure progress bars Copy Chapter

By default, tqdm progress bars will be displayed during dataset download and preprocessing. You can disable them globally by setting HF_DATASETS_DISABLE_PROGRESS_BARS
environment variable. You can also enable/disable them using [~utils.enable_progress_bars] and [~utils.disable_progress_bars]. If set, the environment variable has priority on the helpers.

[[autodoc]] datasets.utils.enable_progress_bars

[[autodoc]] datasets.utils.disable_progress_bars

[[autodoc]] datasets.utils.are_progress_bars_disabled

File: docs/source/about_arrow.md Copy Chapter

Datasets 🤝 Arrow

What is Arrow? Copy Chapter

Arrow enables large amounts of data to be processed and moved quickly. It is a specific data format that stores data in a columnar memory layout. This provides several significant advantages:

• Arrow's standard format allows zero-copy reads which removes virtually all serialization overhead.
• Arrow is language-agnostic so it supports different programming languages.
• Arrow is column-oriented so it is faster at querying and processing slices or columns of data.
• Arrow allows for copy-free hand-offs to standard machine learning tools such as NumPy, Pandas, PyTorch, and TensorFlow.
• Arrow supports many, possibly nested, column types.

Memory-mapping Copy Chapter

🤗 Datasets uses Arrow for its local caching system. It allows datasets to be backed by an on-disk cache, which is memory-mapped for fast lookup.
This architecture allows for large datasets to be used on machines with relatively small device memory.

For example, loading the full English Wikipedia dataset only takes a few MB of RAM:

>>> import os; import psutil; import timeit
>>> from datasets import load_dataset

# Process.memory_info is expressed in bytes, so convert to megabytes 
>>> mem_before = psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024)
>>> wiki = load_dataset("wikimedia/wikipedia", "20220301.en", split="train")
>>> mem_after = psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024)

>>> print(f"RAM memory used: {(mem_after - mem_before)} MB")
RAM memory used: 50 MB

This is possible because the Arrow data is actually memory-mapped from disk, and not loaded in memory.
Memory-mapping allows access to data on disk, and leverages virtual memory capabilities for fast lookups.

Performance Copy Chapter

Iterating over a memory-mapped dataset using Arrow is fast. Iterating over Wikipedia on a laptop gives you speeds of 1-3 Gbit/s:

>>> s = """batch_size = 1000
... for batch in wiki.iter(batch_size):
...     ...
... """

>>> elapsed_time = timeit.timeit(stmt=s, number=1, globals=globals())
>>> print(f"Time to iterate over the {wiki.dataset_size >> 30} GB dataset: {elapsed_time:.1f} sec, "
...       f"ie. {float(wiki.dataset_size >> 27)/elapsed_time:.1f} Gb/s")
Time to iterate over the 18 GB dataset: 31.8 sec, ie. 4.8 Gb/s

File: docs/source/about_cache.mdx Copy Chapter

The cache

The cache is one of the reasons why 🤗 Datasets is so efficient. It stores previously downloaded and processed datasets so when you need to use them again, they are reloaded directly from the cache. This avoids having to download a dataset all over again, or reapplying processing functions. Even after you close and start another Python session, 🤗 Datasets will reload your dataset directly from the cache!

Fingerprint Copy Chapter

How does the cache keeps track of what transforms are applied to a dataset? Well, 🤗 Datasets assigns a fingerprint to the cache file. A fingerprint keeps track of the current state of a dataset. The initial fingerprint is computed using a hash from the Arrow table, or a hash of the Arrow files if the dataset is on disk. Subsequent fingerprints are computed by combining the fingerprint of the previous state, and a hash of the latest transform applied.

[!TIP]
Transforms are any of the processing methods from the How-to Process guides such as [Dataset.map] or [Dataset.shuffle].

Here are what the actual fingerprints look like:

>>> from datasets import Dataset
>>> dataset1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> dataset2 = dataset1.map(lambda x: {"a": x["a"] + 1})
>>> print(dataset1._fingerprint, dataset2._fingerprint)
d19493523d95e2dc 5b86abacd4b42434

In order for a transform to be hashable, it needs to be picklable by dill or pickle.

When you use a non-hashable transform, 🤗 Datasets uses a random fingerprint instead and raises a warning. The non-hashable transform is considered different from the previous transforms. As a result, 🤗 Datasets will recompute all the transforms. Make sure your transforms are serializable with pickle or dill to avoid this!

An example of when 🤗 Datasets recomputes everything is when caching is disabled. When this happens, the cache files are generated every time and they get written to a temporary directory. Once your Python session ends, the cache files in the temporary directory are deleted. A random hash is assigned to these cache files, instead of a fingerprint.

[!TIP]
When caching is disabled, use [Dataset.save_to_disk] to save your transformed dataset or it will be deleted once the session ends.

Hashing Copy Chapter

The fingerprint of a dataset is updated by hashing the function passed to map as well as the map parameters (batch_size, remove_columns, etc.).

You can check the hash of any Python object using the [fingerprint.Hasher]:

>>> from datasets.fingerprint import Hasher
>>> my_func = lambda example: {"length": len(example["text"])}
>>> print(Hasher.hash(my_func))
'3d35e2b3e94c81d6'

The hash is computed by dumping the object using a dill pickler and hashing the dumped bytes.
The pickler recursively dumps all the variables used in your function, so any change you do to an object that is used in your function, will cause the hash to change.

If one of your functions doesn't seem to have the same hash across sessions, it means at least one of its variables contains a Python object that is not deterministic.
When this happens, feel free to hash any object you find suspicious to try to find the object that caused the hash to change.
For example, if you use a list for which the order of its elements is not deterministic across sessions, then the hash won't be the same across sessions either.

File: docs/source/about_dataset_features.mdx Copy Chapter

Dataset features

[Features] defines the internal structure of a dataset. It is used to specify the underlying serialization format. What's more interesting to you though is that [Features] contains high-level information about everything from the column names and types, to the [ClassLabel]. You can think of [Features] as the backbone of a dataset.

The [Features] format is simple: dict[column_name, column_type]. It is a dictionary of column name and column type pairs. The column type provides a wide range of options for describing the type of data you have.

Let's have a look at the features of the MRPC dataset from the GLUE benchmark:

>>> from datasets import load_dataset
>>> dataset = load_dataset('nyu-mll/glue', 'mrpc', split='train')
>>> dataset.features
{'idx': Value('int32'),
 'label': ClassLabel(names=['not_equivalent', 'equivalent']),
 'sentence1': Value('string'),
 'sentence2': Value('string'),
}

The [Value] feature tells 🤗 Datasets:

• The idx data type is int32.
• The sentence1 and sentence2 data types are string.

🤗 Datasets supports many other data types such as bool, float32 and binary to name just a few.

[!TIP]
Refer to [Value] for a full list of supported data types.

The [ClassLabel] feature informs 🤗 Datasets the label column contains two classes. The classes are labeled not_equivalent and equivalent. Labels are stored as integers in the dataset. When you retrieve the labels, [ClassLabel.int2str] and [ClassLabel.str2int] carries out the conversion from integer value to label name, and vice versa.

If your data type contains a list of objects, then you want to use the [List] feature. Remember the SQuAD dataset?

>>> from datasets import load_dataset
>>> dataset = load_dataset('rajpurkar/squad', split='train')
>>> dataset.features
{'id': Value('string'),
 'title': Value('string'),
 'context': Value('string'),
 'question': Value('string'),
 'answers': {'text': List(Value('string')),
  'answer_start': List(Value('int32'))}}

The answers field is constructed using the dict of features because and contains two subfields, text and answer_start, which are lists of string and int32, respectively.

[!TIP]
See the flatten section to learn how you can extract the nested subfields as their own independent columns.

The array feature type is useful for creating arrays of various sizes. You can create arrays with two dimensions using [Array2D], and even arrays with five dimensions using [Array5D].

>>> features = Features({'a': Array2D(shape=(1, 3), dtype='int32')})

The array type also allows the first dimension of the array to be dynamic. This is useful for handling sequences with variable lengths such as sentences, without having to pad or truncate the input to a uniform shape.

>>> features = Features({'a': Array3D(shape=(None, 5, 2), dtype='int32')})

Audio feature Copy Chapter

Audio datasets have a column with type [Audio], which contains three important fields:

• array: the decoded audio data represented as a 1-dimensional array.
• path: the path to the downloaded audio file.
• sampling_rate: the sampling rate of the audio data.

When you load an audio dataset and call the audio column, the [Audio] feature automatically decodes and resamples the audio file:

>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("PolyAI/minds14", "en-US", split="train")
>>> dataset[0]["audio"]
<datasets.features._torchcodec.AudioDecoder object at 0x11642b6a0>

[!WARNING]
Index into an audio dataset using the row index first and then the audio column - dataset[0]["audio"] - to avoid decoding and resampling all the audio files in the dataset. Otherwise, this can be a slow and time-consuming process if you have a large dataset.

With decode=False, the [Audio] type simply gives you the path or the bytes of the audio file, without decoding it into an torchcodec AudioDecoder object,

>>> dataset = load_dataset("PolyAI/minds14", "en-US", split="train").cast_column("audio", Audio(decode=False))
>>> dataset[0]
{'audio': {'bytes': None,
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav'},
 'english_transcription': 'I would like to set up a joint account with my partner',
 'intent_class': 11,
 'lang_id': 4,
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav',
 'transcription': 'I would like to set up a joint account with my partner'}

Image feature Copy Chapter

Image datasets have a column with type [Image], which loads PIL.Image objects from images stored as bytes:

When you load an image dataset and call the image column, the [Image] feature automatically decodes the image file:

>>> from datasets import load_dataset, Image

>>> dataset = load_dataset("AI-Lab-Makerere/beans", split="train")
>>> dataset[0]["image"]

[!WARNING]
Index into an image dataset using the row index first and then the image column - dataset[0]["image"] - to avoid decoding all the image files in the dataset. Otherwise, this can be a slow and time-consuming process if you have a large dataset.

With decode=False, the [Image] type simply gives you the path or the bytes of the image file, without decoding it into an PIL.Image,

>>> dataset = load_dataset("AI-Lab-Makerere/beans", split="train").cast_column("image", Image(decode=False))
>>> dataset[0]["image"]
{'bytes': None,
 'path': '/Users/username/.cache/huggingface/datasets/downloads/extracted/772e7c1fba622cff102b85dd74bcce46e8168634df4eaade7bedd3b8d91d3cd7/train/healthy/healthy_train.265.jpg'}

Depending on the dataset, you may get the path to the local downloaded image, or the content of the image as bytes if the dataset is not made of individual files.

You can also define a dataset of images from numpy arrays:

>>> ds = Dataset.from_dict({"i": [np.zeros(shape=(16, 16, 3), dtype=np.uint8)]}, features=Features({"i": Image()}))

And in this case the numpy arrays are encoded into PNG (or TIFF if the pixels values precision is important).

For multi-channels arrays like RGB or RGBA, only uint8 is supported. If you use a larger precision, you get a warning and the array is downcasted to uint8.
For gray-scale images you can use the integer or float precision you want as long as it is compatible with Pillow. A warning is shown if your image integer or float precision is too high, and in this case the array is downcated: an int64 array is downcasted to int32, and a float64 array is downcasted to float32.

Mesh feature Copy Chapter

Mesh datasets have a column with type [Mesh], which loads 3D mesh files as trimesh objects.

When you load a mesh dataset and call the mesh column, the [Mesh] feature automatically decodes the mesh file:

>>> from datasets import load_dataset, Mesh

>>> dataset = load_dataset("VINAY-UMRETHE/My-Mesh-Dataset", split="train")
>>> dataset[0]["mesh"]

Depending on the file content, trimesh may return a trimesh.Trimesh object or a trimesh.Scene object.

[!WARNING]
Index into a mesh dataset using the row index first and then the mesh column - dataset[0]["mesh"] - to avoid decoding all the mesh files in the dataset. Otherwise, this can be a slow and time-consuming process if you have a large dataset.

With decode=False, the [Mesh] type gives you the path or bytes of the mesh file without decoding it with trimesh:

>>> dataset = load_dataset("VINAY-UMRETHE/My-Mesh-Dataset", split="train").cast_column("mesh", Mesh(decode=False))
>>> dataset[0]["mesh"]
{'bytes': b'...',
 'path': '00001.glb'}

The [Mesh] feature supports .glb, .ply, and .stl files. For embedded bytes, the stored path is used to infer the mesh file type.

Json feature Copy Chapter

Datasets are based on Arrow which is a columnar format, and therefore they expect every example to have the same type and subtypes, and dictionaries to have the same keys and values types.
Loading a dataset errors out when fields have mismatching types, and fills missing fields in dictionaries with None so all dictionaries have the same keys and value types.

To avoid this and allow mixed-types without errors, you can use on_mixed_types="use_json" or specify features= with a [Json] type:

>>> ds = Dataset.from_dict({"a": [0, "foo", {"subfield": "bar"}]})
Traceback (most recent call last):
  ...
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert 'foo' with type str: tried to convert to int64

>>> features = Features({"a": Json()})
>>> ds = Dataset.from_dict({"a": [0, "foo", {"subfield": "bar"}]}, features=features)
>>> ds.features
{'a': Json()}
>>> list(ds["a"])
[0, "foo", {"subfield": "bar"}]

This is also useful for lists of dictionaries with arbitrary keys and values, to avoid filling missing fields with None:

>>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]})
>>> ds.features
{'a': List({'b': Value('int64'), 'c': Value('int64')})}
>>> list(ds["a"])
[[{'b': 0, 'c': None}, {'b': None, 'c': 0}]]  # missing fields are filled with None

>>> features = Features({"a": List(Json())})
>>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]}, features=features)
>>> ds.features
{'a': List(Json())}
>>> list(ds["a"])
[[{'b': 0}, {'c': 0}]]  # OK

Another example with tool calling data and the on_mixed_types="use_json" argument (useful to not have to specify features= manually):

>>> messages = [
...     {"role": "user", "content": "Turn on the living room lights and play my electronic music playlist."},
...     {"role": "assistant", "tool_calls": [
...         {"type": "function", "function": {
...             "name": "control_light",
...             "arguments": {"room": "living room", "state": "on"}
...         }},
...         {"type": "function", "function": {
...             "name": "play_music",
...             "arguments": {"playlist": "electronic"}  # mixed-type here since keys ["playlist"] and ["room", "state"] are different
...         }}]
...     },
...     {"role": "tool", "name": "control_light", "content": "The lights in the living room are now on."},
...     {"role": "tool", "name": "play_music", "content": "The music is now playing."},
...     {"role": "assistant", "content": "Done!"}
... ]
>>> ds = Dataset.from_dict({"messages": [messages]}, on_mixed_types="use_json")
>>> ds.features
{'messages': List({'role': Value('string'), 'content': Value('string'), 'tool_calls': List(Json()), 'name': Value('string')})}
>>> ds[0][1]["tool_calls"][0]["function"]["arguments"]
{"room": "living room", "state": "on"}

File: docs/source/about_dataset_load.mdx Copy Chapter

Build and load

Nearly every deep learning workflow begins with loading a dataset, which makes it one of the most important steps. With 🤗 Datasets, there are more than 900 datasets available to help you get started with your NLP task. All you have to do is call: [load_dataset] to take your first step. This function is a true workhorse in every sense because it builds and loads every dataset you use.

ELI5: `load_dataset` Copy Chapter

Let's begin with a basic Explain Like I'm Five.

A dataset is a directory that contains:

• Some data files in generic formats (JSON, CSV, Parquet, text, etc.)
• A dataset card named README.md that contains documentation about the dataset as well as a YAML header to define the datasets tags and configurations

The [load_dataset] function fetches the requested dataset locally or from the Hugging Face Hub.
The Hub is a central repository where all the Hugging Face datasets and models are stored.

If the dataset only contains data files, then [load_dataset] automatically infers how to load the data files from their extensions (json, csv, parquet, tsfile, txt, etc.).
Under the hood, 🤗 Datasets will use an appropriate [DatasetBuilder] based on the data files format. There exist one builder per data file format in 🤗 Datasets:

• [datasets.packaged_modules.text.Text] for text
• [datasets.packaged_modules.csv.Csv] for CSV and TSV
• [datasets.packaged_modules.json.Json] for JSON and JSONL
• [datasets.packaged_modules.parquet.Parquet] for Parquet
• [datasets.packaged_modules.arrow.Arrow] for Arrow (streaming file format)
• [datasets.packaged_modules.sql.Sql] for SQL databases
• [datasets.packaged_modules.tsfile.TsFile] for TsFile (time-series data)
• [datasets.packaged_modules.imagefolder.ImageFolder] for image folders
• [datasets.packaged_modules.audiofolder.AudioFolder] for audio folders

[!TIP]
Read the Share section to learn more about how to share a dataset.

🤗 Datasets downloads the dataset files from the original URL, generates the dataset and caches it in an Arrow table on your drive.
If you've downloaded the dataset before, then 🤗 Datasets will reload it from the cache to save you the trouble of downloading it again.

Now that you have a high-level understanding about how datasets are built, let's take a closer look at the nuts and bolts of how all this works.

Building a dataset Copy Chapter

When you load a dataset for the first time, 🤗 Datasets takes the raw data file and builds it into a table of rows and typed columns. There are two main classes responsible for building a dataset: [BuilderConfig] and [DatasetBuilder].

BuilderConfig[[datasets-builderconfig]] Copy Section

[BuilderConfig] is the configuration class of [DatasetBuilder]. The [BuilderConfig] contains the following basic attributes about a dataset:

If you want to add additional attributes to your dataset such as the class labels, you can subclass the base [BuilderConfig] class. There are two ways to populate the attributes of a [BuilderConfig] class or subclass:

• Provide a list of predefined [BuilderConfig] class (or subclass) instances in the datasets [DatasetBuilder.BUILDER_CONFIGS] attribute.

• When you call [load_dataset], any keyword arguments that are not specific to the method will be used to set the associated attributes of the [BuilderConfig] class. This will override the predefined attributes if a specific configuration was selected.

You can also set the [DatasetBuilder.BUILDER_CONFIG_CLASS] to any custom subclass of [BuilderConfig].

DatasetBuilder[[datasets-datasetbuilder]] Copy Section

[DatasetBuilder] accesses all the attributes inside [BuilderConfig] to build the actual dataset.

There are three main methods in [DatasetBuilder]:

1. [DatasetBuilder._info] is in charge of defining the dataset attributes. When you call dataset.info, 🤗 Datasets returns the information stored here. Likewise, the [Features] are also specified here. Remember, the [Features] are like the skeleton of the dataset. It provides the names and types of each column.

2. [DatasetBuilder._split_generator] downloads or retrieves the requested data files, organizes them into splits, and defines specific arguments for the generation process. This method has a [DownloadManager] that downloads files or fetches them from your local filesystem. Within the [DownloadManager], there is a [DownloadManager.download_and_extract] method that accepts a dictionary of URLs to the original data files, and downloads the requested files. Accepted inputs include: a single URL or path, or a list/dictionary of URLs or paths. Any compressed file types like TAR, GZIP and ZIP archives will be automatically extracted.

   Once the files are downloaded, [SplitGenerator] organizes them into splits. The [SplitGenerator] contains the name of the split, and any keyword arguments that are provided to the [DatasetBuilder._generate_examples] method. The keyword arguments can be specific to each split, and typically comprise at least the local path to the data files for each split.

3. [DatasetBuilder._generate_examples] reads and parses the data files for a split. Then it yields dataset examples according to the format specified in the features from [DatasetBuilder._info]. The input of [DatasetBuilder._generate_examples] is actually the filepath provided in the keyword arguments of the last method.

   The dataset is generated with a Python generator, which doesn't load all the data in memory. As a result, the generator can handle large datasets. However, before the generated samples are flushed to the dataset file on disk, they are stored in an ArrowWriter buffer. This means the generated samples are written by batch. If your dataset samples consumes a lot of memory (images or videos), then make sure to specify a low value for the DEFAULT_WRITER_BATCH_SIZE attribute in [DatasetBuilder]. We recommend not exceeding a size of 200 MB.

Maintaining integrity Copy Chapter

To ensure a dataset is complete, [load_dataset] will perform a series of tests on the downloaded files to make sure everything is there. This way, you don't encounter any surprises when your requested dataset doesn't get generated as expected. [load_dataset] verifies:

• The number of splits in the generated DatasetDict.
• The number of samples in each split of the generated DatasetDict.
• The list of downloaded files.
• The SHA256 checksums of the downloaded files (disabled by default).

If the dataset doesn't pass the verifications, it is likely that the dataset author made some changes in the data files.

In this case, an error is raised to alert that the dataset has changed.
To ignore the error, one needs to specify verification_mode="no_checks" in [load_dataset].
Anytime you see a verification error, feel free to open a discussion or pull request in the corresponding dataset "Community" tab, so that the integrity checks for that dataset are updated.

Security Copy Chapter

The dataset repositories on the Hub are scanned for malware, see more information here.