## File: README.md

🤗 Datasets is a lightweight library providing **two** main features:

- **one-line dataloaders for many public datasets**: one-liners to download and pre-process any of the  major public datasets (image datasets, audio datasets, text datasets in 467 languages and dialects, 3D medical images, video datasets, agent traces, etc.) provided on the [HuggingFace Datasets Hub](https://huggingface.co/datasets). With a simple command like `squad_dataset = load_dataset("rajpurkar/squad")`, get any of these datasets ready to use in a dataloader for training/evaluating a ML model (Numpy/Pandas/PyTorch/TensorFlow/JAX/Polars),
- **efficient data pre-processing**: simple, fast and reproducible data pre-processing for the public datasets as well as your own local datasets in CSV, JSON, JSONL, Parquet, HDF5, XML, text, PNG, JPEG, WAV, MP3, PDF, NIfTI, and more. With simple commands like `processed_dataset = dataset.map(process_example)`, efficiently prepare the dataset for inspection and ML model evaluation and training.

[🎓 **Documentation**](https://huggingface.co/docs/datasets/) [🔎 **Find a dataset in the Hub**](https://huggingface.co/datasets) [🌟 **Share a dataset on the Hub**](https://huggingface.co/docs/datasets/share)

<h3 align="center">
    [](https://hf.co/course)
</h3>

# 🚀 Key Features

🤗 Datasets is designed to let the community easily add and share new datasets, and provides powerful capabilities for data manipulation:

| Feature | Description |
|---------|-------------|
| 📦 **One-line dataset loading** | Load AI-ready datasets from the [Hugging Face Hub](https://huggingface.co/datasets) or local files with `load_dataset()` |
| 🔍 **Multiple formats** | Native support for CSV, JSON, JSONL, Parquet, Arrow, XML, Text, Webdataset, and more |
| 🖼️ **Multi-modal data** | Built-in support for text, audio, image, video, PDF, and NIfTI (3D medical) data |
| 🚀 **Streaming mode** | Stream datasets without downloading — iterate over data on-the-fly with `streaming=True` (now up to **100x faster** with Xet backend) |
| 💾 **HF Storage Buckets** | Read and write directly from/to [Hugging Face Storage Buckets](https://huggingface.co/docs/hub/storage-buckets) for mutable, large-scale raw data |
| 🧠 **AI Agent Traces** | Load and process AI agent traces (prompts, tool calls, responses) from the Hub |
| ⚡ **Apache Arrow backend** | Zero-copy memory-mapped storage — datasets naturally free you from RAM limitations |
| 🔄 **Smart caching** | Never wait for your data to process twice — cached results are automatically reused |
| 📊 **Multi-framework interoperability** | Native conversion to/from NumPy, Pandas, Polars, Arrow, PyTorch, TensorFlow, JAX, and Spark |
| 🏎️ **Multi-processing** | Fast parallel data processing with `map(num_proc=N)` |
| 🔎 **Search & index** | Built-in FAISS and Elasticsearch index support for similarity search |
| 📦 **JSON type** | Flexible JSON/structured data support with `Json()` feature type |

# Installation

## With pip

🤗 Datasets can be installed from PyPi and should be installed in a virtual environment (venv or conda for instance):

```bash
pip install datasets
```

For the latest development version:

```bash
pip install "datasets @ git+https://github.com/huggingface/datasets.git"
```

## With conda

```bash
conda install -c huggingface -c conda-forge datasets
```

## Optional dependencies

🤗 Datasets supports various optional features via extras:

```bash
# For audio (torchcodec)
pip install datasets[audio]

# For image/video (Pillow, torchcodec)
pip install datasets[vision]

# For PDFs/NIfTI (pdfplumber, nibabel)
pip install datasets[pdfs,nibabel]

# For PyTorch/TensorFlow/JAX integration
pip install datasets[torch,tensorflow,jax]

```

For more details on installation, check the [installation page](https://huggingface.co/docs/datasets/installation).

# Quick Start

🤗 Datasets is made to be very simple to use — the API is centered around a single function, `datasets.load_dataset(dataset_name, **kwargs)`, that instantiates a dataset.

Here is a quick example:

```python
from datasets import load_dataset

# Load a dataset and print the first example in the training set
squad_dataset = load_dataset('rajpurkar/squad')
print(squad_dataset['train'][0])

# Process the dataset - add a column with the length of the context texts
dataset_with_length = squad_dataset.map(lambda x: {"length": len(x["context"])})

# Tokenize the context texts (using a tokenizer from the 🤗 Transformers library)
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')

tokenized_dataset = squad_dataset.map(lambda x: tokenizer(x['context']), batched=True)

# Tokenize chat conversations with a chat template (using a model that supports chat templates)
# This is useful for fine-tuning instruction/chat models

# Load a popular chat dataset (ultrachat_200k contains ~200k AI assistant conversations)
chat_dataset = load_dataset('HuggingFaceH4/ultrachat_200k', split='train_sft')

chat_tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-7B-Instruct')

def tokenize_chat(examples):
    # Apply the chat template and tokenize in one step
    return chat_tokenizer.apply_chat_template(examples["messages"])

tokenized_chat_dataset = chat_dataset.map(tokenize_chat, batched=True)
```

## Streaming mode

If your dataset is bigger than your disk or if you don't want to wait to download the data, you can use streaming:

```python
# Stream the dataset without downloading anything
image_dataset = load_dataset('timm/imagenet-1k-wds', streaming=True)
for example in image_dataset["train"]:
    print(example["image"])
    break
```

## Multi-modal data

🤗 Datasets supports a wide variety of data types out of the box:

```python
# Audio dataset
dataset = load_dataset("openslr/librispeech_asr", "clean")

# Image dataset
dataset = load_dataset("ILSVRC/imagenet-1k")

# Video dataset
dataset = load_dataset("Shofo/shofo-tiktok-general-small")

# PDF documents
dataset = load_dataset("pixparse/pdfa-eng-wds")

# NIfTI (3D medical imaging)
dataset = load_dataset("dartbrains/localizer", "betas")
```

## From local files

```python
# Load from local CSV
dataset = load_dataset('csv', data_files='my_data.csv')

# Load from local Parquet
dataset = load_dataset('parquet', data_files='data/*.parquet')

# Load from a local directory (auto-detect format)
dataset = load_dataset('./path/to/data')
```

## From Python objects

```python
from datasets import Dataset

# From a dictionary
dataset = Dataset.from_dict({"text": ["Hello world", "How are you?"]})

# From a list
dataset = Dataset.from_list([{"text": "Hello world"}, {"text": "How are you?"}])

# From Pandas
import pandas as pd
df = pd.DataFrame({"col1": [1, 2, 3], "col2": ["a", "b", "c"]})
dataset = Dataset.from_pandas(df)

# From a generator
def gen():
    for i in range(10):
        yield {"value": i}
dataset = Dataset.from_generator(gen)
```

For more details on using the library, check the [quick start guide](https://huggingface.co/docs/datasets/quickstart) and the specific pages on:

- [Loading a dataset](https://huggingface.co/docs/datasets/loading)
- [What's in a Dataset](https://huggingface.co/docs/datasets/access)
- [Processing data with 🤗 Datasets](https://huggingface.co/docs/datasets/process)
  - [Processing audio data](https://huggingface.co/docs/datasets/audio_process)
  - [Processing image data](https://huggingface.co/docs/datasets/image_process)
  - [Processing text data](https://huggingface.co/docs/datasets/nlp_process)
  - [Processing PDF data](https://huggingface.co/docs/datasets/pdf_process)
  - [Processing video data](https://huggingface.co/docs/datasets/video_process)
- [Streaming a dataset](https://huggingface.co/docs/datasets/stream)

# Core Classes

The library provides two main dataset classes:

| Class | Description |
|-------|-------------|
| `Dataset` | In-memory / memory-mapped dataset backed by Apache Arrow. Supports indexing, slicing, random access and caching. |
| `IterableDataset` | Lazy, streamable dataset for large-scale / out-of-core processing. Supports streaming and infinite iteration. |

Both are wrapped in `DatasetDict` / `IterableDatasetDict` for multi-split datasets (e.g., train/test/val).

# Add a new dataset to the Hub

We have a very detailed step-by-step guide to add a new dataset to the  datasets already provided on the [HuggingFace Datasets Hub](https://huggingface.co/datasets).

You can find:
- [how to upload a dataset to the Hub using your web browser or Python](https://huggingface.co/docs/datasets/upload_dataset) and also
- [how to upload it using Git](https://huggingface.co/docs/datasets/share).

# Disclaimers

You can use 🤗 Datasets to load datasets based on versioned git repositories maintained by the dataset authors. For reproducibility reasons, we ask users to pin the `revision` of the repositories they use.

If you're a dataset owner and wish to update any part of it (description, citation, license, etc.), or do not want your dataset to be included in the Hugging Face Hub, please get in touch by opening a discussion or a pull request in the Community tab of the dataset page. Thanks for your contribution to the ML community!

# Contributing

We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details on:

- How to submit issues and pull requests
- Code style guidelines (we use [Ruff](https://docs.astral.sh/ruff/))
- Testing requirements
- Documentation standards

# BibTeX

If you want to cite our 🤗 Datasets library, you can use our [paper](https://huggingface.co/papers/2109.02846):

```
/* Detailed source-code truncated for AI context efficiency. */
```

If you need to cite a specific version of our 🤗 Datasets library for reproducibility, you can use the corresponding version Zenodo DOI from this [list](https://zenodo.org/search?q=conceptrecid:%224817768%22&sort=-version&all_versions=True).

---

## File: docs/source/package_reference/builder_classes.mdx

# Builder classes

## Builders

🤗 Datasets relies on two main classes during the dataset building process: [`DatasetBuilder`] and [`BuilderConfig`].

[[autodoc]] datasets.DatasetBuilder

[[autodoc]] datasets.GeneratorBasedBuilder

[[autodoc]] datasets.ArrowBasedBuilder

[[autodoc]] datasets.BuilderConfig

## Download

[[autodoc]] datasets.DownloadManager

[[autodoc]] datasets.StreamingDownloadManager

[[autodoc]] datasets.DownloadConfig

[[autodoc]] datasets.DownloadMode

## Verification

[[autodoc]] datasets.VerificationMode

## Splits

[[autodoc]] datasets.SplitGenerator

[[autodoc]] datasets.Split

[[autodoc]] datasets.NamedSplit

[[autodoc]] datasets.NamedSplitAll

[[autodoc]] datasets.ReadInstruction

## Version

[[autodoc]] datasets.utils.Version

---

## File: docs/source/package_reference/loading_methods.mdx

# Loading methods

Methods for listing and loading datasets:

## Datasets

[[autodoc]] datasets.load_dataset

[[autodoc]] datasets.load_from_disk

[[autodoc]] datasets.load_dataset_builder

[[autodoc]] datasets.get_dataset_config_names

[[autodoc]] datasets.get_dataset_infos

[[autodoc]] datasets.get_dataset_split_names

## From files

Configurations used to load data files.
They are used when loading local files or a dataset repository:

- local files: `load_dataset("parquet", data_dir="path/to/data/dir")`
- dataset repository: `load_dataset("allenai/c4")`

You can pass arguments to `load_dataset` to configure data loading.
For example you can specify the `sep` parameter to define the [`~datasets.packaged_modules.csv.CsvConfig`] that is used to load the data:

```python
load_dataset("csv", data_dir="path/to/data/dir", sep="\t")
```

### Text

[[autodoc]] datasets.packaged_modules.text.TextConfig

[[autodoc]] datasets.packaged_modules.text.Text

### CSV

[[autodoc]] datasets.packaged_modules.csv.CsvConfig

[[autodoc]] datasets.packaged_modules.csv.Csv

### JSON

[[autodoc]] datasets.packaged_modules.json.JsonConfig

[[autodoc]] datasets.packaged_modules.json.Json

### XML

[[autodoc]] datasets.packaged_modules.xml.XmlConfig

[[autodoc]] datasets.packaged_modules.xml.Xml

### Parquet

[[autodoc]] datasets.packaged_modules.parquet.ParquetConfig

[[autodoc]] datasets.packaged_modules.parquet.Parquet

### Arrow

[[autodoc]] datasets.packaged_modules.arrow.ArrowConfig

[[autodoc]] datasets.packaged_modules.arrow.Arrow

### SQL

[[autodoc]] datasets.packaged_modules.sql.SqlConfig

[[autodoc]] datasets.packaged_modules.sql.Sql

### Images

[[autodoc]] datasets.packaged_modules.imagefolder.ImageFolderConfig

[[autodoc]] datasets.packaged_modules.imagefolder.ImageFolder

### Audio

[[autodoc]] datasets.packaged_modules.audiofolder.AudioFolderConfig

[[autodoc]] datasets.packaged_modules.audiofolder.AudioFolder

### Videos

[[autodoc]] datasets.packaged_modules.videofolder.VideoFolderConfig

[[autodoc]] datasets.packaged_modules.videofolder.VideoFolder

### HDF5

[[autodoc]] datasets.packaged_modules.hdf5.HDF5Config

[[autodoc]] datasets.packaged_modules.hdf5.HDF5

### TsFile

[[autodoc]] datasets.packaged_modules.tsfile.TsFileConfig

[[autodoc]] datasets.packaged_modules.tsfile.TsFile

### Pdf

[[autodoc]] datasets.packaged_modules.pdffolder.PdfFolderConfig

[[autodoc]] datasets.packaged_modules.pdffolder.PdfFolder

### Nifti

[[autodoc]] datasets.packaged_modules.niftifolder.NiftiFolderConfig

[[autodoc]] datasets.packaged_modules.niftifolder.NiftiFolder

### WebDataset

[[autodoc]] datasets.packaged_modules.webdataset.WebDataset

---

## File: docs/source/package_reference/main_classes.mdx

# Main classes


## DatasetInfo

[[autodoc]] datasets.DatasetInfo

## Dataset

The base class [`Dataset`] implements a Dataset backed by an Apache Arrow table.

[[autodoc]] datasets.Dataset
    - add_column
    - add_item
    - from_file
    - from_buffer
    - from_pandas
    - from_dict
    - from_list
    - from_generator
    - data
    - cache_files
    - num_columns
    - num_rows
    - column_names
    - shape
    - unique
    - flatten
    - cast
    - cast_column
    - remove_columns
    - rename_column
    - rename_columns
    - select_columns
    - class_encode_column
    - __len__
    - __iter__
    - iter
    - formatted_as
    - set_format
    - set_transform
    - reset_format
    - with_format
    - with_transform
    - __getitem__
    - cleanup_cache_files
    - map
    - filter
    - select
    - sort
    - shuffle
    - skip
    - take
    - train_test_split
    - shard
    - repeat
    - to_tf_dataset
    - push_to_hub
    - save_to_disk
    - load_from_disk
    - flatten_indices
    - to_csv
    - to_pandas
    - to_dict
    - to_json
    - to_parquet
    - to_sql
    - to_iterable_dataset
    - add_faiss_index
    - add_faiss_index_from_external_arrays
    - save_faiss_index
    - load_faiss_index
    - add_elasticsearch_index
    - load_elasticsearch_index
    - list_indexes
    - get_index
    - drop_index
    - search
    - search_batch
    - get_nearest_examples
    - get_nearest_examples_batch
    - info
    - split
    - builder_name
    - citation
    - config_name
    - dataset_size
    - description
    - download_checksums
    - download_size
    - features
    - homepage
    - license
    - size_in_bytes
    - supervised_keys
    - version
    - from_csv
    - from_json
    - from_parquet
    - from_text
    - from_sql
    - align_labels_with_mapping

[[autodoc]] datasets.concatenate_datasets

[[autodoc]] datasets.interleave_datasets

[[autodoc]] datasets.distributed.split_dataset_by_node

[[autodoc]] datasets.enable_caching

[[autodoc]] datasets.disable_caching

[[autodoc]] datasets.is_caching_enabled

[[autodoc]] datasets.Column

## DatasetDict

Dictionary with split names as keys ('train', 'test' for example), and `Dataset` objects as values.
It also has dataset transform methods like map or filter, to process all the splits at once.

[[autodoc]] datasets.DatasetDict
    - data
    - cache_files
    - num_columns
    - num_rows
    - column_names
    - shape
    - unique
    - cleanup_cache_files
    - map
    - filter
    - sort
    - shuffle
    - set_format
    - reset_format
    - formatted_as
    - with_format
    - with_transform
    - flatten
    - cast
    - cast_column
    - remove_columns
    - rename_column
    - rename_columns
    - select_columns
    - class_encode_column
    - push_to_hub
    - save_to_disk
    - load_from_disk
    - from_csv
    - from_json
    - from_parquet
    - from_text

<a id='package_reference_features'></a>

## IterableDataset

The base class [`IterableDataset`] implements an iterable Dataset backed by python generators.

[[autodoc]] datasets.IterableDataset
    - from_file
    - from_pandas
    - from_dict
    - from_list
    - from_generator
    - remove_columns
    - select_columns
    - cast_column
    - cast
    - decode
    - __iter__
    - iter
    - map
    - rename_column
    - filter
    - shuffle
    - batch
    - skip
    - take
    - shard
    - reshard
    - repeat
    - to_csv
    - to_pandas
    - to_dict
    - to_json
    - to_parquet
    - to_sql
    - push_to_hub
    - load_state_dict
    - state_dict
    - info
    - split
    - builder_name
    - citation
    - config_name
    - dataset_size
    - description
    - download_checksums
    - download_size
    - features
    - homepage
    - license
    - size_in_bytes
    - supervised_keys
    - version
    - from_csv
    - from_json
    - from_parquet
    - from_text

[[autodoc]] datasets.IterableColumn

## IterableDatasetDict

Dictionary with split names as keys ('train', 'test' for example), and `IterableDataset` objects as values.

[[autodoc]] datasets.IterableDatasetDict
    - map
    - filter
    - shuffle
    - with_format
    - cast
    - cast_column
    - remove_columns
    - rename_column
    - rename_columns
    - select_columns
    - push_to_hub

## Features

[[autodoc]] datasets.Features

### Scalar

[[autodoc]] datasets.Value

[[autodoc]] datasets.ClassLabel

### Composite

[[autodoc]] datasets.LargeList

[[autodoc]] datasets.List

[[autodoc]] datasets.Sequence

### Translation

[[autodoc]] datasets.Translation

[[autodoc]] datasets.TranslationVariableLanguages

### Arrays

[[autodoc]] datasets.Array2D

[[autodoc]] datasets.Array3D

[[autodoc]] datasets.Array4D

[[autodoc]] datasets.Array5D

### Audio

[[autodoc]] datasets.Audio

### Image

[[autodoc]] datasets.Image

### Video

[[autodoc]] datasets.Video

### Mesh

[[autodoc]] datasets.Mesh

### Json

[[autodoc]] datasets.Json

### Pdf

[[autodoc]] datasets.Pdf

### Nifti

[[autodoc]] datasets.Nifti

## Filesystems

[[autodoc]] datasets.filesystems.is_remote_filesystem

## Fingerprint

[[autodoc]] datasets.fingerprint.Hasher

---

## File: docs/source/package_reference/table_classes.mdx

# Table Classes

Each `Dataset` object is backed by a PyArrow Table.
A Table can be loaded from either the disk (memory mapped) or in memory.
Several Table types are available, and they all inherit from [`table.Table`].

## Table

[[autodoc]] datasets.table.Table
    - validate
    - equals
    - to_batches
    - to_pydict
    - to_pandas
    - to_string
    - field
    - column
    - itercolumns
    - schema
    - columns
    - num_columns
    - num_rows
    - shape
    - nbytes

## InMemoryTable

[[autodoc]] datasets.table.InMemoryTable
    - validate
    - equals
    - to_batches
    - to_pydict
    - to_pandas
    - to_string
    - field
    - column
    - itercolumns
    - schema
    - columns
    - num_columns
    - num_rows
    - shape
    - nbytes
    - column_names
    - slice
    - filter
    - flatten
    - combine_chunks
    - cast
    - replace_schema_metadata
    - add_column
    - append_column
    - remove_column
    - set_column
    - rename_columns
    - select
    - drop
    - from_file
    - from_buffer
    - from_pandas
    - from_arrays
    - from_pydict
    - from_batches

## MemoryMappedTable

[[autodoc]] datasets.table.MemoryMappedTable
    - validate
    - equals
    - to_batches
    - to_pydict
    - to_pandas
    - to_string
    - field
    - column
    - itercolumns
    - schema
    - columns
    - num_columns
    - num_rows
    - shape
    - nbytes
    - column_names
    - slice
    - filter
    - flatten
    - combine_chunks
    - cast
    - replace_schema_metadata
    - add_column
    - append_column
    - remove_column
    - set_column
    - rename_columns
    - select
    - drop
    - from_file

## ConcatenationTable

[[autodoc]] datasets.table.ConcatenationTable
    - validate
    - equals
    - to_batches
    - to_pydict
    - to_pandas
    - to_string
    - field
    - column
    - itercolumns
    - schema
    - columns
    - num_columns
    - num_rows
    - shape
    - nbytes
    - column_names
    - slice
    - filter
    - flatten
    - combine_chunks
    - cast
    - replace_schema_metadata
    - add_column
    - append_column
    - remove_column
    - set_column
    - rename_columns
    - select
    - drop
    - from_blocks
    - from_tables

## Utils

[[autodoc]] datasets.table.concat_tables

[[autodoc]] datasets.table.list_table_cache_files

---

## File: docs/source/package_reference/utilities.mdx

# Utilities

## Configure logging

🤗 Datasets strives to be transparent and explicit about how it works, but this can be quite verbose at times. We have included a series of logging methods which allow you to easily adjust the level of verbosity of the entire library. Currently the default verbosity of the library is set to `WARNING`.

To change the level of verbosity, use one of the direct setters. For instance, here is how to change the verbosity to the `INFO` level:

```py
import datasets
datasets.logging.set_verbosity_info()
```

You can also use the environment variable `DATASETS_VERBOSITY` to override the default verbosity, and set it to one of the following: `debug`, `info`, `warning`, `error`, `critical`:

```bash
DATASETS_VERBOSITY=error ./myprogram.py
```

All the methods of this logging module are documented below. The main ones are:

- [`logging.get_verbosity`] to get the current level of verbosity in the logger
- [`logging.set_verbosity`] to set the verbosity to the level of your choice

In order from the least to the most verbose (with their corresponding `int` values):

1. `logging.CRITICAL` or `logging.FATAL` (int value, 50): only report the most critical errors.
2. `logging.ERROR` (int value, 40): only report errors.
3. `logging.WARNING` or `logging.WARN` (int value, 30): only reports error and warnings. This the default level used by the library.
4. `logging.INFO` (int value, 20): reports error, warnings and basic information.
5. `logging.DEBUG` (int value, 10): report all information.

[[autodoc]] datasets.logging.get_verbosity

[[autodoc]] datasets.logging.set_verbosity

[[autodoc]] datasets.logging.set_verbosity_info

[[autodoc]] datasets.logging.set_verbosity_warning

[[autodoc]] datasets.logging.set_verbosity_debug

[[autodoc]] datasets.logging.set_verbosity_error

[[autodoc]] datasets.logging.disable_propagation

[[autodoc]] datasets.logging.enable_propagation

## Configure progress bars

By default, `tqdm` progress bars will be displayed during dataset download and preprocessing. You can disable them globally by setting `HF_DATASETS_DISABLE_PROGRESS_BARS`
environment variable. You can also enable/disable them using [`~utils.enable_progress_bars`] and [`~utils.disable_progress_bars`]. If set, the environment variable has priority on the helpers.

[[autodoc]] datasets.utils.enable_progress_bars

[[autodoc]] datasets.utils.disable_progress_bars

[[autodoc]] datasets.utils.are_progress_bars_disabled

---

## File: docs/source/about_arrow.md

# Datasets 🤝 Arrow

## What is Arrow?

[Arrow](https://arrow.apache.org/) enables large amounts of data to be processed and moved quickly. It is a specific data format that stores data in a columnar memory layout. This provides several significant advantages:

* Arrow's standard format allows [zero-copy reads](https://en.wikipedia.org/wiki/Zero-copy) which removes virtually all serialization overhead.
* Arrow is language-agnostic so it supports different programming languages.
* Arrow is column-oriented so it is faster at querying and processing slices or columns of data.
* Arrow allows for copy-free hand-offs to standard machine learning tools such as NumPy, Pandas, PyTorch, and TensorFlow.
* Arrow supports many, possibly nested, column types.

## Memory-mapping

🤗 Datasets uses Arrow for its local caching system. It allows datasets to be backed by an on-disk cache, which is memory-mapped for fast lookup.
This architecture allows for large datasets to be used on machines with relatively small device memory.

For example, loading the full English Wikipedia dataset only takes a few MB of RAM:

```python
>>> import os; import psutil; import timeit
>>> from datasets import load_dataset

# Process.memory_info is expressed in bytes, so convert to megabytes 
>>> mem_before = psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024)
>>> wiki = load_dataset("wikimedia/wikipedia", "20220301.en", split="train")
>>> mem_after = psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024)

>>> print(f"RAM memory used: {(mem_after - mem_before)} MB")
RAM memory used: 50 MB
```

This is possible because the Arrow data is actually memory-mapped from disk, and not loaded in memory.
Memory-mapping allows access to data on disk, and leverages virtual memory capabilities for fast lookups.

## Performance

Iterating over a memory-mapped dataset using Arrow is fast. Iterating over Wikipedia on a laptop gives you speeds of 1-3 Gbit/s:

```python
>>> s = """batch_size = 1000
... for batch in wiki.iter(batch_size):
...     ...
... """

>>> elapsed_time = timeit.timeit(stmt=s, number=1, globals=globals())
>>> print(f"Time to iterate over the {wiki.dataset_size >> 30} GB dataset: {elapsed_time:.1f} sec, "
...       f"ie. {float(wiki.dataset_size >> 27)/elapsed_time:.1f} Gb/s")
Time to iterate over the 18 GB dataset: 31.8 sec, ie. 4.8 Gb/s
```

---

## File: docs/source/about_cache.mdx

# The cache

The cache is one of the reasons why 🤗 Datasets is so efficient. It stores previously downloaded and processed datasets so when you need to use them again, they are reloaded directly from the cache. This avoids having to download a dataset all over again, or reapplying processing functions. Even after you close and start another Python session, 🤗 Datasets will reload your dataset directly from the cache!

## Fingerprint

How does the cache keeps track of what transforms are applied to a dataset? Well, 🤗 Datasets assigns a fingerprint to the cache file. A fingerprint keeps track of the current state of a dataset. The initial fingerprint is computed using a hash from the Arrow table, or a hash of the Arrow files if the dataset is on disk. Subsequent fingerprints are computed by combining the fingerprint of the previous state, and a hash of the latest transform applied. 

> [!TIP]
> Transforms are any of the processing methods from the [How-to Process](./process) guides such as [`Dataset.map`] or [`Dataset.shuffle`].

Here are what the actual fingerprints look like:

```py
>>> from datasets import Dataset
>>> dataset1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> dataset2 = dataset1.map(lambda x: {"a": x["a"] + 1})
>>> print(dataset1._fingerprint, dataset2._fingerprint)
d19493523d95e2dc 5b86abacd4b42434
```

In order for a transform to be hashable, it needs to be picklable by [dill](https://dill.readthedocs.io/en/latest/) or [pickle](https://docs.python.org/3/library/pickle). 

When you use a non-hashable transform, 🤗 Datasets uses a random fingerprint instead and raises a warning. The non-hashable transform is considered different from the previous transforms. As a result, 🤗 Datasets will recompute all the transforms. Make sure your transforms are serializable with pickle or dill to avoid this!

An example of when 🤗 Datasets recomputes everything is when caching is disabled. When this happens, the cache files are generated every time and they get written to a temporary directory. Once your Python session ends, the cache files in the temporary directory are deleted. A random hash is assigned to these cache files, instead of a fingerprint. 

> [!TIP]
> When caching is disabled, use [`Dataset.save_to_disk`] to save your transformed dataset or it will be deleted once the session ends.

## Hashing

The fingerprint of a dataset is updated by hashing the function passed to `map` as well as the `map` parameters (`batch_size`, `remove_columns`, etc.).

You can check the hash of any Python object using the [`fingerprint.Hasher`]:

```py
>>> from datasets.fingerprint import Hasher
>>> my_func = lambda example: {"length": len(example["text"])}
>>> print(Hasher.hash(my_func))
'3d35e2b3e94c81d6'
```

The hash is computed by dumping the object using a `dill` pickler and hashing the dumped bytes.
The pickler recursively dumps all the variables used in your function, so any change you do to an object that is used in your function, will cause the hash to change.

If one of your functions doesn't seem to have the same hash across sessions, it means at least one of its variables contains a Python object that is not deterministic.
When this happens, feel free to hash any object you find suspicious to try to find the object that caused the hash to change.
For example, if you use a list for which the order of its elements is not deterministic across sessions, then the hash won't be the same across sessions either.

---

## File: docs/source/about_dataset_features.mdx

# Dataset features

[`Features`] defines the internal structure of a dataset. It is used to specify the underlying serialization format. What's more interesting to you though is that [`Features`] contains high-level information about everything from the column names and types, to the [`ClassLabel`]. You can think of [`Features`] as the backbone of a dataset.

The [`Features`] format is simple: `dict[column_name, column_type]`. It is a dictionary of column name and column type pairs. The column type provides a wide range of options for describing the type of data you have.

Let's have a look at the features of the MRPC dataset from the GLUE benchmark:

```py
>>> from datasets import load_dataset
>>> dataset = load_dataset('nyu-mll/glue', 'mrpc', split='train')
>>> dataset.features
{'idx': Value('int32'),
 'label': ClassLabel(names=['not_equivalent', 'equivalent']),
 'sentence1': Value('string'),
 'sentence2': Value('string'),
}
```

The [`Value`] feature tells 🤗 Datasets:

- The `idx` data type is `int32`.
- The `sentence1` and `sentence2` data types are `string`.

🤗 Datasets supports many other data types such as `bool`, `float32` and `binary` to name just a few.

> [!TIP]
> Refer to [`Value`] for a full list of supported data types.

The [`ClassLabel`] feature informs 🤗 Datasets the `label` column contains two classes. The classes are labeled `not_equivalent` and `equivalent`. Labels are stored as integers in the dataset. When you retrieve the labels, [`ClassLabel.int2str`] and [`ClassLabel.str2int`] carries out the conversion from integer value to label name, and vice versa.

If your data type contains a list of objects, then you want to use the [`List`] feature. Remember the SQuAD dataset?

```py
>>> from datasets import load_dataset
>>> dataset = load_dataset('rajpurkar/squad', split='train')
>>> dataset.features
{'id': Value('string'),
 'title': Value('string'),
 'context': Value('string'),
 'question': Value('string'),
 'answers': {'text': List(Value('string')),
  'answer_start': List(Value('int32'))}}
```

The `answers` field is constructed using the dict of features because and contains two subfields, `text` and `answer_start`, which are lists of `string` and `int32`, respectively.

> [!TIP]
> See the [flatten](./process#flatten) section to learn how you can extract the nested subfields as their own independent columns.

The array feature type is useful for creating arrays of various sizes. You can create arrays with two dimensions using [`Array2D`], and even arrays with five dimensions using [`Array5D`].

```py
>>> features = Features({'a': Array2D(shape=(1, 3), dtype='int32')})
```

The array type also allows the first dimension of the array to be dynamic. This is useful for handling sequences with variable lengths such as sentences, without having to pad or truncate the input to a uniform shape.

```py
>>> features = Features({'a': Array3D(shape=(None, 5, 2), dtype='int32')})
```

## Audio feature

Audio datasets have a column with type [`Audio`], which contains three important fields:

- `array`: the decoded audio data represented as a 1-dimensional array.
- `path`: the path to the downloaded audio file.
- `sampling_rate`: the sampling rate of the audio data.

When you load an audio dataset and call the audio column, the [`Audio`] feature automatically decodes and resamples the audio file:

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("PolyAI/minds14", "en-US", split="train")
>>> dataset[0]["audio"]
<datasets.features._torchcodec.AudioDecoder object at 0x11642b6a0>
```

> [!WARNING]
> Index into an audio dataset using the row index first and then the `audio` column - `dataset[0]["audio"]` - to avoid decoding and resampling all the audio files in the dataset. Otherwise, this can be a slow and time-consuming process if you have a large dataset.

With `decode=False`, the [`Audio`] type simply gives you the path or the bytes of the audio file, without decoding it into an torchcodec `AudioDecoder` object,

```py
>>> dataset = load_dataset("PolyAI/minds14", "en-US", split="train").cast_column("audio", Audio(decode=False))
>>> dataset[0]
{'audio': {'bytes': None,
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav'},
 'english_transcription': 'I would like to set up a joint account with my partner',
 'intent_class': 11,
 'lang_id': 4,
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav',
 'transcription': 'I would like to set up a joint account with my partner'}
```

## Image feature

Image datasets have a column with type [`Image`], which loads `PIL.Image` objects from images stored as bytes:

When you load an image dataset and call the image column, the [`Image`] feature automatically decodes the image file:

```py
>>> from datasets import load_dataset, Image

>>> dataset = load_dataset("AI-Lab-Makerere/beans", split="train")
>>> dataset[0]["image"]

```

> [!WARNING]
> Index into an image dataset using the row index first and then the `image` column - `dataset[0]["image"]` - to avoid decoding all the image files in the dataset. Otherwise, this can be a slow and time-consuming process if you have a large dataset.

With `decode=False`, the [`Image`] type simply gives you the path or the bytes of the image file, without decoding it into an `PIL.Image`,

```py
>>> dataset = load_dataset("AI-Lab-Makerere/beans", split="train").cast_column("image", Image(decode=False))
>>> dataset[0]["image"]
{'bytes': None,
 'path': '/Users/username/.cache/huggingface/datasets/downloads/extracted/772e7c1fba622cff102b85dd74bcce46e8168634df4eaade7bedd3b8d91d3cd7/train/healthy/healthy_train.265.jpg'}
```

Depending on the dataset, you may get the path to the local downloaded image, or the content of the image as bytes if the dataset is not made of individual files.

You can also define a dataset of images from numpy arrays:

```python
>>> ds = Dataset.from_dict({"i": [np.zeros(shape=(16, 16, 3), dtype=np.uint8)]}, features=Features({"i": Image()}))
```

And in this case the numpy arrays are encoded into PNG (or TIFF if the pixels values precision is important).

For multi-channels arrays like RGB or RGBA, only uint8 is supported. If you use a larger precision, you get a warning and the array is downcasted to uint8.
For gray-scale images you can use the integer or float precision you want as long as it is compatible with `Pillow`. A warning is shown if your image integer or float precision is too high, and in this case the array is downcated: an int64 array is downcasted to int32, and a float64 array is downcasted to float32.

## Mesh feature

Mesh datasets have a column with type [`Mesh`], which loads 3D mesh files as `trimesh` objects.

When you load a mesh dataset and call the mesh column, the [`Mesh`] feature automatically decodes the mesh file:

```py
>>> from datasets import load_dataset, Mesh

>>> dataset = load_dataset("VINAY-UMRETHE/My-Mesh-Dataset", split="train")
>>> dataset[0]["mesh"]

```

Depending on the file content, `trimesh` may return a `trimesh.Trimesh` object or a `trimesh.Scene` object.

> [!WARNING]
> Index into a mesh dataset using the row index first and then the `mesh` column - `dataset[0]["mesh"]` - to avoid decoding all the mesh files in the dataset. Otherwise, this can be a slow and time-consuming process if you have a large dataset.

With `decode=False`, the [`Mesh`] type gives you the path or bytes of the mesh file without decoding it with `trimesh`:

```py
>>> dataset = load_dataset("VINAY-UMRETHE/My-Mesh-Dataset", split="train").cast_column("mesh", Mesh(decode=False))
>>> dataset[0]["mesh"]
{'bytes': b'...',
 'path': '00001.glb'}
```

The [`Mesh`] feature supports `.glb`, `.ply`, and `.stl` files. For embedded bytes, the stored `path` is used to infer the mesh file type.

## Json feature

Datasets are based on Arrow which is a columnar format, and therefore they expect every example to have the same type and subtypes, and dictionaries to have the same keys and values types.
Loading a dataset errors out when fields have mismatching types, and fills missing fields in dictionaries with None so all dictionaries have the same keys and value types.

To avoid this and allow mixed-types without errors, you can use `on_mixed_types="use_json"` or specify `features=` with a [`Json`] type:

```python
>>> ds = Dataset.from_dict({"a": [0, "foo", {"subfield": "bar"}]})
Traceback (most recent call last):
  ...
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert 'foo' with type str: tried to convert to int64

>>> features = Features({"a": Json()})
>>> ds = Dataset.from_dict({"a": [0, "foo", {"subfield": "bar"}]}, features=features)
>>> ds.features
{'a': Json()}
>>> list(ds["a"])
[0, "foo", {"subfield": "bar"}]
```

This is also useful for lists of dictionaries with arbitrary keys and values, to avoid filling missing fields with None:

```python
>>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]})
>>> ds.features
{'a': List({'b': Value('int64'), 'c': Value('int64')})}
>>> list(ds["a"])
[[{'b': 0, 'c': None}, {'b': None, 'c': 0}]]  # missing fields are filled with None

>>> features = Features({"a": List(Json())})
>>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]}, features=features)
>>> ds.features
{'a': List(Json())}
>>> list(ds["a"])
[[{'b': 0}, {'c': 0}]]  # OK
```

Another example with tool calling data and the `on_mixed_types="use_json"` argument (useful to not have to specify `features=` manually):

```python
>>> messages = [
...     {"role": "user", "content": "Turn on the living room lights and play my electronic music playlist."},
...     {"role": "assistant", "tool_calls": [
...         {"type": "function", "function": {
...             "name": "control_light",
...             "arguments": {"room": "living room", "state": "on"}
...         }},
...         {"type": "function", "function": {
...             "name": "play_music",
...             "arguments": {"playlist": "electronic"}  # mixed-type here since keys ["playlist"] and ["room", "state"] are different
...         }}]
...     },
...     {"role": "tool", "name": "control_light", "content": "The lights in the living room are now on."},
...     {"role": "tool", "name": "play_music", "content": "The music is now playing."},
...     {"role": "assistant", "content": "Done!"}
... ]
>>> ds = Dataset.from_dict({"messages": [messages]}, on_mixed_types="use_json")
>>> ds.features
{'messages': List({'role': Value('string'), 'content': Value('string'), 'tool_calls': List(Json()), 'name': Value('string')})}
>>> ds[0][1]["tool_calls"][0]["function"]["arguments"]
{"room": "living room", "state": "on"}
```

---

## File: docs/source/about_dataset_load.mdx

# Build and load

Nearly every deep learning workflow begins with loading a dataset, which makes it one of the most important steps. With 🤗 Datasets, there are more than 900 datasets available to help you get started with your NLP task. All you have to do is call: [`load_dataset`] to take your first step. This function is a true workhorse in every sense because it builds and loads every dataset you use.

## ELI5: `load_dataset`

Let's begin with a basic Explain Like I'm Five.

A dataset is a directory that contains:

- Some data files in generic formats (JSON, CSV, Parquet, text, etc.)
- A dataset card named `README.md` that contains documentation about the dataset as well as a YAML header to define the datasets tags and configurations

The [`load_dataset`] function fetches the requested dataset locally or from the Hugging Face Hub.
The Hub is a central repository where all the Hugging Face datasets and models are stored.

If the dataset only contains data files, then [`load_dataset`] automatically infers how to load the data files from their extensions (json, csv, parquet, tsfile, txt, etc.).
Under the hood, 🤗 Datasets will use an appropriate [`DatasetBuilder`] based on the data files format. There exist one builder per data file format in 🤗 Datasets:

* [`datasets.packaged_modules.text.Text`] for text
* [`datasets.packaged_modules.csv.Csv`] for CSV and TSV
* [`datasets.packaged_modules.json.Json`] for JSON and JSONL
* [`datasets.packaged_modules.parquet.Parquet`] for Parquet
* [`datasets.packaged_modules.arrow.Arrow`] for Arrow (streaming file format)
* [`datasets.packaged_modules.sql.Sql`] for SQL databases
* [`datasets.packaged_modules.tsfile.TsFile`] for TsFile (time-series data)
* [`datasets.packaged_modules.imagefolder.ImageFolder`] for image folders
* [`datasets.packaged_modules.audiofolder.AudioFolder`] for audio folders

> [!TIP]
> Read the [Share](./upload_dataset) section to learn more about how to share a dataset.

🤗 Datasets downloads the dataset files from the original URL, generates the dataset and caches it in an Arrow table on your drive.
If you've downloaded the dataset before, then 🤗 Datasets will reload it from the cache to save you the trouble of downloading it again.

Now that you have a high-level understanding about how datasets are built, let's take a closer look at the nuts and bolts of how all this works.

## Building a dataset

When you load a dataset for the first time, 🤗 Datasets takes the raw data file and builds it into a table of rows and typed columns. There are two main classes responsible for building a dataset: [`BuilderConfig`] and [`DatasetBuilder`].



   


### BuilderConfig[[datasets-builderconfig]]

[`BuilderConfig`] is the configuration class of [`DatasetBuilder`]. The [`BuilderConfig`] contains the following basic attributes about a dataset:

| Attribute     | Description                                                  |
|---------------|--------------------------------------------------------------|
| `name`        | Short name of the dataset.                                   |
| `version`     | Dataset version identifier.                                  |
| `data_dir`    | Stores the path to a local folder containing the data files. |
| `data_files`  | Stores paths to local data files.                            |
| `description` | Description of the dataset.                                  |

If you want to add additional attributes to your dataset such as the class labels, you can subclass the base [`BuilderConfig`] class. There are two ways to populate the attributes of a [`BuilderConfig`] class or subclass:

- Provide a list of predefined [`BuilderConfig`] class (or subclass) instances in the datasets [`DatasetBuilder.BUILDER_CONFIGS`] attribute.

- When you call [`load_dataset`], any keyword arguments that are not specific to the method will be used to set the associated attributes of the [`BuilderConfig`] class. This will override the predefined attributes if a specific configuration was selected.

You can also set the [`DatasetBuilder.BUILDER_CONFIG_CLASS`] to any custom subclass of [`BuilderConfig`].

### DatasetBuilder[[datasets-datasetbuilder]]

[`DatasetBuilder`] accesses all the attributes inside [`BuilderConfig`] to build the actual dataset.


   


There are three main methods in [`DatasetBuilder`]:

1. [`DatasetBuilder._info`] is in charge of defining the dataset attributes. When you call `dataset.info`, 🤗 Datasets returns the information stored here. Likewise, the [`Features`] are also specified here. Remember, the [`Features`] are like the skeleton of the dataset. It provides the names and types of each column.

2. [`DatasetBuilder._split_generator`] downloads or retrieves the requested data files, organizes them into splits, and defines specific arguments for the generation process. This method has a [`DownloadManager`] that downloads files or fetches them from your local filesystem. Within the [`DownloadManager`], there is a [`DownloadManager.download_and_extract`] method that accepts a dictionary of URLs to the original data files, and downloads the requested files. Accepted inputs include: a single URL or path, or a list/dictionary of URLs or paths. Any compressed file types like TAR, GZIP and ZIP archives will be automatically extracted.

   Once the files are downloaded, [`SplitGenerator`] organizes them into splits. The [`SplitGenerator`] contains the name of the split, and any keyword arguments that are provided to the [`DatasetBuilder._generate_examples`] method. The keyword arguments can be specific to each split, and typically comprise at least the local path to the data files for each split.

3. [`DatasetBuilder._generate_examples`] reads and parses the data files for a split. Then it yields dataset examples according to the format specified in the `features` from [`DatasetBuilder._info`]. The input of [`DatasetBuilder._generate_examples`] is actually the `filepath` provided in the keyword arguments of the last method.

   The dataset is generated with a Python generator, which doesn't load all the data in memory. As a result, the generator can handle large datasets. However, before the generated samples are flushed to the dataset file on disk, they are stored in an `ArrowWriter` buffer. This means the generated samples are written by batch. If your dataset samples consumes a lot of memory (images or videos), then make sure to specify a low value for the `DEFAULT_WRITER_BATCH_SIZE` attribute in [`DatasetBuilder`]. We recommend not exceeding a size of 200 MB.

## Maintaining integrity

To ensure a dataset is complete, [`load_dataset`] will perform a series of tests on the downloaded files to make sure everything is there. This way, you don't encounter any surprises when your requested dataset doesn't get generated as expected. [`load_dataset`] verifies:

- The number of splits in the generated `DatasetDict`.
- The number of samples in each split of the generated `DatasetDict`.
- The list of downloaded files.
- The SHA256 checksums of the downloaded files (disabled by default).

If the dataset doesn't pass the verifications, it is likely that the dataset author made some changes in the data files.

In this case, an error is raised to alert that the dataset has changed.
To ignore the error, one needs to specify `verification_mode="no_checks"` in [`load_dataset`].
Anytime you see a verification error, feel free to open a discussion or pull request in the corresponding dataset "Community" tab, so that the integrity checks for that dataset are updated.

## Security

The dataset repositories on the Hub are scanned for malware, see more information [here](https://huggingface.co/docs/hub/security#malware-scanning).