# Technical Documentation: Eventual-Inc/Daft > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/Eventual-Inc/Daft) ### Overview (`benchmarking/ai/README.md`) # AI Benchmarks This repository contains performance benchmarks comparing different data processing engines (Daft, Ray Data, and Spark) across various AI and multimodal data processing workloads. ## Overview The benchmarks cover four different workload types, each designed to test different aspects of multimodal data processing: 1. **Audio Transcription** - Transcribing audio files 2. **Document Embedding** - Generating embeddings for PDF documents 3. **Image Classification** - Classify images 4. **Video Object Detection** - Detect objects in videos ## Performance Results Summary | Workload | Data Size | Daft | Ray Data | Spark | |----------|-----------|------|----------|-------| | Audio Transcription | 113,800 audio files | 6m 22s | 29m 20s | 25m 46s | | Document Embedding | 10,000 PDFs | 1m 54s | 14m 32s | 8m 4s | | Image Classification | 803,580 images | 4m 23s | 23m 30s | 45m 7s | | Video Object Detection | 1,000 videos | 11m 46s | 25m 54s | 3h 36m | --- ### Overview (`benchmarking/asof_join/README.md`) ## Summary Adds a self-contained benchmarking suite for Daft's `join_asof` operation: - **`data_generation.py`** — generates reproducible left/right parquet datasets at three scales (`small`, `medium`, `large`) with clustered timestamps and Zipf-skewed entity distribution, written to `benchmarking/data/asof_join/` - **`benchmark.py`** — runs a single asof-join using Daft's native or Ray runner, wrapped in a memray memory tracker, and prints a JSON result with wall time and memray output path - **`cluster.yaml`** — Ray cluster config for AWS (1 `m7i.large` head + 4 `r7i.4xlarge` workers) for distributed runs against S3 ## How to run From inside `benchmarking/asof_join/`: **1. Generate data (one-time)** ```bash python data_generation.py --scale small # or --scale medium / --scale large / --all ``` **2. Run locally (native runner)** ```bash python benchmark.py --scale small # Output: asof_join_memray.bin + JSON result on stdout ``` **3. Inspect memory profile** ```bash memray flamegraph asof_join_memray.bin ``` **Run on a Ray cluster** > Before running: update `DATA_ROOT` to your S3 bucket and uncomment `daft.set_runner_ray()` in `benchmark.py`. Also update the S3 bucket and IAM settings in `cluster.yaml`. Spin up the cluster: ```bash ray up benchmarking/asof_join/cluster.yaml ``` Forward the dashboard in one terminal: ```bash ray dashboard benchmarking/asof_join/cluster.yaml ``` Submit the job in another (after updating `DATA_ROOT` and `daft.set_runner_ray()` in `benchmark.py`): ```bash ray job submit --address "http://localhost:8265" --working-dir benchmarking/asof_join -- python benchmark.py --scale small ``` Tear down when done: ```bash ray down benchmarking/asof_join/cluster.yaml ``` --- ### Overview (`benchmarking/lerobot/README.md`) # LeRobot video decode: per-frame → per-shard The `daft.datasets.lerobot` reader decoded video frames with a **per-row** UDF that re-opened the MP4 shard for every frame. Because `av.open()` on a remote shard re-reads and parses the container index over the network, decoding N frames re-opened the shard N times, paying that cost each time - so cost scaled ~linearly at **~3s/frame** (the slope of the sweep below). This directory holds the benchmarks that diagnosed it and the fix that makes the decode **batched**: rows sharing a shard within a batch are grouped so each shard is opened once per batch, instead of once per frame. ## The fix: batched decode `_decode_lerobot_video_timestamp` in [`daft/datasets/lerobot.py`](../../daft/datasets/lerobot.py) is now a `@daft.func.batch` UDF. Within each batch it groups rows by shard path, opens each shard once, and does a single forward decode assigning the closest frame to every requested timestamp. Output is **byte-identical** to the old per-row decode. ### Original vs batched (rows 1→10) [`sweep.py`](sweep.py) times an end-to-end `lerobot.read(...).limit(n).collect()` with frame decoding for n = 1..10 rows of a remote test dataset (`pepijn223/egodex-test`), run once per reader revision (merge-base vs this branch) - the original grows linearly to ~34s; the batched version stays flat at ~4s (all 10 frames share one shard → one open). All benchmarks in this directory were run on the same machine, an Apple M4 Max (36 GB). | rows | original | batched | | --- | --- | --- | | 1 | 4.2s | 4.4s | | 8 | **25.0s** | **3.9s** | | 10 | 34.4s | 3.9s | 8-frame output hashes matched exactly (`sha 80bdb30c…`) between versions. Beyond this test dataset, the fix was validated on six public LeRobot v3 datasets spanning av1/h264/mp4v, 5-30 fps, 128x128-1280x720, and 1-3 cameras - pixel-identical output everywhere, 4-13x faster - plus a full-dataset decode and a 100-frame comparison. See [real_datasets.md](real_datasets.md). ### Downstream workload: hand tracking [`hand_tracking.py`](hand_tracking.py) times the [daft-physical-ai demo](https://github.com/Eventual-Inc/daft-physical-ai/blob/main/examples/demo.py) workload end to end: decode 12 remote frames of `pepijn223/egodex-test` and run MediaPipe hand tracking as a Daft UDF. [`run_hand_tracking.sh`](run_hand_tracking.sh) runs it once per reader revision (the original pinned to the parent of the #7184 merge commit) and checks the detected hands match. Measured 2026-07-14 on an Apple M4 Max (36 GB, macOS 15.6), remote reads over `hf://`: **44.8s → 9.8s**, identical detections. ## Running ```bash python sweep.py --label batched # rows 1..10 sweep + chart ./run_hand_tracking.sh # hand-tracking workload A/B + chart ``` --- ### Overview (`benchmarking/parquet/README.md`) # Parquet Benchmarks Goals: 1. Find Parquet features that Daft underperforms on 2. Compare Daft against other frameworks ## Setup Create a new virtual environment and install the dependencies ```bash python -m venv .venv source .venv/bin/activate pip install -r benchmark-requirements.txt ``` Now, install the version of Daft you wish to use for benchmarking (either a released wheel, or if you want, a local build) ```bash pip install daft ``` ## Running the benchmarks: ### Goal 1: Find Parquet features that Daft underperforms on ```bash pytest benchmarking/parquet/ -m benchmark --benchmark-group-by=group -k daft ``` ### Goal 2: Compare Daft against other frameworks ```bash pytest benchmarking/parquet/ -m benchmark --benchmark-group-by=param:path ``` ### Check peak memory usage Ensure that you have `pytest-memray` installed. ```bash pytest benchmarking/parquet/ -m benchmark --memray ``` --- ### Overview (`benchmarking/vllm/README.md`) # vLLM Benchmarks This directory contains benchmarks for running batch inference with vLLM. ## Setup 1. Install the dependencies: - `vllm` - `daft` - `ray` 2. Run the `generate_data.ipynb` notebook to generate the datasets. 3. Update `config.py` with the correct dataset path and other parameters. 4. Run the benchmark scripts on a Ray cluster. ## Benchmarks - `naive-batch.py`: Simple batch inference using a Daft batch function and vLLM's `LLM` class. - `naive-batch-sorted.py`: Same as `naive-batch.py`, but with the prompts sorted. - `continuous-batch.py`: Continuous batching using the `vllm-prefix-caching` provider, with prefix routing disabled. - `continuous-batch-sorted.py`: Same as `continuous-batch.py`, but with the prompts sorted. - `prefix-bucketing.py`: Both continuous batching and prefix bucketing using the `vllm-prefix-caching` provider. - `ray-data.py`: Ray Data batching using `ray.data.llm.build_llm_processor`. --- ### Overview (`docs/plugins/README.md`) # Daft MkDocs Plugins Custom MkDocs plugins for the Daft documentation site. ## nav-hide-children A plugin that allows you to conditionally hide navigation children based on configured paths. ### Configuration Add to your `mkdocs.yml`: ```yaml plugins: - nav-hide-children: hide_children: - ["Python API", "Functions"] # Hide children of Python API > Functions - ["SQL Reference"] # Hide children of SQL Reference ``` ### Usage The plugin automatically injects an attribute `hide_children` into [navigation objects](https://www.mkdocs.org/dev-guide/themes/#navigation-objects) based on the config paths, which can be used in Jinja2 templates to conditionally hide navigation elements. --- ### Overview (`examples/dvector/README.md`) # dvector Vector distance functions for [Daft](https://docs.daft.ai), implemented as a native extension. ## Functions | Function | Description | Input | Output | |--------------------------|-------------------------------------------------|-----------------|---------| | `l2_distance(a, b)` | Euclidean distance | float vectors | Float64 | | `inner_product(a, b)` | Negative inner product (pgvector convention) | float vectors | Float64 | | `cosine_distance(a, b)` | Cosine distance (null for zero-norm) | float vectors | Float64 | | `l1_distance(a, b)` | Manhattan distance | float vectors | Float64 | | `hamming_distance(a, b)` | Count of differing positions | boolean vectors | UInt32 | | `jaccard_distance(a, b)` | 1 - intersection/union (null if union is empty) | boolean vectors | Float64 | Vectors can be FixedSizeList, List, or LargeList columns. Float vectors support Float32 and Float64 elements. ## Install ```sh ``` Requires a Rust toolchain and `setuptools-rust`. ## Usage ```python import daft import dvector from dvector import l2_distance # Load the extension into the active session daft.load_extension(dvector) df = daft.from_pydict({ "a": [[1.0, 2.0, 3.0], [0.0, 0.0, 0.0]], "b": [[4.0, 5.0, 6.0], [1.0, 1.0, 1.0]], }) df.select(l2_distance(daft.col("a"), daft.col("b"))).collect() # ╭───────────╮ # │ result │ # │ --- │ # │ Float64 │ # ╞═══════════╡ # │ 5.196152 │ # ├╌╌╌╌╌╌╌╌╌╌╌┤ # │ 1.732051 │ # ╰───────────╯ ``` ## Development ```sh # Build and install with the native library uv pip install -e . # Run tests pytest -v tests/ ``` --- ### Overview (`examples/hello/README.md`) # Hello Native Extension A minimal Daft native extension example. See the [extension guide](../../docs/extensions/index.md) for details. ## Quick start ```bash # Install the extension in the project .venv uv pip install -e . # Run the tests! pytest -v ``` --- ### Overview (`examples/hello_cpp/README.md`) # Hello C++ Native Extension A minimal Daft native extension written in pure C++ using the [Apache Arrow C++ library](https://arrow.apache.org/docs/cpp/). This is the C++ counterpart of the [hello](../hello/) Rust example. It implements a single `greet_cpp` scalar function that prepends `"Hello, "` to every string in a column. ## Prerequisites - CMake >= 3.20 - A C++17 compiler - Apache Arrow C++ (with headers) ### Install Arrow C++ **macOS (Homebrew):** ```bash brew install apache-arrow ``` **Ubuntu / Debian:** ```bash sudo apt install -y libarrow-dev ``` **Conda:** ```bash conda install -c conda-forge arrow-cpp ``` ## Development ### Build the shared library ```bash cmake -B build -DCMAKE_EXPORT_COMPILE_COMMANDS=ON cmake --build build ``` The `CMAKE_EXPORT_COMPILE_COMMANDS` flag generates `build/compile_commands.json` for clangd / LSP support. Symlink it to the project root if your editor needs it: ```bash ln -sf build/compile_commands.json . ``` ### Install as a Python package ```bash uv pip install -e . ``` This uses [scikit-build-core](https://scikit-build-core.readthedocs.io/) to build the C++ library and install it into the Python package. ### Run tests ```bash pytest -v tests/ ``` ## Usage ```python import daft import hello_cpp from hello_cpp import greet from daft import col from daft.session import Session sess = Session() sess.load_extension(hello_cpp) df = daft.from_pydict({"name": ["John", "Paul", "George", None]}) with sess: df.select(greet(col("name"))).show() # ╭──────────────────╮ # │ greet_cpp │ # │ --- │ # │ Utf8 │ # ╞══════════════════╡ # │ Hello, John! │ # ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ # │ Hello, Paul! │ # ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ # │ Hello, George! │ # ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ # │ None │ # ╰──────────────────╯ ``` --- ### Overview (`src/daft-dashboard/README.md`) # daft-dashboard ## Requirements - Node.js ≥22 - npm ≥11.10.0 — required for the `min-release-age` supply-chain setting in `frontend/.npmrc` to take effect. Older npm versions silently ignore it, leaving local `npm install ` unprotected. Upgrade with `npm install npm@latest`. ## Build The frontend (Next.js, under `frontend/`) is statically embedded into the Rust binary at compile time via `include_dir!` (see `src/assets.rs`). There is no separate dev server — **any frontend change requires `make build`** (or `make build-release`) from the repo root to take effect. ## Architecture - **Backend**: Axum server (`src/`) exposes SSE streams and REST endpoints under `/client/`. - **Frontend**: Next.js app (`frontend/`) compiled to static assets, embedded in the binary. - **State**: `DashboardState` (in `src/state.rs`) holds query info in a `DashMap` (in memory, currently no persistence). --- ### Overview (`src/daft-ext-macros/README.md`) # daft-ext-macros Proc macros for [Daft](https://github.com/Eventual-Inc/Daft) extensions. Most users should depend on [`daft-ext`](https://crates.io/crates/daft-ext) instead. ## License Apache-2.0 --- ### Overview (`src/daft-ext/README.md`) # daft-ext Extension SDK for building custom [Daft](https://github.com/Eventual-Inc/Daft) extensions. Daft is a distributed query engine for large-scale data processing. This crate provides the types, traits, and macros needed to build extensions that plug into Daft's execution engine. ## Usage ```toml [dependencies] daft-ext = "0.1" ``` ## License Apache-2.0 --- ### Overview (`src/daft-scan/README.md`) # daft-scan Defines the `ScanOperator` trait, `ScanTask` struct, and built-in operator implementations (`GlobScanOperator`, `AnonymousScanOperator`). Also owns scan-related primitives: `Pushdowns`, `PartitionField`, `Sharder`, and predicate rewriting for partition pruning. --- ### Overview (`tutorials/flyte/README.md`) # Using Daft with Flyte Daft works well with the [Flyte](https://flyte.org/) workflow engine and its Ray plugin. ## Using from Flyte task To run Daft in a Flyte task, use Daft as if you were using Daft in a local machine! ## Using Daft from a Ray Flyte task To run Daft in a Ray Flyte task and connect to a Ray cluster, use Daft as if `ray.init()` has already been called. That is to say, run Daft with: `daft.set_runner_ray()`. See: `app.py` for an example. ## Setup To run this locally, perform Flyte local setup as detailed here: https://github.com/jeevb/flyte-examples/tree/main/ray --- ### Overview (`benchmarking/ai/audio_transcription/README.md`) # Audio Transcription Benchmark Transcribes **113,800 audio files** using Whisper-tiny model. Processes audio resampling, feature extraction, and speech-to-text inference. **Input Dataset**: Common Voice 17 (S3 parquet format) **Output Format**: Parquet with transcriptions and metadata **Cluster**: 8 worker nodes using g6.xlarge instances **Benchmark Date**: September 22, 2024 **Framework Versions**: Daft 0.6.2, Ray Data 2.49.2, AWS EMR Spark 7.10.0 ## Performance Results | Engine | Runtime | |----------|---------| | Daft | 6m 22s | | Ray Data | 29m 20s | | Spark | 25m 46s | --- ### Overview (`benchmarking/ai/document_embedding/README.md`) # Document Embedding Benchmark Generates embeddings for **10,000 PDF documents**. Extracts text, chunks content, and creates 384-dimensional embeddings using sentence-transformers/all-MiniLM-L6-v2 model across distributed nodes. **Input Dataset**: Digital Corpora PDF metadata (S3 parquet format) **Output Format**: Parquet with embeddings, text chunks, and metadata **Cluster**: 8 worker nodes using g6.xlarge instances **Benchmark Date**: September 22, 2024 **Framework Versions**: Daft 0.6.2, Ray Data 2.49.2, AWS EMR Spark 7.10.0 ## Performance Results | Engine | Runtime | |----------|---------| | Daft | 1m 54s | | Ray Data | 14m 32s | | Spark | 8m 4s | --- ### Overview (`benchmarking/ai/image_classification/README.md`) # Image Classification Benchmark Classifies **803,580 rows** (80,358 unique images, each repeated 10x) using ResNet18 model. Downloads images, applies preprocessing transforms, and runs inference to predict ImageNet labels across distributed GPU nodes. **Input Dataset**: ImageNet benchmark dataset (S3 parquet format) **Output Format**: Parquet with image URLs and predicted labels **Cluster**: 8 worker nodes using g6.xlarge instances **Benchmark Date**: September 22, 2024 **Framework Versions**: Daft 0.6.2, Ray Data 2.49.2, AWS EMR Spark 7.10.0 ## Performance Results | Engine | Runtime | |----------|---------| | Daft | 4m 23s | | Ray Data | 23m 30s | | Spark | 45m 7s | --- ### Overview (`benchmarking/ai/video_object_detection/README.md`) # Video Object Detection Benchmark Detects objects in **1,000 videos** using YOLO11n model. Extracts frames, runs object detection, and crops detected objects across distributed GPU nodes. **Input Dataset**: Hollywood2 video dataset (S3 binary files) **Output Format**: Parquet with object detections, bounding boxes, and cropped images **Cluster**: 8 worker nodes using g6.xlarge instances **Benchmark Date**: September 22, 2024 **Framework Versions**: Daft 0.6.2, Ray Data 2.49.2, AWS EMR Spark 7.10.0 ## Performance Results | Engine | Runtime | |----------|---------| | Daft | 11m 46s | | Ray Data | 25m 54s | | Spark | 3h 36m | --- ### Overview (`k8s/charts/quickstart/README.md`) # Daft Quickstart Helm Chart A Helm chart for running Daft data processing workloads on Kubernetes, with support for both simple single-node execution and distributed Ray-based processing. ## Features - **Two deployment modes**: Simple mode for quick jobs, distributed mode for large-scale processing - **Two execution methods**: Script injection for rapid prototyping, custom images for managing complex dependencies - **No operator required**: Uses native Kubernetes resources only ## Prerequisites - Kubernetes 1.19+ - Helm 3.x - kubectl configured to access your cluster ## Quick Start ### Simple Mode (Default) Run a quick Daft job using the native runner: ```bash # Create a simple Python script cat > my_script.py <<'EOF' # /// script # dependencies = ["daft"] # /// import daft df = daft.from_pydict({ "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35] }) df.filter(df["age"] > 25).show() EOF # Deploy helm install my-job oci://ghcr.io/eventual-inc/daft/quickstart \ --set-file job.script=my_script.py # View logs kubectl logs -f job/my-job-quickstart-job # Cleanup helm uninstall my-job ``` ### Distributed Mode Run a Daft job on a Ray cluster with multiple workers: ```bash # Create a simple Python script with Ray cat > my_script.py <<'EOF' # /// script # dependencies = ["daft", "ray[client]==2.46.0"] # /// import daft import ray ray.init(runtime_env={"pip": ["daft"]}) df = daft.from_pydict({ "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35] }) df.filter(df["age"] > 25).show() EOF # Deploy with Ray cluster helm install distributed-job oci://ghcr.io/eventual-inc/daft/quickstart \ --set distributed=true \ --set worker.replicas=3 \ --set-file job.script=my_script.py # Access Ray Dashboard kubectl port-forward service/distributed-job-quickstart-head 8265:8265 # Open http://localhost:8265 # Cleanup helm uninstall distributed-job ``` ### Using a custom image This chart also supports running Daft jobs from custom images. This is useful in scenarios where more complex dependencies are required. ```bash helm install custom-image-job oci://ghcr.io/eventual-inc/daft/quickstart \ --set image=my-image:tag \ --set "job.command={python,my_script.py}" ``` ## Installation See [values.yaml](values.yaml) for detailed descriptions of all supported parameters. ## Monitoring ### View Job Status ```bash # Get job status kubectl get jobs -l app.kubernetes.io/instance=my-release # View job logs kubectl logs -f job/my-release-quickstart-job # Check pod status kubectl get pods -l app.kubernetes.io/instance=my-release ``` ### Access Ray Dashboard (Distributed Mode) ```bash # Port forward to access both Ray Dashboard and Grafana kubectl port-forward service/my-release-quickstart-head 8265:8265 3000:3000 # Open http://localhost:8265 in browser # Grafana panels will be embedded directly in the Ray Dashboard # Optionally open http://localhost:3000 to access Grafana directly # Default credentials: admin/admin ``` ### Check Ray Cluster Status (Distributed Mode) ```bash # Get cluster status kubectl exec deployment/my-release-quickstart-head -- ray status # View cluster resources kubectl exec deployment/my-release-quickstart-head -- python -c "import ray; ray.init(); print(ray.cluster_resources())" ``` ## Scaling ### Manual Scaling (Distributed Mode) ```bash # Scale workers using kubectl kubectl scale deployment my-release-quickstart-worker --replicas=10 ``` ## Troubleshooting ### Job Not Starting ```bash # Check job events kubectl describe job my-release-quickstart-job # Check pod events kubectl describe pod -l app.kubernetes.io/component=job # View pod logs kubectl logs -l app.kubernetes.io/component=job ``` ### Workers Not Connecting (Distributed Mode) ```bash # Check head service kubectl get service -l ray.io/node-type=head # Check head logs kubectl logs deployment/my-release-quickstart-head # Check worker logs kubectl logs deployment/my-release-quickstart-worker ``` ### Out of Memory Errors Increase memory limits in values: ```yaml job: resources: limits: memory: "16Gi" # Increase as needed head: resources: limits: memory: "32Gi" # Increase as needed worker: resources: limits: memory: "32Gi" # Increase as needed ``` ### Script Not Found Error Ensure you're using `--set-file` (not `--set`): ```bash # Correct helm install my-job oci://ghcr.io/eventual-inc/daft/quickstart --set-file job.script=script.py # Incorrect helm install my-job oci://ghcr.io/eventual-inc/daft/quickstart --set job.script=script.py ``` ### Ray Module Not Found (Distributed Mode) Your custom image must include `ray[default]`: ```txt # requirements.txt daft ray[default] ``` ## Uninstalling ```bash helm uninstall my-release ``` This removes all Kubernetes resources created by the chart. ## License Apache 2.0 ## Contributing Contributions are welcome! Please see the [Daft repository](https://github.com/Eventual-Inc/Daft) for contribution guidelines. ## Support - **Issues**: [GitHub Issues](https://github.com/Eventual-Inc/Daft/issues) - **Documentation**: [docs.daft.ai](https://docs.daft.ai) - **Community**: [Daft Discussions](https://github.com/Eventual-Inc/Daft/discussions) --- ### Overview (`tests/integration/gravitino/README.md`) # Gravitino Integration Tests These tests exercise Daft's Gravitino connector end-to-end. To run them: 1. Start the local Gravitino service: ```bash cd tests/integration/gravitino/docker-compose docker compose up -d ``` 2. Export the connection settings (if you use the compose file above the defaults already match): ```bash export GRAVITINO_ENDPOINT=${GRAVITINO_ENDPOINT:-http://127.0.0.1:8090} export GRAVITINO_METALAKE=${GRAVITINO_METALAKE:-metalake_demo} ``` 3. Run the tests: ```bash DAFT_RUNNER=native pytest tests/integration/gravitino -v -m integration ``` ## Test Files - `test_gravitino_fileset.py` - Tests filesets with local file:// storage - `test_gravitino_fileset_s3.py` - Tests filesets with S3 storage (using SeaweedFS) - `test_gravitino_table.py` - Tests table catalog operations ## S3 Storage Tests The S3 fileset tests (`test_gravitino_fileset_s3.py`) test Gravitino filesets backed by S3 storage. ### Current Status The docker-compose setup includes: 1. **SeaweedFS**: S3-compatible storage on port 9000 2. **Update fileset.conf**: To run with S3, set a property in `/root/gravitino/catalogs/fileset/conf/fileset.conf` 3. **Test Infrastructure**: Complete test suite for S3 filesets ### Running S3 Tests with External Gravitino If you have access to a Gravitino deployment with S3 filesets already configured, you can test against it: ```bash # Point to an existing S3-backed fileset export GRAVITINO_TEST_DIR="gvfs://fileset////" export GRAVITINO_TEST_FILE="gvfs://fileset////file.parquet" export GRAVITINO_ENDPOINT="http://your-gravitino-server:8090" # Run specific tests DAFT_RUNNER=native pytest tests/integration/gravitino/test_gravitino_fileset_s3.py::test_read_existing_s3_fileset -v -m integration ``` ### Test Coverage The S3 test suite includes: - Basic read operations from S3-backed filesets - Glob patterns and file listing - Partitioned data handling - Error handling for missing files and empty directories ## Optional Configuration Optionally, configure `GRAVITINO_TEST_FILE` and `GRAVITINO_TEST_DIR` to point at a fileset that exists in your Gravitino deployment so that the gvfs:// IO tests can read concrete data. The catalog integration tests provision temporary catalog/schema/fileset via the Gravitino REST API and write test data to either local storage or SeaweedFS. The tests clean up both the server-side metadata and storage when finished. --- ### Overview (`tools/observability/opentelemetry/README.md`) # OpenTelemetry This directory contains the code for running the OpenTelemetry collector and exporter. The docker compose file located in this directory will start the following services: 1. [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) - an agent for collecting and exporting opentelemetry data 2. [Jaeger](https://www.jaegertracing.io/) - an open source distributed tracing system 3. [Prometheus](https://prometheus.io/) - an open source metrics monitoring system ## Running the collector To start the services, navigate to this directory and run the following command: ```bash docker compose up ``` ## Executing a job with opentelemetry enabled The opentelemetry collector exposes an OTLP gRPC endpoint on port 4317. To execute a job with opentelemetry enabled, set the `DAFT_DEV_OTEL_EXPORTER_OTLP_ENDPOINT` environment variable to `grpc://localhost:4317` and run the job. Example: ```bash DAFT_DEV_OTEL_EXPORTER_OTLP_ENDPOINT=grpc://localhost:4317 python tools/observability/opentelemetry/example.py ``` ## Viewing the metrics ### Traces (Jaeger) To view the traces on the Jaeger UI, navigate to `http://localhost:16686/` and select the `daft` service from the dropdown menu. ### Metrics (Prometheus) To view the metrics on the Prometheus UI, navigate to `http://localhost:9090/`. To query for all metrics provided by Daft, try the following [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query: ```promql {service_name='daft'} ``` --- ### Index (`docs/index.md`) # Introduction Daft is a high-performance data engine designed for AI and multimodal workloads, providing simple, reliable data processing for images, audio, video, and structured data at any scale.
READ ANYTHING

Daft reads raw unstructured and multimodal data collected from application systems

Object storage (AWS S3, GCS, R2)
Event bus (Kafka)
Data lake (Iceberg, Deltalake)

EXPENSIVE TRANSFORMATIONS

Build efficient Daft data pipelines involving heavy transformations

GPU models
User-provided Python code
External LLM APIs

WRITE

Daft lands data into specialized data systems for downstream use cases

Search (full-text search and vector DBs)
Applications (SQL/NoSQL databases)
Analytics (data warehouses)
Model training (S3 object storage)

## Why Daft? **:octicons-image-24: Unified multimodal data processing** While traditional dataframes struggle with anything beyond tables, Daft natively handles tables, images, text, and embeddings through a single Python API. No more stitching together specialized tools for different data types. **:material-language-python: Python-native, no JVM required** Built for modern AI/ML workflows with Python at its core and Rust under the hood. Skip the JVM complexity, version conflicts, and memory tuning to achieve 20x faster start times—get the performance without the Java tax. **:fontawesome-solid-laptop: Seamless scaling, from laptop to cluster** Start local, scale global—without changing a line of code. Daft's Rust-powered engine delivers blazing performance on a single machine and effortlessly extends to distributed clusters when you need more horsepower. ## Key Features * **Native multimodal processing**: Process any data type—from structured tables to unstructured text and rich media—with native support for images, audio, video, and embeddings in a single, unified framework. * **Built-in AI operations**: Transform data with AI natively: run LLM prompts with structured outputs, generate embeddings, and classify images or text using models from OpenAI, Transformers, or your own custom providers, all optimized for batch processing. * **Rust-powered performance**: Experience breakthrough speed with our Rust foundation delivering vectorized execution and non-blocking I/O that processes the same queries with 5x less memory while consistently outperforming industry standards by an order of magnitude. * **Universal data connectivity**: Access data anywhere it lives—cloud storage ([S3](connectors/aws.md), [Azure](connectors/azure.md), [GCS](api/config.md#daft.io.GCSConfig), [Hugging Face](connectors/huggingface.md)), modern table formats ([Apache Iceberg](connectors/iceberg.md), [Delta Lake](connectors/delta_lake.md), [Apache Hudi](connectors/hudi.md)), or enterprise catalogs ([Unity Catalog](connectors/unity_catalog.md), [AWS Glue](connectors/glue.md))—all with zero configuration. * **Push your code to your data**: Bring your Python functions directly to your data with zero-copy UDFs powered by [Apache Arrow](https://arrow.apache.org/), eliminating data movement overhead and accelerating processing speeds. * **Out of the box reliability**: Deploy with confidence—intelligent memory management prevents OOM errors while sensible defaults eliminate configuration headaches, letting you focus on results, not infrastructure. !!! tip "Looking to get started with Daft ASAP?" If you are ready to jump into code, take a look at these resources: 1. [Quickstart](quickstart.md): Itching to run some Daft code? Hit the ground running with our 10 minute quickstart. 2. [Examples](examples/index.md): See Daft in action with use cases across text, images, audio, and more. 3. [API Documentation](api/index.md): Searchable documentation and reference material to Daft's public API. ## Contribute to Daft If you're interested in hands-on learning about Daft internals and would like to contribute to our project, join us [on GitHub](https://github.com/Eventual-Inc/Daft) 🚀 Take a look at the many issues tagged with `good first issue` in our repo. If there are any that interest you, feel free to chime in on the issue itself or join us in our [Distributed Data Slack Community](https://daft.ai/slack) and send us a message in #daft-dev. Daft team members will be happy to assign any issue to you and provide any guidance if needed! --- ### Quickstart (`docs/quickstart.md`) # Quickstart [](https://colab.research.google.com/github/Eventual-Inc/Daft/blob/main/docs/notebooks/quickstart.ipynb) Daft is the best multimodal data processing engine that allows you to load data from anywhere, transform it with a powerful DataFrame API and AI functions, and store it in your destination of choice. In this quickstart, you'll see what this looks like in practice with a realistic e-commerce data workflow. ### Requirements Daft requires **Python 3.10 or higher**. ### Install Daft You can install Daft using `pip`. Run the following command in your terminal or notebook: ```bash pip install -U "daft[openai]" # Includes OpenAI extras needed for this quickstart ``` Additionally, install these packages for image processing (used later in this quickstart): ```bash pip install numpy pillow ``` ### Load Your Data Let's start by loading an e-commerce dataset from Hugging Face. [This dataset](https://huggingface.co/datasets/calmgoose/amazon-product-data-2020) contains 10,000+ Amazon products from diverse categories including electronics, toys, home goods, and more. Each product includes details like names, prices, descriptions, technical specifications, and product images. ```python import daft df_original = daft.read_huggingface("calmgoose/amazon-product-data-2020") ``` !!! note "Load from anywhere" Daft can load data from many sources including [S3](connectors/aws.md), [Iceberg](connectors/iceberg.md), [Delta Lake](connectors/delta_lake.md), [Hudi](connectors/hudi.md), and [more](connectors/index.md). We're using Hugging Face here as a demonstration. ### Inspect Your Data Now let's take a look at what we loaded. You can inspect the DataFrame by simply printing it: ```python df_original ``` ``` {title="Output"} ╭─────────┬──────────────┬──────────┬────────────┬──────────┬─────────────┬──────────────────╮ │ Uniq Id ┆ Product Name ┆ Category ┆ … ┆ Variants ┆ Product Url ┆ Is Amazon Seller │ │ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- │ │ String ┆ String ┆ String ┆ (9 hidden) ┆ String ┆ String ┆ String │ ╰─────────┴──────────────┴──────────┴────────────┴──────────┴─────────────┴──────────────────╯ (No data to display: Dataframe not materialized, use .collect() to materialize) ``` You see the above output because **Daft is lazy by default** - it displays the schema (column names and types) but doesn't actually load or process your data until you explicitly tell it to. This allows Daft to optimize your entire workflow before executing anything. To actually view your data, you have two options: **Option 1: Preview with `.show()`** - View the first few rows: ```python df_original.show(2) ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` This materializes and displays just the first 2 rows, which is perfect for quickly inspecting your data without loading the entire dataset. **Option 2: Materialize with `.collect()`** - Load the entire dataset: ```python # df_original.collect() ``` This would materialize the entire DataFrame (all 10,000+ rows in this case) into memory. Use `.collect()` when you need to work with the full dataset in memory. ### Working with a Smaller Dataset For quick experimentation, let's create a smaller, simplified version of the dataframe with just the essential columns: ```python # Select only the columns we need and limit to 5 rows for faster iteration df = df_original.select("Product Name", "About Product", "Image").limit(5) ``` Now we have a manageable dataset of 5 products with just the product name, description, and image URLs. This simplified dataset lets us explore Daft's features without the overhead of unnecessary columns. ### Downloading Images Let's extract and download product images. The `Image` column contains pipe-separated URLs. We'll extract the first URL and download it: ```python # Extract the first image URL from the pipe-separated list # The pattern captures everything before the first pipe or the entire string if no pipe df = df.with_column( "first_image_url", daft.functions.regexp_extract( df["Image"], r"^([^|]+)", # Extract everything before the first pipe 1 # Get the first capture group ) ) # Download the image data df = df.with_column( "image_data", daft.functions.download(df["first_image_url"], on_error="null") ) # Decode images for visual display (in Jupyter notebooks, this shows actual images!) df = df.with_column( "image", daft.functions.decode_image(df["image_data"], on_error="null") ) # Check what we have - in Jupyter notebooks, the 'image' column shows actual images! df.select("Product Name", "first_image_url", "image_data", "image").show(3) ``` ``` {title="Output"} ╭────────────────────────────────┬────────────────────────────────┬────────────────────────────────┬──────────────╮ │ Product Name ┆ first_image_url ┆ image_data ┆ image │ │ --- ┆ --- ┆ --- ┆ --- │ │ String ┆ String ┆ Binary ┆ Image[MIXED] │ ╞════════════════════════════════╪════════════════════════════════╪════════════════════════════════╪══════════════╡ │ DB Longboards CoreFlex Crossb… ┆ https://images-na.ssl-images-… ┆ b"\xff\xd8\xff\xe0\x00\x10JFI… ┆ │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ Electronic Snap Circuits Mini… ┆ https://images-na.ssl-images-… ┆ b"\xff\xd8\xff\xe0\x00\x10JFI… ┆ │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ 3Doodler Create Flexy 3D Prin… ┆ https://images-na.ssl-images-… ┆ b"\xff\xd8\xff\xe0\x00\x10JFI… ┆ │ ╰────────────────────────────────┴────────────────────────────────┴────────────────────────────────┴──────────────╯ (Showing first 3 rows) ``` !!! note "Visual Display in Notebooks" In Jupyter notebooks, the `image` column will display actual thumbnail images instead of `` text. This demonstrates Daft's multimodal capabilities: - **Native regex support**: Use `regexp_extract()` to parse structured text with Rust-powered regex - **URL handling**: Download content directly with`daft.functions.download()` - **Image decoding**: Convert binary data to images with `decode_image()` for visual display The decoded images are now ready for further processing. ### Batch AI Inference on Images Let's use AI to analyze product materials at scale. Daft automatically parallelizes AI operations across your local machine's cores, making it efficient to process multiple images concurrently. Let's suppose you want to create a new column that shows if each product is made of wood or not. This might be useful for, for example, a filtering feature on your website. If you're running this in Google Colab or Jupyter, run the following cell to set your OpenAI API key. In Colab, first add your key to Secrets (🔑 icon in the left sidebar) with the name `OPENAI_API_KEY`. In Jupyter, you'll be prompted to enter your key and the input will be hidden. ```python import os try: from google.colab import userdata os.environ["OPENAI_API_KEY"] = userdata.get("OPENAI_API_KEY") except ImportError: from getpass import getpass if "OPENAI_API_KEY" not in os.environ: os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key: ") ``` ```python from pydantic import BaseModel, Field from daft.functions import prompt # Define a simple structured output model class WoodAnalysis(BaseModel): is_wooden: bool = Field(description="Whether the product appears to be made of wood") # Run AI inference on each image - Daft automatically batches and parallelizes this df = df.with_column( "wood_analysis", prompt( ["Is this product made of wood? Look at the material.", df["image"]], return_format=WoodAnalysis, model="gpt-4o-mini", # Using mini for cost-efficiency provider="openai", # api_key="your-key-here", # Use OPENAI_API_KEY env var, or uncomment to set manually ) ) # Extract the boolean value from the structured output # The result is a struct, so we extract the 'is_wooden' field df = df.with_column( "is_wooden", df["wood_analysis"]["is_wooden"] ) # Materialize the dataframe to compute all transformations df = df.collect() # View results df.select("Product Name", "image", "is_wooden").show() ``` ``` {title="Output"} ╭────────────────────────────────┬──────────────┬───────────╮ │ Product Name ┆ image ┆ is_wooden │ │ --- ┆ --- ┆ --- │ │ String ┆ Image[MIXED] ┆ Bool │ ╞════════════════════════════════╪══════════════╪═══════════╡ │ DB Longboards CoreFlex Crossb… ┆ ┆ true │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤ │ Electronic Snap Circuits Mini… ┆ ┆ false │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤ │ Guillow Airplane Design Studio ┆ ┆ false │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤ │ Woodstock- Collage 500 pc Puz… ┆ ┆ false │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┤ │ 3Doodler Create Flexy 3D Prin… ┆ ┆ false │ ╰────────────────────────────────┴──────────────┴───────────╯ (Showing first 5 of 5 rows) ``` The AI analyzes each product image to determine if it's made of wood. Notice that the longboard is identified as wooden (true), while the electronic circuits, design studio, puzzle, and 3D printing filament are identified as not wooden (false). !!! note "Improving Accuracy" Looking at the actual product data, the longboard is made of bamboo and fiberglass, not wood. However, this is exactly what a human might categorize from the image alone! To improve accuracy, you could feed additional context to the AI like the product name, category, and description alongside the image. This example demonstrates how to get started with image-based analysis. ### Expanding the Analysis Now, suppose you're satisfied with the results from your small subset and want to scale up. Instead of analyzing just 5 products, let's run the same analysis on 100 products to get more meaningful insights: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ``` {title="Output"} Out of 100 products analyzed: - 4 are made of wood - 96 are not made of wood - Percentage of wooden products: 4.0% ``` !!! note "Results May Vary" AI models are non-deterministic, so you may see slightly different numbers when running this analysis. ### Storing Your Results After processing your data, you'll often want to save it for later use. Let's store our analyzed dataset as Parquet files: ```python # Write the analyzed data to local Parquet files df_large.write_parquet("product_analysis", write_mode="overwrite") ``` This writes your data to the `product_analysis/` directory. Daft automatically handles file naming using UUIDs to prevent conflicts. The `write_mode="overwrite"` parameter ensures that any existing data in the directory is replaced. !!! note "Write anywhere" Just like reading, Daft can write data to many destinations including [S3](connectors/aws.md), [Iceberg](connectors/iceberg.md), [Delta Lake](connectors/delta_lake.md), and [more](connectors/index.md). ### Loading Your Stored Data Let's verify the stored data by loading it back from those Parquet files: ```python # Read the data back from Parquet files df_loaded = daft.read_parquet("product_analysis/*.parquet") # Verify the data loaded correctly df_loaded.show(5) ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### What's Next? Now that you have a basic sense of Daft's functionality and features, here are some more resources to help you get the most out of Daft: !!! tip "Scaling Further" This same pipeline can process thousands or millions of products by leveraging Daft's distributed computing capabilities. Check out our [distributed computing guide](distributed/index.md) to run this analysis at scale on Ray or Kubernetes clusters. **Work with your favorite table and catalog formats:**
- [Apache Hudi](connectors/hudi.md) - [Apache Iceberg](connectors/iceberg.md) - [AWS Glue](connectors/glue.md) - [AWS S3 Tables](connectors/s3tables.md) - [Delta Lake](connectors/delta_lake.md) - [Hugging Face Datasets](connectors/huggingface.md) - [Unity Catalog](connectors/unity_catalog.md) - [Apache Gravitino](connectors/gravitino.md)
**Explore our [Examples](examples/index.md) to see Daft in action:**
- [:material-image-edit: MNIST Digit Classification](examples/mnist.md) - [:octicons-search-16: Running LLMs on the Red Pajamas Dataset](examples/llms-red-pajamas.md) - [:material-image-search: Querying Images with UDFs](examples/querying-images.md) - [:material-image-sync: Image Generation on GPUs](examples/image-generation.md) - [:material-window-closed-variant: Window Functions in Daft](examples/window-functions.md)
--- ### Ai Functions Overview (`docs/ai-functions/overview.md`) # AI Functions Overview Daft is purpose-built for scaling multimodal AI workloads. AI Functions make it easy to run models on data whether you're generating embeddings, classifying text, or prompting language models. Daft's built-in Image and File abstractions simplify data handling whether it's on your laptop or in the cloud. The current list of AI functions includes: - [`prompt`](prompt.md) - Generate text completions from language models - [`embed_text`](embed.md#text-embeddings) - Create vector embeddings from text - [`embed_image`](embed.md#image-embeddings) - Create vector embeddings from images - [`classify_text`](classify.md#text-classification) - Zero-shot text classification - [`classify_image`](classify.md#image-classification) - Zero-shot image classification For more detailed information on the Providers, see [AI Providers Overview](providers.md). If you'd like to contribute a new AI function or expand provider support, check out [Contributing New AI Functions & Providers](../contributing/contributing-ai-functions.md). ## See Also - [Working with Text](../modalities/text.md) - Text processing and chunking strategies - [Working with Images](../modalities/images.md) - Image processing and transformations - [Embeddings](../modalities/embeddings.md) - Vector representations for RAG and search - [Custom Code with `@daft.func`](../custom-code/func.md) - Write custom Python functions for specialized processing - [Batch Inference](../use-case/batch-inference.md) - End-to-end batch inference patterns - [Checkpointing](../use-case/checkpointing.md) - Resume expensive AI pipelines without redoing completed work --- ### Api Index (`docs/api/index.md`) # Daft API Documentation Welcome to Daft Python API Documentation. For Daft User Guide, head to [User Guide](../index.md).
- [**AI**](ai.md) Functions and interfaces for building AI applications with daft. This includes providers and various model inference protocols such as text embedding. - [**I/O**](io.md) Variety of approaches to creating a DataFrame from reading various data sources like in-memory data, files, data catalogs, and integrations and writing to various data sources. - [**DataFrame**](dataframe.md) Available DataFrame methods that are enqueued in the DataFrame's internal query plan and executed when Execution DataFrame methods are called. - [**Datasets**](datasets.md) Simple, performant, and responsible ways to access useful datasets like Common Crawl and DROID. - [**Expressions**](expressions.md) Expressions allow you to express some computation that needs to happen in a DataFrame. - [**Functions**](functions/index.md) Daft provides a set of built-in operations that can be applied to DataFrame columns. - [**User-Defined Functions**](udf.md) User-Defined Functions (UDFs) are a mechanism to run Python code on the data that lives in a DataFrame. - [**Window**](window.md) Window functions allow you to perform calculations across a set of rows that are related to the current row. - [**Sessions**](sessions.md) Sessions enable you to attach catalogs, tables, and create temporary objects which are accessible through both the Python and SQL APIs. - [**Catalogs & Tables**](catalogs_tables.md) Daft integrates with various catalog implementations using its Catalog and Table interfaces to manage catalog objects like tables and namespaces. - [**Schema**](schema.md) Daft can display your DataFrame's schema without materializing it by performing intelligent sampling of your data to determine appropriate schema. - [**Data Types**](datatypes/all_datatypes.md) Daft provides simple DataTypes that are ubiquituous in many DataFrames such as numbers, strings, dates, tensors, and images. - [**Aggregations**](aggregations.md) When performing aggregations such as sum, mean and count, Daft enables you to group data by certain keys and aggregate within those keys. - [**Series**](series.md) Series expose methods which invoke high-performance kernels for manipulation of a column of data. - [**Configuration**](config.md) Configure the execution backend, Daft in various ways during execution, and how Daft interacts with storage. - [**Miscellaneous**](misc.md)
--- ### Architecture Index (`docs/architecture/index.md`) # Daft Architecture Daft is a high-performance data engine designed for any modality and any scale. At a high level, a Daft query moves through 3 main stages: Planning, Optimization, and Execution. ## Planning Users express their workloads via the Python DataFrame API or the SQL interface. These APIs are lazy: operations construct a plan of execution, but they do not read or transform data yet. Internally, the workload is represented as a `LogicalPlan`: a tree of operators that describe *what* work should happen. Example operators include `Source`, `Project`, `Filter`, `GroupBy`, and `Join`. Within each operator, especially the `Project` operator, there is a collection of expressions that define how data should be transformed. When users apply native functions (like `col("a") + 1`) or custom Python UDFs (such as `col("b").apply(my_udf)`), these are represented internally as nodes in an *expression tree*. Each node in the tree can be a column reference, a literal, a function call, or a UDF, and complex expressions are built by chaining these nodes together. For example, consider the following user code: ```python import daft from daft import col df = daft.from_pydict({"a": [1, 2, 3]}) result = df.select((col("a") + 1).alias("a_plus_one")) result.explain() ``` Internally, this becomes a simple expression attached to the `Project` operator: ```{title="Output"} * Project: col(a) + lit(1) as a_plus_one | * Source: | Number of partitions = 1 | Output schema = a#Int64 ``` ## Optimization Executing the logical plan as written would be correct but often inefficient. Daft applies both rule-based and cost-based optimization once execution is requested. - **Rule-based pass.** Classical rewrite rules execute first: filter, projection, limit, and aggregation pushdowns; projection folding and splitting; pruning redundant repartitions; expression simplification; and subquery unnesting. These transform the plan structure without needing runtime statistics. - **Cost-based pass.** Joins are reordered using a brute-force enumerator that evaluates the cheapest ordering based on available statistics calculated from sources. - **Multimodal awareness.** Expensive projections such as Python UDFs, model inference, URL downloads, image decoding, and similar operators are isolated into dedicated logical nodes. They are intentionally *not* pushed into scans so that execution can batch, schedule, and backpressure them independently. Such operations are executed as late as correctness permits (after joins, aggregations, etc.) to reduce wasted work on discarded rows. This hybrid optimizer lets users focus on declarative queries while still obtaining efficient plans tailored to multimodal workloads. ## Execution Execution is triggered by a materializing operation such as `collect()`, `show()`, `to_arrow()`, and write methods such as `write_parquet()`. Once optimization completes, the plan is executed by either the native runner (single-machine) or the Ray runner (distributed). ### Native Runner (aka Swordfish) The native runner is a streaming execution engine implemented in Rust, using the [Tokio](https://tokio.rs/) runtime for async I/O and multithreaded operations. Given a plan, the native runner constructs a graph of operators, where each operator corresponds to a node in the plan. Operators send data up the graph via async channels. Once the graph is constructed, the runner initiates execution, which typically follows this pattern: - Source operators read data from local files or object stores and emit them in batches. - Intermediate operators (e.g. Project, Filter, UDF) receive batches of data from previous operators, immediately transform them, and send them to the next operator. - Sinks receive and accumulate batches of data. Streaming sinks (e.g. Limit) can emit early, while blocking sinks (e.g. Aggregate, Sort) only emit results once all input has been processed. Operators determine their own parallelism and batching, and the engine schedules work across operators on a threadpool (Tokio). ### Ray Runner (aka Flotilla) The Ray runner is a distributed execution engine built on top of Ray. A Ray cluster contains a single head node, and all other nodes are worker nodes. On each worker node, Daft launches a single Ray actor that hosts an instance of the Swordfish engine. The Flotilla scheduler runs on the head node and distributes tasks across workers, monitors progress, and orchestrates data movement when needed. Each task corresponds to a partition of the DataFrame, and tasks are assigned based on data locality and worker load, so computation runs close to the data while keeping cluster resources balanced. The number of partitions typically corresponds to the number of input files. !!! tip "Controlling partitioning" See our [Partitioning and Batching guide](../optimization/partitioning.md) to learn how to control the number and distribution of partitions in your DataFrame Each worker receives tasks and executes them using the Swordfish streaming engine with full access to the machine's resources. Outputs of the tasks are written to Ray's object store, and can be moved between workers for global operations like groupbys and joins. --- ### Benchmarks Index (`docs/benchmarks/index.md`) # Benchmarks ## AI Benchmarks !!! info "Powered by Flotilla" These benchmarks showcase [Flotilla](https://www.daft.ai/blog/introducing-flotilla-simplifying-multimodal-data-processing-at-scale), Daft's distributed engine optimized for multimodal data processing. [Daft vs Spark vs Ray Data: Benchmarks for Multimodal AI Workloads](https://www.daft.ai/blog/benchmarks-for-multimodal-ai-workloads) - Full methodology and technical deep-dive. Modern AI workloads demand simple, fast, and scalable data engines. Daft is purpose-built for these workloads, excelling where traditional big data frameworks fall short. Our benchmarks show how Daft uniquely delivers on the core needs of AI data processing: 1. **Process any modality:** Audio, video, images, and documents as first class citizens. 2. **Scale from 1 to 1000:** Easily scale across machines without rewriting code or configs. 3. **Run models over data:** Seamlessly use models to embed, classify, and generate data. ### Setup The AI benchmarks cover four different workload types, each designed to test different use cases across modalities, and at large scale. Each benchmark runs on AWS g6.xlarge instances with 8 worker nodes. The complete benchmark code is available in our repository: - **[Audio Transcription](https://github.com/Eventual-Inc/Daft/tree/main/benchmarking/ai/audio_transcription)** - Transcribing 113,800 audio files using OpenAI Whisper-tiny model - **[Document Embedding](https://github.com/Eventual-Inc/Daft/tree/main/benchmarking/ai/document_embedding)** - Generating embeddings for 10,000 PDF documents using sentence-transformers - **[Image Classification](https://github.com/Eventual-Inc/Daft/tree/main/benchmarking/ai/image_classification)** - Classifying 803,580 images using ResNet18 model - **[Video Object Detection](https://github.com/Eventual-Inc/Daft/tree/main/benchmarking/ai/video_object_detection)** - Detecting objects in 1,000 videos using YOLO11n model Each benchmark includes implementations for Daft, Ray Data, and EMR Spark, along with cluster configurations and dependencies. **Benchmark Date**: September 22, 2025 **Engine Versions**: Daft 0.6.2, Ray Data 2.49.2, AWS EMR Spark 7.10.0 ### Results
| | Daft | Ray Data | EMR Spark | | -------- | :--: | :------: | :---: | | Audio Transcription | 6m 22s | 29m 20s (4.6x slower) | 25m 46s (4.0x slower) | | Document Embedding | 1m 54s | 14m 32s (7.6x slower) | 8m 4s (4.2x slower) | | Image Classification | 4m 23s | 23m 30s (5.4x slower) | 45m 7s (10.3x slower) | | Video Object Detection | 11m 46s | 25m 54s (2.2x slower) | 3h 36m (18.4x slower) | ### Logs Complete execution logs for all AI benchmark runs are available for transparency and reproducibility: | Workload | Daft | Ray Data | Spark | | -------- | ---- | -------- | ----- | | Audio Transcription | s3://daft-oss-public-data/benchmarking/logs/ai_benchmarking_logs/audio_transcription/daft.txt | s3://daft-oss-public-data/benchmarking/logs/ai_benchmarking_logs/audio_transcription/ray_data.txt | s3://daft-oss-public-data/benchmarking/logs/ai_benchmarking_logs/audio_transcription/spark.txt | | Document Embedding | s3://daft-oss-public-data/benchmarking/logs/ai_benchmarking_logs/document_embedding/daft.txt | s3://daft-oss-public-data/benchmarking/logs/ai_benchmarking_logs/document_embedding/ray_data.txt | s3://daft-oss-public-data/benchmarking/logs/ai_benchmarking_logs/document_embedding/spark.txt | | Image Classification | s3://daft-oss-public-data/benchmarking/logs/ai_benchmarking_logs/image_classification/daft.txt | s3://daft-oss-public-data/benchmarking/logs/ai_benchmarking_logs/image_classification/ray_data.txt | s3://daft-oss-public-data/benchmarking/logs/ai_benchmarking_logs/image_classification/spark.txt | | Video Object Detection | s3://daft-oss-public-data/benchmarking/logs/ai_benchmarking_logs/video_object_detection/daft.txt | s3://daft-oss-public-data/benchmarking/logs/ai_benchmarking_logs/video_object_detection/ray_data.txt | s3://daft-oss-public-data/benchmarking/logs/ai_benchmarking_logs/video_object_detection/spark.txt | --- ## TPC-H Benchmarks Here we compare Daft against some popular Distributed Dataframes such as Spark, Modin, and Dask on the TPC-H benchmark. Our goal for this benchmark is to demonstrate that Daft is able to meet the following development goals: 1. **Solid out of the box performance:** great performance without having to tune esoteric flags or configurations specific to this workload 2. **Reliable out-of-core execution:** highly performant and reliable processing on larger-than-memory datasets, without developer intervention and Out-Of-Memory (OOM) errors 3. **Ease of use:** getting up and running should be easy on cloud infrastructure for an individual developer or in an enterprise cloud setting A great stress test for Daft is the [TPC-H benchmark](https://www.tpc.org/tpch/), which is a standard benchmark for analytical query engines. This benchmark helps ensure that while Daft makes it very easy to work with multimodal data, it can also do a great job at larger scales (terabytes) of more traditional tabular analytical workloads. ### Setup The basic setup for our benchmarks are as follows: 1. We run questions 1 to 10 of the TPC-H benchmarks using Daft and other commonly used Python Distributed Dataframes. 2. The data for the queries are stored and retrieved from AWS S3 as partitioned Apache Parquet files, which is typical of enterprise workloads. No on disk/in-memory caching was performed. 3. We run each framework on a cluster of AWS i3.2xlarge instances that each have: - 8 vCPUs - 61G of memory - 1900G of NVMe SSD space The frameworks that we benchmark against are Spark, Modin, and Dask. We chose these comparable Dataframes as they are the most commonly referenced frameworks for running large scale distributed analytical queries in Python. For benchmarking against Spark, we use AWS EMR which is a hosted Spark service. For other benchmarks, we host our own Ray and Dask clusters on Kubernetes. Please refer to the section on our [Detailed Benchmarking Setup](#detailed-benchmarking-setup) for additional information. ### Results !!! success "Highlights" 1. Out of all the benchmarked frameworks, **only Daft and EMR Spark are able to run terabyte scale queries reliably** on out-of-the-box configurations. 2. **Daft is consistently much faster** (3.3x faster than EMR Spark, 7.7x faster than Dask Dataframes, and 44.4x faster than Modin). !!! note "Note" We were unable to obtain full results for Modin due to cluster OOMs, errors and timeouts (one hour limit per question attempt). Similarly, Dask was unable to provide comparable results for the terabyte scale benchmark. It is possible that these frameworks may perform and function better with additional tuning and configuration. Logs for all the runs are provided in a public AWS S3 bucket. #### 100 Scale Factor First we run TPC-H 100 Scale Factor (around 100GB) benchmark on 4 i3.2xlarge worker instances. In total, these instances add up to 244GB of cluster memory which will require the Dataframe library to perform disk spilling and out-of-core processing for certain questions that have a large join or sort.
| Dataframe | Questions Completed | Total Time (seconds) | Relative to Daft | | --------- | :-----------------: | :------------------: | :--------------: | | Daft | 10/10 | 785 | 1.0x | | Spark | 10/10 | 2648 | 3.3x | | Dask | 10/10 | 6010 | 7.7x | | Modin | 5/10 | Did not finish | 44.4x* | *\* Only for queries that completed.* From the results we see that Daft, Spark, and Dask are able to complete all the questions and Modin completes less than half. We also see that Daft is **3.3x** faster than Spark and **7.7x** faster than Dask including S3 IO. We expect these speed-ups to be much larger if the data is loaded in memory instead of cloud storage, which we will show in future benchmarks. #### 1000 Scale Factor Next we scale up the data size by 10x while keeping the cluster size the same. Since we only have 244GB of memory and 1TB+ of tabular data, the DataFrame library will be required to perform disk spilling and out-of-core processing for all questions at nearly all stages of the query.
| Dataframe | Questions Completed | Total Time (seconds) | Relative to Daft | | --------- | :-----------------: | :------------------: | :--------------: | | Daft | 10/10 | 7774 | 1.0x | | Spark | 10/10 | 27161 | 3.5x | | Dask | 3/10 | Did not finish | 5.8x* | | Modin | 0/10 | Did not finish | No data | *\* Only for queries that completed.* From the results we see that only Daft and Spark are able to complete all the questions. Dask completes less than a third and Modin is unable to complete any due to OOMs and cluster crashes. Since we can only compare to Spark here, we see that Daft is **3.5x** faster including S3 IO. This shows that Daft and Spark are the only Dataframes in this comparison capable of processing data larger than memory, with Daft standing out as the significantly faster option. #### 1000 Scale Factor - Node Count Ablation Finally, we compare how Daft performs on varying size clusters on the terabyte scale dataset. We run the same Daft TPC-H questions on the same dataset as the [previous section](#1000-scale-factor) but sweep the worker node count.
We note two interesting results here: 1. Daft can process 1TB+ of analytical data on a single 61GB instance without being distributed (16x more data than memory). 2. Daft query times scale linearly with the number of nodes (e.g. 4 nodes being 4 times faster than a single node). This allows for faster queries while maintaining the same compute cost! ### Detailed Benchmarking Setup #### Benchmarking Code Our benchmarking scripts and code can be found in the [distributed-query-benchmarks](https://github.com/Eventual-Inc/distributed-query-benchmarking) GitHub repository. - TPC-H queries for Daft were written by us. - TPC-H queries for SparkSQL was adapted from [this repository](https://github.com/bodo-ai/Bodo/blob/ec18c281c450ca463b55194c45ba11bcf986d04f/benchmarks/tpch/pyspark_notebook.ipynb). - TPC-H queries for Dask and Modin were adapted from these repositories for questions [Q1-7](https://github.com/pola-rs/tpch) and [Q8-10](https://github.com/xprobe-inc/benchmarks/tree/main/tpch). ### Infrastructure Our infrastructure runs on an EKS Kubernetes cluster. - **Driver Instance**: i3.2xlarge - **Worker Instance**: i3.2xlarge - **Number of Workers**: 1/4/8 - **Networking**: All instances colocated in the same Availability Zone in the AWS us-west-2 region #### Data Data for the benchmark was stored in AWS S3. No node-level caching was performed, and data is read directly from AWS S3 on every attempt to simulate realistic workloads. - **Storage**: AWS S3 Bucket - **Format**: Parquet - **Region**: us-west-2 - **File Layout**: Each table is split into 32 (for the 100SF benchmark) or 512 (for the 1000SF benchmark) separate Parquet files. Parquet files for a given table have their paths prefixed with that table’s name, and are laid out in a flat folder structure under that prefix. Frameworks are instructed to read Parquet files from that prefix. - **Data Generation**: TPC-H data was generated using the utilities found in the open-sourced [Daft repository](https://github.com/Eventual-Inc/Daft/blob/main/benchmarking/tpch/pipelined_data_generation.py). This data is also available on request if you wish to reproduce any results! #### Cluster Setup ##### Dask and Ray To help us run the Distributed Dataframe libraries, we used Kubernetes for deploying Dask and Ray clusters. The configuration files for these setups can be found in our [open source benchmarking repository](https://github.com/Eventual-Inc/distributed-query-benchmarking/tree/main/cluster_setup). Our benchmarks for Daft and Modin were run on a [KubeRay](https://github.com/ray-project/kuberay) cluster, and our benchmarks for Dask was run on a [Dask-on-Kubernetes](https://github.com/dask/dask-kubernetes) cluster. Both projects are owned and maintained officially by the creators of these libraries as one of the main methods of deploying. ##### Spark For benchmarking Spark we used AWS EMR, the official managed Spark solution provided by AWS. For more details on our setup and approach, please consult our Spark benchmarks [README](https://github.com/Eventual-Inc/distributed-query-benchmarking/tree/main/distributed_query_benchmarking/spark_queries). #### Logs | Dataframe | Scale Factor | Nodes | Links | | --------- | ------------ | ------ | ------------------------- | | Daft | 1000 | 8 | 1. s3://daft-oss-public-data/benchmarking/logs/daft.0_1_3.1tb.8-i32xlarge.log | | Daft | 1000 | 4 | 1. s3://daft-oss-public-data/benchmarking/logs/daft.0_1_3.1tb.4-i32xlarge.log | | Daft | 1000 | 1 | 1. s3://daft-oss-public-data/benchmarking/logs/daft.1tb.1.i3-2xlarge.part1.log
2. s3://daft-oss-public-data/benchmarking/logs/daft.1tb.1.i3-2xlarge.part2.log | | Daft | 100 | 4 | 1. s3://daft-oss-public-data/benchmarking/logs/daft.0_1_3.100gb.4-i32xlarge.log | Spark | 1000 | 4 | 1. s3://daft-oss-public-data/benchmarking/logs/emr-spark.6_10_0.1tb.4-i32xlarge.log | Spark | 100 | 4 | 1. s3://daft-oss-public-data/benchmarking/logs/emr-spark.6_10_0.100gb.4-i32xlarge.log.gz |Dask (failed, multiple retries) | 1000 | 16 | 1. s3://daft-oss-public-data/benchmarking/logs/dask.2023_5_0.1tb.16-i32xlarge.0.log
2. s3://daft-oss-public-data/benchmarking/logs/dask.2023_5_0.1tb.16-i32xlarge.1.log
3. s3://daft-oss-public-data/benchmarking/logs/dask.2023_5_0.1tb.16-i32xlarge.2.log
4. s3://daft-oss-public-data/benchmarking/logs/dask.2023_5_0.1tb.16-i32xlarge.3.log | | Dask (failed, multiple retries)| 1000 | 4 | 1. s3://daft-oss-public-data/benchmarking/logs/dask.2023_5_0.1tb.4-i32xlarge.q126.log | | Dask (multiple retries) | 100 | 4 | 1. s3://daft-oss-public-data/benchmarking/logs/dask.2023_5_0.100gb.4-i32xlarge.0.log
2. s3://daft-oss-public-data/benchmarking/logs/dask.2023_5_0.100gb.4-i32xlarge.0.log
3. s3://daft-oss-public-data/benchmarking/logs/dask.2023_5_0.100gb.4-i32xlarge.1.log | | Modin (failed, multiple retries) | 1000 | 16 | 1. s3://daft-oss-public-data/benchmarking/logs/modin.0_20_1.1tb.16-i32xlarge.0.log
2. s3://daft-oss-public-data/benchmarking/logs/modin.0_20_1.1tb.16-i32xlarge.1.log | | Modin (failed, multiple retries) | 100 | 4 | 1. s3://daft-oss-public-data/benchmarking/logs/modin.0_20_1.100gb.4-i32xlarge.log | --- ### Connectors Index (`docs/connectors/index.md`) # Data Connectors Daft reads from and writes to a wide range of data sources — object storage, open table formats, catalogs, databases, files, and more. Please see [Daft Connectors API docs](../api/io.md) for API details. ## Object Storage Daft natively supports reading and writing data to major cloud object storage providers: | Provider | URL Protocols | Configuration | |----------|---------------|---------------| | [AWS S3](aws.md) | `s3://` | [`S3Config`][daft.io.S3Config] | | [Azure Blob Storage](azure.md) | `az://`, `abfs://` | [`AzureConfig`][daft.io.AzureConfig] | | [Google Cloud Storage](gcs.md) | `gs://`, `gcs://` | [`GCSConfig`][daft.io.GCSConfig] | | [Tencent Cloud COS](cos.md) | `cos://`, `cosn://` | [`CosConfig`][daft.io.CosConfig] | | [GooseFS](goosefs.md) | `goosefs://` | [`GooseFSConfig`][daft.io.GooseFSConfig] | ## Table Formats Daft reads and writes the major open table formats. ### Apache Hudi | Function | Description | |----------------------------------|------------------------------------| | [`read_hudi`][daft.io.read_hudi] | Read a Hudi table into a DataFrame | See also [Apache Hudi](hudi.md) for detailed integration. ### Apache Iceberg | Function | Description | |-----------------------------------------------------------|----------------------------------------| | [`read_iceberg`][daft.io.read_iceberg] | Read an Iceberg table into a DataFrame | | [`write_iceberg`][daft.dataframe.DataFrame.write_iceberg] | Write a DataFrame to an Iceberg table | See also [Apache Iceberg](iceberg.md) for detailed integration. ### Delta Lake | Function | Description | |---------------------------------------------------------------|------------------------------------------| | [`read_deltalake`][daft.io.read_deltalake] | Read a Delta Lake table into a DataFrame | | [`write_deltalake`][daft.dataframe.DataFrame.write_deltalake] | Write a DataFrame to a Delta Lake table | See also [Delta Lake](delta_lake.md) for detailed integration. ### Apache Paimon | Function | Description | |-------------------------------------------------------------|----------------------------------------| | [`read_paimon`][daft.io.read_paimon] | Read a Paimon table into a DataFrame | | [`write_paimon`][daft.dataframe.DataFrame.write_paimon] | Write a DataFrame to a Paimon table | See also [Apache Paimon](paimon.md) for detailed integration. ### Lance | Function | Description | |-------------------------------------------------------|---------------------------------------| | [`read_lance`][daft.io.read_lance] | Read a Lance dataset into a DataFrame | | [`write_lance`][daft.dataframe.DataFrame.write_lance] | Write a DataFrame to a Lance dataset | See also [Lance](lance.md) for detailed integration. ## Catalogs !!! warning "Warning" These APIs are early in their development. Please feel free to [open feature requests and file issues](https://github.com/Eventual-Inc/Daft/issues/new/choose). We'd love to hear what you would like, thank you! 🤘 Catalogs are a centralized place to organize and govern your data. A catalog is often responsible for creating objects such as tables and namespaces, managing transactions, and access control. Most importantly, the catalog abstracts away physical storage details, letting you focus on the logical structure of your data without worrying about file formats, partitioning schemes, or storage locations. Daft integrates with these catalog implementations: | Catalog | Description | |---------|-------------| | [AWS Glue](glue.md) | AWS Glue Data Catalog | | [AWS S3 Tables](s3tables.md) | Amazon S3 Tables catalog | | [Apache Gravitino](gravitino.md) | Apache Gravitino metadata lake | | [Unity Catalog (Databricks)](unity_catalog.md) | Databricks Unity Catalog | Daft integrates with these via its `Catalog` and `Table` interfaces. These are high-level APIs to manage catalog objects (tables and namespaces), while also making it easy to leverage Daft's existing `daft.read_` and `df.write_` APIs for open table formats like [Apache Iceberg](iceberg.md) and [Delta Lake](delta_lake.md). ### Example !!! note "Note" These examples use the Iceberg Catalog from the [Daft Sessions](../configuration/sessions-usage.md) tutorial. ```python import daft from daft import Catalog # iceberg_catalog from the 'Sessions' tutorial iceberg_catalog = load_catalog(...) # create a daft catalog from the pyiceberg catalog instance catalog = Catalog.from_iceberg(iceberg_catalog) # verify catalog """ Catalog('default') """ # we can read as a dataframe catalog.read_table("example.tbl").schema() """ ╭─────────────┬─────────╮ │ column_name ┆ type │ ╞═════════════╪═════════╡ │ x ┆ Boolean │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ y ┆ Int64 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ z ┆ Utf8 │ ╰─────────────┴─────────╯ """ # give a dataframe... df = daft.from_pylist([{ "x": False, "y": -1, "z": "xyz" }]) # we can write to tables catalog.write_table("example.tbl", df, mode="append") # we can get also get table instances t = catalog.get_table("example.tbl") # see 'Working with Tables' for what we can do! t """ Table('tbl') """ ``` ### Usage This section covers detailed usage of the current APIs with some code snippets. #### Working with Catalogs The `Catalog` interface allows you to perform catalog actions like `get_table` and `list_tables`. **Example** ```python import daft from daft import Catalog, Table # create a catalog from a pyiceberg catalog object _ = Catalog.from_iceberg(pyiceberg_catalog) # create a catalog from a unity catalog object _ = Catalog.from_unity(unity_catalog) # we can register various types as tables, note that all are equivalent example_dict = { "x": [ 1, 2, 3 ] } example_df = daft.from_pydict(example_dict) example_table = Table.from_df("temp", example_df) # create a catalog from a pydict mapping names to tables catalog = Catalog.from_pydict( { "R": example_dict, "S": example_df, "T": example_table, } ) # list available tables # Note: Pattern syntax is catalog-dependent. Native/memory and Postgres catalogs use SQL LIKE syntax (%, _). # Other catalogs like Iceberg use prefix matching. catalog.list_tables(pattern=None) """ ['R', 'S', 'T'] """ # get a table by name table_t = catalog.get_table("T") # table_t.show() """ ╭───────╮ │ x │ │ --- │ │ Int64 │ ╞═══════╡ │ 1 │ ├╌╌╌╌╌╌╌┤ │ 2 │ ├╌╌╌╌╌╌╌┤ │ 3 │ ╰───────╯ """ ``` #### Working with Tables The `Table` interface is a bridge from catalogs to dataframes. We can read tables into dataframes, and we can write dataframes to tables. You can work with a table independently of a catalog by using one of the factory methods, but it might not appear to provide that much utility over the existing `daft.read_` and `daft.write_` APIs. You would be correct in assuming that this is what is happening under the hood! The `Table` interface provides indirection over the table format itself and serves as a single abstraction for reading and writing that our catalogs can work with. **Examples** ```python from daft import Table from pyiceberg.table import StaticTable # suppose you have a pyiceberg table pyiceberg_table = StaticTable("metadata.json") # we can make it a daft table to use daft's table APIS table = Table.from_iceberg(pyiceberg_table) # we can read a dataframe like `daft.read_iceberg(pyiceberg_table)` df = table.read() # you can also create temporary tables from dataframes daft.create_temp_table("my_temp_table", daft.from_pydict({ ... })) # these will be resolved just like other tables df = daft.read_table("my_temp_table") ``` !!! note "Note" Today you can read from `pyiceberg` and `daft.unity` table objects. ### Reference !!! note "Note" For complete documentation, please see the [Catalog & Table API docs](../api/catalogs_tables.md). * [Catalog][daft.catalog.Catalog] - Interface for creating and accessing both tables and namespaces * [Identifier][daft.catalog.Identifier] - Paths to objects e.g. `catalog.namespace.table` * [Table][daft.catalog.Table] - Interface for reading and writing dataframes ## Databases ### Google Cloud Bigtable !!! warning "Experimental" This connector is experimental and the API may change. | Function | Description | |---------------------------------------------------------------|-----------------------------------------| | [`write_bigtable`][daft.dataframe.DataFrame.write_bigtable] | Write a DataFrame to Google Cloud Bigtable | See also [Bigtable](bigtable.md) for detailed integration. ### ClickHouse | Function | Description | |-----------------------------------------------------------------|--------------------------------------| | [`write_clickhouse`][daft.dataframe.DataFrame.write_clickhouse] | Write a DataFrame to ClickHouse | See also [ClickHouse](clickhouse.md) for detailed integration. ### PostgreSQL | Function | Description | |---------------------------------------------------------|---------------------------------------------------| | [`Catalog.from_postgres`][daft.catalog.Catalog.from_postgres] | Create a catalog from a PostgreSQL database | See also [Postgres](postgres.md) for detailed integration. ### SQL Databases | Function | Description | |---------------------------------------------------|------------------------------------------------| | [`read_sql`][daft.io.read_sql] | Read data from a SQL database into a DataFrame | | [`write_sql`][daft.dataframe.DataFrame.write_sql] | Write a DataFrame to a SQL database | See also [SQL Databases](sql.md) for detailed integration. ### Turbopuffer | Function | Description | |-----------------------------------------------------------------------|----------------------------------------------| | [`write_turbopuffer`][daft.dataframe.DataFrame.write_turbopuffer] | Write a DataFrame to a Turbopuffer namespace | See also [Turbopuffer](turbopuffer.md) for detailed usage. ## Files ### File References | Function | Description | |---------------------------------------------------|---------------------------------------------------------| | [`from_files`][daft.io.from_files] | Create a DataFrame of lazy file references from a glob pattern | | [`from_glob_path`][daft.from_glob_path] | Create a DataFrame of file paths from a glob pattern | See also [Files](files.md) for detailed usage. ### CSV | Function | Description | |---------------------------------------------------|--------------------------------------------------------| | [`read_csv`][daft.io.read_csv] | Read a CSV file or multiple CSV files into a DataFrame | | [`write_csv`][daft.dataframe.DataFrame.write_csv] | Write a DataFrame to CSV files | ### JSON | Function | Description | |-----------------------------------------------------|----------------------------------------------------------| | [`read_json`][daft.io.read_json] | Read a JSON file or multiple JSON files into a DataFrame | | [`write_json`][daft.dataframe.DataFrame.write_json] | Write a DataFrame to JSON files | ### Parquet | Function | Description | |-----------------------------------------------------------|----------------------------------------------------------------| | [`read_parquet`][daft.io.read_parquet] | Read a Parquet file or multiple Parquet files into a DataFrame | | [`write_parquet`][daft.dataframe.DataFrame.write_parquet] | Write a DataFrame to Parquet files | ### Text and Blob | Function | Description | |----------------------------------|--------------------------------------------------------------------| | [`read_text`][daft.io.read_text] | Read text files into a DataFrame as strings, with one row per line | | [`read_blob`][daft.io.read_blob] | Read files into a DataFrame as raw bytes, with one row per file | See also [Text Files](text.md) and [Blob Files](blob.md) for detailed usage. ### WARC | Function | Description | |----------------------------------|----------------------------------------------------------| | [`read_warc`][daft.io.read_warc] | Read a WARC file or multiple WARC files into a DataFrame | ### ORC | Function | Description | | --- | --- | | [`read_orc`][daft.io.read_orc] | Read raw ORC files into a DataFrame | See also [ORC Files](orc.md) for schema and file-reading behavior. ### WebDataset | Function | Description | |------------------------------------------------|----------------------------------------------------------------| | [`read_webdataset`][daft.io.read_webdataset] | Read multimodal WebDataset TAR shards into a DataFrame | See also [WebDataset](webdataset.md) for format details and lazy media reads. ### Video | Function | Description | |--------------------------------------------------|------------------------------------| | [`read_video_frames`][daft.io.read_video_frames] | Read video frames into a DataFrame | ### Generic File Source Options Options shared across the file-based readers above (compression, schema hints, and more). See [Generic File Source Options](generic-file-source-options.md) for the full list. ## Other Sources ### Apache Kafka !!! warning "Experimental" This connector is experimental. Currently only bounded batch reads are supported — there is no streaming/unbounded mode and no offset commit management. | Function | Description | |------------------------------------|------------------------------------------------------| | [`read_kafka`][daft.io.read_kafka] | Read messages from Kafka topic(s) into a DataFrame | See also [Apache Kafka](kafka.md) for detailed integration. ### MCAP !!! warning "Experimental" This connector is experimental. See [MCAP](mcap.md) for details. | Function | Description | |------------------------------------|----------------------------------------------| | [`read_mcap`][daft.io.read_mcap] | Read MCAP files into a DataFrame | See also [MCAP](mcap.md) for detailed integration. ### Hugging Face Datasets | Function | Description | |---------------------------------------------------|---------------------------------------------------| | [`read_huggingface`][daft.read_huggingface] | Read a Hugging Face dataset into a DataFrame | See also [Hugging Face Datasets](huggingface.md) for detailed usage. ## Custom ### Custom Connectors Implement your own source or sink to read from and write to systems Daft does not support natively: | Function | Description | |-----------------------------------------------------|--------------------------------------------------------------------| | [`DataSink`][daft.io.sink.DataSink] | Interface for writing data from DataFrames | | [`DataSource`][daft.io.source.DataSource] | Interface for reading data into DataFrames | | [`DataSourceTask`][daft.io.source.DataSourceTask] | Represents a partition of data that can be processed independently | | [`WriteResult`][daft.io.sink.WriteResult] | Wrapper for intermediate results written by a DataSink | | [`write_sink`][daft.dataframe.DataFrame.write_sink] | Write a DataFrame to the given DataSink | See also [Custom Connectors](custom.md) for detailed usage. ### Custom Catalogs Implement your own catalog by extending Daft's `Catalog` and `Table` interfaces. See [Custom Catalogs](custom-catalogs.md) for detailed usage. ## In-Memory Create DataFrames directly from in-memory Python objects: | Function | Description | |---------------------------------------------------|---------------------------------------------------------| | [`from_arrow`][daft.from_arrow] | Create a DataFrame from PyArrow Tables or RecordBatches | | [`from_dask_dataframe`][daft.from_dask_dataframe] | Create a DataFrame from a Dask DataFrame | | [`from_pandas`][daft.from_pandas] | Create a DataFrame from a Pandas DataFrame | | [`from_pydict`][daft.from_pydict] | Create a DataFrame from a python dictionary | | [`from_pylist`][daft.from_pylist] | Create a DataFrame from a python list | | [`from_ray_dataset`][daft.from_ray_dataset] | Create a DataFrame from a Ray Dataset | --- ### Custom Code Index (`docs/custom-code/index.md`) # Running Custom Python Code in Daft When Daft's built-in functions aren't sufficient for your needs, the `@daft.func` and `@daft.cls` decorators let you run your own Python code over each row of data. For custom aggregations, `@daft.udaf` lets you define aggregate functions with a three-stage pipeline. Simply decorate a Python function or class, and it becomes usable in Daft DataFrame operations. If you are looking to migrate from Legacy UDFs to Daft's next generation UDF API, please see our [migration guide](migration.md). ## Quick Example ```python import daft @daft.func def add_and_format(a: int, b: int) -> str: return f"Sum: {a + b}" df = daft.from_pydict({"x": [1, 2, 3], "y": [4, 5, 6]}) df = df.select(add_and_format(df["x"], df["y"])) df.show() ``` ``` ╭───────────╮ │ x │ │ --- │ │ Utf8 │ ╞═══════════╡ │ Sum: 5 │ ├╌╌╌╌╌╌╌╌╌╌╌┤ │ Sum: 7 │ ├╌╌╌╌╌╌╌╌╌╌╌┤ │ Sum: 9 │ ╰───────────╯ ``` ## Function Variants Daft supports multiple function variants to optimize for different use cases: - **Row-wise** (default): Regular Python functions process one row at a time - **Async row-wise**: Async Python functions process rows concurrently - **Generator**: Generator functions produce multiple output rows per input row - **Batch**: Process entire batches of data with `daft.Series` for high performance Daft automatically detects which variant to use for regular functions based on your function signature. For batch functions, you must use the `@daft.func.batch` decorator. ## End to End Example with Return Type Inference ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Distributed Index (`docs/distributed/index.md`) # Distributed Execution Daft is designed to scale seamlessly from your laptop to distributed clusters. By default, Daft runs using your local machine's resources with the **native runner (codenamed Swordfish)**, but when you need more compute power, you can easily scale to distributed execution across multiple machines. There are a few ways to run Daft on multiple machines: - For the easiest way to get started, check out our guide for [running on Kubernetes](kubernetes.md) - Alternatively, if you have an existing Ray setup, check out our guide for [running on Ray](ray.md) --- ### Examples Index (`docs/examples/index.md`) # Examples
- [Hand Tracking for Physical AI](hand-tracking-physical-ai.md) Read a LeRobot dataset, run hand tracking as a Daft UDF, and score the results against EgoDex ground truth. - [Reward Scoring for Physical AI](reward-scoring-physical-ai.md) Read LIBERO episodes with Daft's LeRobot reader and score each one with a reward model - per-frame task progress plus success probability - then filter low-quality episodes with a Daft query. - [Motion Trimming for Physical AI](motion-trimming-physical-ai.md) Read DROID episodes with Daft's LeRobot reader, score per-frame motion from the robot's own joint positions - no video decode - and reduce it to one trim window per episode. - [Multimodal Structured Outputs: Evaluating Image Understanding](mm_structured_outputs.md) Leverage image ablation to analyze textual bias in image understanding datasets. - [Voice AI Analytics with Faster-Whisper and embed_text](voice-ai-analytics.md) Transcribe audio files into segments with timestamps and embed content. - [MinHash Deduplication on Common Crawl](minhash-dedupe.md) Clean web text at scale with MinHash, LSH Banding, and Connected Components. - [Getting Started with Common Crawl](common-crawl-daft-tutorial.md) Daft provides a simple, performant, and responsible way to access Common Crawl data. - [Audio Transcription with Whisper](audio-transcription.md) Effortlessly transcribe audio to text at scale. - [Build a 100% GPU Utilization Text Embedding Pipeline featuring spaCy and Turbopuffer](text-embeddings.md) Generate and store millions of text embeddings in vector databases using distributed GPU processing and state-of-the-art models. - [Generate Images with Stable Diffusion](image-generation.md) Open Source image generation model on your own GPUs using Daft UDFs. - [Daft's Four UDF Pattern Tutorial](udf-patterns.md) One notebook, four UDF patterns, one dataset. Row-wise, generator, async, and stateful -- learn when to use each. - [Window Functions: The Great Chocolate Race](window-functions.md) Explore how window functions can reduce complex joins and groupby's to just a few simple operations. - [Running LLMs on the Red Pajamas Dataset](llms-red-pajamas.md) Perform similarity search on Stack Exchange questions using language models and embeddings.
!!! tip "More Examples" For more examples, check out our new [daft-examples](https://github.com/Eventual-Inc/daft-examples) repository! --- ### Extensions Overview (`docs/extensions/overview.md`) # Extensions Daft extensions are reusable libraries that add domain-specific functionality on top of Daft without requiring every specialized function, datatype, integration, or workflow to live directly in Daft core. There are two broad ways to build extensions: - **Python UDF-based extensions**: Python packages built on Daft's [custom-code APIs](../custom-code/func.md), including [`@daft.func`](../api/udf.md), [`@daft.func.batch`](../api/udf.md), [`@daft.cls`](../api/udf.md), and [`@daft.method.batch`](../api/udf.md). - **Native ABI extensions**: shared-library-backed packages that use Daft's Arrow C Data Interface ABI for lower-level, vectorized native execution. Both models can expose clean Python APIs that feel like ordinary Daft expression functions. The implementation details can stay hidden behind functions, classes, and expressions. ## Python UDF-Based Extensions The fastest way to build a Daft extension is often pure Python. Using [`@daft.func`](../api/udf.md), [`@daft.func.batch`](../api/udf.md), [`@daft.cls`](../api/udf.md), and [`@daft.method.batch`](../api/udf.md), contributors can package reusable Python logic that runs inside Daft's distributed execution engine. This is ideal for libraries that orchestrate existing Python ecosystems, external services, ML models, GPUs, data maintenance tasks, or domain-specific workflows. These extensions do not require a native shared library or the Arrow C ABI. They are normal Python packages that expose higher-level Daft APIs. [`daft-lance`](https://github.com/daft-engine/daft-lance) is an example of this model. It extends Daft with Lance-specific distributed operations such as file compaction, scalar indexing, column merging, and REST catalog operations. Internally, it uses Daft's Python UDF and class-UDF APIs to distribute Lance tasks across a Daft query, while users interact with functions like `compact_files`, `create_scalar_index`, and `merge_columns_df`. This pattern also exists inside Daft itself. The [`daft.functions.ai`](../api/ai.md) module exposes high-level functions like [`prompt`](../api/functions/prompt.md), [`embed_text`](../api/functions/embed_text.md), [`embed_image`](../api/functions/embed_image.md), [`classify_text`](../api/functions/classify_text.md), and [`classify_image`](../api/functions/classify_image.md). To users, these look like normal Daft expression functions. Under the hood, they use Daft's UDF and class-UDF machinery, including batching, concurrency controls, retries, and [GPU resource hints](../custom-code/gpu.md). The file APIs follow a similar product pattern. [`daft.File`](../api/datatypes/file_types.md) values and helpers like [`file`](../api/functions/file.md), [`audio_file`](../api/functions/audio_file.md), [`video_file`](../api/functions/video_file.md), [`file_path`](../api/functions/file_path.md), and [`file_size`](../api/functions/file_size.md) give users expression-level building blocks for working with files. Those file objects can then flow into Python UDFs, model pipelines, and domain-specific libraries without users needing to think about execution details. ## Native ABI Extensions For contributors who need lower-level, vectorized performance, Daft also supports native extensions through a C ABI based on the [Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html). Native ABI extensions can add high-performance scalar functions, aggregate functions, Python expression wrappers, and extension-backed datatypes. They ship as pip-installable Python packages that bundle a native shared library. Users import the package, load it into a Daft [`Session`](../api/sessions.md), and call ordinary Python wrappers in their DataFrame expressions. Rust currently has the most ergonomic SDK through [`daft-ext`](https://github.com/Eventual-Inc/Daft/tree/main/src/daft-ext), while C++ is demonstrated through the raw ABI. Other systems languages are possible if they can produce a shared library, export the expected C ABI, and read/write Arrow C Data Interface arrays. Daft's own examples include: - [`examples/hello`](https://github.com/Eventual-Inc/Daft/tree/main/examples/hello): a minimal Rust native extension that registers a `greet` scalar function. - [`examples/dvector`](https://github.com/Eventual-Inc/Daft/tree/main/examples/dvector): a pgvector-style native extension for vector distance functions such as `l2_distance`, `cosine_distance`, `inner_product`, and `jaccard_distance`. - [`examples/hello_cpp`](https://github.com/Eventual-Inc/Daft/tree/main/examples/hello_cpp): a pure C++ native extension using Apache Arrow C++ and the raw Daft C ABI. ## What Can Contributors Build? Contributors can build: - Pure-Python UDF extension libraries using [`@daft.func`](../api/udf.md), [`@daft.func.batch`](../api/udf.md), [`@daft.cls`](../api/udf.md), and [`@daft.method.batch`](../api/udf.md) - Distributed task libraries like [`daft-lance`](https://github.com/daft-engine/daft-lance) - GPU-backed model inference extensions - AI, file, media, and multimodal processing libraries that hide UDFs behind expression APIs - Native scalar functions - Native aggregate functions / UDAFs - Python expression wrappers - Extension-backed logical datatypes such as [`DataType.extension(...)`](../api/datatypes/all_datatypes.md) ## Where to Start - Browse installable projects in [Community Extensions](community.md). - Learn how to build a native extension in the [Authoring Guide](authoring.md). - Browse projects using Daft in [Built on Daft](projects.md). - Learn more about Daft's [Python UDFs](../custom-code/func.md). --- ### Modalities Overview (`docs/modalities/overview.md`) # Modalities Overview **Daft is designed to work with any modality.** Artificial Intelligence now natively understands text, images, audio, video, and documents, but legacy engines were never designed to feed these formats to large models. Daft closes that gap, giving you one distributed engine that processes any modality, respects memory limits, and keeps GPUs fed so you can build the pipeline once and scale it anywhere.
- 🔠 [**Text**](text.md) Normalize, chunk, dedupe, prompt, and embed text data. - 🌄 [**Images**](images.md) Work with visual data and image processing. Use `daft.ImageFile` for lazy metadata extraction and filtered decoding. - 🔉 [**Audio**](audio.md) Read, extract metadata, resample audio files. - 🎥 [**Video**](videos.md) Working with video files and metadata. - 📄 [**Documents**](documents.md) Extract text and image data from PDF documents. - {} [**JSON and Nested Data**](json.md) Parse, query, and manipulate semi-structured and hierarchical data. - ⊹ [**Embeddings**](embeddings.md) Generate vector representations for RAG and AI search. - 📁 [**Generic Files and URLs**](files.md) Take advantage of Daft's built-in URL functions and `daft.File` types
### **[Custom Modalities](custom.md)** The most important modality might be one we haven't explored yet. Daft makes it easy to define your own modality with [custom connectors](../connectors/custom.md) to read and write any kind of data, and use [user-defined functions](../custom-code/index.md) to process custom Python code efficiently and reliably at scale. --- ### Sql Index (`docs/sql/index.md`) # Daft SQL # SQL Reference Welcome to Daft SQL Reference. Daft's [SQL](https://en.wikipedia.org/wiki/SQL) dialect closely follows both DuckDB and PostgreSQL. ## Example Please see [Sessions and Catalogs](../configuration/sessions-usage.md) for a detailed look at connecting data sources to Daft SQL. ```python import daft from daft import Session # create a session sess = Session() # create temp tables sess.create_temp_table("T", daft.from_pydict({ "a": [ 0, 1 ] })) sess.create_temp_table("S", daft.from_pydict({ "b": [ 1, 0 ] })) # execute sql sess.sql("SELECT * FROM T, S").show() """ ╭───────┬───────╮ │ a ┆ b │ │ --- ┆ --- │ │ Int64 ┆ Int64 │ ╞═══════╪═══════╡ │ 0 ┆ 1 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 1 ┆ 1 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 0 ┆ 0 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 1 ┆ 0 │ ╰───────┴───────╯ """ ``` ## Usage ### SQL with DataFrames Daft's [`daft.sql`][daft.sql.sql.sql] function automatically detects any [`daft.DataFrame`][daft.DataFrame] objects in your current Python environment to let you query them easily by name. === "⚙️ SQL" ```python # Note the variable name `my_special_df` my_special_df = daft.from_pydict({"A": [1, 2, 3], "B": [1, 2, 3]}) # Use the SQL table name "my_special_df" to refer to the above DataFrame! sql_df = daft.sql("SELECT A, B FROM my_special_df") sql_df.show() ``` ``` {title="Output"} ╭───────┬───────╮ │ A ┆ B │ │ --- ┆ --- │ │ Int64 ┆ Int64 │ ╞═══════╪═══════╡ │ 1 ┆ 1 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 2 ┆ 2 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 3 ┆ 3 │ ╰───────┴───────╯ (Showing first 3 of 3 rows) ``` In the above example, we query the DataFrame called `"my_special_df"` by simply referring to it in the SQL command. This produces a new DataFrame `sql_df` which can natively integrate with the rest of your Daft query. You can also use table functions to query sources directly. === "🐍 Python" ```python daft.sql("SELECT * FROM read_parquet('s3://...')") daft.sql("SELECT * FROM read_iceberg('s3://.../metadata.json')") ``` ### Table Function Options Table functions take the path as the first positional argument, plus any of the options below. Named arguments use either `=>` or `:=`: === "🐍 Python" ```python daft.sql("SELECT * FROM read_csv('data.csv', has_headers => false)") daft.sql("SELECT * FROM read_csv('data.csv', has_headers := false)") ``` `read_parquet`, `read_csv`, `read_json`, and `read_deltalake` also accept the path itself as a named `path` argument: === "🐍 Python" ```python daft.sql("SELECT * FROM read_csv(path => 'data.csv')") ``` A list of paths is written as a SQL array: === "🐍 Python" ```python daft.sql("SELECT * FROM read_parquet(['a.parquet', 'b.parquet'])") ``` | Function | Options | |---|---| | `read_parquet` | `infer_schema`, `schema`, `coerce_int96_timestamp_unit`, `chunk_size`, `multithreaded`, `io_config`, `file_path_column`, `hive_partitioning`, `ignore_corrupt_files` | | `read_csv` | `infer_schema`, `schema`, `has_headers`, `delimiter`, `double_quote`, `quote`, `escape_char`, `comment`, `allow_variable_columns`, `io_config`, `file_path_column`, `hive_partitioning`, `buffer_size`, `chunk_size`, `ignore_corrupt_files` | | `read_json` | `infer_schema`, `schema`, `io_config`, `file_path_column`, `hive_partitioning`, `buffer_size`, `chunk_size`, `skip_empty_files` | | `read_iceberg` | `snapshot_id`, `branch`, `tag`, `io_config`, `ignore_corrupt_files` | | `read_deltalake` | `io_config` | These mirror the corresponding Python reader arguments — see [`read_parquet`][daft.read_parquet], [`read_csv`][daft.read_csv], [`read_json`][daft.read_json], [`read_iceberg`][daft.read_iceberg], and [`read_deltalake`][daft.read_deltalake] for what each one does, and [Generic File Source Options](../connectors/generic-file-source-options.md) for `ignore_corrupt_files` in depth. A `schema` is given as a struct literal mapping column names to type names: === "🐍 Python" ```python daft.sql(""" SELECT * FROM read_csv('data.csv', schema := {'a': 'int64', 'b': 'string'}) """) ``` For Iceberg, `snapshot_id`, `branch`, and `tag` select which version to read and are mutually exclusive: === "🐍 Python" ```python daft.sql("SELECT * FROM read_iceberg('/warehouse/db/t/metadata/v3.metadata.json', branch => 'audit')") ``` `ignore_corrupt_files` skips unreadable files instead of failing the query, and `df.skipped_corrupt_files` reports what was skipped once the DataFrame is materialized: === "🐍 Python" ```python df = daft.sql("SELECT * FROM read_csv('s3://my-bucket/data/**/*.csv', ignore_corrupt_files => true)") df.collect() print(df.skipped_corrupt_files) ``` ### SQL Expressions SQL has the concept of expressions as well. Here is an example of a simple addition expression, adding columns `A` and `B` in SQL to produce a new column `C`. We also present here the equivalent query for SQL and DataFrame. Notice how similar the concepts are! === "⚙️ SQL" ```python df = daft.from_pydict({"A": [1, 2, 3], "B": [1, 2, 3]}) df = daft.sql("SELECT A + B as C FROM df") df.show() ``` === "🐍 Python" ``` python expr = (daft.col("A") + daft.col("B")).alias("C") df = daft.from_pydict({"A": [1, 2, 3], "B": [1, 2, 3]}) df = df.select(expr) df.show() ``` ``` {title="Output"} ╭───────╮ │ C │ │ --- │ │ Int64 │ ╞═══════╡ │ 2 │ ├╌╌╌╌╌╌╌┤ │ 4 │ ├╌╌╌╌╌╌╌┤ │ 6 │ ╰───────╯ (Showing first 3 of 3 rows) ``` In the above query, both the SQL version of the query and the DataFrame version of the query produce the same result. Under the hood, they run the same Expression `col("A") + col("B")`! One really cool trick you can do is to use the [`daft.sql_expr`][daft.sql.sql.sql_expr] function as a helper to easily create Expressions. The following are equivalent: === "⚙️ SQL" ```python sql_expr = daft.sql_expr("A + B as C") print("SQL expression:", sql_expr) ``` === "🐍 Python" ``` python py_expr = (daft.col("A") + daft.col("B")).alias("C") print("Python expression:", py_expr) ``` ``` {title="Output"} SQL expression: col(A) + col(B) as C Python expression: col(A) + col(B) as C ``` This means that you can pretty much use SQL anywhere you use Python expressions, making Daft extremely versatile at mixing workflows which leverage both SQL and Python. As an example, consider the filter query below and compare the two equivalent Python and SQL queries: === "⚙️ SQL" ```python df = daft.from_pydict({"A": [1, 2, 3], "B": [1, 2, 3]}) # Daft automatically converts this string using `daft.sql_expr` df = df.where("A < 2") df.show() ``` === "🐍 Python" ``` python df = daft.from_pydict({"A": [1, 2, 3], "B": [1, 2, 3]}) # Using Daft's Python Expression API df = df.where(df["A"] < 2) df.show() ``` ``` {title="Output"} ╭───────┬───────╮ │ A ┆ B │ │ --- ┆ --- │ │ Int64 ┆ Int64 │ ╞═══════╪═══════╡ │ 1 ┆ 1 │ ╰───────┴───────╯ (Showing first 1 of 1 rows) ``` Pretty sweet! Of course, this support for running Expressions on your columns extends well beyond arithmetic as we'll see in the next section on SQL Functions. ### SQL Functions SQL also has access to all of Daft's powerful [`daft.Expression`][daft.Expression] functionality through SQL functions. However, unlike the Python Expression API which encourages method-chaining (e.g. `col("a").download().decode_image()`), in SQL you have to do function nesting instead (e.g. `"image_decode(url_download(a))"`). Here is an example of an equivalent function call in SQL vs Python: === "⚙️ SQL" ```python df = daft.from_pydict({"urls": [ "https://user-images.githubusercontent.com/17691182/190476440-28f29e87-8e3b-41c4-9c28-e112e595f558.png", "https://user-images.githubusercontent.com/17691182/190476440-28f29e87-8e3b-41c4-9c28-e112e595f558.png", "https://user-images.githubusercontent.com/17691182/190476440-28f29e87-8e3b-41c4-9c28-e112e595f558.png", ]}) df = daft.sql("SELECT image_decode(url_download(urls)) FROM df") df.show() ``` === "🐍 Python" ``` python df = daft.from_pydict({"urls": [ "https://user-images.githubusercontent.com/17691182/190476440-28f29e87-8e3b-41c4-9c28-e112e595f558.png", "https://user-images.githubusercontent.com/17691182/190476440-28f29e87-8e3b-41c4-9c28-e112e595f558.png", "https://user-images.githubusercontent.com/17691182/190476440-28f29e87-8e3b-41c4-9c28-e112e595f558.png", ]}) df = df.select(daft.col("urls").download().decode_image()) df.show() ``` ``` {title="Output"} ╭──────────────╮ │ urls │ │ --- │ │ Image[MIXED] │ ╞══════════════╡ │ │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ │ ╰──────────────╯ (Showing first 3 of 3 rows) ``` --- ### SUMMARY (`docs/SUMMARY.md`) * Guide * [Overview](index.md) * [Quickstart](quickstart.md) * [Installation](install.md) * AI Functions * [Overview](ai-functions/overview.md) * [Prompt](ai-functions/prompt.md) * [Embed](ai-functions/embed.md) * [Classify](ai-functions/classify.md) * [Jev](ai-functions/jev.md) * [Providers](ai-functions/providers.md) * Modalities * [Overview](modalities/overview.md) * [Text](modalities/text.md) * [Images](modalities/images.md) * [Audio](modalities/audio.md) * [Videos](modalities/videos.md) * [Documents](modalities/documents.md) * [JSON and Nested Data](modalities/json.md) * [Files and URLs](modalities/files.md) * [Tensors](modalities/tensors.md) * [Embeddings](modalities/embeddings.md) * [Custom Modalities](modalities/custom.md) * User Defined Functions * [Overview](custom-code/index.md) * [Functions](custom-code/func.md) * [Classes & Methods](custom-code/cls.md) * [Aggregate UDFs](custom-code/udaf.md) * [Working with GPUs](custom-code/gpu.md) * [Legacy UDF Migration Guide](custom-code/migration.md) * [Legacy UDFs](custom-code/udfs.md) * Common Use Cases * [Batch Inference](use-case/batch-inference.md) * Datasets * [Common Crawl](datasets/common-crawl.md) * [LeRobot](datasets/lerobot.md) * [DROID](datasets/droid.md) * Data Connectors * [Overview](connectors/index.md) * Object Storage * [AWS S3](connectors/aws.md) * [Azure Blob Store](connectors/azure.md) * [Google Cloud Storage](connectors/gcs.md) * [COS (Tencent Cloud)](connectors/cos.md) * [GooseFS](connectors/goosefs.md) * Table Formats * [Apache Hudi](connectors/hudi.md) * [Apache Iceberg](connectors/iceberg.md) * [Delta Lake](connectors/delta_lake.md) * [Apache Paimon](connectors/paimon.md) * [Lance](connectors/lance.md) * Catalogs * [AWS Glue](connectors/glue.md) * [AWS S3 Tables](connectors/s3tables.md) * [Apache Gravitino](connectors/gravitino.md) * [Unity Catalog (Databricks)](connectors/unity_catalog.md) * Databases * [Bigtable](connectors/bigtable.md) * [ClickHouse](connectors/clickhouse.md) * [Postgres](connectors/postgres.md) * [SQL Databases](connectors/sql.md) * [Turbopuffer](connectors/turbopuffer.md) * Files * [Files](connectors/files.md) * [Text Files](connectors/text.md) * [Blob Files](connectors/blob.md) * [WebDataset](connectors/webdataset.md) * [ORC Files](connectors/orc.md) * [Generic File Source Options](connectors/generic-file-source-options.md) * Other Sources * [Apache Kafka](connectors/kafka.md) * [MCAP](connectors/mcap.md) * [Hugging Face Datasets](connectors/huggingface.md) * Custom * [Custom Connectors](connectors/custom.md) * [Custom Catalogs](connectors/custom-catalogs.md) * Extensions * [Overview](extensions/overview.md) * [Community Extensions](extensions/community.md) * [Built on Daft](extensions/projects.md) * [Authoring Guide](extensions/authoring.md) * [Architecture](architecture/index.md) * Scaling & Performance * Scaling Out and Deployment * [Distributed Execution](distributed/index.md) * [Running on Kubernetes](distributed/kubernetes.md) * [Running on Ray](distributed/ray.md) * Optimization * [Managing Memory Usage](optimization/memory.md) * [Partitioning and Batching](optimization/partitioning.md) * [Shuffle Algorithms](optimization/shuffle.md) * [Join Strategies](optimization/join-strategies.md) * [Checkpointing](use-case/checkpointing.md) * Observability * [Dashboard](observability/dashboard.md) * [Progress Indicators](observability/progress-indicators.md) * [Logging](observability/logging.md) * [Telemetry](observability/telemetry.md) * [Sessions, Catalogs, and Tables](configuration/sessions-usage.md) * Resources * [Roadmap](roadmap.md) * [Benchmarks](benchmarks/index.md) * [Community ↗](https://daft.ai/slack) * [Release Notes ↗](https://github.com/Eventual-Inc/Daft/releases) * [Usage Telemetry](telemetry.md) * Examples * [Examples](examples/index.md) * [Hand Tracking for Physical AI](examples/hand-tracking-physical-ai.md) * [Reward Scoring for Physical AI](examples/reward-scoring-physical-ai.md) * [Motion Trimming for Physical AI](examples/motion-trimming-physical-ai.md) * [Multimodal Structured Outputs: Evaluating Image Understanding](examples/mm_structured_outputs.md) * [Voice AI Analytics with Faster-Whisper and embed_text](examples/voice-ai-analytics.md) * [Web Text Deduplication](examples/minhash-dedupe.md) * [Audio Transcription](examples/audio-transcription.md) * [Generate Text Embeddings for Turbopuffer](examples/text-embeddings.md) * [Running LLMs on the Red Pajamas Dataset](examples/llms-red-pajamas.md) * [Generate Images from Text with Stable Diffusion](examples/image-generation.md) * [Querying Image Data](examples/querying-images.md) * [MNIST Digit Classification](examples/mnist.md) * [UDF Patterns](examples/udf-patterns.md) * [Window Functions](examples/window-functions.md) * [Working with Common Crawl Data](examples/common-crawl-daft-tutorial.md) * [Document Processing](examples/document-processing.md) * Python API * [Python API](api/index.md) * [AI](api/ai.md) * [I/O](api/io.md) * [DataFrame](api/dataframe.md) * [Datasets](api/datasets.md) * [Expressions](api/expressions.md) * [Functions](api/functions/) * [User-Defined Functions](api/udf.md) * Data Types * [DataType](api/datatypes/all_datatypes.md) * [File Types](api/datatypes/file_types.md) * [Type Conversions](api/datatypes/type_conversions.md) * [Casting](api/datatypes/casting.md) * [Window](api/window.md) * [Sessions](api/sessions.md) * [Catalogs & Tables](api/catalogs_tables.md) * [Schema](api/schema.md) * [Aggregations](api/aggregations.md) * [Series](api/series.md) * [Configuration](api/config.md) * [Miscellaneous](api/misc.md) * SQL Reference * [SQL Reference](sql/index.md) * Statements * [SELECT](sql/statements/select.md) * [DESCRIBE](sql/statements/describe.md) * [SHOW](sql/statements/show.md) * [USE](sql/statements/use.md) * [Data Types](sql/datatypes.md) * [Identifiers](sql/identifiers.md) * [Window Functions](sql/window_functions.md) * Contributing * [Overview](contributing/overview.md) * [Development Guide](contributing/development.md) * [Contributing AI Functions](contributing/contributing-ai-functions.md) * [Daft Skills](skills.md) --- ### Install (`docs/install.md`) # How to Install Daft This guide helps you install Daft based on your specific needs and environment. For most users, install Daft with a single command: ```bash pip install -U daft ``` ## Optional Dependencies Depending on your use case, you may need to install Daft with additional dependencies.

Use Cases:

Installation Command:

1
 
pip install -U daft
You can also install Daft with all extra dependencies: ```bash pip install -U "daft[all]" ``` ## Nightly Builds Nightly builds are published daily from the latest `main` branch. These are useful for testing upcoming features and bug fixes before they are released on PyPI. ```bash pip install daft --pre --extra-index-url https://nightly.daft.ai ``` !!! warning "Stability" Nightly builds may contain unstable or experimental changes. They are not recommended for production use. ## Troubleshooting Legacy CPU Support If you encounter `Illegal instruction` errors, your CPU may lack support for advanced instruction sets like [AVX](https://en.wikipedia.org/wiki/Advanced_Vector_Extensions). Use the LTS version instead: ```bash pip install -U daft-lts ``` !!! warning "Performance Impact" The LTS version uses limited CPU instructions and cannot leverage vectorized operations, resulting in slower performance. Only use this if the standard package fails to run. --- ### Roadmap (`docs/roadmap.md`) # Roadmap *Last updated: March 2026* This roadmap outlines the big picture of what the Daft team plans to work on in the coming year, as well as some of the features to expect from these. Please note that items on this roadmap are subject to change at any time. If there are features you would like to implement, we welcome and encourage open-source contributions. Our team is happy to provide guidance, help scope the work, and review PRs. Feel free to open an issue or PR on [GitHub ↗](https://github.com/Eventual-Inc/Daft) or join our [Daft Slack Community ↗](https://daft.ai/slack). ## Performance 1. **Performant large scale shuffles** - Support for native shuffling services for data transfer in our distributed engine (Flotilla). This includes a specialized peer-to-peer service based on Arrow Flight RPC. ([discussion #6472 ↗](https://github.com/Eventual-Inc/Daft/discussions/6472)) 2. **Distributed engine optimization** - Optimizing our distributed engine (Flotilla) to dynamically schedule and execute tasks to minimize worker setup overhead and contention, add dynamic partitioning. 3. **Memory management** - Tracking and managing memory usage within our local runner to reduce chances of OOMs, inform backpressure and dynamic batching. 4. **Hash-based operator improvements** - Improve partitioning and hash-table building techniques within various hash-based operators, particularly groupby aggregations and hash joins in low-cardinality cases. ## Key Features 1. **Checkpointing** - Ability to checkpoint and resume long running workloads from a given checkpoint. ([discussion #6446](https://github.com/Eventual-Inc/Daft/discussions/6446)) 2. **Kubernetes support** - Distributed Daft running directly on Kubernetes. 3. **Multiple distributed backends** - Enable Daft to run distributed on multiple backends with a generic interface and continued support of Ray as a backend. ## Observability 1. **Daft Dashboard** - Observability dashboard with support for Flotilla, providing cluster, task and partition level observability. 2. **Improved Debugging** - Support for capturing and retrieving debugging information and run history. 3. **Memory and CPU observability** - Ability to capture and retrieve resource utilization metrics. ## Extensibility 1. **Native extensions** - Support for Daft native extensions via the Arrow C interfaces; enabling developers to extend Daft in any language which supports C-based interop. 2. **Data Source refactor** - Simplify the source pipeline to make adding native Rust sources straightforward. 3. **Arrow2 deprecation** ([discussion #5741](https://github.com/Eventual-Inc/Daft/discussions/5741)) ## Future Work The following features are in consideration, but are not currently on our roadmap. We're sharing these to highlight opportunities for open source contributions, invite discussion around implementation approaches, and provide further visibility. These features have been tagged with `help wanted` and `good first issue` on [Daft repo ↗](https://github.com/Eventual-Inc/Daft). 1. **Improved Delta Lake support** (see [roadmap for Delta Lake ↗](https://github.com/Eventual-Inc/Daft/issues/2457)) - Support for reading tables with deletion vectors ([issue #1954 ↗](https://github.com/Eventual-Inc/Daft/issues/1954)) - Support for reading tables with column mappings ([issue #1955 ↗](https://github.com/Eventual-Inc/Daft/issues/1955)) 2. **VARIANT type** - Support for the VARIANT type into Daft's type system which is compatible with the parquet VARIANT. 3. **`Result` type** - Add a `Result` type (Either) to Daft's type system to support graceful handling of fallible operations. If you are interested in working on any of these features, feel free to open an issue or start a discussion on [GitHub ↗](https://github.com/Eventual-Inc/Daft) or join our [Daft Slack Community ↗](https://daft.ai/slack). Our team can provide technical direction and help scope the work appropriately. Thank you in advance 💜 --- ### Robots (`docs/robots.txt`) User-agent: * Disallow: /en/ Allow: /en/stable Sitemap: https://docs.daft.ai/en/stable/sitemap.xml --- ### Skills (`docs/skills.md`) # Daft Skills Daft ships with built-in skills that help AI coding agents work effectively with Daft. ## Available skills ### [daft-udf-tuning](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-udf-tuning/SKILL.md) Optimize User-Defined Functions for performance. **Triggers:** GPU inference setup, slow UDFs, async or batch processing questions. **What it helps with:** - Choosing between `@daft.func` (stateless), `@daft.cls` (stateful), and `@daft.func.batch` (vectorized) - Async I/O patterns for web APIs - GPU batch inference with PyTorch/model loading via `@daft.cls(gpus=N)` - Tuning `max_concurrency`, `batch_size`, `gpus`, and `into_batches(N)` ### [daft-distributed-scaling](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-distributed-scaling/SKILL.md) Scale single-node workflows to distributed execution. **Triggers:** Performance optimization, handling large datasets, distributed execution. **What it helps with:** - Choosing between shuffle (`repartition(N)`) and streaming (`into_batches(N)`) strategies - Calculating optimal partition counts for your cluster - Avoiding OOM errors with heavy data (images, tensors) - Advanced tuning formulas for repartitioning and streaming ### [daft-docs-navigation](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-docs-navigation/SKILL.md) Navigate Daft's documentation for APIs, concepts, and examples. **Triggers:** General questions about Daft APIs, concepts, or searching the docs. **What it helps with:** - Navigating key doc locations: API reference, optimization, distributed, UDFs, connectors - Searching for API usage in the `docs/` directory - Browsing the docs structure via `docs/SUMMARY.md` ## Installation ### Already cloned Daft (automatic) If you're working inside a Daft clone, the skills are already available. No setup needed - Claude Code picks them up from `.claude/skills/`. ### Install with `npx skills add` To use these Daft skills outside the repo (across many agents including Claude Code, Cursor, Codex, Gemini CLI, Copilot, and more): ```bash # Install all skills npx skills add Eventual-Inc/Daft # Install a specific skill npx skills add Eventual-Inc/Daft --skill daft-udf-tuning ``` Follow the prompts to choose skills, agents, scope (project or global), and whether to symlink or copy. ### Claude Code plugin These Daft skills are also available as a Claude Code plugin. ```bash claude plugin marketplace add Eventual-Inc/Daft claude plugin install daft@Eventual-Inc ``` --- ### Telemetry (`docs/telemetry.md`) # Telemetry To help core developers improve Daft, we collect non-identifiable statistics on Daft usage in order to better understand how Daft is used, common bugs and performance bottlenecks. Data is collected via [Scarf](https://scarf.sh) and [osstelemetry.io](https://osstelemetry.io). We take the privacy of our users extremely seriously, and telemetry in Daft is built to be: 1. Easy to opt-out: To disable telemetry, set the environment variable `DO_NOT_TRACK=true` 2. Non-identifiable: No session IDs or user identifiers are collected 3. Metadata-only: We do not collect any of our users' proprietary code or data We **do not** sell or buy any of the data that is collected in telemetry. !!! info "*Daft telemetry is enabled in versions >= v0.0.21*" ## What data do we collect? To audit what data is collected, please see `scarf_telemetry.py`. In short, we collect the following: 1. On import, we track the version of Daft, OS, Python version, and system architecture. 2. On DataFrame executions, we track the same system information as we do on imports, in addition to the runner being used, i.e. native or Ray. --- ### Ai Functions Classify (`docs/ai-functions/classify.md`) # Text and Image Classification with Daft Daft provides powerful classification functions for both text and images, making it easy to label and categorize your data at scale. Whether you're working with sentiment analysis, content categorization, or visual recognition tasks, Daft's classification functions integrate seamlessly with popular models from Hugging Face and other providers. !!! tip "Choosing the Right Approach" - Use `classify_text` and `classify_image` for fast, efficient classification with pre-defined labels - Use `prompt` for more flexible classification that requires explanations or complex reasoning - For custom models, Daft supports both Hugging Face Transformers and OpenAI-compatible providers ## Text Classification The `classify_text` function provides fast, efficient text classification using zero-shot models or fine-tuned classifiers. It's ideal for tasks like sentiment analysis, topic categorization, and intent detection where you have predefined labels. ```python import daft from daft.functions import classify_text df = daft.from_pydict({ "text": [ "One day I will see the world", "I've always enjoyed preparing dinner for my family", ], }) df = df.with_column( "label", classify_text( daft.col("text"), labels=['travel', 'cooking', 'dancing'], provider="transformers", model="facebook/bart-large-mnli", ), ) df.show() ``` ```{title="Output"} ╭────────────────────────────────┬─────────╮ │ text ┆ label │ │ --- ┆ --- │ │ String ┆ String │ ╞════════════════════════════════╪═════════╡ │ One day I will see the world ┆ travel │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ I've always enjoyed preparing… ┆ cooking │ ╰────────────────────────────────┴─────────╯ (Showing first 2 of 2 rows) ``` ## Image Classification The `classify_image` function uses vision models like CLIP for zero-shot image classification. This example shows a complete pipeline including image preprocessing steps: ```python import daft from daft.functions import classify_image, decode_image, convert_image, resize df = ( # Discover a few images from HuggingFace daft.from_glob_path("hf://datasets/datasets-examples/doc-image-3/images") # Read the 4 PNG, JPEG, TIFF, WEBP Images .with_column("image_bytes", daft.col("path").download()) # Decode the image bytes into a daft Image DataType .with_column("image_type", decode_image(daft.col("image_bytes"))) # Convert Image to RGB and resize the image to 224x224 .with_column( "image_resized", resize(convert_image(daft.col("image_type"), "RGB"), 224, 224) ) # Classify the image .with_column( "image_label", classify_image( daft.col("image_resized"), provider="transformers", labels=["bulbasaur", "catapie", "voltorb", "electrode"], model="openai/clip-vit-base-patch32", ), ) ) df.show() ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` !!! note "When to Use Prompts for Classification" Use `classify_image` and `classify_text` when you have a fixed set of labels and need fast, efficient classification. Use `prompt` when you need: - Explanations for classification decisions - Complex reasoning or multi-step logic - Dynamic or unstructured label sets - Additional context beyond a simple label ## Flexible Classification with Prompts When you need more flexibility—such as generating explanations, handling complex reasoning, or working with unstructured labels—use the `prompt` function. This approach trades some speed for significantly more capability. ```python import os import daft from daft.functions import prompt daft.set_provider( "openai", base_url="https://openrouter.ai/api/v1", api_key=os.environ.get("OPENROUTER_API_KEY") ) df = daft.from_pydict({ "quote": [ "I am going to be the king of the pirates!", "I'm going to be the next Hokage!", ], }) df = ( df .with_column( "response", prompt( daft.col("quote"), system_message="Classify the anime from the quote and return the show, character name, and explanation.", provider="openai", model="nvidia/nemotron-nano-9b-v2:free" ) ) ) df.show(format="fancy", max_width=120) ``` ```{title="Output"} ╭───────────────────────────────────────────┬──────────────────────────────────────────────────────────────╮ │ quote ┆ response │ ╞═══════════════════════════════════════════╪══════════════════════════════════════════════════════════════╡ │ I am going to be the king of the pirates! ┆ **Show:** *One Piece* │ │ ┆ **Character Name:** Monkey D. Luffy │ │ ┆ **Explanation:** │ │ ┆ The quote "I am going to be the king… │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ I'm going to be the next Hokage! ┆ **Show:** *Naruto* │ │ ┆ **Character Name:** Naruto Uzumaki │ │ ┆ **Explanation:** The quote "I'm going to be the next Hokage… │ ╰───────────────────────────────────────────┴──────────────────────────────────────────────────────────────╯ (Showing first 2 of 2 rows) ``` --- ### Ai Functions Embed (`docs/ai-functions/embed.md`) # Generate Embeddings from Text and Images with Daft Embeddings are dense vector representations of data that capture semantic meaning. Daft provides fast, scalable embedding generation for both text and images, with support for various providers including OpenAI, Transformers, and local models. ## Text Embeddings The `embed_text` function generates embeddings from text using various models. You can use AI functions without explicitly configuring a provider—if you have an `OPENAI_API_KEY` environment variable set, simply specify `provider="openai"` and it will automatically create a provider in your session. ```python import daft from daft.functions import embed_text df = daft.from_pydict({"text": ["Hello World"]}) df = df.with_column( "embeddings", embed_text( daft.col("text"), provider="openai", model="text-embedding-3-small" ) ) df.show() ``` ```{title="Output"} ╭─────────────┬──────────────────────────╮ │ text ┆ embeddings │ │ --- ┆ --- │ │ String ┆ Embedding[Float32; 1536] │ ╞═════════════╪══════════════════════════╡ │ Hello World ┆ ▆█▆▆▆▃▆▆▂▄▃▂▃▃▄▁▃▅▂▃▂▂▂▂ │ ╰─────────────┴──────────────────────────╯ ``` ### Local Embeddings with LM Studio For privacy-sensitive workloads or offline environments, you can generate embeddings locally with [LM Studio](https://lmstudio.ai/). This lets you run embedding models like Nomic, Qwen, or Mistral on your own machine, taking advantage of hardware accelerators like [Apple's Metal Performance Shaders (MPS)](https://developer.apple.com/documentation/metalperformanceshaders). ```python import daft from daft.functions import embed_text df = daft.from_pydict({ "text": ["Hello, world!"] }) df = df.with_column( "embedding", embed_text(daft.col("text"), model="text-embedding-nomic-embed-text-v1.5", provider="lm_studio") ) df.show() ``` ``` ╭───────────────┬──────────────────────────╮ │ text ┆ embedding │ │ --- ┆ --- │ │ String ┆ Embedding[Float32; 768] │ ╞═══════════════╪══════════════════════════╡ │ Hello, world! ┆ █▄▄▃▃▄▃▄▃▆▂▁▂▅▅▂▂▃▅▄▄▃▂▄ │ ╰───────────────┴──────────────────────────╯ (Showing first 1 of 1 rows) ``` ### Provider-specific request parameters OpenAI-compatible embedding servers can accept additional JSON request properties through `extra_body`. For example, vLLM can truncate long inputs from the right, keeping the first 4,096 tokens: ```python df = df.with_column( "embedding", embed_text( daft.col("text"), provider=provider, model="BAAI/bge-m3", extra_body={ "truncate_prompt_tokens": 4096, "truncation_side": "right", }, ), ) ``` ## Image Embeddings The `embed_image` function generates embeddings from images using vision models. Daft natively supports images with the [Image DataType](../modalities/images.md), making multimodal preprocessing straightforward with built-in utilities like `decode_image`, `convert_image`, and `resize`. This example shows a complete pipeline: reading multiple image formats (PNG, JPEG, TIFF, WEBP) from HuggingFace, preprocessing them to the required format, and generating embeddings using Apple's [aimv2-large-patch14-224-lit](https://huggingface.co/apple/aimv2-large-patch14-224-lit) model. ```python import daft from daft.functions import embed_image, decode_image, convert_image, resize df = ( # Discover a few images from HuggingFace daft.from_glob_path("hf://datasets/datasets-examples/doc-image-3/images") # Read the 4 PNG, JPEG, TIFF, WEBP Images .with_column("image_bytes", daft.col("path").download()) # Decode the image bytes into a daft Image DataType .with_column("image_type", decode_image(daft.col("image_bytes"))) # Convert Image to RGB and resize the image to 288x288 .with_column("image_resized", resize(convert_image(daft.col("image_type"), "RGB"), 288, 288)) # Embed the image .with_column( "image_embeddings", embed_image( daft.col("image_resized"), provider="transformers", model="apple/aimv2-large-patch14-224-lit" ) ) ) # Show the dataframe df.show() ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` !!! tip "Image Preprocessing" When working with images for embedding, it's important to preprocess them to the right size and format. Be sure to check the model card for specific requirements regarding image format, size, and normalization. ## Semantic Search with Cosine Distance Once you have embeddings, you can use cosine distance to find similar items. This example demonstrates a simple RAG (Retrieval-Augmented Generation) pattern where we find the most relevant documents for a query. ```python import daft from daft.functions import embed_text, cosine_distance # Create a knowledge base with documents documents = daft.from_pydict({ "doc_id": [1, 2, 3, 4], "text": [ "Python is a high-level programming language", "Machine learning models require training data", "Daft is a distributed dataframe library", "Embeddings capture semantic meaning of text", ] }) # Embed all documents documents = documents.with_column( "embedding", embed_text(daft.col("text"), provider="openai", model="text-embedding-3-small") ) # Create a query query = daft.from_pydict({ "query_text": ["What is Daft?"] }) # Embed the query query = query.with_column( "query_embedding", embed_text( daft.col("query_text"), provider="openai", # Requires OPENAI_API_KEY model="text-embedding-3-small" ) ) # Cross join to compare query against all documents results = query.join(documents, how="cross") # Calculate cosine distance (lower is more similar) results = results.with_column( "distance", cosine_distance(daft.col("query_embedding"), daft.col("embedding")) ) # Sort by distance and show top results results = results.sort("distance").select("query_text", "text", "distance") results.show() ``` ```{title="Output"} ╭───────────────┬────────────────────────────────┬─────────────────────╮ │ query_text ┆ text ┆ distance │ │ --- ┆ --- ┆ --- │ │ String ┆ String ┆ Float64 │ ╞═══════════════╪════════════════════════════════╪═════════════════════╡ │ What is Daft? ┆ Daft is a distributed datafra… ┆ 0.36203420173101486 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ What is Daft? ┆ Python is a high-level progra… ┆ 0.91644507004277 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ What is Daft? ┆ Embeddings capture semantic m… ┆ 0.937416795759487 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ What is Daft? ┆ Machine learning models requi… ┆ 0.9697065433855928 │ ╰───────────────┴────────────────────────────────┴─────────────────────╯ (Showing first 4 of 4 rows) ``` !!! tip "Scaling Similarity Search" For large-scale similarity search, consider: - Pre-computing and storing embeddings to avoid re-generation - Using approximate nearest neighbor (ANN) algorithms for huge datasets - Filtering documents before computing similarity to reduce comparisons - Writing results to a vector database like [Turbopuffer](../connectors/turbopuffer.md) for efficient retrieval --- ### Ai Functions Jev (`docs/ai-functions/jev.md`) # Filtering, Classifying, and Scoring Rows with Jev Daft's `jev` functions let you ask plain-English questions about each row of a DataFrame: does this ticket come from an angry customer, which team should handle it, how luxurious is this product? Questions are answered by TypeSafe's Jev model, and the functions are modelled after DuckDB's `jev` community extension. | Function | Answers | Returns | | ----------------------------------------------- | ------------------------------------------------------------- | ---------------------------- | | [`jev`](#filtering-with-jev) | Does the row satisfy a condition? | `Boolean` | | [`jev_prob`](#probabilities-with-jev_prob) | How likely is it that the row satisfies a condition? | `Float64` in [0, 1] | | [`jev_choice`](#classification-with-jev_choice) | Which of these options best answers a question about the row? | `String` | | [`jev_score`](#scoring-with-jev_score) | Where does the row fall on an ordered rubric? | `Float64` in [0, levels - 1] | !!! tip "Choosing the Right Approach" - Use `jev` and `jev_prob` to filter or rank rows by a condition that is easy to state but hard to code - Use `jev_choice` and `jev_score` to label rows without hosting a model; use [`classify_text`](classify.md#text-classification) to run a classifier locally - Use [`prompt`](prompt.md) when you need free-form text, explanations, or structured outputs ## Setup Install Daft with the `jev` extra and set your TypeSafe API key: ```bash pip install "daft[jev]" export TYPESAFE_API_KEY="..." ``` Unlike the other AI functions, the `jev` functions don't use a [provider](providers.md). Each function takes its settings as keyword arguments, falling back to environment variables: | Argument | Environment variable | Default | | ---------- | ------------------------ | ------------------------- | | `api_key` | `TYPESAFE_API_KEY` | None; required | | `model` | `TYPESAFE_DEFAULT_MODEL` | `jev-latest` | | `base_url` | `TYPESAFE_BASE_URL` | `https://api.typesafe.ai` | | `timeout` | | 60 seconds per request | Settings are read when you build the expression, so workers on a distributed cluster don't need the environment variables. ## Filtering with `jev` The `jev` function returns `True` for rows that satisfy a plain-English condition, which makes it a natural fit for `where`. To judge several columns at once, pack them into a struct with `to_struct`; the field names are visible to the model. ```python import daft from daft.functions import jev, to_struct df = daft.from_pydict({ "first": ["Hans", "Mei", "Giulia", "Kwame"], "last": ["Müller", "Chen", "Rossi", "Mensah"], }) df = df.where( jev(to_struct(daft.col("first"), daft.col("last")), "the name is European") ) df.show() ``` A row counts as `True` when its probability is at least `threshold`, which defaults to 0.5. Raise it to keep only confident matches: ```python df.where(jev(daft.col("ticket"), "the customer is angry", threshold=0.9)) ``` ## Probabilities with `jev_prob` The `jev_prob` function returns the probability behind `jev`, so you can rank rows or pick a threshold after looking at the data. `jev(row, condition, threshold)` is equivalent to `jev_prob(row, condition) >= threshold`. ```python import daft from daft.functions import jev_prob df = daft.from_pydict({ "ticket": [ "This is the third time you've double charged me. I am furious and cancelling today!", "Thanks so much, the new dashboard is lovely.", "I was billed twice for my subscription this month.", "The app crashes every time I try to log in.", ], }) df = df.with_column("p_angry", jev_prob(daft.col("ticket"), "the customer is angry")) df.sort("p_angry", desc=True).show() ``` ## Classification with `jev_choice` The `jev_choice` function picks the most probable of a list of options as the answer to a question about each row. It takes between 1 and 255 options, and duplicate options are dropped. ```python import daft from daft.functions import jev_choice df = daft.from_pydict({ "ticket": [ "I was billed twice for my subscription this month.", "The app crashes every time I try to log in.", "Someone logged into my account from another country.", "Do you offer discounts for teams of 50 or more?", ], }) df = df.with_column( "team", jev_choice( daft.col("ticket"), "which team should handle this?", ["billing", "technical", "security", "sales"], ), ) df.show() ``` ## Scoring with `jev_score` The `jev_score` function places each row on an ordered rubric of 2 to 10 levels, listed lowest first. The result is the index of a level, weighted by the model's probability for each level, so it can fall between levels: with the levels below, a score of 2.5 sits between `premium` and `luxury`. ```python import daft from daft.functions import jev_score df = daft.from_pydict({ "product": [ "Plastic garden stool", "Solid oak dining chair", "Hand-stitched Italian leather armchair", ], }) df = df.with_column( "tier", jev_score( daft.col("product"), "how luxurious is this product?", ["budget", "mid-range", "premium", "luxury"], ), ) df.sort("tier", desc=True).show() ``` !!! note "Nulls and Batching" Null rows, including `NaN`, are never sent to the model and produce a null result. The remaining rows are sent 25 per request with up to 16 requests in flight, and requests that time out, are rate-limited, or fail on the server are retried. --- ### Ai Functions Prompt (`docs/ai-functions/prompt.md`) # Prompting Language Models with Daft A key strength of Daft is its ability to parallelize data processing on structured data. The `prompt` function is a powerful tool for use-cases like knowledge extraction, synthetic data generation, and batch tool-calling.
- [**Prompt Basics**](#prompt-basics) Learn how to compose prompt templates using DataFrame syntax, configure models, and pass optional parameters like temperature and max_tokens. - [**Structured Outputs**](#structured-outputs) Generate structured data from language models using Pydantic models that automatically convert to Daft DataTypes with native struct support. - [**Multimodal Inputs (Text, Images, Files)**](#multimodal-inputs) Pass text, images, and files (PDF, Markdown, HTML, CSV) as multimodal inputs to language models with automatic content formatting. - [**Tool Calling**](#tools) Use OpenAI's built-in tools (web search, file search, code interpreter) or define custom functions for agentic workflows at scale.
!!! note `prompt` is under active development, so make sure to stay up to date with our [latest releases on github](https://github.com/Eventual-Inc/Daft/releases) or through announcements in our [slack community](https://daft.ai/slack). ## Quick Start The `prompt` function enables you to call language models on DataFrame columns, making it easy to scale LLM operations across thousands or millions of rows with automatic batching and parallelization. The default provider for `prompt` is **OpenAI**. This means you can use OpenAI's latest models without specifying a provider. Just make sure your `OPENAI_API_KEY` environment variable is set and an OpenAIProvider will be created and attached to your `daft.Session` for you. ```python import daft from daft.functions import prompt # read some sample data df = daft.from_pydict({ "input": [ "What sound does a cat make?", "What sound does a dog make?", "What sound does a cow make?", "What does the fox say?", ] }) # generate responses using a chat model df = df.with_column("response", prompt(daft.col("input"), model="gpt-5.4-mini")) df.show() ``` ``` ╭─────────────────────────────┬─────────────────────────────╮ │ input ┆ response │ │ --- ┆ --- │ │ String ┆ String │ ╞═════════════════════════════╪═════════════════════════════╡ │ What sound does a cat make? ┆ A cat makes a "meow" sound. │ │ What sound does a dog make? ┆ A dog makes a "woof" sound. │ │ What sound does a cow make? ┆ A cow makes a "moo" sound. │ │ What does the fox say? ┆ The fox says "ring-ding-... │ ╰─────────────────────────────┴─────────────────────────────╯ ``` ### Supported Providers for `prompt` 1. **OpenAI** (Default) - Leverage models from OpenAI or any OpenAI-compatible provider. 2. **Google** - Take advantage of Google's latests models like *Gemini 3 Pro*. 3. **vLLM Prefix Caching** (Beta) - Distributed batch inference on vLLM [built by the Daft Cloud team](https://www.daft.ai/blog/cutting-llm-batch-inference-time-in-half-dynamic-prefix-bucketing-at-scale). 4. **Transformers** - Run open-weight instruction-tuned models locally via [HuggingFace Transformers](https://huggingface.co/docs/transformers). ## Prompt Basics The `prompt` function supports a wide range of tasks, from simple text generation to complex agentic workflows with tool calling. Daft's native multimodal data handling and scalable execution make it easy to process large datasets efficiently. This section covers core patterns for working with prompts, including templates, parameters, and model configuration. !!! note The default endpoint for OpenAI providers is `v1/responses`. You can pass `use_chat_completions=True` as an option to use the Chat Completions API instead of the new Responses API. ### Working with OpenAI-compatible providers This example shows how to use the `prompt` function with an OpenAI-compatible provider like OpenRouter to classify anime quotes. ```python import os import daft from daft.functions import prompt daft.set_provider( "openai", base_url="https://openrouter.ai/api/v1", api_key=os.environ.get("OPENROUTER_API_KEY") ) df = daft.from_pydict({ "quote": [ "I am going to be the king of the pirates!", "I'm going to be the next Hokage!", ], }) df = ( df .with_column( "response", prompt( daft.col("quote"), system_message="Classify the anime from the quote and return the show, character name, and explanation.", provider="openai", model="nvidia/nemotron-nano-9b-v2:free" ) ) ) df.show(format="fancy", max_width=120) ``` ```{title="Output"} ╭───────────────────────────────────────────┬──────────────────────────────────────────────────────────────╮ │ quote ┆ response │ ╞═══════════════════════════════════════════╪══════════════════════════════════════════════════════════════╡ │ I am going to be the king of the pirates! ┆ **Show:** *One Piece* │ │ ┆ **Character Name:** Monkey D. Luffy │ │ ┆ **Explanation:** │ │ ┆ The quote "I am going to be the king… │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ I'm going to be the next Hokage! ┆ **Show:** *Naruto* │ │ ┆ **Character Name:** Naruto Uzumaki │ │ ┆ **Explanation:** The quote "I'm going to be the next Hokage… │ ╰───────────────────────────────────────────┴──────────────────────────────────────────────────────────────╯ (Showing first 2 of 2 rows) ``` For more details on how to use OpenAI-compatible providers with the `prompt` function like OpenRouter, Hugging Face Inference Providers, Databricks, and more, check out the [Prompting OpenAI-Compatible Providers](providers.md#prompting-with-openai-compatible-providers) section. Additionally, check out the [Prompting with vLLM Online Serving](providers.md#prompting-with-vllm-online-serving) section for more details on how to use vLLM Online Serving. ### Running models locally with the Transformers provider The `transformers` provider runs open-weight instruction-tuned models locally via HuggingFace [`pipeline("text-generation")`](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.TextGenerationPipeline). Useful for offline batch inference without standing up a separate inference service. ```bash pip install 'daft[transformers]' ``` ```python import daft from daft.functions import prompt df = daft.from_pydict({"quote": ["I am going to be the king of the pirates!"]}) df = df.with_column( "response", prompt( daft.col("quote"), provider="transformers", model="HuggingFaceTB/SmolLM2-360M-Instruct", system_message="Classify the anime from the quote in one short sentence.", max_new_tokens=100, ), ) df.show() ``` Generation kwargs (`max_new_tokens`, `temperature`, `top_p`, ...) are forwarded to the pipeline call. Constructor-time options (`dtype`, `device_map`, `trust_remote_code`, `device`, ...) go through `pipeline_kwargs`. The provider auto-selects `cuda` > `mps` > `cpu`. You can also pass text-file columns (`daft.File` or raw `bytes` with a `text/*` MIME type) directly. The file contents are read and inlined into the prompt wrapped in a `...` tag derived from the MIME type, so the model can distinguish file content from your instructions: ```python df = daft.from_files("s3://my-bucket/docs/*.md") df = df.with_column( "summary", prompt( [daft.col("file"), "Summarize this document in 3 bullets."], provider="transformers", model="HuggingFaceTB/SmolLM2-360M-Instruct", ), ) ``` ### Building Prompt Templates with the `format` function In addition to passing plain strings, you can build dynamic prompt templates using Daft's `format` function. This works similarly to Python's `format`, but works with Daft expressions. You can also define prompt templates with user-defined functions. With `@daft.func` you can pass in expressions or plain python strings for more flexible composition. ```python import daft from daft.functions import prompt, format # Define a prompt template with "format" def answer_in_another_language(language: str, column_name: str) -> daft.Expression: return format( "Answer the following text in {}: {}", daft.lit(language), daft.col(column_name), ) # Alternatively define a row-wise UDF @daft.func def answer_in_another_language_func(language: str, question: str) -> str: return f"Answer the following text in {language}: {question}" # Define some toy data df = daft.from_pydict({ "text": [ "In what state(s) does water reach 101 degrees celsius?", "What are the total number of planets in our solar system?", "Who was the first person to discover radioactivity?", ] }) # Lazily plan to create two new columns df = ( df .with_column( "answer1", prompt( answer_in_another_language("Chinese", "text"), model="google/gemma-3-4b", provider="lm_studio" # Make sure you have google/gemma-3-4b loaded & running in lm_studio (local) ) ) .with_column( "answer2", prompt( answer_in_another_language_func(daft.lit("Spanish"), daft.col("text")), model="google/gemma-3-4b", provider="lm_studio" ) ) ) df.show(format="fancy", max_width=80) ``` ### Optional parameters Passing additional arguments like `temperature` or `max_tokens` can simply be added to any prompt call as a keyword argument. All `**kwargs` are routed to the inference request unless documented otherwise. Instructions for the model can also be added via the `system_message` argument which automatically prepends user content messages: ```json {"role": "system", "content": "You are an expert Chinese translation interpreter"} ``` For instance, if we wanted to call OpenAI via the Chat Completions endpoint, specifying `temperature` and `max_tokens` we would define a script like: ```python import daft from daft.functions import prompt df = daft.from_pydict({ "prompt": [ "What is the triple point of water?", ] }) # Prompt Usage for GPT-4.1 Chat Completions df = df.with_column( "result", prompt( messages=daft.col("prompt"), system_message="You are a helpful assistant.", model="gpt-4.1-2025-04-14", provider="openai", temperature=0.717, # Specify temperature max_tokens=200, # Specify max_tokens use_chat_completions=True, ) ) ``` However, the Responses API uses different parameter names. For GPT-5 models, use `max_output_tokens` instead of `max_tokens`, and note that temperature is not supported: ```python import daft from daft.functions import prompt df = daft.from_pydict({ "prompt": [ "What is the triple point of water?", ] }) # Prompt Usage for GPT-5 Responses df = df.with_column( "result", prompt( messages=daft.col("prompt"), system_message="You are a helpful assistant.", model="gpt-5", provider="openai", max_output_tokens=200, # Specify max_output_tokens ) ) ``` ### Reasoning Models GPT-5 and other reasoning-capable models support the `reasoning` parameter to control compute allocation. Pass it as you would in a `client.responses.create()` request: ```python import daft from daft.functions import prompt df = daft.from_pydict({ "prompt": [ "Find me buy-one-get-one-free burrito deals in SF right now.", ] }) # Prompt Usage for GPT-5 Responses df = df.with_column( "result", prompt( messages=daft.col("prompt"), system_message="You are a helpful assistant.", model="gpt-5-2025-08-07", provider="openai", reasoning={"effort": "high"}, # Adjust reasoning level ) ) ``` **See also**: [Providers Overview](providers.md). ### Tools The easiest way to get started with tool calling is to use OpenAI's built-in tools like web-search. ```python import daft from daft.functions import prompt from datetime import datetime df = daft.from_pydict({ "query": [ "Buy one get one free burritos in SF.", ] }) df = df.with_column( "search_results", prompt( daft.col("query"), model="gpt-5", tools=[{"type": "web_search"}], ) ) df.write_csv(f"../.data/prompt/oai_web_search_{str(datetime.now())}") ``` ## Multimodal Inputs Daft is purpose-built for multimodal AI workloads. The `prompt` function accepts text, images, and files as a list of expressions in the `messages` parameter, making it easy to work with mixed content types. ```python import daft from daft.functions import prompt df = daft.from_pydict({ "prompt": ["What's in this image and file?"], "my_image": ["/Users/MyName/Downloads/SigmundTheCat.png"], "my_file": ["/Users/MyName/Downloads/report.pdf"], }) # Decode the image and file paths df = df.with_column( "my_image", daft.functions.decode_image(daft.col("my_image").download()) ) df = df.with_column( "my_file", daft.functions.file(daft.col("my_file")) ) # Prompt Usage for GPT-5 Responses df = df.with_column( "result", prompt( messages=[daft.col("prompt"), daft.col("my_image"), daft.col("my_file")], system_message="You are a helpful assistant.", model="gpt-5-2025-08-07", provider="openai", reasoning={"effort": "high"}, # Adjust reasoning level tools=[{"type": "web_search"}], # Leverage internal OpenAI Tools ) ) ``` ### Working with Files The `prompt` function supports multiple file input methods depending on the provider and file type: - **PDF files**: Passed directly as file inputs (native OpenAI support) - **Text files** (Markdown, HTML, CSV, etc.): Content is automatically extracted and injected into prompts - **Images**: Supported via `daft.DataType.Image()` or file paths #### PDF Files To include PDF files, create a `daft.File` column and pass it in your messages list: ```python import daft from daft import lit, col from daft.functions import prompt, file, format from dotenv import load_dotenv load_dotenv() # Discover PDF files from HuggingFace df = daft.from_glob_path("hf://datasets/Eventual-Inc/sample-files/papers/*.pdf") df = ( df # Create a daft.File column from the path .with_column("file", file(col("path"))) # Prompt GPT-5-nano with PDF files as context .with_column( "response", prompt( [lit("What are in the contents of this file?\n"), col("file")], model="gpt-5-nano", provider="openai", ) ) ) df.show(format="fancy", max_width=80) ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` #### Text Files (Markdown, HTML, CSV) While OpenAI's native API only supports PDFs directly, attempting to pass other file types like Markdown, CSV, or HTML will throw errors like: ``` Invalid input: Expected file type to be a supported format: .pdf but got .jsonl” ``` To work around this, Daft automatically extracts content from any `text/*` MIME type file and injects it into the prompt with XML-wrapped tags. For example, to use Markdown files as context: ```python import daft from daft import lit, col from daft.functions import prompt, file, format # Discover Markdown Files in your Documents Folder df = daft.from_glob_path("/Users/MyName/Documents/*.md") df = ( df # Create a daft.File column from the path .with_column("file", file(col("path"))) # Prompt GPT-5-nano with markdown files as context .with_column( "response", prompt( messages=[lit("What are in the contents of this file?\n"), col("file")], model="gpt-5-nano", provider="openai", ) ) ) df.show(format="fancy", max_width=80) ``` When you pass `[lit("What are in the contents of this file?\n"), col("file")]`, Daft automatically formats the content. The resulting prompt will look like: ```text What are in the contents of this file? ## Some Title Example Markdown content with a [link](https://somelink.com). ``` ## Structured Outputs Daft natively supports converting Pydantic Models to Daft Datatypes. This simplifies structured generation in a vectorized context where nested fields materialize as structs. ```python import os from dotenv import load_dotenv import daft from daft.functions import prompt, unnest from pydantic import BaseModel, Field load_dotenv() class Anime(BaseModel): show: str = Field(description="The name of the anime show") character: str = Field(description="The name of the character who says the quote") explanation: str = Field(description="Why the character says the quote") daft.set_provider( "openai", base_url="https://openrouter.ai/api/v1", api_key=os.environ.get("OPENROUTER_API_KEY"), ) # Create a dataframe with the quotes df = daft.from_pydict( { "quote": [ "I am going to be the king of the pirates!", "I'm going to be the next Hokage!", ], } ) # Use the prompt function to classify the quotes df = df.with_column( "nemotron-response", prompt( daft.col("quote"), system_message="You are an anime expert. Classify the anime based on the text and return the name, character, and quote.", return_format=Anime, provider="openai", model="nvidia/nemotron-nano-9b-v2:free", ), ).select("quote", unnest(daft.col("nemotron-response"))) df.show(format="fancy", max_width=120) ``` ### Structured Outputs with vLLM OpenAI Online Server Sometimes Pydantic structured outputs is overkill. [vLLM's structured outputs](https://docs.vllm.ai/en/latest/features/structured_outputs/#online-serving-openai-api) syntax uses the `extra_body` param which we can use thanks to how Daft wires `kwargs` to the model request. Keep in mind the normal Pydantic structured outputs via `return_format` also works with vLLM online serving. One of the fastest ways to get started with online serving with vLLM is with Google Colab. To launch an OpenAI compatible server on vLLM run the following command in your terminal on an L4 GPU instance or larger: ```bash python -m vllm.entrypoints.openai.api_server \ --model google/gemma-3-4b-it \ --guided-decoding-backend guidance \ --dtype bfloat16 \ --gpu-memory-utilization 0.85 \ --host 0.0.0.0 --port 8000 ``` If you are in Google Colab, you can open a terminal by clicking the terminal icon in the bottom left of the ui. It usually takes at least **7.5** minutes before the vLLM server is ready. #### Setting up vLLM under the OpenAI provider ```python import daft from daft.functions import prompt, format daft.set_provider( "openai", api_key="none", base_url="http://localhost:8000/v1", ) ``` #### Guided Choice (Classification) ```python df = daft.from_pydict({ "statement": [ "vLLM is wonderful", "Daft is wicked fast", "Slow inference sucks", ] }) df = df.with_column( "sentiment", prompt( messages=format("Classify this sentiment: {}", daft.col("statement")), model="google/gemma-3-4b-it", extra_body={"structured_outputs": {"choice": ["positive", "negative"]}}, ) ) ``` #### Regex ```python df = daft.from_pydict({ "name": [ "Alan Turing", "Steve Jobs", "Sammy Sidhu", "Jay Chia", ], "company": [ "enigma.uk", "apple.com", "daft.ai", "daft.ai", ] }) df = df.with_column( "email", prompt( messages=format( "Generate an example email address for {}, who works at {}. Example result: alan.turing@enigma.uk\nMake sure to finish with a new line symbol\n", daft.col("name"), daft.col("company") ), model="google/gemma-3-4b-it", extra_body={"structured_outputs": {"regex": r"\w+@\w+\.com\n"}, "stop": ["\n"]}, ) ) ``` --- ### Ai Functions Providers (`docs/ai-functions/providers.md`) # AI Providers Overview Providers in Daft give you flexible control over where and how your AI inference runs. Whether you're using commercial APIs like OpenAI, running local models with LM Studio, or leveraging open-source models from Hugging Face, providers offer a unified interface for configuration. This guide covers: - [Quick provider setup](#quick-provider-setup) - [Setting named providers within sessions](#setting-a-named-openai-provider-within-a-session) - [Working with multiple providers](#end-to-end-usage-with-multiple-providers) - [Prompting with OpenAI-Compatible Providers](#prompting-with-openai-compatible-providers) - [Prompting with vLLM Online Serving](#prompting-with-vllm-online-serving) ## Quick provider setup The simplest way to use a provider is with `daft.set_provider()`, which configures a provider globally for your Daft session: ```python import os import daft # Set OpenAI Provider for Global Daft Session daft.set_provider( "openai", api_key=os.environ.get("OPENAI_API_KEY"), timeout=30.0, max_retries=3 ) # Retrieve Provider for an AI Function provider = daft.get_provider("openai") ``` ## Supported Providers Daft supports the following [AI Providers](../api/ai.md#providers): - **[OpenAI](https://platform.openai.com/docs/api-reference/introduction)** - API for GPT models and other AI capabilities - **[Transformers](https://huggingface.co/docs/transformers/index)** - Hugging Face models for local inference - **[Google](https://ai.google.dev/gemini-api/docs)** - Google's Gemini API - **[LM Studio](https://lmstudio.ai/)** - Run local models privately on your computer ## Setting a named OpenAI Provider within a Session For more control, you can create named providers within a session. This is useful when working with OpenAI-compatible APIs like OpenRouter, or when you need to switch between different configurations. You can pass a `Provider` instance directly to `set_provider()`, which will automatically attach it to the session: ```python import os import daft from daft.ai.openai.provider import OpenAIProvider sess = daft.Session() openrouter_provider = OpenAIProvider( name="OpenRouter", base_url="https://openrouter.ai/api/v1", api_key=os.environ.get("OPENROUTER_API_KEY") ) sess.set_provider(openrouter_provider) # Auto-attaches and sets as current # Retrieve Provider for an AI Function provider = sess.get_provider("OpenRouter") ``` This is equivalent to the two-step approach of calling `attach_provider()` followed by `set_provider()`: ```python # The two-step approach (still supported): sess.attach_provider(openrouter_provider) sess.set_provider("OpenRouter") ``` ## Using the Google Provider The Google provider enables you to use Google's Gemini models for text generation, multimodal processing, and structured outputs. ```python import daft import os # Set up the Google provider with your API key with daft.session() as session: session.set_provider("google", api_key=os.environ["GOOGLE_API_KEY"]) # Create a DataFrame with questions df = daft.from_pydict({ "question": [ "What is the capital of France?", "Explain quantum computing in simple terms.", "What are the benefits of fusion energy?" ] }) # Use the prompt function with Google's Gemini model df = df.with_column( "answer", daft.functions.prompt( daft.col("question"), provider="google", model="gemini-2.5-flash" # or "gemini-3-pro-preview" ) ) df.show() ``` The Google provider supports all the same features as other providers: - **Structured outputs** with Pydantic models - **Multimodal inputs** (text, images, PDFs, documents) - **System messages** for custom instructions - **Custom generation config** (temperature, max_output_tokens, etc.) ## End-to-End Usage with Multiple Providers For complex workflows, you might need to use different providers for different tasks—for example, using GPT-5 for validation while using a cheaper model for initial classification. Daft makes it easy to manage multiple providers in a single session. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` This example demonstrates: - **Multiple API keys**: Different providers for development and production environments - **Cost optimization**: Use cheaper models for initial processing, premium models for validation - **Provider routing**: Explicitly specify which provider to use for each operation !!! tip "When to Use Multiple Providers" Multiple providers are useful when you need to: - **Balance cost and quality**: Use cheaper models for bulk processing, premium models for critical tasks - **Ensure redundancy**: Fall back to alternative providers if one fails - **Compare model outputs**: Run the same prompt through different models for quality assessment - **Manage rate limits**: Distribute load across multiple API keys or providers ## Prompting with OpenAI-Compatible Providers You can use OpenAI-compatible providers with the `prompt` function like OpenRouter, HuggingFace Inference Providers, Databricks, and more. ```python import os import daft # For OpenRouter # See: https://openrouter.ai/docs/quickstart#using-the-openai-sdk daft.set_provider( "openai", base_url="https://openrouter.ai/api/v1", api_key=os.getenv("OPENROUTER_API_KEY") ) # For Hugging Face Inference Providers # See: https://huggingface.co/inference/get-started daft.set_provider( "openai", base_url="https://router.huggingface.co/v1", api_key=os.getenv("HF_TOKEN") ) # For Databricks # How to get your Databricks token: https://docs.databricks.com/en/dev-tools/auth/pat.html # More info on model serving: https://docs.databricks.com/aws/en/machine-learning/model-serving/score-foundation-models daft.set_provider( "openai", base_url="https://.cloud.databricks.com/serving-endpoints", api_key=os.getenv("DATABRICKS_TOKEN"), ) ``` ## Prompting with vLLM Online Serving For vLLM Online Serving, you can set the provider as demonstrated in the example below. **Note:** vLLM's OpenAI-compatible endpoint expects requests in the Chat Completions API format. Daft's `prompt()` function defaults to using the newer Responses API format, which is not supported by vLLM. To ensure compatibility, set `use_chat_completions=True` in `prompt()` so that your requests use the Chat Completions API format required by vLLM. ```python import os import daft # For vLLM Online Serving daft.set_provider( "openai", api_key="none", base_url="http://localhost:8000/v1", ) # Make sure to set use_chat_completions=True in prompt() df = df.with_column( "response", prompt( daft.col("input"), use_chat_completions=True, model="google/gemma-3-4b-it", ) ) df.show() ``` The following sample cli launch command is optimized for Google Colab's A100 High-RAM instance and Google's gemma-3-4b-it model. ```bash python -m vllm.entrypoints.openai.api_server \ --model google/gemma-3-4b-it \ --enable-chunked-prefill \ --guided-decoding-backend guidance \ --dtype bfloat16 \ --gpu-memory-utilization 0.85 \ --host 0.0.0.0 --port 8000 ``` * Server readiness may take ~7–8 minutes * For vLLM online serving, set `api_key = "none"` and `base_url = "http://0.0.0.0:8000/v1"` * `guided-decoding-backend guidance` is required for structured outputs. ## More Resources - **[AI Functions Overview](overview.md)** - **[Embedding Text and Images](embed.md)** - **[Classify Text and Images](classify.md)** - **[Contributing AI Functions](../contributing/contributing-ai-functions.md)** --- ### Api Aggregations (`docs/api/aggregations.md`) # Aggregations When performing aggregations such as sum, mean and count, Daft enables you to group data by certain keys and aggregate within those keys. Calling [`df.groupby()`][daft.DataFrame.groupby] returns a `GroupedDataFrame` object which is a view of the original DataFrame but with additional context on which keys to group on. You can then call various aggregation methods to run the aggregation within each group, returning a new DataFrame. ::: daft.dataframe.dataframe.GroupedDataFrame options: filters: ["!^_"] --- ### Api Ai (`docs/api/ai.md`) # AI Feature coverage table: | Feature | OpenAI | Google | vLLM Prefix Caching | LM Studio | Transformers | | --- | --- | --- | --- | --- | --- | | Prompt | ✅ | ✅ | ✅ | ✅ | x | | Embedding | ✅ | x | x | ✅ | ✅ | | Classification | x | x | x | x | ✅ | | Image Classification | x | x | x | x | ✅ | !!! note Open an issue on our [GitHub](https://github.com/Eventual-Inc/Daft/issues/new/choose) if you would like to see support for a feature. ## Functions ::: daft.functions.ai.prompt options: heading_level: 3 ::: daft.functions.ai.embed_text options: heading_level: 3 ::: daft.functions.ai.embed_image options: heading_level: 3 ::: daft.functions.ai.classify_text options: heading_level: 3 ::: daft.functions.ai.classify_image options: heading_level: 3 ## Model Protocols ::: daft.ai.protocols.Prompter options: heading_level: 3 ::: daft.ai.protocols.TextEmbedder options: heading_level: 3 ::: daft.ai.protocols.TextEmbedderDescriptor options: heading_level: 3 ::: daft.ai.protocols.ImageEmbedder options: heading_level: 3 ::: daft.ai.protocols.ImageEmbedderDescriptor options: heading_level: 3 ::: daft.ai.protocols.TextClassifier options: heading_level: 3 ::: daft.ai.protocols.TextClassifierDescriptor options: heading_level: 3 ## Providers ::: daft.ai.provider.Provider options: filters: ["!^_"] heading_level: 3 ::: daft.ai.provider.load_provider options: heading_level: 3 ::: daft.ai.provider.load_google options: heading_level: 3 ::: daft.ai.provider.load_lm_studio options: heading_level: 3 ::: daft.ai.provider.load_openai options: heading_level: 3 ::: daft.ai.provider.load_transformers options: heading_level: 3 ::: daft.ai.provider.load_vllm_prefix_caching options: heading_level: 3 --- ### Api Catalogs Tables (`docs/api/catalogs_tables.md`) # Catalogs and Tables Daft integrates with various catalog implementations using its `Catalog` and `Table` interfaces. These are high-level APIs to manage catalog objects (tables and namespaces), while also making it easy to leverage Daft's existing `daft.read_` and `df.write_` APIs for open table formats like [Iceberg](../connectors/iceberg.md) and [Delta Lake](../connectors/delta_lake.md). ::: daft.catalog.Catalog options: filters: ["!^_"] ::: daft.catalog.Identifier options: filters: ["!^_"] ::: daft.catalog.Table options: filters: ["!^_"] --- ### Api Config (`docs/api/config.md`) # Configuration Configure the execution backend, Daft in various ways during execution, and how Daft interacts with storage. ## Setting the Runner Control the execution backend that Daft will run on by calling these functions once at the start of your application. ::: daft.set_runner_native options: heading_level: 3 ::: daft.set_runner_ray options: heading_level: 3 ::: daft.get_or_create_runner options: heading_level: 3 ## Checking the Runner Check the execution backend that Daft is currently using. ::: daft.get_or_infer_runner_type options: heading_level: 3 ## Setting Configurations Configure Daft in various ways during execution. ::: daft.context.set_planning_config options: heading_level: 3 ::: daft.context.planning_config_ctx options: heading_level: 3 ::: daft.context.set_execution_config options: heading_level: 3 ::: daft.context.execution_config_ctx options: heading_level: 3 ## I/O Configurations Configure behavior when Daft interacts with storage (e.g. credentials, retry policies and various other knobs to control performance/resource usage) These configurations are most often used as inputs to Daft when reading I/O functions such as in [I/O](io.md). ::: daft.daft.IOConfig options: filters: ["!^_"] ::: daft.io.S3Config options: filters: ["!^_"] ::: daft.io.S3Credentials options: filters: ["!^_"] ::: daft.io.GCSConfig options: filters: ["!^_"] ::: daft.io.AzureConfig options: filters: ["!^_"] ::: daft.io.HTTPConfig options: filters: ["!^_"] ::: daft.io.UnityConfig options: filters: ["!^_"] ::: daft.io.HuggingFaceConfig options: filters: ["!^_"] ::: daft.io.TosConfig options: filters: ["!^_"] ::: daft.io.CosConfig options: filters: ["!^_"] --- ### Api Dataframe (`docs/api/dataframe.md`) # DataFrame Most DataFrame methods are **lazy**, meaning that they do not execute computation immediately when invoked. Instead, these operations are enqueued in the DataFrame's internal query plan, and are only executed when Execution DataFrame methods are called. ::: daft.DataFrame options: filters: ["!^_[^_]", "!__repr__", "!__builder"] --- ### Api Datasets (`docs/api/datasets.md`) # Datasets Daft provides simple, performant, and responsible ways to access useful datasets like [Common Crawl](https://commoncrawl.org/get-started) and [DROID](https://droid-dataset.github.io/). ## Common Crawl Check out our [Common Crawl dataset guide](../datasets/common-crawl.md) for more examples! ::: daft.datasets.common_crawl.common_crawl options: filters: ["!^_"] heading_level: 3 ## LeRobot See the [LeRobot dataset guide](../datasets/lerobot.md) for v2.1 / v3 layouts, episode vs frame workflows, and Hub/local paths. ::: daft.datasets.lerobot.read options: filters: ["!^_"] heading_level: 3 ::: daft.datasets.lerobot.read_episodes options: filters: ["!^_"] heading_level: 3 ::: daft.datasets.lerobot.load_episode_frames options: filters: ["!^_"] heading_level: 3 ::: daft.datasets.lerobot.read_tasks ## DROID Check out our [DROID dataset guide](../datasets/droid.md) for more examples! ::: daft.datasets.droid.raw options: filters: ["!^_"] heading_level: 3 ::: daft.datasets.droid.scenes options: filters: ["!^_"] heading_level: 3 ::: daft.datasets.droid.trajectory options: filters: ["!^_"] heading_level: 3 ::: daft.datasets.droid.camera_frames options: filters: ["!^_"] heading_level: 3 --- ### Api Expressions (`docs/api/expressions.md`) # Expressions Daft Expressions allow you to express some computation that needs to happen in a DataFrame. This page provides an overview of all the functionality that is provided by Daft Expressions. ## Constructors ::: daft.expressions.col options: heading_level: 3 ::: daft.expressions.lit options: heading_level: 3 ::: daft.sql.sql.sql_expr options: heading_level: 3 ::: daft.expressions.Expression options: filters: ["!^_[^_]", "!over", "!lag", "!lead"] ::: daft.expressions.WhenExpr options: filters: ["!^_"] ::: daft.expressions.visitor.ExpressionVisitor options: filters: ["!^_"] --- ### Api Io (`docs/api/io.md`) # I/O Daft offers a variety of approaches to creating a DataFrame from reading various data sources (in-memory data, files, data catalogs, and integrations) and writing to various data sources. See more about other [Connectors](../connectors/index.md) in Daft User Guide. ## Input ::: daft.from_arrow options: heading_level: 3 ::: daft.from_dask_dataframe options: heading_level: 3 ::: daft.from_glob_path options: heading_level: 3 ::: daft.from_pandas options: heading_level: 3 ::: daft.from_pydict options: heading_level: 3 ::: daft.from_pylist options: heading_level: 3 ::: daft.from_ray_dataset options: heading_level: 3 ::: daft.read_blob options: heading_level: 3 ::: daft.read_csv options: heading_level: 3 ::: daft.read_deltalake options: heading_level: 3 ::: daft.read_hudi options: heading_level: 3 ::: daft.read_iceberg options: heading_level: 3 ::: daft.read_json options: heading_level: 3 ::: daft.read_kafka options: heading_level: 3 ::: daft.read_lance options: heading_level: 3 ::: daft.read_parquet options: heading_level: 3 ::: daft.read_orc options: heading_level: 3 ::: daft.read_sql options: heading_level: 3 ::: daft.read_video_frames options: heading_level: 3 ::: daft.read_warc options: heading_level: 3 ::: daft.read_webdataset options: heading_level: 3 ::: daft.read_huggingface options: heading_level: 3 ::: daft.sql.sql.sql options: heading_level: 3 ## Output ::: daft.dataframe.DataFrame.write_csv options: heading_level: 3 ::: daft.dataframe.DataFrame.write_json options: heading_level: 3 ::: daft.dataframe.DataFrame.write_parquet options: heading_level: 3 ::: daft.dataframe.DataFrame.write_deltalake options: heading_level: 3 ::: daft.dataframe.DataFrame.write_iceberg options: heading_level: 3 ::: daft.dataframe.DataFrame.write_lance options: heading_level: 3 ::: daft.dataframe.DataFrame.write_sql options: heading_level: 3 ::: daft.dataframe.DataFrame.write_bigtable options: heading_level: 3 ::: daft.dataframe.DataFrame.write_clickhouse options: heading_level: 3 ::: daft.dataframe.DataFrame.write_huggingface options: heading_level: 3 ::: daft.dataframe.DataFrame.write_turbopuffer options: heading_level: 3 ::: daft.dataframe.DataFrame.write_sink options: heading_level: 3 ## User-Defined Daft supports diverse input sources and output sinks, this section covers lower-level APIs which we are evolving for more advanced usage. !!! warning "Warning" These APIs are considered experimental. ::: daft.io.source.DataSource options: filters: ["!^_"] heading_level: 3 ::: daft.io.source.DataSourceTask options: filters: ["!^_"] heading_level: 3 ::: daft.io.sink.DataSink options: filters: ["!^_"] heading_level: 3 ::: daft.io.sink.WriteResult options: filters: ["!^_"] heading_level: 3 ## Pushdowns Daft supports predicate, projection, and limit pushdowns. ::: daft.io.pushdowns.Pushdowns options: filters: ["!^_"] heading_level: 3 ::: daft.io.scan.ScanOperator options: filters: ["!^_"] --- ### Api Misc (`docs/api/misc.md`) # Miscellaneous ## Image Types ::: daft.daft.ImageMode options: filters: ["!^_"] ::: daft.daft.ImageFormat options: filters: ["!^_"] --- ### Api Schema (`docs/api/schema.md`) # Schema Daft can display your DataFrame's schema without materializing it. Under the hood, it performs intelligent sampling of your data to determine the appropriate schema, and if you make any modifications to your DataFrame it can infer the resulting types based on the operation. ::: daft.schema.Schema options: filters: ["!^_"] !!! warning "" Schema has been moved to `daft.schema` but is still accessible at `daft.logical.schema`. ::: daft.logical.schema.Schema options: filters: ["!^_"] --- ### Api Series (`docs/api/series.md`) # Series Each column in a Table is a Series. Series expose methods which invoke high-performance kernels for manipulation of a column of data. ::: daft.series.Series options: filters: ["^to", "^from"] --- ### Api Sessions (`docs/api/sessions.md`) # Sessions Sessions enable you to attach objects such as catalogs, providers, models, functions, and tables which are reference-able in DataFrame operations. The session also enables creating temporary objects which are accessible through both the Python and SQL APIs. Sessions hold configuration state such as current_catalog and current_namespace which are used in name resolution and can simplify your workflows. Learn more about [Sessions](../configuration/sessions-usage.md) in the Daft User Guide. ::: daft.session.session options: heading_level: 3 ::: daft.session.current_session options: heading_level: 3 ::: daft.session.Session options: filters: ["!^_"] --- ### Api Udf (`docs/api/udf.md`) # User-Defined Functions User-Defined Functions (UDFs) are a mechanism to run Python code on the data that lives in a DataFrame. A UDF can be used just like [Expressions](expressions.md), allowing users to express computation that should be executed by Daft lazily. To write a UDF, use the `@daft.func` or `@daft.cls` decorators to turn Python functions into Daft operations that work seamlessly with DataFrame expressions. Daft also supports custom aggregation functions, or UDAFs with `@daft.udaf` with a three-stage pipeline (aggregate, combine, finalize) that plugs into Daft's distributed aggregation engine. Learn more about user-defined functions in the user guide: [`daft.func`](../custom-code/func.md), [`daft.cls`](../custom-code/cls.md), [`daft.udaf`](../custom-code/udaf.md). ## Stateless Function UDFs ::: daft.func ::: daft.udf._FuncDecorator ## Stateful Class UDFs ::: daft.cls ::: daft.method ::: daft.udf._MethodDecorator ::: daft.udf.udf_v2.Func ## Aggregate UDFs ::: daft.udaf --- ### Api Window (`docs/api/window.md`) # Window Functions Window functions allow you to perform calculations across a set of rows that are related to the current row. They operate on a group of rows (called a window frame) and return a result for each row based on the values in its window frame, without collapsing the result into a single row like aggregate functions do. See [Window Functions Tutorial](../examples/window-functions.md) for a step-by-step tutorial. ::: daft.window.Window ## Applying Window Functions ::: daft.expressions.Expression.over options: heading_level: 3 ## Aggregate Functions Standard aggregate functions (e.g., [`sum`][daft.expressions.Expression.sum], [`mean`][daft.expressions.Expression.mean], [`count`][daft.expressions.Expression.count], [`min`][daft.expressions.Expression.min], [`max`][daft.expressions.Expression.max], etc) can be used as window functions by applying them with [`.over`][daft.expressions.Expression.over]. They work with all valid window specifications (partition by only, partition + order by, partition + order by + frame). Refer to the [Expressions API](./expressions.md) for a full list of aggregate functions. !!! note "Note" When using aggregate functions with both partition by and order by, the default window frame includes all rows from the start of the partition up to the current row — equivalent to rows between unbounded preceding and current row. ## Ranking Functions These functions compute ranks within a window partition. They require an [`order_by`][daft.window.Window.order_by] clause without a [`rows_between`][daft.window.Window.rows_between] or [`range_between`][daft.window.Window.range_between] clause in the window specification. * [daft.functions.row_number](functions/row_number.md) * [daft.functions.rank](functions/rank.md) * [daft.functions.dense_rank](functions/dense_rank.md) ## Lead/Lag Functions These functions access data from preceding or succeeding rows within a window partition. They require an [`order_by`][daft.window.Window.order_by] clause without a [`rows_between`][daft.window.Window.rows_between] or [`range_between`][daft.window.Window.range_between] clause in the window specification. ::: daft.expressions.Expression.lag options: heading_level: 3 ::: daft.expressions.Expression.lead options: heading_level: 3 --- ### Configuration Sessions Usage (`docs/configuration/sessions-usage.md`) # Daft Sessions, Catalogs, and Tables Sessions enable you to attach catalogs, tables, and create temporary objects which are accessible through both the Python and SQL APIs. Sessions hold configuration state such as `current_catalog` and `current_namespace` which are used in name resolution and can simplify your workflows. ## Example ```python import daft # `import daft` defines an implicit session `daft.current_session()` from daft.session import Session # create a new session sess = Session() # create a temp table from a DataFrame sess.create_temp_table("T", daft.from_pydict({ "x": [1,2,3] })) # read table as dataframe from the session _ = sess.read_table("T") # get the table instance t = sess.get_table("T") # read table instance as a datadrame _ = t.read() # execute sql against the session sess.sql("SELECT * FROM T").show() ╭───────╮ │ x │ │ --- │ │ Int64 │ ╞═══════╡ │ 1 │ ├╌╌╌╌╌╌╌┤ │ 2 │ ├╌╌╌╌╌╌╌┤ │ 3 │ ╰───────╯ ``` ## Usage This section covers detailed usage of the current APIs with some code snippets. ### Setup !!! note "Note" For these examples, we are using sqlite Iceberg which requires `pyiceberg[sql-sqlite]`. ```python from daft import Catalog from pyiceberg.catalog.sql import SqlCatalog # don't forget to `mkdir -p /tmp/daft/example` tmpdir = "/tmp/daft/example" # create a pyiceberg catalog backed by sqlite iceberg_catalog = SqlCatalog( "default", **{ "uri": f"sqlite:///{tmpdir}/catalog.db", "warehouse": f"file://{tmpdir}", }, ) # creating a daft catalog from the pyiceberg catalog implementation catalog = Catalog.from_iceberg(iceberg_catalog) # check catalog.name """ 'default' """ ``` ### Session State Let's get started by creating an empty session and checking the state. ```python import daft from daft.session import Session # create a new empty session sess = Session() # check the current catalog (None) sess.current_catalog() # get the current namespace (None) sess.current_namespace() ``` ### Attach & Detach The attach and detach methods make it easy to use catalogs and tables in a session. This example shows how we can attach our newly created catalog. When you attach a catalog to an empty session, it automatically becomes the current active catalog. ```python # attach makes it possible to use existing catalogs in the session sess.attach(catalog) # check the current catalog was set automatically sess.current_catalog() """ Catalog('default') """ # detach would remove the catalog # sess.detach_catalog("default") ``` ### Create & Drop We can create tables and namespaces directly through a catalog or via our session. ```python # create a namespace 'example' sess.create_namespace("example") # verify it was created sess.list_namespaces() """ [Identifier('example')] """ # you can create an empty table with a schema # sess.create_table("example.tbl", schema) # but suppose we have some data.. df = daft.from_pydict({ "x": [ True, True, False ], "y": [ 1, 2, 3 ], "z": [ "abc", "def", "ghi" ], }) # create a table from the dataframe, which will create + append sess.create_table("example.tbl", df) # you can also create temporary tables from dataframes # > echo "x,y,z\nFalse,4,jkl" > /tmp/daft/row.csv sess.create_temp_table("temp", daft.read_csv("/tmp/daft/row.csv")) # you can drop too # sess.drop_table("example.tbl") # sess.drop_namespace("example") ``` ### Read & Write Using sessions abstracts away underlying catalog and table implementations so you can easily read and write DataFrames. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Using SQL The session enables executing Daft SQL against your catalogs. ```python # use the given catalog and namespace (like the use/set_ methods) sess.sql("USE default.example") # we support both qualified and unqualified names by leveraging the session state sess.sql("SELECT * FROM tbl LIMIT 1").show() ╭─────────┬───────┬──────╮ │ x ┆ y ┆ z │ │ --- ┆ --- ┆ --- │ │ Boolean ┆ Int64 ┆ String │ ╞═════════╪═══════╪══════╡ │ true ┆ 4 ┆ jkl │ ╰─────────┴───────┴──────╯ # we can even combine our queries with the temp table from earlier sess.sql("SELECT * FROM example.tbl, temp LIMIT 1").show() ╭─────────┬───────┬──────┬────────────┬────────┬────────╮ │ x ┆ y ┆ z ┆ … ┆ temp.y ┆ temp.z │ │ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- │ │ Boolean ┆ Int64 ┆ String ┆ (1 hidden) ┆ Int64 ┆ String │ ╞═════════╪═══════╪══════╪════════════╪════════╪════════╡ │ true ┆ 4 ┆ jkl ┆ … ┆ 4 ┆ jkl │ ╰─────────┴───────┴──────┴────────────┴────────┴────────╯ ``` !!! note "Note" We aim to support SQL DDL in future releases! ## Reference For complete documentation, please see the [Session API docs](../api/sessions.md). | Method | Description | |---------------------------------------------------------------------------------|--------------------------------------------------------------------| | [`attach`][daft.Session.attach] | Attaches a catalog, table, or function to this session. | | [`attach_catalog`][daft.Session.attach_catalog] | Attaches an exiting catalog to this session | | [`attach_table`][daft.Session.attach_table] | Attaches an existing table to this session | | [`attach_function`][daft.Session.attach_function] | Attaches a user-defined function to this session | | [`create_namespace`][daft.Session.create_namespace] | Creates a new namespace | | [`create_namespace_if_not_exists`][daft.Session.create_namespace_if_not_exists] | Creates a new namespace if it doesn't already exist | | [`create_table`][daft.Session.create_table] | Creates a new table from the source | | [`create_table_if_not_exists`][daft.Session.create_table_if_not_exists] | Creates a new table from the source if it doesn't already exist | | [`create_temp_table`][daft.Session.create_temp_table] | Creates a temp table scoped to this session from an existing view. | | [`current_catalog`][daft.Session.current_catalog] | Returns the session's current catalog. | | [`current_namespace`][daft.Session.current_namespace] | Returns the session's current namespace. | | [`detach_catalog`][daft.Session.detach_catalog] | Detaches the catalog from this session | | [`detach_table`][daft.Session.detach_table] | Detaches the table from this session | | [`drop_namespace`][daft.Session.drop_namespace] | Drop the namespace in the session's current catalog | | [`drop_table`][daft.Session.drop_table] | Drop the table in the session's current catalog | | [`get_catalog`][daft.Session.get_catalog] | Returns the catalog or an object not found error. | | [`get_table`][daft.Session.get_table] | Returns the table or an object not found error. | | [`has_catalog`][daft.Session.has_catalog] | Returns true iff the session has access to a matching catalog. | | [`has_namespace`][daft.Session.has_namespace] | Returns true iff the session has access to a matching namespace. | | [`has_table`][daft.Session.has_table] | Returns true iff the session has access to a matching table. | | [`list_catalogs`][daft.Session.list_catalogs] | Lists all catalogs matching the pattern. | | [`list_namespaces`][daft.Session.list_namespaces] | Lists all namespaces matching the pattern. | | [`list_tables`][daft.Session.list_tables] | Lists all tables matching the pattern. Pattern syntax is catalog-dependent: native/memory and Postgres catalogs use SQL LIKE syntax (`%`, `_`, `\`); other catalogs may use different pattern matching. | | [`read_table`][daft.Session.read_table] | Reads a table from the session. | | [`write_table`][daft.Session.write_table] | Writes a dataframe to the table. | | [`set_catalog`][daft.Session.set_catalog] | Sets the current catalog. | | [`set_namespace`][daft.Session.set_namespace] | Sets the current namespace. | | [`sql`][daft.Session.sql] | Executes SQL against the session. | | [`use`][daft.Session.use] | Sets the current catalog and namespace. | --- ### Connectors Aws (`docs/connectors/aws.md`) # Reading from and Writing to AWS S3 Daft has [record-setting](https://www.daft.ai/blog/announcing-daft-02) performance when reading from and writing to AWS S3, and understands natively the URL protocol `s3://` as referring to data that resides in S3. ## Authorization/Authentication In AWS S3, data is stored under the hierarchy of: 1. Bucket: The container for data storage, which is the top-level namespace for data storage in S3. 2. Object Key: The unique identifier for a piece of data within a bucket. URLs to data in S3 come in the form: `s3://{BUCKET}/{OBJECT_KEY}`. ### Rely on Environment You can configure the AWS [CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) to have Daft automatically discover credentials. Alternatively, you may specify your credentials in [environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html): `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN`. Please be aware that when doing so in a distributed environment such as Ray, Daft will pick these credentials up from worker machines and thus each worker machine needs to be appropriately provisioned. If instead you wish to have Daft use credentials from the "driver", you may wish to manually specify your credentials. ### Manually specify credentials You may also choose to pass these values into your Daft I/O function calls using an [`daft.io.S3Config`][daft.io.S3Config] config object. [`daft.set_planning_config`][daft.context.set_planning_config] is a convenient way to set your [`daft.io.IOConfig`][daft.io.IOConfig] as the default config to use on any subsequent Daft method calls. === "🐍 Python" ```python from daft.io import IOConfig, S3Config # Supply actual values: key_id is the AWS Access Key ID, access_key is the AWS Secret Access Key io_config = IOConfig(s3=S3Config(key_id="key_id", session_token="session_token", access_key="access_key")) # Globally set the default IOConfig for any subsequent I/O calls daft.set_planning_config(default_io_config=io_config) # Perform some I/O operation df = daft.read_parquet("s3://my_bucket/my_path/**/*") ``` Alternatively, Daft supports overriding the default IOConfig per-operation by passing it into the `io_config=` keyword argument. This is extremely flexible as you can pass a different [`daft.io.S3Config`][daft.io.S3Config] per function call if you wish! === "🐍 Python" ```python # Perform some I/O operation but override the IOConfig df2 = daft.read_csv("s3://my_bucket/my_other_path/**/*", io_config=io_config) ``` --- ### Connectors Azure (`docs/connectors/azure.md`) # Reading from and Writing to Azure Blob Store Daft is able to read/write data to/from Azure Blob Store, and understands natively the URL protocols `az://` and `abfs://` as referring to data that resides in Azure Blob Store. !!! warning "Warning" Daft currently only supports globbing and listing files in storage accounts with [hierarchical namespaces](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-namespace) enabled. Hierarchical namespaces enable Daft to use its embarrassingly parallel globbing algorithm to improve performance of listing large nested directories of data. Please file an [issue](https://github.com/Eventual-Inc/Daft/issues) if you need support for non-hierarchical namespace buckets! We'd love to support your use-case. ## Authorization/Authentication In Azure Blob Service, data is stored under the hierarchy of: 1. Storage Account 2. Container (sometimes referred to as "bucket" in S3-based services) 3. Object Key URLs to data in Azure Blob Store come in the form: `az://{CONTAINER_NAME}/{OBJECT_KEY}`. Given that the Storage Account is not a part of the URL, you must provide this separately. ### Rely on Environment You can rely on Azure's [environment variables](https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-data-operations-cli#set-environment-variables-for-authorization-parameters) to have Daft automatically discover credentials. Please be aware that when doing so in a distributed environment such as Ray, Daft will pick these credentials up from worker machines and thus each worker machine needs to be appropriately provisioned. If instead you wish to have Daft use credentials from the "driver", you may wish to manually specify your credentials. ### Manually specify credentials You may also choose to pass these values into your Daft I/O function calls using an [`daft.io.AzureConfig`][daft.io.AzureConfig] config object. [`daft.set_planning_config`][daft.context.set_planning_config] is a convenient way to set your [`daft.io.IOConfig`][daft.io.IOConfig] as the default config to use on any subsequent Daft method calls. === "🐍 Python" ```python from daft.io import IOConfig, AzureConfig # Supply actual values for the storage_account and access key here io_config = IOConfig(azure=AzureConfig(storage_account="***", access_key="***")) # Globally set the default IOConfig for any subsequent I/O calls daft.set_planning_config(default_io_config=io_config) # Perform some I/O operation df = daft.read_parquet("az://my_container/my_path/**/*") ``` Alternatively, Daft supports overriding the default IOConfig per-operation by passing it into the `io_config=` keyword argument. This is extremely flexible as you can pass a different [`daft.io.AzureConfig`][daft.io.AzureConfig] per function call if you wish! === "🐍 Python" ```python # Perform some I/O operation but override the IOConfig df2 = daft.read_csv("az://my_container/my_other_path/**/*", io_config=io_config) ``` ### Connect to Microsoft Fabric/OneLake If you are connecting to storage in OneLake or another Microsoft Fabric service, set the `use_fabric_endpoint` parameter to `True` in the [`daft.io.AzureConfig`][daft.io.AzureConfig] object. === "🐍 Python" ```python from daft.io import IOConfig, AzureConfig io_config = IOConfig( azure=AzureConfig( storage_account="onelake", use_fabric_endpoint=True, # Set credentials as needed ) ) df = daft.read_deltalake('abfss://[WORKSPACE]@onelake.dfs.fabric.microsoft.com/[LAKEHOUSE].Lakehouse/Tables/[TABLE]', io_config=io_config) ``` --- ### Connectors Bigtable (`docs/connectors/bigtable.md`) # Writing to Google Cloud Bigtable !!! warning "Experimental" This connector is experimental and the API may change in future releases. [Google Cloud Bigtable](https://cloud.google.com/bigtable) is a fully managed, scalable NoSQL database service. Daft can write DataFrames to Bigtable tables using [`df.write_bigtable()`][daft.dataframe.DataFrame.write_bigtable]. ## Installing Dependencies Bigtable support requires the `google-cloud-bigtable` package: ```bash pip install google-cloud-bigtable ``` ## Basic Usage ```python import daft # Create a DataFrame df = daft.from_pydict({ "user_id": ["user_001", "user_002", "user_003"], "name": ["Alice", "Bob", "Charlie"], "age": [30, 25, 35], "email": ["alice@example.com", "bob@example.com", "charlie@example.com"], }) # Write to Bigtable result = df.write_bigtable( project_id="my-gcp-project", instance_id="my-bigtable-instance", table_id="users", row_key_column="user_id", column_family_mappings={ "name": "profile", "age": "profile", "email": "contact", }, ) ``` ## Key Concepts ### Row Keys Every Bigtable row requires a unique row key. Use `row_key_column` to specify which DataFrame column should be used as the row key: ```python df.write_bigtable( ..., row_key_column="user_id", # This column becomes the row key ) ``` ### Column Families Bigtable organizes columns into column families. Use `column_family_mappings` to specify which family each column belongs to: ```python df.write_bigtable( ..., column_family_mappings={ "name": "user_data", # 'name' column goes to 'user_data' family "age": "user_data", # 'age' column goes to 'user_data' family "email": "contact_info", # 'email' column goes to 'contact_info' family }, ) ``` The column families must already exist in the Bigtable table. ## Parameters | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `project_id` | `str` | Yes | Google Cloud project ID | | `instance_id` | `str` | Yes | Bigtable instance ID | | `table_id` | `str` | Yes | Bigtable table ID | | `row_key_column` | `str` | Yes | Column name to use as the row key | | `column_family_mappings` | `dict[str, str]` | Yes | Mapping of column names to column families | | `client_kwargs` | `dict` | No | Additional arguments for the Bigtable Client | | `write_kwargs` | `dict` | No | Additional arguments for MutationsBatcher | | `serialize_incompatible_types` | `bool` | No | Auto-convert incompatible types to JSON (default: True) | ## Data Type Handling Bigtable cells only accept data that can be converted to bytes. By default, Daft automatically serializes incompatible types to JSON: ```python df = daft.from_pydict({ "id": ["row1"], "data": [{"nested": "object"}], # Complex type }) # Complex types are automatically serialized to JSON df.write_bigtable( ..., serialize_incompatible_types=True, # Default behavior ) ``` To disable automatic serialization (will raise an error for incompatible types): ```python df.write_bigtable( ..., serialize_incompatible_types=False, ) ``` ## Advanced Configuration ### Client Options Pass additional options to the Bigtable client: ```python result = df.write_bigtable( ..., client_kwargs={ "admin": True, "channel": custom_channel, }, ) ``` ### Write Options Configure the MutationsBatcher for write operations: ```python result = df.write_bigtable( ..., write_kwargs={ "flush_count": 1000, "max_row_bytes": 5 * 1024 * 1024, # 5MB }, ) ``` ## Use Cases ### IoT Data Storage ```python import daft from daft import col # Read sensor data df = daft.read_parquet("s3://bucket/sensors/*.parquet") # Prepare for Bigtable (create composite row key) df = df.with_column( "row_key", col("device_id").concat("#").concat(col("timestamp").cast(str)), ) # Write to Bigtable df.write_bigtable( project_id="iot-project", instance_id="sensor-data", table_id="readings", row_key_column="row_key", column_family_mappings={ "temperature": "metrics", "humidity": "metrics", "device_id": "metadata", "timestamp": "metadata", }, ) ``` ### User Profile Storage ```python import daft df = daft.from_pydict({ "user_id": ["u001", "u002"], "preferences": [{"theme": "dark"}, {"theme": "light"}], "last_login": ["2024-01-15", "2024-01-16"], }) df.write_bigtable( project_id="my-project", instance_id="user-store", table_id="profiles", row_key_column="user_id", column_family_mappings={ "preferences": "settings", "last_login": "activity", }, ) ``` ## Notes - The Bigtable table and column families must exist before writing - Row keys should be designed carefully for efficient access patterns - Consider Bigtable's row key design best practices for optimal performance --- ### Connectors Blob (`docs/connectors/blob.md`) # Reading Files as Blobs Daft can read files as raw bytes into DataFrames using [`daft.read_blob()`][daft.io.read_blob], similar to DuckDB's `read_blob` function. This is useful for loading non-tabular files such as images, audio, PDFs, or any other binary data into a DataFrame, with one row per file. ## Basic Usage Read files matching a path or glob pattern, where each file becomes a row: === "Local Files" ```python import daft df = daft.read_blob("/path/to/files/*.jpeg") df.show() ``` === "Remote Files (S3)" ```python import daft from daft.io import S3Config, IOConfig io_config = IOConfig(s3=S3Config(region_name="us-west-2", anonymous=True)) df = daft.read_blob("s3://my-bucket/images/*.jpeg", io_config=io_config) df.show() ``` === "Remote Files (GCS)" ```python import daft from daft.io import GCSConfig, IOConfig io_config = IOConfig(gcs=GCSConfig(anonymous=True)) df = daft.read_blob("gs://my-bucket/images/*.jpeg", io_config=io_config) df.show() ``` ## Output Schema The `read_blob` function returns a DataFrame with the following columns: | Column | Type | Description | |--------|------|-------------| | `path` | `string` | Path to the file | | `size` | `int64` | Size of the file in bytes | | `content` | `binary` | Raw bytes contents of the file | ## Glob Patterns `read_blob` supports the same wildcards as other Daft readers: ```python # All .png files in a directory df = daft.read_blob("/data/*.png") # Recursive search df = daft.read_blob("/data/**/*.png") # Multiple paths df = daft.read_blob(["/data/a.bin", "/data/b.bin"]) ``` ## Error Handling By default, a download error for any file raises immediately. Set `on_error="null"` to log the error and fall back to a null `content` value instead: ```python df = daft.read_blob("/data/*.bin", on_error="null") ``` ## Common Use Cases ### Decoding Images Read image files and decode them for downstream processing: ```python import daft from daft.functions import decode_image df = daft.read_blob("s3://my-bucket/images/*.jpeg", io_config=io_config) df = df.with_column("image", decode_image(df["content"])) ``` ### Detecting File Types Inspect magic bytes to determine the MIME type of each file: ```python import daft from daft.functions import guess_mime_type df = daft.read_blob("/data/**/*") df = df.with_column("mime", guess_mime_type(df["content"])) ``` ## Relationship to Other APIs `read_blob` is a convenience API composed of [`daft.from_glob_path()`][daft.from_glob_path] (which lists files with their `path` and `size`) and the [`download`][daft.functions.download] expression (which fetches file contents as bytes). Use those building blocks directly if you need more control, such as downloading only a filtered subset of files. --- ### Connectors Clickhouse (`docs/connectors/clickhouse.md`) # Writing to ClickHouse [ClickHouse](https://clickhouse.com/) is a fast, open-source columnar database for real-time analytics. Daft can write DataFrames to ClickHouse tables using [`df.write_clickhouse()`][daft.dataframe.DataFrame.write_clickhouse]. ## Installing Dependencies ClickHouse support requires the `clickhouse-connect` package: ```bash pip install clickhouse-connect ``` ## Basic Usage ```python import daft # Create a DataFrame df = daft.from_pydict({ "id": [1, 2, 3, 4], "name": ["Alice", "Bob", "Charlie", "Diana"], "value": [100.5, 200.3, 150.7, 300.2], }) # Write to ClickHouse result = df.write_clickhouse( table="my_table", host="localhost", port=8123, user="default", password="", ) result.show() ``` ## Output Schema The write operation returns a DataFrame with write statistics: | Column | Type | Description | |--------|------|-------------| | `total_written_rows` | `int64` | Total number of rows written | | `total_written_bytes` | `int64` | Total number of bytes written | ## Connection Parameters | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `table` | `str` | Yes | Name of the ClickHouse table to write to | | `host` | `str` | Yes | ClickHouse server hostname | | `port` | `int` | No | ClickHouse HTTP port (default: 8123) | | `user` | `str` | No | ClickHouse username | | `password` | `str` | No | ClickHouse password | | `database` | `str` | No | ClickHouse database name | ## Advanced Configuration ### Client Options Pass additional options to the ClickHouse client: ```python result = df.write_clickhouse( table="my_table", host="localhost", client_kwargs={ "secure": True, "verify": True, "connect_timeout": 30, }, ) ``` ### Write Options Pass additional options to the write operation: ```python result = df.write_clickhouse( table="my_table", host="localhost", write_kwargs={ "settings": { "async_insert": 1, "wait_for_async_insert": 1, }, }, ) ``` ## Use Cases ### Analytics Pipeline ```python import daft from daft import col # Process data df = daft.read_parquet("s3://bucket/events/*.parquet") aggregated = df.groupby("event_type").agg( col("value").sum().alias("total_value"), col("user_id").count().alias("event_count"), ) # Write aggregated results to ClickHouse for dashboards aggregated.write_clickhouse( table="event_aggregates", host="clickhouse.example.com", database="analytics", user="writer", password="secret", ) ``` ### Real-Time Data Ingestion ```python import daft # Read streaming batch df = daft.read_json("/data/batch/*.json") # Transform and load to ClickHouse df.write_clickhouse( table="events", host="localhost", write_kwargs={ "settings": {"async_insert": 1}, }, ) ``` ## Notes - The target table must exist in ClickHouse before writing - Column names and types in the DataFrame should match the table schema - ClickHouse will perform type coercion where possible --- ### Connectors Cos (`docs/connectors/cos.md`) # Reading from and Writing to Tencent Cloud COS Daft supports reading and writing data to Tencent Cloud COS (Cloud Object Storage), and understands natively the URL protocols `cos://` and `cosn://` (Hadoop CosN compatible) as referring to data that resides in COS. ## Authorization/Authentication In Tencent Cloud COS, data is stored under the hierarchy of: 1. Bucket: The container for data storage, which is the top-level namespace for data storage in COS. 2. Object Key: The unique identifier for a piece of data within a bucket. URLs to data in COS come in the form: `cos://{BUCKET}/{OBJECT_KEY}`. !!! note "Hadoop CosN Compatibility" Daft also supports the `cosn://` URL scheme for compatibility with Hadoop CosN. Both `cos://` and `cosn://` are treated identically. ### Rely on Environment You can configure Daft to automatically discover credentials from environment variables. Daft supports the following environment variable prefixes: | Environment Variable | Description | |---|---| | `COS_ENDPOINT` | Endpoint of the COS service | | `COS_REGION` or `TENCENTCLOUD_REGION` | Region of the COS service | | `COS_SECRET_ID` or `TENCENTCLOUD_SECRET_ID` | SecretId for COS authentication | | `COS_SECRET_KEY` or `TENCENTCLOUD_SECRET_KEY` | SecretKey for COS authentication | | `COS_SECURITY_TOKEN` or `TENCENTCLOUD_SECURITY_TOKEN` | Security token for temporary credentials (STS) | Please be aware that when doing so in a distributed environment such as Ray, Daft will pick these credentials up from worker machines and thus each worker machine needs to be appropriately provisioned. If instead you wish to have Daft use credentials from the "driver", you may wish to manually specify your credentials. ### Manually specify credentials You may also choose to pass these values into your Daft I/O function calls using a [`daft.io.CosConfig`][daft.io.CosConfig] config object. [`daft.set_planning_config`][daft.context.set_planning_config] is a convenient way to set your [`daft.io.IOConfig`][daft.io.IOConfig] as the default config to use on any subsequent Daft method calls. === "Using Permanent Keys" ```python from daft.io import IOConfig, CosConfig io_config = IOConfig( cos=CosConfig( region="ap-guangzhou", secret_id="your-secret-id", secret_key="your-secret-key", ) ) # Globally set the default IOConfig for any subsequent I/O calls daft.set_planning_config(default_io_config=io_config) # Perform some I/O operation df = daft.read_parquet("cos://my-bucket/my_path/**/*") ``` === "Using STS Temporary Credentials" ```python from daft.io import IOConfig, CosConfig io_config = IOConfig( cos=CosConfig( region="ap-guangzhou", secret_id="your-tmp-secret-id", secret_key="your-tmp-secret-key", security_token="your-security-token", ) ) # Perform some I/O operation df = daft.read_parquet("cos://my-bucket/my_path/**/*", io_config=io_config) ``` === "Using Custom Endpoint" ```python from daft.io import IOConfig, CosConfig # Use a custom endpoint (region will be auto-derived from the endpoint) io_config = IOConfig( cos=CosConfig( endpoint="https://cos.ap-beijing.myqcloud.com", secret_id="your-secret-id", secret_key="your-secret-key", ) ) df = daft.read_parquet("cos://my-bucket/data/**/*", io_config=io_config) ``` === "Anonymous Access" ```python from daft.io import IOConfig, CosConfig # Access public buckets without credentials io_config = IOConfig(cos=CosConfig(anonymous=True)) # Read from a public bucket df = daft.read_parquet("cos://public-bucket/data/**/*", io_config=io_config) ``` === "Using CosN URL Scheme" ```python from daft.io import IOConfig, CosConfig io_config = IOConfig( cos=CosConfig( region="ap-guangzhou", secret_id="your-secret-id", secret_key="your-secret-key", ) ) # Use cosn:// scheme (Hadoop CosN compatible) df = daft.read_parquet("cosn://my-bucket/my_path/**/*", io_config=io_config) ``` Alternatively, Daft supports overriding the default IOConfig per-operation by passing it into the `io_config=` keyword argument. This is extremely flexible as you can pass a different [`daft.io.CosConfig`][daft.io.CosConfig] per function call if you wish! === "Per-Operation Config" ```python from daft.io import IOConfig, CosConfig io_config = IOConfig( cos=CosConfig( region="ap-guangzhou", secret_id="your-secret-id", secret_key="your-secret-key", ) ) # Perform some I/O operation but override the IOConfig df2 = daft.read_csv("cos://my-bucket/my_other_path/**/*", io_config=io_config) ``` ## Writing Data Daft supports writing data to COS using the same `CosConfig`: ```python import daft from daft.io import IOConfig, CosConfig io_config = IOConfig( cos=CosConfig( region="ap-guangzhou", secret_id="your-secret-id", secret_key="your-secret-key", ) ) df = daft.from_pydict({"a": [1, 2, 3], "b": ["x", "y", "z"]}) # Write to COS as Parquet df.write_parquet("cos://my-bucket/output/", io_config=io_config) ``` ## Configuration Options The [`daft.io.CosConfig`][daft.io.CosConfig] object supports the following options: | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `region` | `str` | `None` | Region name, e.g. `"ap-guangzhou"`, `"ap-beijing"`, `"ap-shanghai"` | | `endpoint` | `str` | `None` | Custom endpoint URL, e.g. `"https://cos.ap-guangzhou.myqcloud.com"`. If not provided, it will be derived from the region. | | `secret_id` | `str` | `None` | Tencent Cloud SecretId | | `secret_key` | `str` | `None` | Tencent Cloud SecretKey | | `security_token` | `str` | `None` | Security token for temporary credentials (STS) | | `anonymous` | `bool` | `False` | Whether to use anonymous access (for public buckets) | | `max_retries` | `int` | `3` | Maximum number of retries for failed requests | | `retry_timeout_ms` | `int` | `30000` | Timeout duration for retry attempts in milliseconds | | `connect_timeout_ms` | `int` | `10000` | Connection timeout in milliseconds | | `read_timeout_ms` | `int` | `30000` | Read timeout in milliseconds | | `max_concurrent_requests` | `int` | `50` | Maximum number of concurrent requests | | `max_connections` | `int` | `50` | Maximum number of connections per IO thread | !!! tip "Region and Endpoint Auto-Derivation" You only need to specify either `region` or `endpoint` — Daft will automatically derive the other: - If only `region` is provided, the endpoint is derived as `https://cos.{region}.myqcloud.com` - If only `endpoint` is provided, the region is extracted from the endpoint URL - If both are provided, both values are used as-is ## Supported Operations Daft supports the following operations with COS: - **Read**: `read_parquet`, `read_csv`, `read_json`, and other file readers - **Write**: `write_parquet`, `write_csv`, `write_json` (including multipart uploads) - **List**: Listing objects with glob pattern matching - **Delete**: Deleting objects --- ### Connectors Custom Catalogs (`docs/connectors/custom-catalogs.md`) # Custom Catalogs Daft ships with built-in catalog integrations for [Iceberg](iceberg.md), [Unity](unity_catalog.md), [AWS Glue](glue.md), and others. But if your tables live behind a proprietary metadata service, an internal API, or a convention like "every subdirectory in S3 is a Delta table," you can implement your own catalog and plug it into Daft. This also applies if you use a public catalog system that Daft doesn't have a built-in integration for yet - you can bridge the gap yourself rather than waiting for native support! The `Catalog` and `Table` abstract base classes in `daft.catalog` are public interfaces designed for this. Subclass them to teach Daft how to discover and read your tables. Once registered, your catalog works everywhere - the Python API, SQL queries, and Sessions all resolve tables through it. Tables are resolved lazily, so you can expose thousands of tables without paying the cost of eagerly constructing DataFrames for all of them upfront. ## Overview Daft's catalog system has two core interfaces: - **`Catalog`** - discovers and manages tables and namespaces - **`Table`** - reads and writes data for a single table Both are abstract base classes. You implement the private `_`-prefixed abstract methods, and the base classes provide the public API (argument normalization, `if_not_exists` variants, etc.) on top. Once implemented, custom catalogs work with Sessions and SQL just like the built-in ones: ```python import daft from daft.session import Session sess = Session() sess.attach(my_catalog) sess.sql("SELECT * FROM my_namespace.my_table").show() ``` ## Catalog Interface A `Catalog` subclass implements these abstract methods: | Method | Description | |---|---| | `name` (property) | Returns the catalog's name | | `_get_table(ident)` | Look up a table by identifier | | `_has_table(ident)` | Check if a table exists | | `_list_tables(pattern)` | List tables, optionally filtered by pattern | | `_has_namespace(ident)` | Check if a namespace exists | | `_list_namespaces(pattern)` | List namespaces, optionally filtered by pattern | | `_create_namespace(ident)` | Create a namespace | | `_create_table(ident, schema, ...)` | Create a table | | `_drop_namespace(ident)` | Drop a namespace | | `_drop_table(ident)` | Drop a table | The base class provides the public methods (`get_table`, `list_tables`, `create_table_if_not_exists`, `read_table`, `write_table`, etc.) on top of these. ## Table Interface A `Table` subclass implements these abstract methods: | Method | Description | |---|---| | `name` (property) | Returns the table's name | | `schema()` | Returns the table's `Schema` | | `read(**options)` | Returns a `DataFrame` for this table | | `append(df, **options)` | Appends a DataFrame to this table | | `overwrite(df, **options)` | Overwrites this table with a DataFrame | The base class provides `write(df, mode)`, `select(*columns)`, and `show(n)` on top of these. ## Walkthrough: Iceberg Catalog Implementation Daft's [Iceberg Catalog integration](https://github.com/Eventual-Inc/Daft/blob/main/daft/catalog/__iceberg.py) is a complete built-in implementation that covers every abstract method. The snippets below are trimmed to highlight the pattern - see the full source for details. ### IcebergCatalog The catalog wraps a PyIceberg `Catalog` client. Each abstract method converts between Daft's `Identifier` type and PyIceberg's tuple-based identifiers, then delegates to the inner client: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` The key pattern: convert `Identifier` to your backend's native format (here, a tuple via `tuple(identifier)`), call the backend, and translate exceptions to `NotFoundError`. ### IcebergTable The table wraps a PyIceberg `Table` object. `read()` delegates to `daft.read_iceberg` and write methods delegate to `DataFrame.write_iceberg`: ```python from daft.catalog import Table from daft.dataframe import DataFrame from daft.logical.schema import Schema class IcebergTable(Table): _inner: InnerTable # pyiceberg.table.Table @property def name(self) -> str: return self._inner.name()[-1] def schema(self) -> Schema: # Iceberg derives schema from the lazy read plan here, but if your # backend exposes schema metadata directly, prefer that instead. return self.read().schema() def read(self, **options) -> DataFrame: return daft.read_iceberg(self._inner, **options) def append(self, df: DataFrame, **options) -> None: df.write_iceberg(self._inner, mode="append") def overwrite(self, df: DataFrame, **options) -> None: df.write_iceberg(self._inner, mode="overwrite") ``` ## Using a Custom Catalog Custom catalogs integrate with the full Daft API - Sessions, SQL, and the Python API: ```python import daft from daft.session import Session # Create your catalog catalog = MyCatalog(...) # Use it directly catalog.list_tables() catalog.read_table("my_namespace.my_table") # Or attach it to a session for SQL support sess = Session() sess.attach(catalog) sess.set_namespace("my_namespace") # SQL queries resolve tables through your catalog sess.sql("SELECT * FROM my_table").show() ``` ## Reference For complete API documentation, see the [Catalogs & Tables API reference](../api/catalogs_tables.md). Other built-in implementations for reference: - [Unity Catalog](https://github.com/Eventual-Inc/Daft/blob/main/daft/catalog/__unity.py) - read-only, simplest implementation - [AWS Glue Catalog](https://github.com/Eventual-Inc/Daft/blob/main/daft/catalog/__glue.py) - AWS Glue metadata service - [PostgreSQL Catalog](https://github.com/Eventual-Inc/Daft/blob/main/daft/catalog/__postgres.py) - full read/write with PostgreSQL --- ### Connectors Custom (`docs/connectors/custom.md`) # Reading from and Writing to Custom Connectors This guide shows you how to create custom connectors to read from or write to data sources that aren't currently implemented in Daft, which includes proprietary data sources that can't be implemented in open source. ## Overview Daft provides two main interfaces for custom connectors: - [**Reading from a custom `DataSource`**](#reading-from-a-custom-data-source) - [**Writing to a custom `DataSink`**](#writing-to-a-custom-data-sink) You can also take a look at actual code references on how we implemented: - [Reads from Video Frames](https://github.com/Eventual-Inc/Daft/blob/main/daft/io/av/_read_video_frames.py) - [Writes to Hugging Face 🤗](https://github.com/Eventual-Inc/Daft/blob/main/daft/io/huggingface/sink.py) - [More examples below](#more-examples) ## Reading from a Custom Data Source ### Step 1: Implement the `DataSource` and `DataSourceTask` Interfaces Create a class that inherits from [`DataSource`](../api/io.md#daft.io.source.DataSource), a class that inherits from [`DataSourceTask`](../api/io.md#daft.io.source.DataSourceTask), and implement the required methods. Here's a simple example doing this with a custom local file reader that reads each line from a file as a single String row. === "🐍 Python" ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Step 2: Use Your Custom Data Source to Create a Daft DataFrame === "🐍 Python" ```python import daft # Create a sample text file and read from it. sample_file = "sample_text.txt" with open(sample_file, "w") as f: f.write("Alice was beginning to get very tired of sitting by her sister on the bank.\n") f.write("So she was considering in her own mind (as well as she could, for the hot day made her feel very sleepy and stupid),\n") f.write("when suddenly a White Rabbit with pink eyes ran close by her.\n") f.write("There was nothing so very remarkable in that;\n") f.write("nor did Alice think it so very much out of the way to hear the Rabbit say to itself, 'Oh dear! Oh dear! I shall be late!'\n") data_source = TextFileDataSource([sample_file]) ( data_source .read() .show() ) ``` **Output:** ```bash ╭────────────────────────────────╮ │ line │ │ --- │ │ Utf8 │ ╞════════════════════════════════╡ │ Alice was beginning to get ve… │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ So she was considering in her… │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ when suddenly a White Rabbit … │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ There was nothing so very rem… │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ nor did Alice think it so ver… │ ╰────────────────────────────────╯ (Showing first 5 of 5 rows) ``` ### Optional: Declaring Clustering to Skip Shuffles Clustering is an *execution-time* property: it describes how rows are distributed across the in-memory partitions a query runs over. It is distinct from *storage* partitioning (the on-disk layout declared via `get_partition_fields()`, e.g. Hive/Iceberg directories) — a source can be laid out one way on disk yet emit partitions clustered another way. Two clustering hints are available: #### Hash clustering If your source already emits data that is hash-partitioned by some keys — for example, each [`DataSourceTask`](../api/io.md#daft.io.source.DataSourceTask) corresponds to exactly one `(producer, hour)` group — you can tell Daft by overriding `get_clustering_keys()`. Daft then skips the shuffle it would otherwise insert before a downstream `groupby`, `Window.partition_by`, or `distinct` whose keys are *covered by* (equal to, or a superset of) the declared clustering. ```python from daft import col from daft.io.clustering import ClusteringKeys from daft.io.source import DataSource class ClusteredSource(DataSource): # ... name / schema / get_tasks as above ... def get_clustering_keys(self) -> ClusteringKeys | None: # Each task emits exactly one (a, b) group, so the output is hash-partitioned by (a, b). return ClusteringKeys.hash("a", "b") ``` Keys may be column names or arbitrary [`Expression`](../api/expressions.md)s. An expression-valued key follows a projection that materializes it as a derived column, so the clustering is preserved even after a `with_column`: ```python def hour_bucket(ts: "daft.Expression") -> "daft.Expression": # bucket a unix-epoch timestamp (in seconds) into hourly buckets return ts // 3600 class EventSource(DataSource): def get_clustering_keys(self) -> ClusteringKeys | None: return ClusteringKeys.hash(col("producer"), hour_bucket(col("ts"))) df = ( EventSource().read() .with_column("hour", hour_bucket(col("ts"))) # materialize the expression key as a column .groupby("producer", "hour") # covered by the declared clustering => no shuffle .sum("value") ) ``` #### Range clustering If your source already emits data where each task covers a non-overlapping range of values for those columns, you can tell Daft by returning `ClusteringKeys.range()`. Pass `descending=True` if partitions are ordered from highest to lowest values. If any key values are null, declare where those rows live with `nulls_first` (default: nulls in the last partition when ascending, the first when descending, matching sort semantics). ```python class TimeSeriesSource(DataSource): # ... name / schema / get_tasks as above ... def get_clustering_keys(self) -> ClusteringKeys | None: # Each task holds a non-overlapping ts range (ascending partition order). return ClusteringKeys.range("ts") # Or, if partitions are ordered high-to-low: # return ClusteringKeys.range("ts", descending=True) ``` !!! note "Shuffle elision applies to the distributed runner" Daft only inserts these shuffles when running distributed (e.g. on Ray); the single-node runner already executes the operators locally. The declaration is therefore a no-op for local execution and only changes plans for distributed runs. !!! warning "Clustering must hold for every task" Daft trusts the declaration. For hash clustering, every row with the same hash of the declared keys must be produced within a single task. For range clustering, each task must cover a genuinely non-overlapping range of values for the declared columns. Incorrectly overriding `get_clustering_keys()` will likely lead to incorrect results. ## Writing to a Custom Data Sink ### Step 1: Implement the `DataSink` Interface Create a class that inherits from [`DataSink`](../api/io.md#daft.io.sink.DataSink) and implements the required methods. Here's a simple example doing this with a custom local file writer. === "🐍 Python" ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Step 2: Use Your Custom Data Sink === "🐍 Python" ```python import daft from custom_datasink_example import LocalFileDataSink data = { "id": list(range(1, 1001)), "name": [f"User_{i}" for i in range(1, 1001)], } local_file_data_sink = LocalFileDataSink( output_dir="./output_folder", filename_prefix="users", max_rows_per_file=10 ) ( daft.from_pydict(data) .write_sink(local_file_data_sink) .show() ) ``` **Sample File Output:** ``` {'id': 111, 'name': 'User_111'} {'id': 112, 'name': 'User_112'} {'id': 113, 'name': 'User_113'} {'id': 114, 'name': 'User_114'} {'id': 115, 'name': 'User_115'} {'id': 116, 'name': 'User_116'} {'id': 117, 'name': 'User_117'} {'id': 118, 'name': 'User_118'} {'id': 119, 'name': 'User_119'} {'id': 120, 'name': 'User_120'} ``` ## More Examples For further reference, feel free to check out some of our notable data connectors that are implemented using the `DataSource` and `DataSink` interfaces: - [Reads from Video Frames](https://github.com/Eventual-Inc/Daft/blob/main/daft/io/av/_read_video_frames.py) - [Reads from LanceDB](https://github.com/daft-engine/daft-lance/blob/main/daft_lance/lance_scan.py) - [Writes to Hugging Face 🤗](https://github.com/Eventual-Inc/Daft/blob/main/daft/io/huggingface/sink.py) - [Writes to ClickHouse](https://github.com/Eventual-Inc/Daft/blob/main/daft/io/clickhouse/clickhouse_data_sink.py) - [Writes to Turbopuffer](https://github.com/Eventual-Inc/Daft/blob/main/daft/io/turbopuffer/turbopuffer_data_sink.py) - [Writes to LanceDB](https://github.com/daft-engine/daft-lance/blob/main/daft_lance/lance_data_sink.py) --- ### Connectors Delta Lake (`docs/connectors/delta_lake.md`) # Reading from and Writing to Delta Lake [Delta Lake](https://delta.io/) is an open-source storage framework for data analytics on data lakes. It provides ACID transactions, scalable metadata handling, and a unification of streaming and batch data processing, all on top of Parquet files in cloud storage. Daft currently supports: 1. **Parallel + Distributed Reads:** Daft parallelizes Delta Lake table reads over all cores of your machine, if using the default multithreading runner, or all cores + machines of your Ray cluster, if using the [distributed Ray runner](../distributed/index.md). 2. **Skipping Filtered Data:** Daft ensures that only data that matches your [`df.where()`][daft.DataFrame.where] filter will be read, often skipping entire files/partitions. 3. **Multi-cloud Support:** Daft supports reading Delta Lake tables from AWS S3, Azure Blob Store, and GCS, as well as local files. A detailed Delta Lake roadmap for Daft can be found on [our GitHub issues](https://github.com/Eventual-Inc/Daft/issues/2457). For the overall Daft development plan, see [Daft Roadmap](../roadmap.md). ## Installing Daft with Delta Lake Support Daft internally uses the [deltalake](https://pypi.org/project/deltalake/) Python package to fetch metadata about the Delta Lake table, such as paths to the underlying Parquet files and table statistics. The `deltalake` package therefore must be installed to read Delta Lake tables with Daft, either manually or with the below `daft[deltalake]` extras install of Daft. ```bash pip install -U "daft[deltalake]" ``` ## Reading a Table A Delta Lake table can be read by providing [`daft.read_deltalake`][daft.read_deltalake] with the URI for your table. The below example uses the [deltalake](https://pypi.org/project/deltalake/) Python package to create a local Delta Lake table for Daft to read, but Daft can also read Delta Lake tables from all of the major cloud stores. === "🐍 Python" ```python # Create a local Delta Lake table. from deltalake import write_deltalake import pandas as pd df = pd.DataFrame({ "group": [1, 1, 2, 2, 3, 3, 4, 4], "num": list(range(8)), "letter": ["a", "b", "c", "d", "e", "f", "g", "h"], }) # This will write out separate partitions for group=1, group=2, group=3, group=4. write_deltalake("some-table", df, partition_by="group") ``` After writing this local example table, we can easily read it into Daft. === "🐍 Python" ```python # Read a Delta Lake table import daft df = daft.read_deltalake("some-table") ``` ## Data Skipping Optimizations Subsequent filters on the partition column `group` will efficiently skip data that doesn't match the predicate. In the below example, the `group != 2` partitions (files) will be pruned, i.e. they will never be read into memory. === "🐍 Python" ```python # Filter on partition columns will result in efficient partition pruning; non-matching partitions will be skipped. df2 = df.where(df["group"] == 2) df2.show() ``` Filters on non-partition columns will still benefit from automatic file pruning via file-level statistics. In the below example, the `group=2` partition (file) will have `2 <= df["num"] <= 3` lower/upper bounds for the `num` column, and since the filter predicate is `df["num"] < 2`, Daft will prune the file from the read. Similar is true for `group=3` and `group=4` partitions, with none of the data from those files being read into memory. === "🐍 Python" ```python # Filter on non-partition column, relying on file-level column stats to efficiently prune unnecessary file reads. df3 = df.where(df["num"] < 2) df3.show() ``` ## Write to Delta Lake You can use [`df.write_deltalake()`][daft.DataFrame.write_deltalake] to write a DataFrame to a Delta table: === "🐍 Python" ```python df.write_deltalake("tmp/daft-recordbatch", mode="overwrite") ``` Daft supports multiple write modes. See the API docs for [`df.write_deltalake()`][daft.DataFrame.write_deltalake] for more details. When writing Delta Lake tables to S3, Daft relies on the native conditional write support in `deltalake` versions >=0.23.0. DynamoDB locking is not required by default for AWS S3 writes. To explicitly use a DynamoDB locking provider, pass `dynamo_table_name="..."` to [`df.write_deltalake()`][daft.DataFrame.write_deltalake]. For deltalake versions less than 0.23.0, if dynamo_table_name is not provided and allow_unsafe_rename is False, ValueError exception will be raised. ## Checkpointing Daft supports idempotent writes to Delta Lake via the `checkpoint=` parameter on [`df.write_deltalake()`][daft.DataFrame.write_deltalake]. Retries of the same logical commit — after a crash, a transient catalog error, or a deliberate re-invocation — produce the same Delta state without duplicate commits. See the [Checkpointing user guide](../use-case/checkpointing.md) for concepts; the sections below cover Delta-specific behavior. ### Example The pattern: one `CheckpointStore` paired into both the source (via `CheckpointConfig`) and the sink (via `IdempotentCommit`). The source records which inputs were processed; the sink stamps the resulting Delta commit with the idempotence key. === "🐍 Python" ```python import daft # One store, paired into both the source and the sink. store = daft.CheckpointStore("s3://my-bucket/ckpt/") df = daft.read_parquet( "s3://input/", checkpoint=daft.CheckpointConfig(store, on="file_id"), ) # Any map-only operations work here: filter, project, UDF, explode, ... df = df.where(df["status"] == "active") written = df.write_deltalake( "s3://my-bucket/my-table/", checkpoint=daft.IdempotentCommit(store, idempotence_key="job-2026-05-21-001"), ) ``` A fresh run produces one new Delta commit tagged with `daft.idempotence-key=job-2026-05-21-001`. Retries with the same `idempotence_key` recognize the prior commit and exit cleanly — no duplicate commit, no reprocessing of inputs already handled. ### Inspecting the Marker Every idempotent commit tags its Delta commit info with `daft.idempotence-key`. To verify which logical commit produced the current state of a table: === "🐍 Python" ```python # Inspect via the deltalake library. from deltalake import DeltaTable table = DeltaTable("s3://my-bucket/my-table/") print(table.history()[0].get("daft.idempotence-key")) # → "job-2026-05-21-001" ``` For older commits, walk `table.history()` and check each entry. ### Constraints These are constraints specific to the Delta Lake connector. Daft-wide constraints (Ray runner, map-only pipelines, single-writer concurrency, etc.) are in the [user guide's Limitations section](../use-case/checkpointing.md#limitations). - **`mode='append'` only.** Other modes raise `NotImplementedError` when `checkpoint=` is set. - **Reserved `daft.idempotence-*` property prefix.** Keys in `custom_metadata` that start with this prefix raise `ValueError` — Daft uses this namespace internally. ### Idempotence-Key Contract The user picks the `idempotence_key`. Daft uses it as the marker that identifies a logical commit. The key must be stable across retries and unique across distinct logical commits — both halves matter: - **Same key, different inputs → silent no-op (data loss).** Daft sees the existing marker and skips the commit. The new data is dropped without an error. - **Different key on a retry of the same logical commit → duplicate commit.** Daft doesn't recognize the prior attempt and lands a second commit. ### Recovery The [user guide's Recovery section](../use-case/checkpointing.md#recovery) walks through the full chronology. The Delta-specific steps: - **Commit-from-store.** When a run crashes after the pipeline finishes but before the Delta commit lands, Daft reads the staged file references from the store on rerun and commits them as a single new Delta transaction with `daft.idempotence-key` in `custom_metadata`. - **Marker recognition.** When a run crashes after the Delta commit lands but before the store is marked done — or when the user deliberately re-runs with the same key — Daft walks `table.history()`, finds the marker, marks the store, and exits. No second commit. Returned DataFrame is empty. **Orphan files on crash.** A worker that crashed mid-task may leave parquet files unreferenced by any Delta commit. Daft doesn't auto-clean these — use the deltalake `vacuum` operation. ### Delta-Specific Notes - **Parallel history walk.** `table.history()` is already concurrent under the hood (deltalake-rs buffers up to `num_cpus * 4` log reads), so marker recognition is fast even on tables with long commit histories. - **Fresh-table case handled.** If the target table doesn't exist yet at the configured URI, the first idempotent write creates it. - **Transient retry on `CommitFailedError`.** Daft retries up to twice on this exception (concurrent-writer conflicts, lock contention). Other exceptions — schema mismatches, metadata-construction errors, network errors — propagate immediately. Wrap the call in your own retry policy if you need broader coverage. ## Type System Daft and Delta Lake have compatible type systems. Here are how types are converted across the two systems. When reading from a Delta Lake table into Daft: | Delta Lake | Daft | | ----------------------------------- | -------------------------------------------------------------------------------------------- | | `BOOLEAN` | [`daft.DataType.bool()`][daft.datatype.DataType.bool] | | `BYTE` | [`daft.DataType.int8()`][daft.datatype.DataType.int8] | | `SHORT` | [`daft.DataType.int16()`][daft.datatype.DataType.int16] | | `INT` | [`daft.DataType.int32()`][daft.datatype.DataType.int32] | | `LONG` | [`daft.DataType.int64()`][daft.datatype.DataType.int64] | | `FLOAT` | [`daft.DataType.float32()`][daft.datatype.DataType.float32] | | `DOUBLE` | [`daft.DataType.float64()`][daft.datatype.DataType.float64] | | `DECIMAL(precision, scale)` | [`daft.DataType.decimal128(precision, scale)`][daft.datatype.DataType.decimal128] | | `DATE` | [`daft.DataType.date()`][daft.datatype.DataType.date] | | `TIMESTAMP_NTZ` | [`daft.DataType.timestamp(timeunit="us", timezone=None)`][daft.datatype.DataType.timestamp] | | `TIMESTAMP` | [`daft.DataType.timestamp(timeunit="us", timezone="UTC")`][daft.datatype.DataType.timestamp] | | `STRING` | [`daft.DataType.string()`][daft.datatype.DataType.string] | | `BINARY` | [`daft.DataType.binary()`][daft.datatype.DataType.binary] | | `MAP` | [`daft.DataType.map(key_type, value_type)`][daft.datatype.DataType.map] | | `STRUCT<[field_name: field_type,]>` | [`daft.DataType.struct(fields)`][daft.datatype.DataType.struct] | | `ARRAY` | [`daft.DataType.list(element_type)`][daft.datatype.DataType.list] | References: * [Python `unitycatalog` type name code reference](https://github.com/unitycatalog/unitycatalog-python/blob/main/src/unitycatalog/types/table_info.py) * [Spark types documentation](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/data_types.html) * [Databricks types documentation](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-datatypes) --- ### Connectors Files (`docs/connectors/files.md`) # Creating File References with from_files Daft provides [`daft.from_files()`][daft.io.from_files] to create a DataFrame of lazy file references from glob patterns. Unlike other read functions that immediately load file contents, `from_files` creates `File` objects that can be read on demand. ## Basic Usage === "Local Files" ```python import daft df = daft.from_files("/path/to/files/*.jpeg") df.show() ``` === "Remote Files (S3)" ```python import daft df = daft.from_files("s3://my-bucket/images/*.png") df.show() ``` === "Remote Files (GCS)" ```python import daft df = daft.from_files("gs://my-bucket/images/*.png") df.show() ``` ## Output Schema The `from_files` function returns a DataFrame with a single column: | Column | Type | Description | |--------|------|-------------| | `file` | `File` | Lazy file references that can be read on demand | ## Wildcard Patterns `from_files` supports standard glob patterns: | Pattern | Description | |---------|-------------| | `*` | Matches any number of characters | | `?` | Matches any single character | | `[...]` | Matches any single character in the brackets | | `**` | Recursively matches directories | === "Examples" ```python # All JPEG files in a directory df = daft.from_files("/images/*.jpeg") # Recursive search df = daft.from_files("/images/**/*.png") # Multiple patterns df = daft.from_files(["/images/*.jpeg", "/photos/*.jpeg"]) ``` ## Working with File Objects The `File` type is a lazy reference that provides access to file metadata and content: ```python import daft from daft import col df = daft.from_files("/path/to/files/*") # Access file properties df = df.select( col("file").file_path().alias("path"), col("file").file_size().alias("size_bytes"), ) df.show() ``` ## Use Cases ### Image Processing Pipeline For image processing, use [`daft.from_glob_path()`][daft.from_glob_path] with `.download()` and `decode_image()`: ```python import daft from daft import col from daft.functions import decode_image df = ( daft.from_glob_path("s3://bucket/images/**/*.jpg") .with_column("image_bytes", col("path").download()) .with_column("image", decode_image(col("image_bytes"))) ) df.show() ``` ### Batch File Operations ```python import daft from daft import col # Get file references with metadata df = daft.from_files("/data/**/*") # Filter by file properties before reading content large_files = df.where(col("file").file_size() > 1_000_000) large_files.show() ``` ## Comparison with from_glob_path `from_files` is similar to [`daft.from_glob_path()`][daft.from_glob_path] but returns `File` objects instead of path strings: | Function | Returns | Use Case | |----------|---------|----------| | `from_glob_path` | `path` column (string) | When you need file paths only | | `from_files` | `file` column (File) | When you need to read file content or access file properties | ```python # from_glob_path returns paths paths_df = daft.from_glob_path("/images/*.jpg") # Column: path (string) # from_files returns File objects files_df = daft.from_files("/images/*.jpg") # Column: file (File) ``` ## Empty Results If no files match the glob pattern(s), an empty DataFrame is returned instead of raising an error: ```python df = daft.from_files("/nonexistent/*.txt") df.show() # Empty DataFrame with "file" column ``` --- ### Connectors Gcs (`docs/connectors/gcs.md`) # Reading from and Writing to Google Cloud Storage Daft is able to read/write data to/from Google Cloud Storage (GCS), and understands natively the URL protocols `gs://` and `gcs://` as referring to data that resides in GCS. ## Authorization/Authentication In GCS, data is stored under the hierarchy of: 1. Project: The Google Cloud project that owns the storage resources. 2. Bucket: The container for data storage. 3. Object Key: The unique identifier for a piece of data within a bucket. URLs to data in GCS come in the form: `gs://{BUCKET}/{OBJECT_KEY}`. ### Rely on Environment You can configure [Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials) to have Daft automatically discover credentials. Common methods include: - Setting the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to point to a service account key file - Running `gcloud auth application-default login` for local development - Using the default service account when running on Google Cloud (GCE, GKE, Cloud Run, etc.) Please be aware that when doing so in a distributed environment such as Ray, Daft will pick these credentials up from worker machines and thus each worker machine needs to be appropriately provisioned. If instead you wish to have Daft use credentials from the "driver", you may wish to manually specify your credentials. ### Manually specify credentials You may also choose to pass these values into your Daft I/O function calls using a [`daft.io.GCSConfig`][daft.io.GCSConfig] config object. [`daft.set_planning_config`][daft.context.set_planning_config] is a convenient way to set your [`daft.io.IOConfig`][daft.io.IOConfig] as the default config to use on any subsequent Daft method calls. === "Using Service Account Credentials" ```python from daft.io import IOConfig, GCSConfig # Path to your service account JSON key file io_config = IOConfig(gcs=GCSConfig(credentials="/path/to/service-account-key.json")) # Globally set the default IOConfig for any subsequent I/O calls daft.set_planning_config(default_io_config=io_config) # Perform some I/O operation df = daft.read_parquet("gs://my_bucket/my_path/**/*") ``` === "Using OAuth2 Token" ```python from daft.io import IOConfig, GCSConfig # Use an OAuth2 access token directly io_config = IOConfig(gcs=GCSConfig(token="your-oauth2-access-token")) # Perform some I/O operation df = daft.read_parquet("gs://my_bucket/my_path/**/*", io_config=io_config) ``` === "Anonymous Access" ```python from daft.io import IOConfig, GCSConfig # Access public buckets without credentials io_config = IOConfig(gcs=GCSConfig(anonymous=True)) # Read from a public bucket df = daft.read_parquet("gs://public_bucket/data/**/*", io_config=io_config) ``` Alternatively, Daft supports overriding the default IOConfig per-operation by passing it into the `io_config=` keyword argument. This is extremely flexible as you can pass a different [`daft.io.GCSConfig`][daft.io.GCSConfig] per function call if you wish! === "Per-Operation Config" ```python from daft.io import IOConfig, GCSConfig io_config = IOConfig(gcs=GCSConfig(credentials="/path/to/service-account-key.json")) # Perform some I/O operation but override the IOConfig df2 = daft.read_csv("gs://my_bucket/my_other_path/**/*", io_config=io_config) ``` ## Configuration Options The [`daft.io.GCSConfig`][daft.io.GCSConfig] object supports the following options: | Parameter | Type | Description | |-----------|------|-------------| | `project_id` | `str` | Google Cloud project ID | | `credentials` | `str` | Path to service account JSON key file | | `token` | `str` | OAuth2 access token | | `anonymous` | `bool` | Whether to use anonymous access (for public buckets) | | `max_connections` | `int` | Maximum number of concurrent connections | | `retry_initial_backoff_ms` | `int` | Initial backoff time in milliseconds for retries | | `connect_timeout_ms` | `int` | Connection timeout in milliseconds | | `read_timeout_ms` | `int` | Read timeout in milliseconds | | `num_tries` | `int` | Number of retry attempts | --- ### Connectors Generic File Source Options (`docs/connectors/generic-file-source-options.md`) # Generic File Source Options These options apply to `read_parquet`, `read_csv`, and `read_iceberg`. They are not tied to any single connector or format. Other readers (`read_json`, `read_warc`, `read_text`) do not support these options. ## Ignoring Corrupt Files When reading large collections of files, some files may be unreadable — corrupt, truncated, or deleted between the time Daft lists them and the time it reads them. By default, Daft raises an error and halts the query. The `ignore_corrupt_files` option changes that behavior: qualifying files are silently skipped and the query continues with the remaining data. ### Enabling `ignore_corrupt_files` Pass `ignore_corrupt_files=True` to any of the supported reader functions: ```python import daft # Parquet / CSV (glob-based) df = daft.read_parquet("s3://my-bucket/data/**/*.parquet", ignore_corrupt_files=True) df = daft.read_csv("s3://my-bucket/data/**/*.csv", ignore_corrupt_files=True) # Iceberg import pyiceberg table = pyiceberg.table.StaticTable.from_metadata("s3://bucket/iceberg/metadata.json") df = daft.read_iceberg(table, ignore_corrupt_files=True) df.collect() ``` ### What counts as "corrupt" Daft skips a file when it encounters a problem that is specific to the file itself and cannot be resolved by retrying: | Category | Examples | |---|---| | **Invalid format** | Bad Parquet magic bytes, truncated footer, mismatched row/column counts | | **Corrupt data** | Unreadable row group, invalid CSV encoding, wrong field count in a row | | **Missing file** | File deleted between listing and reading (e.g. concurrent compaction or partition overwrite) | Daft does **not** skip files for transient infrastructure problems, because those can and should be retried: | Category | Examples | |---|---| | **Network errors** | Connection reset, read timeout, throttled I/O | | **Permission errors** | Access denied, insufficient credentials | This distinction matters. Silently retrying a permission error would mask a misconfiguration that needs human attention. ### Observability: knowing what was skipped `ignore_corrupt_files` is designed around the principle that **errors should be visible, not hidden**. Daft provides two complementary observability mechanisms. #### Python warning logs Daft emits a `WARNING`-level log message for every skipped file, including the file path and the reason: ``` WARNING daft.io - Skipping corrupt Parquet file s3://my-bucket/data/bad.parquet: ... WARNING daft.io - Skipping corrupt CSV chunk in s3://my-bucket/data/partial.csv: ... ``` You can see these with standard Python logging: ```python import logging logging.basicConfig(level=logging.WARNING) ``` #### `df.skipped_corrupt_files` — programmatic access After materializing the dataframe with `.collect()`, the `skipped_corrupt_files` property returns the list of skipped `(path, reason)` pairs as structured data, so your pipeline code can act on them: ```python df = daft.read_parquet("s3://my-bucket/data/**/*.parquet", ignore_corrupt_files=True) df.collect() skipped = df.skipped_corrupt_files # list[tuple[str, str, bool]] for path, reason, partial in skipped: tag = " (partial)" if partial else "" print(f"Skipped{tag}: {path}\n Reason: {reason}") ``` Each entry is a `(path, reason, partial)` tuple. When `partial` is `True`, some batches from the file were already emitted before the corruption was detected — the file was not fully skipped. This can happen when corruption appears in a later row group. `skipped_corrupt_files` is available after calling `.collect()` on the dataframe. Other execution methods such as `.count_rows()` do not populate this property, because they operate on an internal dataframe rather than materializing the original one. ### Handling skipped files in production Because `skipped_corrupt_files` is plain Python data, you can plug it directly into your existing alerting or data-quality workflows: ```python import daft df = daft.read_parquet("s3://my-bucket/nightly/**/*.parquet", ignore_corrupt_files=True) df.write_parquet("s3://my-bucket/processed/") skipped = df.skipped_corrupt_files if skipped: # Option 1: send an alert send_alert(f"{len(skipped)} file(s) skipped during nightly run", details=skipped) # Option 2: push to a dead-letter queue for later reprocessing for path, reason, partial in skipped: dead_letter_queue.put({"path": path, "reason": reason, "partial": partial, "run": TODAY}) ``` This pattern — **errors visible, impact contained, tooling to fix** — lets automated batch jobs complete reliably while still surfacing problems for human review. !!! warning "Do not use `ignore_corrupt_files` as a catch-all" This option is designed for files that are genuinely unreadable. It should not be used to suppress transient I/O errors (network issues, throttling) — Daft already retries those automatically. If you find yourself needing `ignore_corrupt_files` for a large fraction of your files, investigate the root cause rather than silencing the errors. ### Supported formats | Format | File-level skip | Within-file error skip | |---|---|---| | Parquet (`read_parquet`) | Yes (bad footer, wrong magic bytes, file too small) | Yes (corrupt row group data) | | CSV (`read_csv`) | Yes (unreadable file, truncated) | Yes (bad encoding, wrong field count in chunk) | | Iceberg (`read_iceberg`) | Yes (data files go through the Rust Parquet reader) | Yes | All three are supported from both the Python API and the SQL table functions. See [Table Function Options](../sql/index.md#table-function-options) for the full list of named arguments each SQL reader accepts. !!! note "Iceberg delete files" Corruption in Iceberg *delete files* is not covered. If a delete file is unreadable, Daft will raise an error regardless of `ignore_corrupt_files`. Delete files are small metadata structures and corruption there generally indicates a more serious catalog inconsistency. !!! note "Count pushdown" When `ignore_corrupt_files` is enabled for Parquet, count pushdown is disabled. This means `df.count()` will read all row-group data instead of using the metadata-only optimization, which may be slower on large datasets. --- ### Connectors Glue (`docs/connectors/glue.md`) # Reading from and Writing to AWS Glue !!! warning "Warning" These APIs are early in their development. Please feel free to [open feature requests and file issues](https://github.com/Eventual-Inc/Daft/issues/new/choose). We'd love hear want you would like, thank you! 🤘 Daft integrates with [AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html) using the [Daft Catalog](../api/catalogs_tables.md) interface. Daft provides builtin support for a handful of tables (details below), but you may provide your own `GlueTable` implementations and register them to your `GlueCatalog` instance. Note that in Daft, the AWS Glue service-level *Data Catalog* maps to a Daft *Catalog* and a Glue *Database* is a *Namespace* — this can be confusing and is important to remember. | Daft | Glue | |-----------|--------------| | Catalog | Data Catalog | | Namespace | Database | | Table | Table | ## Example === "🐍 Python" ```python from daft.catalog.__glue import load_glue # Load a glue catalog instance catalog = load_glue( name="my_glue_catalog", region_name="us-west-2" ) # Load a glue table tbl = catalog.get_table("my_namespace.my_table") # Read the table as a DataFrame df = tbl.read() df.show() ``` ## Support Glue supports many different table [classifications](https://docs.aws.amazon.com/glue/latest/dg/add-classifier.html#classifier-built-in) along with various table formats like [Iceberg](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-iceberg.html), [Delta Lake](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-delta-lake.html), and [Hudi](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-hudi.html). ### Formats Daft has preliminary read support for the CSV and Parquet formats, but does not yet support reading with Hive-style partitioning. Daft does support reading and writing both Iceberg and Delta Lake. We do not currently support creating Glue tables. | Table Format | Support | AWS Documentation | |--------------|-------------|---------------------------------------------------------------------------------------------------------------| | CSV | read | [Documentation](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-csv-home.html) | | Parquet | read | [Documentation](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-parquet-home.html) | | Iceberg | read, write | [Documentation](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-iceberg.html) | | Delta Lake | read, write | [Documentation](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-delta-lake.html) | ## Type System The [Glue Catalog Type System](https://docs.aws.amazon.com/glue/latest/dg/glue-types.html#glue-types-catalog) is based on the [Apache Hive Type System](https://cwiki.apache.org/confluence/display/hive/languagemanual+types); however, the exact types are not validated and are dependent upon the underlying table format. > The Data Catalog does not validate types written to type fields. When AWS Glue components read and write to the Data Catalog, they will be compatible with each other. AWS Glue components also aim to preserve a high degree of compatibility with the Hive types. However, AWS Glue components do not guarantee compatibility with all Hive types. | Table Format | Type System | |--------------|------------------------------------------------------------------------------------------------------------------------------------------| | CSV | [Glue CSV Type Reference](https://docs.aws.amazon.com/glue/latest/webapi/API_CsvClassifier.html#Glue-Type-CsvClassifier-CustomDatatypes) | | Parquet | [Arrow Type System Reference](https://arrow.apache.org/docs/python/api/datatypes.html) | | Iceberg | [Iceberg Type System Reference](iceberg.md#type-system) | | Delta Lake | [Delta Lake Type System Reference](delta_lake.md#type-system) | ## Custom Table Implementation !!! warning "Warning" This is not considered a stable API, it is just a patch technique! To implement a custom table format, you must implement the `GlueTable` abstract class and register to your `GlueCatalog` instance. To register, you must append your custom table's class to the catalog's `_table_impls` field. The Daft `GlueCatalog._table_impls` field holds a list of `GlueTable` implementation classes. When we resolve a table, we call Glue's `GetTable` API and call `from_table_info` with the [Glue Table object](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-tables.html#aws-glue-api-catalog-tables-Table). It is expected that each `GlueTable` implementation will throw a `ValueError` if the table metadata does not match. ```python from typing import Any, Literal from daft.catalog.__glue import GlueCatalog, GlueTable, load_glue class GlueTestTable(GlueTable): """GlueTestTable shows how we register custom table implementations.""" @classmethod def from_table_info(cls, catalog: GlueCatalog, table: dict[str, Any]) -> GlueTable: if bool(table["Parameters"].get("pytest")): return cls(catalog, table) raise ValueError("Expected Parameter pytest='True'") def read(self, **options) -> DataFrame: raise NotImplementedError def write(self, df: DataFrame, mode: Literal['append'] | Literal['overwrite'] = "append", **options) -> None: raise NotImplementedError gc = load_glue("my_glue_catalog", region_name="us-west-2") gc._table_impls.append(GlueTestTable) # Register custom table implementation ``` --- ### Connectors Goosefs (`docs/connectors/goosefs.md`) # Reading from and Writing to GooseFS Daft supports reading and writing data to [GooseFS](https://cloud.tencent.com/document/product/1424) — a distributed caching file system — and understands natively the URL protocol `goosefs://` as referring to data that resides in a GooseFS cluster. Under the hood, Daft speaks GooseFS's native gRPC protocol via [OpenDAL](https://opendal.apache.org/)'s `services-goosefs` backend. This means that no HDFS/JVM gateway is required — Daft talks to GooseFS masters and workers directly. ## URL Format URLs to data in GooseFS come in the form: ```text goosefs://{MASTER_HOST}:{MASTER_PORT}/{PATH} ``` - `MASTER_HOST:MASTER_PORT` identifies a GooseFS master. If no explicit `master_addr` is provided in the config, Daft will use this URL authority as the master address. - `PATH` is the absolute path to the file or directory inside the GooseFS namespace. !!! note "High Availability (HA) masters" For a GooseFS deployment with multiple masters, encode all masters as a comma-separated list in the [`GooseFSConfig`][daft.io.GooseFSConfig] `master_addr` field, e.g. `"10.0.0.1:9200,10.0.0.2:9200,10.0.0.3:9200"`. When `master_addr` is set, the URL authority is ignored — you can then use a symbolic host in the URL, for example `goosefs://ha/path/to/data`. ## Authorization/Authentication GooseFS supports two authentication modes: `simple` (username-based, the default) and `nosasl` (no authentication). Anonymous access forces `nosasl`. ### Rely on Environment You can configure Daft to automatically discover GooseFS settings from environment variables. Daft supports the following variables when [`GooseFSConfig.from_env()`][daft.io.GooseFSConfig] is used: | Environment Variable | Description | |---|---| | `GOOSEFS_MASTER_ADDR` | Master address(es) for the GooseFS cluster (single `host:port`, or comma-separated list for HA) | | `GOOSEFS_AUTH_USERNAME` | Authentication username (falls back to `USER` / `USERNAME` when unset) | | `GOOSEFS_AUTH_PASSWORD` | Authentication password | | `GOOSEFS_AUTH_TYPE` | Authentication type — `"simple"` or `"nosasl"` | | `GOOSEFS_WRITE_TYPE` | Default write type for new files (`"must_cache"`, `"cache_through"`, `"through"`, `"async_through"`) | | `GOOSEFS_ROOT` | Root path of the backend | Please be aware that when doing so in a distributed environment such as Ray, Daft will pick these values up from worker machines and thus each worker machine needs to be appropriately provisioned. If instead you wish to have Daft use configuration from the "driver", you may wish to manually specify your credentials. ### Manually specify credentials You can pass a [`GooseFSConfig`][daft.io.GooseFSConfig] into your Daft I/O function calls using a [`daft.io.IOConfig`][daft.io.IOConfig] config object. [`daft.set_planning_config`][daft.context.set_planning_config] is a convenient way to set your [`daft.io.IOConfig`][daft.io.IOConfig] as the default config to use on any subsequent Daft method calls. === "Using Simple Authentication" ```python import daft from daft.io import IOConfig, GooseFSConfig io_config = IOConfig( goosefs=GooseFSConfig( master_addr="10.0.0.1:9200", auth_type="simple", auth_username="alice", ) ) # Globally set the default IOConfig for any subsequent I/O calls daft.set_planning_config(default_io_config=io_config) # Perform some I/O operation df = daft.read_parquet("goosefs://10.0.0.1:9200/my_path/**/*") ``` === "Using HA Masters" ```python import daft from daft.io import IOConfig, GooseFSConfig io_config = IOConfig( goosefs=GooseFSConfig( # Comma-separated HA master list; the URL authority is ignored. master_addr="10.0.0.1:9200,10.0.0.2:9200,10.0.0.3:9200", auth_type="simple", auth_username="alice", ) ) df = daft.read_parquet("goosefs://ha/my_path/**/*", io_config=io_config) ``` === "Anonymous Access" ```python import daft from daft.io import IOConfig, GooseFSConfig # Anonymous mode forces `nosasl` and skips credential forwarding. io_config = IOConfig(goosefs=GooseFSConfig(anonymous=True)) df = daft.read_parquet( "goosefs://10.0.0.1:9200/public/**/*", io_config=io_config, ) ``` === "Custom Root and Write Type" ```python import daft from daft.io import IOConfig, GooseFSConfig io_config = IOConfig( goosefs=GooseFSConfig( master_addr="10.0.0.1:9200", root="/warehouse", # One of: must_cache, cache_through, through, async_through write_type="cache_through", ) ) df = daft.read_parquet("goosefs://10.0.0.1:9200/table/**/*", io_config=io_config) ``` Alternatively, Daft supports overriding the default `IOConfig` per-operation by passing it into the `io_config=` keyword argument. This is extremely flexible as you can pass a different [`GooseFSConfig`][daft.io.GooseFSConfig] per function call if you wish! === "Per-Operation Config" ```python import daft from daft.io import IOConfig, GooseFSConfig io_config = IOConfig( goosefs=GooseFSConfig( master_addr="10.0.0.1:9200", auth_username="alice", ) ) # Perform some I/O operation but override the IOConfig df2 = daft.read_csv( "goosefs://10.0.0.1:9200/my_other_path/**/*", io_config=io_config, ) ``` ## Writing Data Daft supports writing data to GooseFS using the same `GooseFSConfig`: ```python import daft from daft.io import IOConfig, GooseFSConfig io_config = IOConfig( goosefs=GooseFSConfig( master_addr="10.0.0.1:9200", auth_username="alice", # Persist writes through to the underlying UFS. write_type="cache_through", ) ) df = daft.from_pydict({"a": [1, 2, 3], "b": ["x", "y", "z"]}) # Write to GooseFS as Parquet df.write_parquet("goosefs://10.0.0.1:9200/output/", io_config=io_config) ``` ## Configuration Options The [`GooseFSConfig`][daft.io.GooseFSConfig] object supports the following options: | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `root` | `str` | `None` (`"/"`) | Root path of the backend. All operations happen under this root. | | `master_addr` | `str` | `None` | Master address(es) in `host:port` format. Comma-separated for HA, e.g. `"10.0.0.1:9200,10.0.0.2:9200"`. If not set, the URL authority is used. | | `block_size` | `int` | `None` (64 MiB) | Block size in bytes for new files. | | `chunk_size` | `int` | `None` (1 MiB) | Chunk size in bytes for streaming RPCs. | | `write_type` | `str` | `None` (`"cache_through"`) | Default write type for new files. One of `"must_cache"`, `"cache_through"`, `"through"`, `"async_through"`. | | `auth_type` | `str` | `None` (`"simple"`) | Authentication type. One of `"nosasl"`, `"simple"`. | | `auth_username` | `str` | `None` (current OS user) | Authentication username used in SIMPLE mode. | | `auth_password` | `str` | `None` | Optional authentication password. | | `anonymous` | `bool` | `False` | Whether to use anonymous access. Forces `auth_type="nosasl"` and skips credential forwarding. | | `max_retries` | `int` | `3` | Maximum number of retries for failed requests. | | `retry_timeout_ms` | `int` | `30000` | Timeout duration for retry attempts in milliseconds. | | `connect_timeout_ms` | `int` | `10000` | Connection timeout in milliseconds. | | `read_timeout_ms` | `int` | `30000` | Read timeout in milliseconds. | | `max_concurrent_requests` | `int` | `50` | Maximum number of concurrent requests. | | `max_connections` | `int` | `50` | Maximum number of connections per IO thread. | !!! tip "Master address resolution" You do not have to duplicate the master address in both the URL and the config: - If only the URL authority (`goosefs://host:port/...`) is provided, it is used as `master_addr`. - If `master_addr` is set explicitly, it takes precedence over the URL authority (this is required for HA setups). - If neither is provided, the operation will fail because there is no master to talk to. !!! tip "Choosing a `write_type`" - `must_cache` — write only to workers' cache; not persisted to UFS. - `cache_through` — write to workers' cache and synchronously to UFS. This is the safest default for durability. - `through` — bypass workers' cache and write directly to UFS. - `async_through` — write to workers' cache and asynchronously to UFS. ## Supported Operations Daft supports the following operations with GooseFS: - **Read**: `read_parquet`, `read_csv`, `read_json`, and other file readers - **Write**: `write_parquet`, `write_csv`, `write_json` - **List**: Listing objects with glob pattern matching - **Delete**: Deleting objects --- ### Connectors Gravitino (`docs/connectors/gravitino.md`) # Reading from Apache Gravitino [Apache Gravitino](https://gravitino.apache.org/) is an open-source data catalog that provides unified metadata management for various data sources and storage systems. Users of Gravitino can work with data assets such as tables (Iceberg, Hive, etc.) and filesets (storing raw files, on s3, gcs, azure blob, etc). To use Daft with Gravitino, you will need to install Daft with the `gravitino` option specified like so: ```bash pip install daft[gravitino] ``` !!! warning "Warning" These APIs are in beta and may be subject to change as the Gravitino connector continues to be developed. ## Features - **Catalog Navigation**: List catalogs, schemas, and tables - **Multi-Format Tables**: Read Iceberg and Hive/Parquet tables via `catalog.get_table("...").read()` - **Table Management**: Load existing tables or create new external tables - **Fileset Support**: Access Gravitino filesets for file storage - **GVFS Protocol**: Read and write files using `gvfs://` URLs for seamless fileset access - **Authentication**: Supports simple and OAuth2 authentication methods - **Daft Catalog Integration**: Integration with Daft's catalog system via `Catalog.from_gravitino()` ## Connecting to Gravitino === "🐍 Python" ```python import daft from daft.catalog import Catalog catalog = Catalog.from_gravitino( endpoint="http://localhost:8090", metalake_name="my_metalake", username="admin", ) # List all available tables tables = catalog.list_tables("my_catalog.my_schema") # Read a table (format detected automatically) df = catalog.get_table("my_catalog.my_schema.my_table").read() df.show() ``` ## Configuration ### Authentication `Catalog.from_gravitino` supports two authentication methods: 1. **Simple Authentication**: Uses username/password or just username 2. **OAuth2**: Uses bearer token authentication === "🐍 Python" ```python from daft.catalog import Catalog # Simple auth with username only catalog = Catalog.from_gravitino( endpoint="http://localhost:8090", metalake_name="my_metalake", auth_type="simple", username="admin", ) # OAuth2 auth catalog = Catalog.from_gravitino( endpoint="http://localhost:8090", metalake_name="my_metalake", auth_type="oauth2", token="my-bearer-token", ) ``` ### Storage Credentials Gravitino manages storage credentials through table and fileset properties. The client automatically extracts and configures: - **S3**: Access key, secret key, and session token ## GVFS Protocol Support Daft supports reading and writing files directly from Gravitino filesets using the `gvfs://` protocol. This provides a unified interface for accessing files stored in various cloud storage systems through Gravitino's metadata management. ### GVFS URL Format GVFS URLs follow this format: ``` gvfs://fileset//// ``` Where: - `` - Name of the Gravitino catalog - `` - Name of the schema within the catalog - `` - Name of the fileset - `` - Optional path to specific files within the fileset ### Reading Files with GVFS === "🐍 Python" ```python import daft from daft.io import IOConfig, GravitinoConfig # Build an IOConfig for GVFS fileset access io_config = IOConfig( gravitino=GravitinoConfig( endpoint="http://localhost:8090", metalake_name="my_metalake", username="admin", ) ) # Read parquet files from a fileset df = daft.read_parquet( "gvfs://fileset/my_catalog/my_schema/my_fileset/**/*.parquet", io_config=io_config ) # Read specific file df = daft.read_parquet( "gvfs://fileset/my_catalog/my_schema/my_fileset/data.parquet", io_config=io_config ) # Read CSV files df = daft.read_csv( "gvfs://fileset/my_catalog/my_schema/my_fileset/*.csv", io_config=io_config ) # Use glob patterns for file discovery files_df = daft.from_glob_path( "gvfs://fileset/my_catalog/my_schema/my_fileset/**/*.json", io_config=io_config ) ``` ### Writing Files with GVFS === "🐍 Python" ```python import daft from daft.io import IOConfig, GravitinoConfig io_config = IOConfig( gravitino=GravitinoConfig( endpoint="http://localhost:8090", metalake_name="my_metalake", username="admin", ) ) # Create sample data df = daft.from_pydict({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35] }) # Write parquet files to a fileset df.write_parquet( "gvfs://fileset/my_catalog/my_schema/my_fileset/output.parquet", io_config=io_config ) # Write CSV files df.write_csv( "gvfs://fileset/my_catalog/my_schema/my_fileset/output.csv", io_config=io_config ) # Write JSON files df.write_json( "gvfs://fileset/my_catalog/my_schema/my_fileset/output.json", io_config=io_config ) ``` ### GVFS Benefits - **Unified Access**: Use the same URL format for reading and writing - **Storage Abstraction**: Access files without knowing underlying storage details (S3, GCS, etc.) - **Metadata Integration**: Leverage Gravitino's catalog metadata for data discovery - **Credential Management**: Gravitino handles storage credentials automatically - **Multi-format Support**: Works with Parquet, CSV, JSON, and other file formats ## API Reference ### Catalog.from_gravitino(...) Creates a Daft Catalog from a Gravitino metalake. === "🐍 Python" ```python from daft.catalog import Catalog catalog = Catalog.from_gravitino( endpoint="http://localhost:8090", metalake_name="my_metalake", username="admin", ) catalog.list_tables("my_catalog.my_schema") ``` #### Table.from_gravitino(table) Creates a Daft Table from a GravitinoTable. === "🐍 Python" ```python from daft.catalog import Catalog, Table catalog = Catalog.from_gravitino( endpoint="http://localhost:8090", metalake_name="my_metalake", username="admin", ) table = catalog.get_table("my_catalog.my_schema.my_table") df = table.read() ``` ## Requirements - Apache Gravitino server (0.9.0+) - Python requests library - Appropriate cloud storage credentials configured in Gravitino ## Compatibility This integration supports both legacy and current Gravitino API formats: - **Legacy format** (pre-1.0): Storage location in `properties.location` - **Current format** (1.0+): Multiple storage locations in `storageLocations` with configurable default The client automatically detects and handles both formats for seamless compatibility. ## Limitations - Credential vending is not yet implemented - This version directly calls Gravitino RESTful API, not using Gravitino Python client - GVFS write support currently works with S3-backed filesets (other storage backends coming soon) - Some advanced Gravitino features may not be exposed through this client ## Roadmap 1. Support for read/write Iceberg tables from Gravitino ✓ 2. Support for Hive/Parquet tables ✓ 3. Support for additional table formats (Hudi) 4. Support for more storages (gcs, azure adls, oss, etc) 5. Support for credential vending Please open issues on the [Daft repository](https://github.com/Eventual-Inc/Daft) or [Gravitino repository](https://github.com/apache/gravitino) if you have any use-cases that Daft Gravitino connector does not currently cover! --- ### Connectors Hudi (`docs/connectors/hudi.md`) # Reading from Apache Hudi [Apache Hudi](https://hudi.apache.org/) is an open-sourced transactional data lake platform that brings database and data warehouse capabilities to data lakes. Hudi supports transactions, efficient upserts/deletes, advanced indexes, streaming ingestion services, data clustering/compaction optimizations, and concurrency all while keeping your data in open source file formats. Daft currently supports: 1. **Parallel + Distributed Reads:** Daft parallelizes Hudi table reads over all cores of your machine, if using the default multithreading runner, or all cores + machines of your Ray cluster, if using the [distributed Ray runner](../distributed/index.md). 2. **Skipping Filtered Data:** Daft ensures that only data that matches your [`df.where()`][daft.DataFrame.where] filter will be read, often skipping entire files/partitions. 3. **Multi-cloud Support:** Daft supports reading Hudi tables from AWS S3, Azure Blob Store, and GCS, as well as local files. A detailed Apache Hudi roadmap for Daft can be found on [our GitHub Issues page](https://github.com/Eventual-Inc/Daft/issues/4389). For the overall Daft development plan, see [Daft Roadmap](../roadmap.md). ## Reading a Table To read from an Apache Hudi table, use the [`daft.read_hudi()`][daft.read_hudi] function. The following is an example snippet of loading an example table into Daft: === "🐍 Python" ```python # Read Apache Hudi table import daft df = daft.read_hudi("some-table-uri") df = df.where(df["foo"] > 5) df.show() ``` Currently there are limitations of reading Hudi tables: - Only support snapshot read of Copy-on-Write tables - Only support reading table version 5 & 6 (tables created using release 0.12.x - 0.15.x) - Table must not have `hoodie.datasource.write.drop.partition.columns=true` ## Type System Daft and Hudi have compatible type systems. Here are how types are converted across the two systems. When reading from a Hudi table into Daft: | Apachi Hudi | Daft | | --------------------- | ----------------------------- | | **Primitive Types** | | `boolean` | [`daft.DataType.bool()`][daft.datatype.DataType.bool] | | `byte` | [`daft.DataType.int8()`][daft.datatype.DataType.int8] | | `short` | [`daft.DataType.int16()`][daft.datatype.DataType.int16]| | `int` | [`daft.DataType.int32()`][daft.datatype.DataType.int32] | | `long` | [`daft.DataType.int64()`][daft.datatype.DataType.int64] | | `float` | [`daft.DataType.float32()`][daft.datatype.DataType.float32] | | `double` | [`daft.DataType.float64()`][daft.datatype.DataType.float64] | | `decimal(precision, scale)` | [`daft.DataType.decimal128(precision, scale)`][daft.datatype.DataType.decimal128] | | `date` | [`daft.DataType.date()`][daft.datatype.DataType.date] | | `timestamp` | [`daft.DataType.timestamp(timeunit="us", timezone=None)`][daft.datatype.DataType.timestamp] | | `timestampz` | [`daft.DataType.timestamp(timeunit="us", timezone="UTC")`][daft.datatype.DataType.timestamp] | | `string` | [`daft.DataType.string()`][daft.datatype.DataType.string] | | `binary` | [`daft.DataType.binary()`][daft.datatype.DataType.binary] | | **Nested Types** | | `struct(fields)` | [`daft.DataType.struct(fields)`][daft.datatype.DataType.struct] | | `list(child_type)` | [`daft.DataType.list(child_type)`][daft.datatype.DataType.list] | | `map(K, V)` | [`daft.DataType.struct({"key": K, "value": V})`][daft.datatype.DataType.struct] | --- ### Connectors Huggingface (`docs/connectors/huggingface.md`) # Hugging Face Datasets Daft has native support for reading from and writing to [Hugging Face datasets](https://huggingface.co/datasets). To install all dependencies required for Daft's Hugging Face integrations, use the `huggingface` feature: ``` pip install 'daft[huggingface]' ``` ## Reading From a Dataset Daft is able to read datasets directly from Hugging Face using the [`daft.read_huggingface()`][daft.read_huggingface] function or via the `hf://datasets/` protocol. ### Reading an Entire Dataset Using [`daft.read_huggingface()`][daft.read_huggingface], you can easily read a Hugging Face dataset. === "🐍 Python" ```python import daft df = daft.read_huggingface("username/dataset_name") ``` This will read the entire dataset into a DataFrame. For repositories stored as uncompressed [WebDataset](https://huggingface.co/docs/hub/en/datasets-webdataset) TAR shards, select the WebDataset reader explicitly: === "🐍 Python" ```python import daft df = daft.read_huggingface( "laion/conceptual-captions-12m-webdataset", format="webdataset", ) ``` Media members remain lazy, range-backed file references, so selecting metadata columns does not download image, audio, or video payloads. !!! warning "Warning" `daft.read_huggingface()` is currently limited to WebDataset repositories or datasets that Hugging Face [automatically convert](https://huggingface.co/docs/dataset-viewer/en/parquet) to Parquet. When `format=None`, Daft currently uses the Parquet reader. For other datasets, you will need to manually specify the path or glob pattern to the files you want to read, similar to how you would read from a local file system. See the next section for an example. ### Reading Specific Files Not only can you read entire datasets, but you can also read individual files from a dataset. Using a read function that takes in a path (such as [`daft.read_parquet()`][daft.read_parquet], [`daft.read_csv()`][daft.read_csv], or [`daft.read_json()`][daft.read_json]), specify a Hugging Face dataset path via the `hf://datasets/` prefix: === "🐍 Python" ```python import daft # read a specific Parquet file df = daft.read_parquet("hf://datasets/username/dataset_name/file_name.parquet") # or a csv file df = daft.read_csv("hf://datasets/username/dataset_name/file_name.csv") # or a set of Parquet files using a glob pattern df = daft.read_parquet("hf://datasets/username/dataset_name/**/*.parquet") ``` ## Writing to a Dataset Daft is able to write Parquet files to Hugging Face datasets using [`DataFrame.write_huggingface`][daft.DataFrame.write_huggingface]. Daft supports [Content-Defined Chunking](https://huggingface.co/blog/parquet-cdc) and [Xet](https://huggingface.co/blog/xet-on-the-hub) for faster, deduplicated writes. Basic usage: === "🐍 Python" ```python import daft df: daft.DataFrame = ... df.write_huggingface("username/dataset_name") ``` See the [`DataFrame.write_huggingface`][daft.DataFrame.write_huggingface] API page for more info. ### Configuring Writes [`DataFrame.write_huggingface`][daft.DataFrame.write_huggingface] accepts an [`IOConfig`][daft.io.IOConfig] which can be used to configure the write behavior. Here's an example of how to use it: === "🐍 Python" ```python import daft from daft.io import IOConfig, HuggingFaceConfig df: daft.DataFrame = ... io_config = IOConfig(hf=HuggingFaceConfig( use_content_defined_chunking=..., row_group_size=..., target_filesize=..., max_operations_per_commit=..., )) df.write_huggingface("username/dataset_name", io_config=io_config) ``` See the [`HuggingFaceConfig`][daft.io.HuggingFaceConfig] API page for more information about each argument. ### Xet-Accelerated Reads When [`HuggingFaceConfig.use_xet`][daft.io.HuggingFaceConfig] is enabled (the default), Daft probes whether a file is stored on Hugging Face's [Xet](https://huggingface.co/docs/hub/xet/index) backend and downloads via the Xet protocol when possible. Non-Xet files (small git blobs, etc.) continue to use the standard HTTP resolve path. To disable Xet reads and always use HTTP: === "🐍 Python" ```python from daft.io import IOConfig, HuggingFaceConfig io_config = IOConfig(hf=HuggingFaceConfig(use_xet=False)) df = daft.read_parquet("hf://datasets/username/dataset_name", io_config=io_config) ``` You can also set the `HF_HUB_DISABLE_XET=1` environment variable to force HTTP-only reads. ## Authentication The `token` parameter in [`HuggingFaceConfig`][daft.io.HuggingFaceConfig] can be used to specify a Hugging Face access token for requests that require authentication (e.g. reading private datasets or writing to a dataset). Example of reading a dataset with a specified token: === "🐍 Python" ```python from daft.io import IOConfig, HuggingFaceConfig io_config = IOConfig(hf=HuggingFaceConfig(token="your_token")) df = daft.read_parquet("hf://datasets/username/dataset_name", io_config=io_config) ``` --- ### Connectors Iceberg (`docs/connectors/iceberg.md`) # Reading from and Writing to Apache Iceberg [Apache Iceberg](https://iceberg.apache.org/) is an open-source table format originally developed at Netflix for large-scale analytical tables and datasets. It provides a way to manage and organize data files (like Parquet and ORC) as tables, offering benefits like ACID transactions, time travel, and schema evolution. Daft currently natively supports: 1. **Distributed Reads:** Daft will fully distribute the I/O of reads over your compute resources (whether Ray or on local multithreading) 2. **Skipping Filtered Data:** Daft uses [`df.where()`][daft.DataFrame.where] filter calls to only read data that matches your predicates 3. **All Catalogs From PyIceberg:** Daft is natively integrated with PyIceberg, and supports all the catalogs that PyIceberg does A detailed Iceberg roadmap for Daft can be found on [our GitHub issues](https://github.com/Eventual-Inc/Daft/issues/2458). For the overall Daft development plan, see [Daft Roadmap](../roadmap.md). ## Tutorial ### Reading a Table To read from the Apache Iceberg table format, use the [`daft.read_iceberg`][daft.read_iceberg] function. We integrate closely with [PyIceberg](https://py.iceberg.apache.org/) (the official Python implementation for Apache Iceberg) and allow the reading of DataFrames easily from PyIceberg's Table objects. The following is an example snippet of loading an example table, but for more information please consult the [PyIceberg Table loading documentation](https://py.iceberg.apache.org/api/#load-a-table). === "🐍 Python" ```python # Access a PyIceberg table as per normal from pyiceberg.catalog import load_catalog catalog = load_catalog("my_iceberg_catalog") table = catalog.load_table("my_namespace.my_table") ``` After a table is loaded as the `table` object, reading it into a DataFrame is extremely easy. === "🐍 Python" ```python # Create a DataFrame import daft df = daft.read_iceberg(table) ``` Daft can also read a named Iceberg branch or tag. Branch, tag, and snapshot ID reads are mutually exclusive. === "🐍 Python" ```python # Create branch/tag references with PyIceberg table.refresh() snapshot = table.current_snapshot() with table.manage_snapshots() as snapshot_manager: snapshot_manager.create_branch(snapshot.snapshot_id, "audit") snapshot_manager.create_tag(snapshot.snapshot_id, "v1") # Read the branch or tag with Daft branch_df = daft.read_iceberg(table, branch="audit") tag_df = daft.read_iceberg(table, tag="v1") ``` Any subsequent filter operations on the Daft `df` DataFrame object will be correctly optimized to take advantage of Iceberg features such as hidden partitioning and file-level statistics for efficient reads. === "🐍 Python" ```python # Filter which takes advantage of partition pruning capabilities of Iceberg df = df.where(df["partition_key"] < 1000) df.show() ``` ### Writing to a Table To write to an Apache Iceberg table, use the [`df.write_iceberg()`][daft.DataFrame.write_iceberg] method. The following is an example of appending data to an Iceberg table: === "🐍 Python" ```python written_df = df.write_iceberg(table, mode="append") written_df.show() ``` This call will then return a DataFrame containing the operations that were performed on the Iceberg table, like so: ``` {title="Output"} ╭───────────┬───────┬───────────┬────────────────────────────────╮ │ operation ┆ rows ┆ file_size ┆ file_name │ │ --- ┆ --- ┆ --- ┆ --- │ │ Utf8 ┆ Int64 ┆ Int64 ┆ Utf8 │ ╞═══════════╪═══════╪═══════════╪════════════════════════════════╡ │ ADD ┆ 5 ┆ 707 ┆ 2f1a2bb1-3e64-49da-accd-1074e… │ ╰───────────┴───────┴───────────┴────────────────────────────────╯ ``` ### Overwriting a Table `mode="overwrite"` replaces the entire table: the existing rows are deleted, then the DataFrame's rows are appended. === "🐍 Python" ```python written_df = df.write_iceberg(table, mode="overwrite") ``` ### Overwriting a Partition To replace only part of a table, known as a *static partition overwrite*, pass an `overwrite_filter` along with `mode="overwrite"`. Only the rows matching the predicate are deleted, and the delete and the append land in a single Iceberg transaction. === "🐍 Python" ```python # Replace only the `dt = '2024-01-01'` partition and leave the others in place. written_df = df.write_iceberg(table, mode="overwrite", overwrite_filter="dt = '2024-01-01'") ``` The filter accepts an Iceberg predicate string or a Daft expression, so these are equivalent: === "🐍 Python" ```python from daft import col df.write_iceberg(table, mode="overwrite", overwrite_filter="dt = '2024-01-01'") df.write_iceberg(table, mode="overwrite", overwrite_filter=col("dt") == "2024-01-01") ``` Daft checks that the rows being written fall inside the filter, so an overwrite cannot leave behind rows its delete did not cover: ``` {title="Output"} ValueError: Cannot write rows that do not match overwrite_filter EqualTo(term=Reference(name='dt'), literal=literal('2024-01-01')): .../dt=2024-01-02/....parquet holds rows outside the filter, or its statistics are not sufficient to prove otherwise. Filter the DataFrame down to the rows the filter covers, or pass validate_overwrite_filter=False to write anyway. ``` The check reads the written files' partition values and column statistics rather than the rows themselves. Statistics only ever widen, since truncated bounds cover more than the real range, so the check may refuse a write it cannot prove but never admits a row the filter does not cover. It runs before anything commits, so a rejected write leaves the table unchanged. Two cases are not provable: an equality predicate over a **non-partition** column whose values exceed the bounds truncation length (16 bytes by default), and a predicate over a column whose table properties disable statistics. Partition values are stored whole and a range predicate only needs the bounds to sit inside it, so neither of those is affected. Pass `validate_overwrite_filter=False` for the cases that cannot be proven. To drop a partition without writing anything, pass an empty DataFrame with the filter: === "🐍 Python" ```python empty = df.limit(0) empty.write_iceberg(table, mode="overwrite", overwrite_filter="dt = '2024-01-01'") ``` !!! warning "Warning" A filter aligned with partition boundaries is a metadata-only delete. A finer predicate makes PyIceberg fall back to copy-on-write and rewrite the partially matched data files. The same option is available through the [Catalog](../api/catalogs_tables.md) API: === "🐍 Python" ```python catalog.write_table("default.my_table", df, mode="overwrite", overwrite_filter="dt = '2024-01-01'") ``` ## Checkpointing Daft supports idempotent writes to Iceberg via the `checkpoint=` parameter on [`df.write_iceberg()`][daft.DataFrame.write_iceberg]. Retries of the same logical commit — after a crash, a transient catalog error, or a deliberate re-invocation — produce the same Iceberg state without duplicate snapshots. See the [Checkpointing user guide](../use-case/checkpointing.md) for concepts; the sections below cover Iceberg-specific behavior. ### Example The pattern: one `CheckpointStore` paired into both the source (via `CheckpointConfig`) and the sink (via `IdempotentCommit`). The source records which inputs were processed; the sink stamps the resulting Iceberg snapshot with the idempotence key. === "🐍 Python" ```python import daft # One store, paired into both the source and the sink. store = daft.CheckpointStore("s3://my-bucket/ckpt/") df = daft.read_parquet( "s3://input/", checkpoint=daft.CheckpointConfig(store, on="file_id"), ) # Any map-only operations work here: filter, project, UDF, explode, ... df = df.where(df["status"] == "active") written = df.write_iceberg( table, checkpoint=daft.IdempotentCommit(store, idempotence_key="job-2026-05-21-001"), ) ``` A fresh run produces one new Iceberg snapshot tagged with `daft.idempotence-key=job-2026-05-21-001`. Retries with the same `idempotence_key` recognize the prior commit and exit cleanly — no duplicate snapshot, no reprocessing of inputs already handled. ### Inspecting the Marker Every idempotent commit tags its Iceberg snapshot summary with `daft.idempotence-key`. To verify which logical commit produced the current state of a table: === "🐍 Python" ```python # Inspect via pyiceberg's Table API — `table` is the same pyiceberg.table.Table # loaded in the "Reading a Table" section above. summary = table.current_snapshot().summary print(summary.get("daft.idempotence-key")) # → "job-2026-05-21-001" ``` For older commits, walk `table.metadata.snapshots` (order isn't guaranteed — sort by `timestamp_ms` if you need chronological order) and check each snapshot's `summary`. ### Constraints These are constraints specific to the Iceberg connector. Daft-wide constraints (Ray runner, map-only pipelines, single-writer concurrency, etc.) are in the [user guide's Limitations section](../use-case/checkpointing.md#limitations). - **`mode='append'` only.** `mode='overwrite'` with `checkpoint=` raises `NotImplementedError`. - **Reserved `daft.idempotence-*` property prefix.** Keys in `snapshot_properties` that start with this prefix raise `ValueError` — Daft uses this namespace internally. ### Idempotence-Key Contract The user picks the `idempotence_key`. Daft uses it as the marker that identifies a logical commit. The key must be stable across retries and unique across distinct logical commits — both halves matter: - **Same key, different inputs → silent no-op (data loss).** Daft sees the existing marker and skips the commit. The new data is dropped without an error. - **Different key on a retry of the same logical commit → duplicate snapshot.** Daft doesn't recognize the prior attempt and lands a second snapshot. ### Recovery The [user guide's Recovery section](../use-case/checkpointing.md#recovery) walks through the full chronology. The Iceberg-specific steps: - **Commit-from-store.** When a run crashes after the pipeline finishes but before the snapshot commits, Daft reads the staged file references from the store on rerun and commits them as a single new Iceberg snapshot tagged with `daft.idempotence-key`. - **Marker recognition.** When a run crashes after the snapshot commits but before the store is marked done — or when the user deliberately re-runs with the same key — Daft walks `table.metadata.snapshots`, finds the marker, marks the store, and exits. No second snapshot. Returned DataFrame is empty. **Orphan files on crash.** A worker that crashed mid-task may leave parquet files unreferenced by any snapshot. Daft doesn't auto-clean these — use your Iceberg engine's orphan-file cleanup procedures. ### Iceberg-Specific Notes - **In-memory history walk.** Marker recognition scans `table.metadata.snapshots` directly — fast even on tables with many snapshots, since the metadata is already in memory after refresh. - **`commit.manifest-merge.enabled` is honored.** If the table is configured for manifest merging, Daft uses `merge_append`; otherwise `fast_append`. Same behavior as a non-checkpoint write. - **Partitioned tables include a `partitioning` struct column in the result.** Whether `checkpoint=` is set or not — callers don't see schema drift when they toggle the flag. - **Transient retry on `CommitFailedException`.** Daft retries up to twice on this exception (concurrent-writer conflicts, lock contention). Other exceptions — REST 5xx, network errors, auth — propagate immediately. Wrap the call in your own retry policy if you need broader coverage. ## Type System | Iceberg | Daft | |-------------------------------------|----------------------------------------------------------------------------------------------| | `BOOLEAN` | [`daft.DataType.bool()`][daft.datatype.DataType.bool] | | `INT` | [`daft.DataType.int32()`][daft.datatype.DataType.int32] | | `LONG` | [`daft.DataType.int64()`][daft.datatype.DataType.int64] | | `FLOAT` | [`daft.DataType.float32()`][daft.datatype.DataType.float32] | | `DOUBLE` | [`daft.DataType.float64()`][daft.datatype.DataType.float64] | | `DECIMAL(precision, scale)` | [`daft.DataType.decimal128(precision, scale)`][daft.datatype.DataType.decimal128] | | `DATE` | [`daft.DataType.date()`][daft.datatype.DataType.date] | | `TIME` | [`daft.DataType.int64()`][daft.datatype.DataType.int64] | | `TIMESTAMP` | [`daft.DataType.timestamp(timeunit="us", timezone=None)`][daft.datatype.DataType.timestamp] | | `TIMESTAMPZ` | [`daft.DataType.timestamp(timeunit="us", timezone="UTC")`][daft.datatype.DataType.timestamp] | | `STRING` | [`daft.DataType.string()`][daft.datatype.DataType.string] | | `UUID` | [`daft.DataType.binary()`][daft.datatype.DataType.binary] | | `FIXED(size)` | [`daft.DataType.fixed_size_binary(size)`][daft.datatype.DataType.fixed_size_binary] | | `BINARY` | [`daft.DataType.binary()`][daft.datatype.DataType.binary] | | `STRUCT<[field_name: field_type,]>` | [`daft.DataType.struct(fields)`][daft.datatype.DataType.struct] | | `LIST` | [`daft.DataType.list(element_type)`][daft.datatype.DataType.list] | | `MAP` | [`daft.DataType.map(key_type, value_type)`][daft.datatype.DataType.map] | See also [Iceberg Schemas and Data Types](https://iceberg.apache.org/spec/#schemas-and-data-types). ## Reference Daft has high-level [Session](../api/sessions.md) and [Catalog](../api/catalogs_tables.md) APIs to read and write Iceberg tables; however it is the [`daft.read_iceberg`][daft.read_iceberg] and [`df.write_iceberg`][daft.DataFrame.write_iceberg] API which is ultimately the entry-point to Iceberg reads and writes respectively. This section gives a short reference on those APIs and how they relate to both DataFrames and Iceberg. Daft's DataFrames are an abstraction over relational algebra operators like filter, project, and join. DataFrames start with a *data source* and are built upwards, via composition with additional operators, to form a tree with sources as the leaves. We typically call these leaves *tables* or *sources* and their algebraic operator is called a *scan*. ### [`read_iceberg`][daft.read_iceberg] Daft's [`daft.read_iceberg`][daft.read_iceberg] method creates a DataFrame from the given PyIceberg table. It produces rows by traversing the table's metadata tree to locate all the data files for the given snapshot which is handled by our `IcebergDataSource`. Daft's `IcebergDataSource` initializes itself by fetching the latest schema, or the schema of the given snapshot, along with setting up the partition key metadata. The data source's primary method, `get_tasks`, accepts pushdowns (projections, predicates, partition filters) and yields an iterator of tasks. Each task holds a data file, optional delete files, and the associated pushdowns. Finally, we read each data file's parquet to produce a stream of record batches which later operators consume and transform. ### [`write_iceberg`][daft.DataFrame.write_iceberg] Daft's [`write_iceberg`][daft.DataFrame.write_iceberg] method writes the DataFrame's contents to the given PyIceberg table. It works by creating a special *sink* operator which consumes all inputs and writes data files to the table's location. Daft's sink operator will apply the Iceberg partition transform and distribute records to an appropriate data file writer. Each writer is responsible for actually writing the parquet to storage and keeping track of metadata like total bytes written. Once the sink has exhausted its input, it will close all open writers. Finally, we update the Iceberg table's metadata to include these new data files, and use a transaction to update the latest metadata pointer. ### Iceberg Architecture !!! note "Note" Please see the [Iceberg Table Specification](https://iceberg.apache.org/spec/) for full details. Iceberg tables are a tree where metadata files are the inner nodes, and data files are the leaves. The data files are typically parquet, but are not *necessarily* so. To write data, the new data files are inserted into the tree by creating the necessary inner nodes (metadata files) and re-rooting the parent. To read data, we choose a root (table version) and use the metadata files to collect all relevant data files (their children). The *data layer* is composed of data files and delete files; where as the *metadata layer* is composed of manifest files, manifest lists, and metadata files. #### Manifest Files Manifest files (avro) keep track of data files, delete files, and statistics. These lists track the leaves of the iceberg tree. While all manifest files use the same schema, a manifest file contains either exclusively data files or exclusively delete files. #### Manifest List Manifests lists (snapshots) contain all manifest file locations along with their partitions, and partition columns upper and lower bounds. Each entry in the manifest list has the form, | Field | Description | |-------------------|------------------------------------------------------| | `manifest_path` | location | | `manifest_length` | file length in bytes | | `content` | flag where `0=data` and `1=deletes` | | `partitions` | array of field summaries e.g. nullable, upper, lower | These are not all of the fields, but gives us an idea of what a manifest list looks like. #### Metadata Files Metadata files store the table schema, partition information, and a list of all snapshots including which one is the current. Each time an Iceberg table is modified, a new metadata file is created; this is what is meant earlier by "re-rooting" the tree. The *catalog* is responsible for atomically updating the current metadata file. #### Puffin Files Puffin files store arbitrary metadata as blobs along with the necessary metadata to use these blobs. ### Table Writes Iceberg can efficiently insert (append) and update rows. To insert data, the new data files (parquet) are written to object storage along with a new manifest file. Then a new manifest list, along with all existing manifest files, is added to the new metadata file. The catalog then marks this new metadata file as the current one. An upsert / merge into query is a bit more complicated because its action is conditional. If a record already exists, then it is updated. Otherwise, a new record is inserted. This write operation is accomplished by reading all matched records into memory, then updating each match by either copying with updates applied (COW) or deleting (MOR) then including the updated in the insert. All records which were not matched (did not exist) are inserted like a normal insert. ### Table Reads Iceberg reads begin by fetching the latest metadata file to then locate the "current snapshot id". The current snapshot is a manifest list which has the relevant manifest files which ultimately gives us a list of all data files pertaining to the query. For each metadata layer, we can leverage the statistics to prune both manifest files and data files. ### Other #### COW vs. MOR When data is modified or deleted, we can either rewrite the relevant portion (copy-on-write) or save the modifications for later reconciliation (merge-on-read). Copy-on-write optimizes for read performance; however, modifications are more expensive. Merge-on-read optimizes for write performance; however, reads are more expensive. #### Delete File Delete files are used for the merge-on-read strategy in which tables updates are written to a "delete file" which is applied or "merged" with the data files *while* reading. A delete file can contain either *positional* or *equality* deletes. Positional deletes denote which rows (filepath+row) have been deleted, whereas equality deletes have an equality condition i.e. `WHERE x = 100` to filter rows. #### Partitioning When a table is partitioned on some field, Iceberg will write separate data files for each record, grouped by the partitioned field's value. That is, each data file will contain records for a single partition. This enables efficient scanning of a partition because all other data files can be ignored. You may partition by a column's value (identity) or use a *partition transform* to derive a partition value. ##### [Apache Iceberg Partition Transforms](https://iceberg.apache.org/spec/#partition-transforms) | Transform | Description | |---------------|-----------------------------| | `identity` | Column value unmodified. | | `bucket(n)` | Hash of value, mod `n`. | | `truncate(w)` | Truncated value, width `w`. | | `year` | Timestamp year value. | | `month` | Timestamp month value. | | `day` | Timestamp day value. | | `hour` | Timestamp hour value. | ## FAQs 1. **How does Daft read Iceberg tables?** *Daft reads Iceberg tables by reading a snapshot's data files into its arrow-based record batches. For more detail, please see the [`read_iceberg`](#read_iceberg) reference.* 2. **How does Daft write Iceberg tables?** *Daft writes Iceberg tables by writing the new, possibly partitioned, data files to storage then atomically committing an updated Iceberg metadata file. For more detail, please see [`write_iceberg`](#write_iceberg)*. 3. **How do Daft's data types compare to Iceberg's data types?** *The type systems are quite similar because they are both based around Apache Arrow's types. Please see our comprehensive type comparison table.* 4. **Does Daft support Iceberg REST catalog implementations?** *Yes! Daft uses PyIceberg to interface with Iceberg catalogs which has extensive Iceberg REST catalog support.* 5. **How does Daft handle positional deletes vs equality deletes?** *Daft currently only supports positional deletes, but V2 equality deletes are on the roadmap.* 6. **Can Daft leverage Iceberg's metadata for predicate pushdown?** *Yes, uses min/max statistics from manifest files for partition pruning.* 7. **Does Daft support time travel queries?** *Daft supports reading by snapshot id, and snapshot slices are on the roadmap*. 8. **Which complex data types does Daft support in Iceberg tables?** *Daft supports arrays, structs, and maps.* 9. **How does Daft handle schema evolution?** *Daft currently does not expose any data definition operators beyond [create_table][daft.session.Session.create_table].* 10. **Does Daft support reading table metadata like snapshot information?** *Daft does not have native APIs for this, and we recommend using PyIceberg for reading metadata.* 11. **How does Daft handle Iceberg's hidden partitioning?** *Transparently leverages partition information without user specification.* 12. **Can Daft read and write partition transforms like truncate, bucket, or hour?** *Daft can read and write all Iceberg partition transform types: identity, bucket, date transforms (year/month/day), and truncate.* 13. **How does Daft optimize queries against partitioned data?** *Daft will shuffle partitioned data when possible and based upon your execution environment.* 14. **Which writes operations does Daft support?** *Daft supports append, full overwrite, and static partition overwrite (`overwrite_filter`); it does not support upserts like copy-on-write updates.* --- ### Connectors Kafka (`docs/connectors/kafka.md`) # Reading from Kafka !!! warning "Experimental" This connector is experimental and the API may change in future releases. Current limitations: - **Bounded batch reads only** — no streaming or unbounded mode - **No offset commit management** — consumer group offsets are not committed - **No write support** — Daft cannot produce messages to Kafka [Apache Kafka](https://kafka.apache.org/) is a distributed event streaming platform. Daft can read bounded ranges of messages from Kafka topics into DataFrames using [`daft.read_kafka()`][daft.io.read_kafka]. Daft currently supports: 1. **Bounded batch reads**: Read a fixed range of messages defined by offsets, timestamps, or earliest/latest markers 2. **Multi-topic reads**: Read from one or more topics in a single call 3. **Partition filtering**: Read from specific partitions within a topic 4. **Flexible bounds**: Specify start/end using `"earliest"`/`"latest"`, timestamps (datetime, ISO-8601, or epoch ms), or explicit per-partition offset maps 5. **Limit pushdown**: Use `.limit(N)` to stop reading early once enough rows are collected ## Installing Daft with Kafka Support Daft integrates with Kafka through the [confluent-kafka](https://github.com/confluentinc/confluent-kafka-python) package: ```bash pip install -U "daft[kafka]" ``` Or install the dependency manually: ```bash pip install confluent-kafka ``` ## Output Schema Every `read_kafka` call returns a DataFrame with the following fixed schema: | Column | Type | Description | |----------------|----------|-------------------------------------------------------| | `topic` | `string` | The topic the message was read from | | `partition` | `int32` | The partition ID within the topic | | `offset` | `int64` | The offset of the message within the partition | | `timestamp_ms` | `int64` | Message timestamp in milliseconds (null if unavailable) | | `key` | `binary` | The message key as raw bytes (null if not present) | | `value` | `binary` | The message value as raw bytes | ## Reading Messages ### Basic Usage Read all messages from a topic (earliest to latest): === "🐍 Python" ```python import daft df = daft.read_kafka(bootstrap_servers="localhost:9092", topics="my-topic") df.show() ``` ### Timestamp Bounds Read messages within a time range: === "🐍 Python" ```python import daft from datetime import datetime, timezone start = datetime(2024, 1, 1, tzinfo=timezone.utc) end = datetime(2024, 1, 2, tzinfo=timezone.utc) df = daft.read_kafka( bootstrap_servers="localhost:9092", topics="my-topic", start=start, end=end, ) ``` You can also use ISO-8601 strings or epoch milliseconds: === "🐍 Python" ```python df = daft.read_kafka( bootstrap_servers="localhost:9092", topics="my-topic", start="2024-01-01T00:00:00Z", end=1704153600000, # epoch ms ) ``` ### Offset Bounds Read a specific offset range from a partition: === "🐍 Python" ```python df = daft.read_kafka( bootstrap_servers="localhost:9092", topics="my-topic", start={0: 100}, # partition 0, offset 100 end={0: 200}, # partition 0, offset 200 (exclusive) ) ``` ### Partition Filtering Read only from specific partitions: === "🐍 Python" ```python df = daft.read_kafka( bootstrap_servers="localhost:9092", topics="my-topic", partitions=[0, 2], ) ``` ### Multiple Topics Read from multiple topics at once: === "🐍 Python" ```python df = daft.read_kafka( bootstrap_servers="localhost:9092", topics=["topic-a", "topic-b"], ) ``` With per-topic offset maps: === "🐍 Python" ```python df = daft.read_kafka( bootstrap_servers="localhost:9092", topics=["topic-a", "topic-b"], start={"topic-a": {0: 100}, "topic-b": {0: 50}}, end="latest", ) ``` ### Using Limit Stop reading early once enough rows are collected: === "🐍 Python" ```python df = daft.read_kafka(bootstrap_servers="localhost:9092", topics="my-topic").limit(1000) df.show() ``` ## Kafka Client Configuration Pass additional [librdkafka configuration](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) options via `kafka_client_config`: === "🐍 Python" ```python df = daft.read_kafka( bootstrap_servers="localhost:9092", topics="my-topic", kafka_client_config={ "security.protocol": "SASL_SSL", "sasl.mechanism": "PLAIN", "sasl.username": "my-api-key", "sasl.password": "my-api-secret", }, ) ``` !!! note The following keys are managed internally and cannot be overridden via `kafka_client_config`: `bootstrap.servers`, `group.id`, `enable.auto.commit`, `enable.auto.offset.store`. These are configured through the dedicated `bootstrap_servers` and `group_id` parameters of `read_kafka()` instead. --- ### Connectors Lance (`docs/connectors/lance.md`) # Reading from and Writing to Lance [Lance](https://lance.org/) is a next-generation columnar storage format for multimodal datasets (images, video, audio, and general columnar data). It supports local POSIX filesystems and cloud object stores (e.g., S3/GCS). Lance is known for extremely fast random access, zero-copy reads, deep integration with PyArrow/DuckDB, and strong performance for vector retrieval workloads. Daft currently supports: 1. **Parallel and distributed reads**: Daft parallelizes reads on the default multithreading runner or on the [distributed Ray runner](../distributed/index.md) 2. **Skipping filtered data (Data Skipping)**: Daft uses [`df.where()`][daft.DataFrame.where] predicates and file/fragment statistics to skip non-matching data 3. **Multi-cloud and local access**: Read from S3, GCS, Azure Blob Storage, and local filesystems with unified IO configuration 4. **Version/time-slice reads**: Use `version` and `asof` parameters to read a specific version or the latest version as of a given timestamp 5. **Scan optimization**: Configure `fragment_group_size` to group fragments and improve scan efficiency 6. **Cache tuning**: Configure `index_cache_size` and `metadata_cache_size_bytes` to optimize index page caching and metadata retrieval for large datasets ## Installing Daft with Lance Support Daft integrates Lance through an optional dependency: ```bash pip install -U "daft[lance]" ``` ## Reading a Table Use [`daft.read_lance`][daft.read_lance] to read a Lance dataset from a local path or a cloud object store URI. === "🐍 Python" ```python # Read a Lance dataset from local or object storage import daft # Local path example df_local = daft.read_lance("/data/my_lance_dataset") # Read a specific version or a time slice df_v1 = daft.read_lance("/data/my_lance_dataset", version=1) ``` To access public S3/GCS buckets, configure IO options for authentication and endpoints: === "🐍 Python" ```python import daft from daft.io import IOConfig, S3Config, GCSConfig # Public S3 example (anonymous access) s3_config = S3Config(region_name="us-west-2", anonymous=True) gcs_config = GCSConfig(credentials="/path/to/gcp-service-account.json") io_config = IOConfig(s3=s3_config, gcs=gcs_config) df_s3 = daft.read_lance("s3://daft-oss-public-data/lance/words-test-dataset", io_config=io_config) # GCS example (service account credentials) df_gcs = daft.read_lance("gs://my-bucket/lance/my-dataset", io_config=io_config) ``` For S3-compatible services (e.g. Volcengine TOS), configure IO options for authentication and endpoints: === "🐍 Python" ```python import daft from daft.io import IOConfig, S3Config region = "cn-beijing" # Region of your TOS bucket bucket_name = "my-tos-bucket" # Name of your TOS bucket io_config = IOConfig( s3=S3Config( endpoint_url=f"https://tos-s3-{region}.ivolces.com", region_name=region, force_virtual_addressing=True, verify_ssl=True, key_id="your-access-key-id", access_key="your-secret-access-key", ) ) df = daft.read_lance(f"s3://{bucket_name}/lance-path", io_config=io_config) ``` For datasets with many fragments, group fragments to reduce scheduling and metadata overhead: === "🐍 Python" ```python # Process multiple fragments in a single scan task df_grouped = daft.read_lance("/data/my_lance_dataset", fragment_group_size=8) ``` Filter operations on the Daft `df` DataFrame object will be pushed down to the Lance data source for efficient data skipping. === "🐍 Python" ```python # Enable strict filter pushdown for more efficient data skipping daft.context.set_planning_config(enable_strict_filter_pushdown=True) filtered = df_local.where(df_local["score"] >= 0.8) filtered.show() ``` ### Filtering with Custom SQL Expressions Daft supports passing raw SQL filter strings directly to the Lance scanner via `default_scan_options`. This allows you to leverage Lance's native SQL capabilities, including GeoSpatial functions, which might not be directly expressible in Daft's DataFrame API yet. You can also use this mechanism to perform SQL-based projections (calculations) during the scan. === "🐍 Python" ```python import daft # Example: Using GeoSpatial functions for filtering and projection # Assume we have a dataset with 'point' and 'linestring' columns (GeoArrow types) df = daft.read_lance( "/path/to/geo/dataset", default_scan_options={ # Calculate distance during scan and project it as a new column "columns": {"distance": "st_distance(point, linestring)"}, # Filter rows where linestring intersects with a specific geometry "filter": "st_intersects(linestring, st_geomfromtext('LINESTRING ( 2 0, 0 2 )'))", "with_row_id": True } ) df.show() ``` ## Writing to Lance Use [`df.write_lance()`][daft.dataframe.DataFrame.write_lance] to write a DataFrame to a Lance dataset. Supported modes include `create`, `append`, `overwrite`, and `insert_overwrite`. === "🐍 Python" ```python import daft df = daft.from_pydict({"a": [1, 2, 3, 4]}) meta = df.write_lance("/tmp/lance/my_table.lance") meta.show() # Contains metadata such as num_fragments / num_deleted_rows / num_small_files / version # Overwrite existing table with extra parameters (forwarded to the Lance writer) meta2 = df.write_lance("/tmp/lance/my_table.lance", mode="overwrite", max_bytes_per_file=1024) meta2.show() # Append rows (must be compatible with the existing table schema) meta3 = df.write_lance("/tmp/lance/my_table.lance", mode="append") ``` ### Conditional overwrite `insert_overwrite` removes rows matching a Lance SQL predicate and adds the input rows in one commit. The target table must already exist; it is not compatible with `use_mem_wal=True`. ```python replacement = daft.from_pydict({"day": ["2026-09-15"], "value": [42]}) replacement.write_lance( "/tmp/lance/events.lance", mode="insert_overwrite", overwrite_where="day = '2026-09-15'", ) ``` ### Lance Namespace tables Address a table through a Lance Namespace by supplying `table_id`, `namespace_impl`, and `namespace_properties` instead of a URI. Namespace resolution and any catalog-vended storage credentials are handled by `daft-lance`. ```python namespace = { "namespace_impl": "dir", "namespace_properties": {"root": "/tmp/lance_tables"}, "table_id": ["orders"], } daft.from_pydict({"id": [1, 2]}).write_lance(mode="create", **namespace) orders = daft.read_lance(**namespace) ``` URI and Namespace targets are mutually exclusive. ### Migrating from `mode="merge"` `DataFrame.write_lance(mode="merge")` is deprecated in v0.8.0 and will be removed in v0.9.0. Its behavior depended on the target table and input schema; migrate each case to an explicit operation: ```python # No target table: create it explicitly. df.write_lance("/tmp/lance/my_table.lance", mode="create") # No new columns: append rows explicitly. df.write_lance("/tmp/lance/my_table.lance", mode="append") ``` For a per-fragment column merge, call `daft_lance.merge_columns_df` directly. Read `fragment_id` and the join key from Lance first. The default join key is `_rowaddr`; pass `left_on` and `right_on` for a business key. ```python from daft_lance import merge_columns_df source = daft.read_lance( "/tmp/lance/my_table.lance", default_scan_options={"with_row_address": True}, include_fragment_id=True, ) updates = ( source.select("fragment_id", "_rowaddr", "score") .with_column("normalized_score", daft.col("score") / 100) .select("fragment_id", "_rowaddr", "normalized_score") ) merge_columns_df(updates, "/tmp/lance/my_table.lance") ``` For example, to use `id` as the join key, call `merge_columns_df(updates, "/tmp/lance/my_table.lance", left_on="id", right_on="id")` after retaining `fragment_id` and `id` in `updates`. ### For S3-compatible services (e.g. Volcengine TOS), configure IO options for authentication and endpoints: === "🐍 Python" ```python import daft from daft.io import IOConfig, S3Config df = daft.from_pydict({"user_id": [1, 2, 3], "score": [0.5, 0.8, 0.9]}) region = "cn-beijing" # Region of your TOS bucket bucket_name = "my-tos-bucket" # Name of your TOS bucket io_config = IOConfig( s3=S3Config( endpoint_url=f"https://tos-s3-{region}.ivolces.com", region_name=region, force_virtual_addressing=True, verify_ssl=True, key_id="your-access-key-id", access_key="your-secret-access-key", ) ) meta = df.write_lance(f"s3://{bucket_name}/lance/my_table.lance", io_config=io_config) ``` Note: the `{region}` should match the region of your TOS bucket. e.g. if your bucket is in `cn-beijing`, you should set `region_name="cn-beijing"` and `endpoint_url="https://tos-s3-cn-beijing.ivolces.com"`. ### Writing with a specific schema You can provide a `pyarrow.Schema` to control the on-disk Lance schema. Daft will align the DataFrame to this schema (column order, types, and nullability) before writing: === "🐍 Python" ```python import daft import pyarrow as pa df = daft.from_pydict( { "id": [1, 2, 3], "score": [0.5, 0.8, 0.9], } ) target_schema = pa.schema( [ pa.field("id", pa.int64(), nullable=False), pa.field("score", pa.float32(), nullable=True), ] ) meta = df.write_lance("/tmp/lance/typed_table.lance", schema=target_schema) ``` This ensures that the resulting Lance table uses the exact schema you specify, even if the in-memory DataFrame has compatible but different types or column ordering. !!! note "Write Schema Control" - If `schema` is not provided, Daft uses the current DataFrame schema. - If a `pyarrow.Schema` is provided, data will be aligned to that schema before writing (type/order/nullability). - If the target dataset already exists and the write is not an overwrite, data is converted to the existing table schema for compatibility. ## Advanced Usage ### Vector Search Daft can push Lance's vector search options through `default_scan_options`. This lets you express nearest-neighbor queries at scan time while keeping the rest of your pipeline in Daft. === "🐍 Python" ```python import daft import pyarrow as pa # Example: dataset with a fixed-size list or embedding column named "vector" query = pa.array([0.0, 0.0], type=pa.float32()) df = daft.read_lance( "/data/my_lance_vectors", # Forward Lance's `nearest` options into the underlying scanner default_scan_options={ "nearest": { "column": "vector", # Lance vector column name "q": query, # Query vector (any Lance-compatible QueryVectorLike) "k": 5, # Top-K neighbors to return }, }, ) # The resulting DataFrame contains the K nearest rows according to Lance, # and you can continue working with it using normal Daft APIs. df.select("vector").show() ``` ### Data Evolution If you need to add derived columns in-place to an existing Lance dataset (e.g., apply a UDF across batches and persist the result), use `daft_lance.merge_columns` from the [`daft-lance`](https://github.com/daft-engine/daft-lance) package (installed as part of the `daft[lance]` extra): === "🐍 Python" ```python from daft_lance import merge_columns # Example: double the values in column c and write to a new column new_column import pyarrow.compute as pc def double_score(batch): return batch.append_column("new_column", pc.multiply(batch["c"], 2)) merge_columns( "/tmp/lance/my_table.lance", transform=double_score, read_columns=["c"], ) ``` ### Compaction Compaction is the process of rewriting a Lance dataset to optimize its structure for query performance. This operation can: - **Merge small files**: Combine multiple small data fragments into fewer, larger fragments to reduce metadata overhead and improve read throughput. - **Materialize deletions**: Physically remove rows that have been marked for deletion, which can reduce storage and accelerate scans. - **Improve data layout**: Reorganize data to improve compression and read efficiency. Use compaction when a dataset has undergone many small appends, has a high number of deleted rows, or contains a large number of small files. Daft provides a distributed implementation of compaction through `daft_lance.compact_files` (from the [`daft-lance`](https://github.com/daft-engine/daft-lance) package, installed as part of the `daft[lance]` extra). ```python from daft_lance import compact_files compact_files( uri, compaction_options=None, io_config=None, # --- other options --- index_cache_size=None, block_size=None, default_scan_options=None, metadata_cache_size_bytes=None, partition_num=None, ) ``` - **`uri`**: Path to the Lance dataset. - **`compaction_options`**: A dictionary of options to control compaction behavior (e.g., `target_rows_per_fragment`, `materialize_deletions`), see [Lance documentation](https://lance-format.github.io/lance-python-doc/all-modules.html#lance.dataset.DatasetOptimizer.compact_files) for more details. - **Returns**: A `CompactionMetrics` object with statistics if compaction was performed, or `None` if no action was needed. The `CompactionMetrics` object contains the following fields: - `fragments_removed`: Number of fragments removed during compaction. - `fragments_added`: Number of fragments added during compaction. - `files_removed`: Number of files removed during compaction. - `files_added`: Number of files added during compaction. - **`partition_num`**: On the Ray Runner, this controls the number of parallel compaction tasks. Defaults to 1. On the native runner, this option is ignored. This example compacts a dataset with multiple fragments into a single, larger fragment, and uses `partition_num` to control the number of parallel compaction tasks. === "🐍 Python" ```python import daft import lance import pandas as pd import pyarrow as pa from daft_lance import compact_files daft.set_runner_ray() # Create a dataset and delete some rows data = pa.table({"a": range(800), "b": range(800)}) dataset_path = "/tmp/dataset.lance" dataset = lance.write_dataset(data, dataset_path, max_rows_per_file=200) # The initial row count is 800, and the initial fragment count is 4 print(f"Initial row count: {dataset.count_rows()}, and initial fragment count: {len(dataset.get_fragments())}") metrics = compact_files( dataset_path, compaction_options={ "target_rows_per_fragment": 400, }, partition_num=2, ) dataset = lance.dataset(dataset_path) # The final row count is 800, and the final fragment count is 2 print(f"Final row count: {dataset.count_rows()}, and final fragment count: {len(dataset.get_fragments())}") ``` --- ### Connectors Mcap (`docs/connectors/mcap.md`) # Reading MCAP Files [MCAP](https://mcap.dev/) is an open-source container file format for multimodal log data, commonly used in robotics and autonomous systems. Daft can read MCAP files using [`daft.read_mcap()`][daft.io.read_mcap]. ## Basic Usage === "Local File" ```python import daft df = daft.read_mcap("/path/to/recording.mcap") df.show() ``` === "Remote File (S3)" ```python import daft from daft.io import IOConfig, S3Config io_config = IOConfig(s3=S3Config(region_name="us-west-2")) df = daft.read_mcap("s3://bucket/recordings/data.mcap", io_config=io_config) df.show() ``` === "Remote File (HTTP/HTTPS)" ```python import daft df = daft.read_mcap("https://example.com/recordings/session.mcap") df.show() ``` === "Directory of Files" ```python import daft # Read all MCAP files in a directory df = daft.read_mcap("/recordings/") df.show() ``` ## Output Schema `output_format` controls the column types returned by `read_mcap`. The default, `output_format="raw"`, returns: | Column | Type | Description | |--------|------|-------------| | `source_path` | `string` | Path of the MCAP file containing the message | | `topic` | `string` | The topic name the message was published on | | `log_time` | `uint64` | Timestamp when the message was logged (nanoseconds) | | `publish_time` | `uint64` | Timestamp when the message was published (nanoseconds) | | `sequence` | `uint32` | Sequence number of the message | | `data` | `binary` | Raw message payload | `output_format="raw_legacy"` matches the legacy (before v0.9.0) implementation: | Column | Type | Description | |--------|------|-------------| | `source_path` | `string` | Path of the MCAP file containing the message | | `topic` | `string` | The topic name the message was published on | | `log_time` | `int64` | Timestamp when the message was logged (nanoseconds) | | `publish_time` | `int64` | Timestamp when the message was published (nanoseconds) | | `sequence` | `int32` | Sequence number of the message | | `data` | `string` | Message payload | !!! warning "Deprecated `output_format='raw_legacy'`" `output_format="raw_legacy"` is deprecated and will be removed in v0.9.0. Update your script to use the currently emitted column types. ## Filtering Options ### Time Range Filter messages by time range. The time unit matches the MCAP `log_time` field (typically nanoseconds): ```python df = daft.read_mcap( "/path/to/recording.mcap", start_time=1609459200000000000, # 2021-01-01 UTC in nanoseconds end_time=1609545600000000000, # 2021-01-02 UTC in nanoseconds ) df.show() ``` !!! warning "Unsigned timestamp bounds" Ensure that `start_time`, `end_time`, and times returned by `topic_start_time_resolver` must be between `0` and `2**64 - 1` to match the internal MCAP timestamp format. Negative time values raise `OverflowError`. ### Topic Filtering Read only specific topics: ```python df = daft.read_mcap( "/path/to/recording.mcap", topics=["/camera/image", "/lidar/points"], ) df.show() ``` ### Batch Size Control memory usage by adjusting the batch size: ```python df = daft.read_mcap( "/path/to/recording.mcap", batch_size=500, # Number of messages per batch (default: 1000) ) ``` ## Advanced: Topic Start Time Resolver For advanced use cases, you can provide a callable that computes per-file, per-topic start times. This is useful for resuming reads from specific positions: ```python def get_topic_start_times(file_path: str) -> dict[str, int]: """Return a mapping of topic names to start times.""" # Your logic to determine start times per topic return { "/camera/image": 1609459200000000000, "/lidar/points": 1609459200500000000, } df = daft.read_mcap( "/path/to/recording.mcap", topic_start_time_resolver=get_topic_start_times, ) ``` When a resolver is provided: - One scan task is created per (file, topic) combination - The task's start_time is set to `max(start_time, resolver(file)[topic])` ## Use Cases ### Robotics Data Processing ```python import daft from daft import col # Read sensor data df = daft.read_mcap("/recordings/*.mcap", topics=["/imu/data"]) # Filter and process recent_data = df.where(col("log_time") > 1609459200000000000) recent_data.show() ``` ### Multi-Topic Analysis ```python import daft from daft import col # Read multiple topics df = daft.read_mcap( "/recordings/session.mcap", topics=["/camera/image", "/gps/fix", "/vehicle/speed"], ) # Group by topic topic_counts = df.groupby("topic").count() topic_counts.show() ``` --- ### Connectors Orc (`docs/connectors/orc.md`) # ORC Files Daft reads ORC files from local storage and remote URLs using [`daft.read_orc`][daft.read_orc]. This can be used to process raw ORC files produced by tools such as Apache Spark and Hive. ## Read ORC Files ```python import daft df = daft.read_orc("/path/to/file.orc") df = daft.read_orc("/path/to/directory") df = daft.read_orc("/path/to/files-*.orc") df = daft.read_orc(["/path/to/first.orc", "/path/to/second.orc"]) ``` Directories are searched recursively for files matching `*.orc`. An explicit file path can have a different extension. Overlapping paths and glob patterns are deduplicated. Empty inputs, unmatched paths, unreadable files, and corrupt ORC files raise errors. Paths follow Daft's native glob syntax, including `*`, `?`, `[...]`, and `{...}`. A path containing unescaped glob metacharacters is treated as a pattern even when a file with that name exists. For example, `a[1].orc` matches `a1.orc`; use `a[[]1[]].orc` to match the literal filename `a[1].orc`. Schema inference occurs when the DataFrame is created. Rows are read when an action such as `collect`, `show`, or `write_parquet` executes. Each file is processed by one task, so multiple files can be read in parallel. Stripes within one file are not scheduled as separate distributed tasks. ## Remote Storage Use the same [`IOConfig`][daft.io.IOConfig] as other Daft file readers. If `io_config` is omitted, the reader uses the planning context's default configuration. ```python from daft.io import IOConfig, S3Config io_config = IOConfig(s3=S3Config(region_name="us-east-1", anonymous=True)) df = daft.read_orc("s3://my-public-bucket/data/*.orc", io_config=io_config) ``` ## Query and Batch Size Select columns and filter rows using the DataFrame API: ```python df = daft.read_orc("/path/to/data/*.orc", batch_size=65536) result = df.where(daft.col("score") > 0.5).select("id", "label").limit(100) ``` Column projection reduces the fields requested from the reader. Row filters and limits preserve Daft query semantics; ORC-native predicate pruning and footer-based aggregation are not supported. `limit()` restricts the returned rows but does not guarantee early termination of file reads. The shared Python source bridge can continue reading and buffering batches after the returned-row limit is reached, so even `.limit(1)` can read beyond the first batch and retain multiple batches in memory. `batch_size` controls the maximum number of rows in each emitted batch and defaults to 131072. It does not set a memory limit: field sizes, decoder buffers, and queued batches also affect memory usage. ## Schema and Compatibility The schema is inferred from the first matched file and remains fixed for that DataFrame. Later files are aligned to this schema: missing fields become nulls, extra fields are excluded, and fields are cast using Daft's conversion rules, as with Parquet reads. Unsupported type conversions raise an error; some value conversions, such as an invalid numeric string, can produce null. Files with zero rows and a valid schema are supported. The reader uses the ORC types supported by PyArrow and Daft's Arrow conversion, including numeric values, strings, binary values, dates, timestamps, decimals, lists, structs, and maps. It does not provide full schema merging or an explicit schema override. This API reads raw ORC files. It does not interpret Hive ACID layouts or table transactions, and does not add ORC support to `read_iceberg` or provide an ORC writer. --- ### Connectors Paimon (`docs/connectors/paimon.md`) # Reading from and Writing to Apache Paimon [Apache Paimon](https://paimon.apache.org/) is an open-source lakehouse storage format designed for high-throughput streaming and batch analytics. It supports ACID transactions, primary-key tables with upserts (via an LSM-tree merge engine), append-only tables, and flexible partition strategies — making it a popular choice on top of object stores like HDFS, OSS, and S3. Daft integrates with Paimon through [pypaimon](https://pypi.org/project/pypaimon/), the official Apache Paimon Python SDK. Daft currently supports: 1. **Distributed Reads:** Daft distributes I/O across all available compute resources (Ray workers or local threads). 2. **Predicate & Partition Pushdown:** `df.where()` filter expressions are pushed down to Paimon's scan planner for partition pruning and file-level skipping. 3. **Column Projection:** Only the requested columns are read from disk. 4. **Append-only and Primary-Key Tables:** Both table types are supported; append-only tables use Daft's native high-performance Parquet reader, while PK tables that require LSM merge fall back to pypaimon's built-in reader. 5. **Catalog Abstraction:** Paimon catalogs integrate with Daft's unified `Catalog` / `Table` interfaces, enabling SQL queries and `daft.read_table()` access. ## Installation ```bash pip install daft pip install pypaimon ``` For S3 / S3-compatible storage (e.g. MinIO, Ceph), also install the AWS extra: ```bash pip install "daft[aws]" ``` OSS (Alibaba Cloud Object Storage) is supported via Daft's built-in OpenDAL backend — no extra is required beyond `daft` and `pypaimon`. ## Tutorial ### Reading a Table Use [`daft.read_paimon`][daft.read_paimon] to create a DataFrame from a Paimon table. First, obtain a table object through pypaimon: === "Python" ```python import pypaimon catalog = pypaimon.CatalogFactory.create({"warehouse": "/path/to/warehouse"}) table = catalog.get_table("my_db.my_table") ``` Then create a DataFrame: === "Python" ```python import daft df = daft.read_paimon(table) df.show() ``` Filter operations are pushed down to Paimon's scan planner for efficient partition pruning: === "Python" ```python # Partition pruning — only reads the matching partition files df = df.where(df["dt"] == "2024-01-01") df.show() ``` For tables on object stores, pass an `IOConfig` to supply credentials: === "Python" ```python from daft.io import IOConfig, S3Config catalog = pypaimon.CatalogFactory.create({ "warehouse": "s3://my-bucket/warehouse", }) table = catalog.get_table("my_db.my_table") io_config = IOConfig( s3=S3Config( key_id="...", access_key="...", region_name="us-west-2", ) ) df = daft.read_paimon(table, io_config=io_config) df.show() ``` ### Writing to a Table Use [`df.write_paimon()`][daft.DataFrame.write_paimon] to write a DataFrame back to Paimon. Two modes are supported: #### Append === "Python" ```python result = df.write_paimon(table, mode="append") result.show() ``` The returned DataFrame summarises the written files: ``` ╭───────────┬───────┬───────────┬────────────────────────────────╮ │ operation ┆ rows ┆ file_size ┆ file_name │ │ --- ┆ --- ┆ --- ┆ --- │ │ Utf8 ┆ Int64 ┆ Int64 ┆ Utf8 │ ╞═══════════╪═══════╪═══════════╪════════════════════════════════╡ │ APPEND ┆ 1000 ┆ 48291 ┆ data-00001.parquet │ ╰───────────┴───────┴───────────┴────────────────────────────────╯ ``` #### Overwrite === "Python" ```python result = df.write_paimon(table, mode="overwrite") result.show() ``` ### Using the Catalog Abstraction Daft's `Catalog` and `Table` interfaces let you access Paimon tables through the same API used by Iceberg, Unity, Glue, and other built-in integrations. #### Creating a Catalog === "Python" ```python import pypaimon from daft.catalog import Catalog inner = pypaimon.CatalogFactory.create({"warehouse": "/path/to/warehouse"}) catalog = Catalog.from_paimon(inner, name="my_paimon") ``` You can also wrap a single pypaimon table directly: === "Python" ```python from daft.catalog import Table inner_table = inner.get_table("my_db.my_table") table = Table.from_paimon(inner_table) ``` #### Browsing the Catalog === "Python" ```python # List all namespaces (databases) catalog.list_namespaces() # [Identifier('my_db'), ...] # List all tables catalog.list_tables() # [Identifier('my_db', 'my_table'), ...] # Check existence catalog.has_namespace("my_db") # True catalog.has_table("my_db.my_table") # True ``` #### Reading and Writing Through the Catalog === "Python" ```python # Read into a DataFrame df = catalog.read_table("my_db.my_table") df.show() # Append data catalog.write_table("my_db.my_table", df, mode="append") # Or use the Table object directly table = catalog.get_table("my_db.my_table") table.append(df) table.overwrite(df) ``` #### Creating Tables === "Python" ```python import daft from daft.io.partitioning import PartitionField # Infer schema from an existing DataFrame schema = daft.from_pydict({"id": [1], "name": ["a"], "dt": ["2024-01-01"]}).schema() # Provide partition fields (identity transform — Paimon's native model) dt_field = schema["dt"] partition_fields = [PartitionField.create(dt_field)] table = catalog.create_table( "my_db.new_table", schema, partition_fields=partition_fields, ) ``` To create a primary-key table, pass `primary_keys` in the `properties` dict: === "Python" ```python table = catalog.create_table( "my_db.pk_table", schema, properties={"primary_keys": ["id", "dt"]}, partition_fields=partition_fields, ) ``` ### Session and SQL Integration Once attached to a `Session`, your Paimon catalog is available from SQL queries and `daft.read_table()`: === "Python" ```python from daft.session import Session sess = Session() sess.attach(catalog, "my_paimon") sess.set_namespace("my_db") # SQL query resolves through the Paimon catalog sess.sql("SELECT id, name FROM my_table WHERE dt = '2024-01-01'").show() # Fully-qualified read sess.read_table("my_paimon.my_db.my_table").show() ``` ## Type System Paimon types are mapped through PyArrow to Daft types: | Paimon | Daft | |---------------------------------|---------------------------------------------------------| | `BOOLEAN` | `daft.DataType.bool()` | | `TINYINT` | `daft.DataType.int8()` | | `SMALLINT` | `daft.DataType.int16()` | | `INT` | `daft.DataType.int32()` | | `BIGINT` | `daft.DataType.int64()` | | `FLOAT` | `daft.DataType.float32()` | | `DOUBLE` | `daft.DataType.float64()` | | `DECIMAL(precision, scale)` | `daft.DataType.decimal128(precision, scale)` | | `DATE` | `daft.DataType.date()` | | `TIME(precision)` | `daft.DataType.int64()` | | `TIMESTAMP(precision)` | `daft.DataType.timestamp(timeunit=..., timezone=None)` | | `TIMESTAMP_LTZ(precision)` | `daft.DataType.timestamp(timeunit=..., timezone="UTC")` | | `CHAR(n)` / `VARCHAR(n)` / `STRING` | `daft.DataType.string()` | | `BINARY(n)` / `VARBINARY(n)` / `BYTES` | `daft.DataType.binary()` | | `ARRAY` | `daft.DataType.list(element_type)` | | `MAP` | `daft.DataType.map(key_type, value_type)` | | `ROW<[field_name: field_type]>` | `daft.DataType.struct(fields)` | ## FAQs 1. **Do I need to install pypaimon separately?** *Yes. Run `pip install pypaimon`. pypaimon is not bundled with Daft.* 2. **Which Paimon catalog types are supported?** *pypaimon currently ships two catalog implementations: `FileSystemCatalog` (local / OSS / S3, selected by `metastore=filesystem` or by default) and `RESTCatalog` (selected by `metastore=rest`). Both can be wrapped with `Catalog.from_paimon()`.* 3. **Can I use Daft with an existing Paimon warehouse?** *Yes. Create a pypaimon catalog pointing at your warehouse and pass it to `Catalog.from_paimon()` or use `catalog.get_table()` and `daft.read_paimon()` directly.* 4. **Does Daft support schema evolution for Paimon tables?** *Reading tables with evolved schemas is handled by pypaimon's reader. Daft does not currently expose DDL operators beyond `create_table`.* 5. **How do I configure credentials for OSS or S3?** *Pass an `IOConfig` to `daft.read_paimon()`, or include the relevant `fs.*` options in the catalog options dict when creating the pypaimon catalog (Daft will infer an `IOConfig` automatically from these).* 6. **Can I use the Daft Paimon connector inside a Ray cluster?** *Yes. Daft's distributed execution on Ray works with Paimon the same way it does with Iceberg — scan tasks are distributed across workers automatically.* --- ### Connectors Postgres (`docs/connectors/postgres.md`) # Reading from and Writing to PostgreSQL [PostgreSQL](https://www.postgresql.org/) is an open-source object-relational database system known for its reliability, feature robustness, and performance. ## Installing Daft with PostgreSQL Support Install Daft with the `daft[postgres]` extra, or manually install the required packages: [psycopg](https://www.psycopg.org/) and [pgvector](https://github.com/pgvector/pgvector-python) for vector support. ```bash pip install -U "daft[postgres]" ``` ## Tutorial ### Reading a Table To read from PostgreSQL tables, use Daft's [Catalog API](../api/catalogs_tables.md). Daft provides high-level APIs for connecting to PostgreSQL databases and accessing tables as DataFrames. To connect to a PostgreSQL database, provide a connection string to `Catalog.from_postgres()` and a list of PostgreSQL extensions to install. By default, the pgvector `vector` extension is automatically installed if available for vector support. === "🐍 Python" ```python import daft from daft.catalog import Catalog # Connect to PostgreSQL database # Note: The pgvector extension is enabled by default for vector support catalog = Catalog.from_postgres("postgresql://user:password@localhost:5432/mydb", extensions=["vector"]) # List available schemas schemas = catalog.list_namespaces() print(schemas) # List tables in a schema tables = catalog.list_tables() print(tables) ``` After connecting to your database, reading a table is extremely easy: === "🐍 Python" ```python # Get a table from the catalog table = catalog.get_table("public.users") # Read the table as a DataFrame df = table.read() df.show() ``` Daft supports parallel reads by partitioning on a column: === "🐍 Python" ```python # Read with partitioning for better performance on large tables df = table.read(partition_col="id", num_partitions=4) df.show() ``` ### Writing to a Table To write DataFrames to PostgreSQL tables, you can create tables and use the append or overwrite operations. === "🐍 Python" ```python # Create a sample DataFrame df = daft.from_pydict({ "id": [1, 2, 3, 4], "name": ["Alice", "Bob", "Charlie", "Diana"], "age": [25, 30, 35, 28], "score": [95.5, 87.2, 91.8, 88.9] }) # Create a table in PostgreSQL table = catalog.create_table("public.scores", df.schema()) # Write data to the table table.append(df) ``` You can also overwrite existing tables: === "🐍 Python" ```python # Overwrite the table with new data new_df = daft.from_pydict({ "id": [5, 6], "name": ["Eve", "Frank"], "age": [32, 29], "score": [92.1, 89.3] }) table.overwrite(new_df) ``` ## Type System | Daft Type | PostgreSQL Type | |-----------|-----------------| | [`daft.DataType.bool()`][daft.datatype.DataType.bool] | `BOOLEAN` | | [`daft.DataType.int8()`][daft.datatype.DataType.int8] | `SMALLINT` | | [`daft.DataType.int16()`][daft.datatype.DataType.int16] | `SMALLINT` | | [`daft.DataType.int32()`][daft.datatype.DataType.int32] | `INTEGER` | | [`daft.DataType.int64()`][daft.datatype.DataType.int64] | `BIGINT` | | [`daft.DataType.float32()`][daft.datatype.DataType.float32] | `REAL` | | [`daft.DataType.float64()`][daft.datatype.DataType.float64] | `DOUBLE PRECISION` | | [`daft.DataType.decimal128(precision, scale)`][daft.datatype.DataType.decimal128] | `NUMERIC` | | [`daft.DataType.string()`][daft.datatype.DataType.string] | `TEXT` | | [`daft.DataType.binary()`][daft.datatype.DataType.binary] | `BYTEA` | | [`daft.DataType.date()`][daft.datatype.DataType.date] | `DATE` | | [`daft.DataType.timestamp(timeunit="us", timezone=None)`][daft.datatype.DataType.timestamp] | `TIMESTAMP` | | [`daft.DataType.timestamp(timeunit="us", timezone="UTC")`][daft.datatype.DataType.timestamp] | `TIMESTAMPTZ` | | [`daft.DataType.list(element_type)`][daft.datatype.DataType.list] | `element_type[]` | | [`daft.DataType.struct(fields)`][daft.datatype.DataType.struct] | `JSONB` | | [`daft.DataType.map(key_type, value_type)`][daft.datatype.DataType.map] | `JSONB` | | [`daft.DataType.embedding(size)`][daft.datatype.DataType.embedding] | `VECTOR(size)` | ## Reference Daft provides high-level [Session](../api/sessions.md) and [Catalog](../api/catalogs_tables.md) APIs for reading and writing PostgreSQL tables. However, the [`Catalog.from_postgres`][daft.catalog.Catalog.from_postgres] and table operations are the primary entry points for PostgreSQL interactions. ### Catalog Operations The PostgreSQL catalog supports the full range of catalog operations: - **Namespaces (Schemas):** `create_namespace()`, `drop_namespace()`, `list_namespaces()`, `has_namespace()` - **Tables:** `create_table()`, `create_table_if_not_exists()`, `drop_table()`, `get_table()`, `has_table()`, `list_tables()`, `read_table()`, `write_table()` === "🐍 Python" ```python # Create a new schema catalog.create_namespace("analytics") # Create a table with custom properties, enabling Row Level Security (default behavior) schema = daft.Schema.from_pydict({ "user_id": daft.DataType.int64(), "event": daft.DataType.string(), }) table = catalog.create_table( "analytics.events", schema, properties={"enable_rls": True} ) ``` ### Table Operations PostgreSQL tables support read, append, and overwrite operations: === "🐍 Python" ```python # Read operations df = table.read() df_partitioned = table.read(partition_col="id", num_partitions=4) # Write operations table.append(df) table.overwrite(df) ``` ## FAQs 1. **Does Daft support PostgreSQL extensions?** Yes! Daft can automatically install and configure PostgreSQL extensions if they're available. The pgvector extension `vector` is installed by default, if available. 2. **How does Daft handle PostgreSQL data types?** Daft provides comprehensive type mapping between Daft's Arrow-based types and PostgreSQL types. Complex types like arrays and JSON are supported, with fallbacks to JSONB for unsupported structures. 3. **Does Daft support PostgreSQL Row Level Security?** Yes, Daft automatically enables Row Level Security on newly created tables for enhanced security. This can be controlled via the `enable_rls` property when creating tables. 4. **How does Daft handle PostgreSQL schemas and search paths?** Daft respects PostgreSQL's schema search path. When no schema is specified, PostgreSQL uses its configured search path to resolve table names. 5. **Can Daft work with PostgreSQL in distributed environments?** Yes, when using Daft's distributed runner, PostgreSQL writes can be distributed across multiple machines for maximum performance. 6. **How does Daft handle vector embeddings in PostgreSQL?** Daft has native support for pgvector's VECTOR type. Embedding columns are automatically converted between Daft's embedding type and PostgreSQL's vector type during reads and writes. --- ### Connectors S3tables (`docs/connectors/s3tables.md`) # Reading from and Writing to AWS S3 Tables Daft integrates with [S3 Tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html) using its [Catalog](../api/catalogs_tables.md) interface which supports both reading and writing S3 Tables via Iceberg. ## Example ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### S3 Tables AWS API You can use the S3 Tables AWS API via a `boto3` client or `boto3` session. ### S3 Tables Iceberg REST API S3 Tables offers an [Iceberg REST compatible endpoint](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-open-source.html) which Daft uses when you create a catalog via [`from_s3tables`][daft.Catalog.from_s3tables]. > You can connect your Iceberg REST client to the Amazon S3 Tables Iceberg REST endpoint and make REST API calls to create, update, or query tables in S3 table buckets. The endpoint implements a set of standardized Iceberg REST APIs specified in the Apache Iceberg REST Catalog Open API specification. The endpoint works by translating Iceberg REST API operations into corresponding S3 Tables operations. ### Glue and LakeFormation You may also use the AWS Glue Iceberg REST endpoint, which provides unified table management, centralized governance, and fine-grained access control (table level). However this requires service enabling integrations in AWS Glue and LakeFormation. --- ### Connectors Sql (`docs/connectors/sql.md`) # Reading from and Writing to SQL Databases You can read the results of SQL queries from databases, data warehouses, and query engines via the [`daft.read_sql()`][daft.read_sql] function. Daft currently supports: 1. **20+ SQL Dialects:** Daft supports over 20 databases, data warehouses, and query engines by using [SQLGlot](https://sqlglot.com/sqlglot.html) to convert SQL queries across dialects. See the full list of supported dialects [here](https://sqlglot.com/sqlglot/dialects.html). 2. **Parallel + Distributed Reads:** Daft parallelizes SQL reads by using all local machine cores with its default multithreading runner, or all cores across multiple machines if using the [distributed Ray runner](../distributed/index.md). 3. **Skipping Filtered Data:** Daft ensures that only data that matches your [`df.select()`][daft.DataFrame.select], [`df.limit()`][daft.DataFrame.limit], and [`df.where()`][daft.DataFrame.where] expressions will be read, often skipping entire partitions/columns. ## Installing Daft with SQL Support Install Daft with the `daft[sql]` extra, or manually install the required packages: [ConnectorX](https://sfu-db.github.io/connector-x/databases.html), [SQLAlchemy](https://docs.sqlalchemy.org/en/20/orm/quickstart.html) and [SQLGlot](https://sqlglot.com/sqlglot.html). ```bash pip install -U "daft[sql]" ``` ## Reading a SQL query To read a SQL query, provide [`daft.read_sql()`][daft.read_sql] with the **SQL query** and a **URL** for the data source. The example below creates a local SQLite table for Daft to read. === "🐍 Python" ```python import sqlite3 connection = sqlite3.connect("example.db") connection.execute( "CREATE TABLE IF NOT EXISTS books (title TEXT, author TEXT, year INTEGER)" ) connection.execute( """ INSERT INTO books (title, author, year) VALUES ('The Great Gatsby', 'F. Scott Fitzgerald', 1925), ('To Kill a Mockingbird', 'Harper Lee', 1960), ('1984', 'George Orwell', 1949), ('The Catcher in the Rye', 'J.D. Salinger', 1951) """ ) connection.commit() connection.close() ``` After writing this local example table, we can easily read it into Daft as a DataFrame. === "🐍 Python" ```python # Read SQL query import daft df = daft.read_sql( "SELECT * FROM books", "sqlite:///example.db", ) ``` Daft uses [ConnectorX](https://sfu-db.github.io/connector-x/databases.html) under the hood to read SQL data. ConnectorX is a fast, Rust based SQL connector that reads directly into Arrow Tables, enabling zero-copy transfer into Daft. If the database is not supported by ConnectorX (list of supported databases [here](https://sfu-db.github.io/connector-x/intro.html#supported-sources-destinations)), Daft will fall back to using [SQLAlchemy](https://docs.sqlalchemy.org/en/20/orm/quickstart.html). You can also directly provide a SQL alchemy connection via a **connection factory**. This way, you have the flexibility to provide additional parameters to the engine. === "🐍 Python" ```python # Read SQL query using a connection factory import daft from sqlalchemy import create_engine def create_connection(): return create_engine("sqlite:///example.db", echo=True).connect() df = daft.read_sql("SELECT * FROM books", create_connection) ``` ## Parallel + Distributed Reads For large datasets, Daft can parallelize SQL reads by using all local machine cores with its default multithreading runner, or all cores across multiple machines if using the [distributed Ray runner](../distributed/index.md). Supply the [`daft.read_sql()`][daft.read_sql] function with a **partition column** and optionally the **number of partitions** to enable parallel reads. === "🐍 Python" ```python # Read SQL query with parallel reads import daft df = daft.read_sql( "SELECT * FROM table", "sqlite:///big_table.db", partition_col="col", num_partitions=3, ) ``` Behind the scenes, Daft will partition the data by appending a `WHERE col > ... AND col <= ...` clause to the SQL query, and then reading each partition in parallel. ## Data Skipping Optimizations Filter, projection, and limit pushdown optimizations can be used to reduce the amount of data read from the database. In the example below, Daft reads the top ranked terms from the BigQuery Google Trends dataset. The `where` and `select` expressions in this example will be pushed down into the SQL query itself, we can see this by calling the [`df.explain()`][daft.DataFrame.explain] method. === "🐍 Python" ```python import daft, sqlalchemy, datetime def create_conn(): engine = sqlalchemy.create_engine( "bigquery://", credentials_path="path/to/service_account_credentials.json" ) return engine.connect() df = daft.read_sql("SELECT * FROM `bigquery-public-data.google_trends.top_terms`", create_conn) df = df.where((df["refresh_date"] >= datetime.date(2024, 4, 1)) & (df["refresh_date"] < datetime.date(2024, 4, 8))) df = df.where(df["rank"] == 1) df = df.select(df["refresh_date"].alias("Day"), df["term"].alias("Top Search Term"), df["rank"]) df = df.distinct() df = df.sort(df["Day"], desc=True) df.explain(show_all=True) ``` ``` {title="Output"} .. == Physical Plan == .. | SQL Query = SELECT refresh_date, term, rank FROM (SELECT * FROM `bigquery-public-data.google_trends.top_terms`) AS subquery WHERE rank = 1 AND refresh_date >= CAST('2024-04-01' AS DATE) AND refresh_date < CAST('2024-04-08' AS DATE) ``` The second last line labeled 'SQL Query =' shows the query that Daft executed. Filters such as `rank = 1` and projections such as `SELECT refresh_date, term, rank` have been injected into the query. Without these pushdowns, Daft would execute the unmodified `SELECT * FROM 'bigquery-public-data.google_trends.top_terms'` query and read in the entire dataset/table. We tested the code above on Google Colab (12GB RAM): - With pushdowns, the code ran in **8.87s** with a peak memory of **315.97 MiB** - Without pushdowns, the code took over **2 mins** before crashing with an **out of memory** error. You could modify the SQL query to add the filters and projections yourself, but this may become lengthy and error-prone, particularly with many expressions. That's why Daft automatically handles it for you. ## Roadmap Here are the SQL features that are on our roadmap. Please let us know if you would like to see support for any of these features! 1. Write support into SQL databases. 2. Reads via [ADBC (Arrow Database Connectivity)](https://arrow.apache.org/docs/format/ADBC.html). --- ### Connectors Text (`docs/connectors/text.md`) # Reading Text Files Daft can read line-oriented text files into DataFrames using [`daft.read_text()`][daft.io.read_text]. This is useful for processing log files, configuration files, or any other plain text data. ## Basic Usage Read a text file where each line becomes a row: === "Local File" ```python import daft df = daft.read_text("/path/to/file.txt") df.show() ``` === "Remote File (S3)" ```python import daft from daft.io import S3Config, IOConfig io_config = IOConfig(s3=S3Config(region_name="us-west-2", anonymous=True)) df = daft.read_text("s3://my-bucket/logs/*.txt", io_config=io_config) df.show() ``` === "Remote File (GCS)" ```python import daft from daft.io import GCSConfig, IOConfig io_config = IOConfig(gcs=GCSConfig(anonymous=True)) df = daft.read_text("gs://my-bucket/logs/*.txt", io_config=io_config) df.show() ``` ## Output Schema The `read_text` function returns a DataFrame with a single column: | Column | Type | Description | |--------|------|-------------| | `text` | `string` | Lines from the input files (or entire file content when `whole_text=True`) | ## Reading Options ### Whole File Mode Read each file as a single row instead of splitting by lines: ```python # Each file becomes a single row df = daft.read_text("/path/to/files/*.txt", whole_text=True) df.show() ``` This is useful when you need to process entire file contents at once, such as for document processing or when the file content shouldn't be split by newlines. ### Skip Blank Lines By default, empty lines (after stripping whitespace) are skipped. To include them: ```python df = daft.read_text("/path/to/file.txt", skip_blank_lines=False) ``` When `whole_text=True`, this option skips files that are entirely blank. ### File Encoding Specify the character encoding of input files (defaults to UTF-8): ```python df = daft.read_text("/path/to/file.txt", encoding="latin-1") ``` ### Include File Path Add a column with the source file path: ```python df = daft.read_text("/path/to/files/*.txt", file_path_column="source_file") df.show() ``` ### Hive Partitioning Infer partition columns from the file path: ```python # For paths like /data/year=2024/month=01/file.txt df = daft.read_text("/data/**/*.txt", hive_partitioning=True) df.show() # Includes 'year' and 'month' columns ``` ## Wildcard Patterns `read_text` supports glob patterns for reading multiple files: | Pattern | Description | |---------|-------------| | `*` | Matches any number of characters | | `?` | Matches any single character | | `[...]` | Matches any single character in the brackets | | `**` | Recursively matches directories | === "Examples" ```python # All .txt files in a directory df = daft.read_text("/logs/*.txt") # All .log files recursively df = daft.read_text("/logs/**/*.log") # Files matching a pattern df = daft.read_text("/logs/server-[0-9].txt") ``` ## Use Cases ### Processing Log Files ```python import daft from daft import col # Read log files and filter for errors df = daft.read_text("/var/log/app/*.log", file_path_column="log_file") errors = df.where(col("text").contains("ERROR")) errors.show() ``` ### Document Processing ```python import daft # Read entire documents df = daft.read_text("/documents/*.txt", whole_text=True, file_path_column="doc_path") # Process with embeddings or other analysis df.show() ``` --- ### Connectors Turbopuffer (`docs/connectors/turbopuffer.md`) # Writing to turbopuffer We recommend [turbopuffer](https://turbopuffer.com/) as a fast search engine that combines vector and full-text search using object storage, making all your data easily searchable. ## Example This example does the following: 1. Reads the [`Open-Orca/OpenOrca`](https://huggingface.co/datasets/Open-Orca/OpenOrca) dataset from [Hugging Face](https://huggingface.co/) 2. Computes text embeddings on the `response` column, using either an [OpenAI model](https://platform.openai.com/docs/models/text-embedding-3-small) or the [`BAAI/bge-base-en-v1.5`](https://huggingface.co/BAAI/bge-base-en-v1.5) open model on Hugging Face with [Sentence Transformers](https://sbert.net/index.html) 3. Writes the results to turbopuffer === "🐍 Python" ```python # $ pip install -U "daft[turbopuffer]" # $ pip install -U "daft[openai]" # OR # $ pip install -U "daft[transformers]" import os import daft from daft.functions.ai import embed_text # Create an embedding with OpenAI, or Sentence Transformers # Requires OPENAI_API_KEY to be set (https://platform.openai.com/settings/organization/api-keys) if os.getenv("OPENAI_API_KEY"): provider = "openai" model = "text-embedding-3-small" else: provider = "sentence_transformers" model = "BAAI/bge-base-en-v1.5" turbopuffer_config = { "namespace": "daft-tpuf-example", "region": "gcp-us-central1", "distance_metric": "cosine_distance", "schema": { "text": { "type": "string", "full_text_search": True, } } } ( daft.read_huggingface("Open-Orca/OpenOrca") .limit(8) .with_column("vector", embed_text(daft.col("response"), provider=provider, model=model)) .write_turbopuffer(**turbopuffer_config) # Requires TURBOPUFFER_API_KEY to be set (https://turbopuffer.com/dashboard) ) ``` For a more a detailed example, check out [our guide on generating text embeddings](../examples/text-embeddings.md). Check out our [`DataFrame.write_turbopuffer` documentation][daft.DataFrame.write_turbopuffer] for more customization options. --- ### Connectors Unity Catalog (`docs/connectors/unity_catalog.md`) # Reading from and Writing to Unity Catalog [Unity Catalog](https://github.com/unitycatalog/unitycatalog/) is an open-sourced catalog developed by Databricks. Users of Unity Catalog are able to work with data assets such as tables (Parquet, CSV, Iceberg, Delta), volumes (storing raw files), functions and models. To use Daft with the Unity Catalog, you will need to install Daft with the `unity` option specified like so: ```bash pip install daft[unity] ``` !!! warning "Warning" These APIs are in beta and may be subject to change as the Unity Catalog continues to be developed. ## Connecting to the Unity Catalog Daft includes an abstraction for the Unity Catalog. For more information, see also [Unity Catalog Documentation](https://docs.unitycatalog.io/integrations/unity-catalog-daft/). ### Authentication options You can authenticate either with a personal access token (PAT) or with OAuth credentials. To use OAuth, create a Databricks service principal and generate an OAuth secret for it. See the Databricks docs for the exact steps: [Databricks OAuth M2M docs](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m). === "🐍 Python (PAT)" ```python from daft.catalog.__unity import UnityCatalogClient as UnityCatalog unity = UnityCatalog( endpoint="https://.cloud.databricks.com", # Authentication can be retrieved from your provider of Unity Catalog token="my-token", ) # See all available catalogs print(unity.list_catalogs()) # See available schemas in a given catalog print(unity.list_schemas("my_catalog_name")) # See available tables in a given schema print(unity.list_tables("my_catalog_name.my_schema_name")) ``` === "🐍 Python (OAuth client credentials)" ```python from daft.catalog.__unity import UnityCatalogClient as UnityCatalog from daft.catalog.__unity import OAuth2Credentials unity = UnityCatalog( endpoint="https://.cloud.databricks.com", oauth=OAuth2Credentials( client_id="your-client-id", client_secret="your-client-secret", ), ) # See all available catalogs print(unity.list_catalogs()) # See available schemas in a given catalog print(unity.list_schemas("my_catalog_name")) # See available tables in a given schema print(unity.list_tables("my_catalog_name.my_schema_name")) ``` ## Loading a Dataframe from a Delta Lake table in Unity Catalog === "🐍 Python" ```python unity_table = unity.load_table("my_catalog_name.my_schema_name.my_table_name") df = daft.read_deltalake(unity_table) df.show() ``` Any subsequent filter operations on the Daft `df` DataFrame object will be correctly optimized to take advantage of DeltaLake features: === "🐍 Python" ```python # Filter which takes advantage of partition pruning capabilities of Delta Lake df = df.where(df["partition_key"] < 1000) df.show() ``` See also [Delta Lake](../connectors/delta_lake.md) for more information about how to work with the Delta Lake tables provided by the Unity Catalog. ## Downloading files in Unity Catalog volumes Daft supports downloading from Unity Catalog volumes using [`Expression.download()`][daft.expressions.expressions.download]. File paths that start with `vol+dbfs:/` or `dbfs:/` will be downloaded using the configurations in [`IOConfig.unity`][daft.daft.IOConfig.unity]. These configurations can be created using `UnityCatalog.to_io_config`, or automatically derived from the global session. === "🐍 Python" ```python df = daft.from_pydict({ "files": [ "/Volumes/my_catalog/my_schema_name/my_volume_name/file1.txt", "/Volumes/my_catalog/my_schema_name/my_volume_name/file2.txt" ] }) # explicitly specify the unity catalog io_config = unity.to_io_config() data_df = df.select(("vol+dbfs:" + df["files"]).download(io_config=io_config)) data_df.show() # use the global session from daft.catalog import Catalog import daft.session catalog = Catalog.from_unity(unity) daft.session.attach(catalog) data_df = df.select(df["files"].download()) data_df.show() ``` ## Roadmap 1. Unity Iceberg integration for reading tables using the Iceberg interface instead of the Delta Lake interface Please make issues on the [Daft repository](https://github.com/Eventual-Inc/Daft) if you have any use-cases that Daft does not currently cover! For the overall Daft development plan, see [Daft Roadmap](../roadmap.md). --- ### Connectors Webdataset (`docs/connectors/webdataset.md`) # WebDataset [WebDataset](https://github.com/webdataset/webdataset) stores multimodal samples as consecutive members of TAR shards. Daft reads these shards with [`daft.read_webdataset()`][daft.read_webdataset]. ## Reading shards Pass a TAR file, directory, glob, or list of paths: ```python import daft df = daft.read_webdataset("/datasets/images/*.tar") ``` Each row represents one sample. Consecutive members with the same prefix are grouped together, and the suffix becomes the column name. For example, `000001.jpg`, `000001.json`, and `000001.txt` produce one row with `jpg`, `json`, and `txt` columns. Daft also includes: - `__key__`: the sample prefix - `__url__`: the shard containing the sample JSON, text, and class/index sidecars are decoded eagerly. Image, audio, video, and other binary members are represented as lazy [`File`][daft.File] references into their TAR shard. Selecting only metadata columns therefore avoids reading media payloads: ```python metadata = df.select("__key__", "json", "txt") images = df.select("__key__", "jpg") ``` ## Reading from Hugging Face Use `format="webdataset"` with [`daft.read_huggingface()`][daft.read_huggingface]: ```python df = daft.read_huggingface( "laion/conceptual-captions-12m-webdataset", format="webdataset", ) ``` You can also read explicit Hugging Face paths: ```python df = daft.read_webdataset( "hf://datasets/laion/conceptual-captions-12m-webdataset/data/*.tar" ) ``` ## Format requirements Daft currently requires: - Uncompressed `.tar` shards, because lazy member reads use byte offsets. - Non-sparse TAR members, because sparse file contents are not stored in one contiguous byte range. - Consistent member suffixes and JSON shapes across shards. Schema inference uses the first five samples in the first shard. If a later sample introduces an incompatible suffix or JSON shape, Daft raises an error instead of silently discarding data. --- ### Contributing Contributing Ai Functions (`docs/contributing/contributing-ai-functions.md`) # Contributing New AI Functions This page contains references and guides for developing new AI Functions in daft. These steps will guide you through implementing a model expression like: - `embed_text` - `embed_image` - `classify_text` - `prompt` ### Step 1. Define the Protocol and Descriptor All model expressions are backed by a Protocol and Descriptor. These protocols are based on the verb and modality, and the descriptor is used to instantiate the model at runtime. ```python # daft.ai.protocols @runtime_checkable class TextClassifier(Protocol): """Protocol for text classification implementations.""" def classify_text(self, text: list[str], labels: list[str]) -> list[str]: """Classifies a batch of text strings using the given label(s).""" ... class TextClassifierDescriptor(Descriptor[TextClassifier]): """Descriptor for a TextClassifier implementation.""" ``` ### Step 2. Add to the Provider Interface You must update the Provider interface with a new method to create your descriptor. This should have a default implementation which simply raises; this makes it so that you need not update all existing providers. ```python # daft.ai.provider class Provider(ABC): # ... existing code def get_text_classifier(self, model: str | None = None, **options: Any) -> TextClassifierDescriptor: """Returns a TextClassifierDescriptor for this provider.""" raise not_implemented_err(self, method="classify_text") ``` ### Step 3. Define the Function. In `daft.functions.ai` you can add the function, and then re-export it in `daft.functions.__init__.py`. The implementation is responsible for resolving the provider from the given arguments, then you will call the appropriate provider method to get the relevant descriptor. ```python import daft from daft import DataType, Series from daft.ai.protocols import TextClassifier, TextClassifierDescriptor def classify_text( text: Expression, labels: list[str], *, provider: str | Provider | None = None, model: str | None = None, ) -> Expression: # Load a TextClassifierDescriptor from the resolved provider text_classifier = _resolve_provider(provider, "transformers").get_text_classifier(model) # Create the stateful class UDF classifier = _TextClassifierExpression(text_classifier, labels) # Return the expression return classifier.classify(text) @daft.cls class _TextClassifierExpression: """Function expression implementation for a TextClassifier protocol.""" def __init__(self, descriptor: TextClassifierDescriptor, labels: list[str]): # Instantiate from the descriptor in __init__ self.text_classifier = descriptor.instantiate() self.labels = labels @daft.method.batch(return_dtype=DataType.string()) def classify(self, text: Series) -> list[str]: text_list = text.to_pylist() if not text_list: return [] return self.text_classifier.classify_text(text_list, self.labels) ``` ## Step 4. Implement the Protocol for some Provider. Here is a simplified example implementation of embed_text for OpenAI. This should give you and idea of where you actual logic should live, and the previous steps are to properly hook your new expression into the provider/model system. ```python @dataclass class OpenAITextEmbedderDescriptor(TextEmbedderDescriptor): model: str # store some metadata # We can use the stored metadata to instantiate the protocol implementation def instantiate(self) -> TextEmbedder: return OpenAITextEmbedder(client=OpenAI(), model=self.model) @dataclass class OpenAITextEmbedder(TextEmbedder): client: OpenAI model: str # This is a simple version using the batch API. The full implementation # uses dynamic batching and has error handling mechanisms. def embed_text(self, text: list[str]) -> list[Embedding]: response = self.client.embeddings.create( input=text, model=self.model, encoding_format="float", ) return [np.array(embedding.embedding) for embedding in response.data] ``` ## Step 5. Expression Usage You can now use this like any other expression. ```python import daft df = daft.read_parquet("/path/to/file.parquet") # assuming has some column 'text' df = df.with_column("embedding", embed_text(df["text"], provider="openai")) # <- set provider to 'openai' df.show() ``` --- ### Contributing Development (`docs/contributing/development.md`) # Development Guide > **💡 Already set up?** > > See our quick [tutorial](#adding-new-expressions) on how to add a new expression to Daft. ## Development Environment To set up your development environment: 1. Install [uv](https://docs.astral.sh/uv/). You can run `curl -LsSf https://astral.sh/uv/install.sh | sh` on macOS and Linux. 2. [Install the Rust compilation toolchain](https://www.rust-lang.org/tools/install) 3. Install [Node.js](https://nodejs.org/) (22.x LTS) and npm in order to build docs and the daft-dashboard functionality. 4. Install [cmake](https://cmake.org/). If you use [homebrew](https://brew.sh), you can run `brew install cmake`. 5. Install [protoc](https://protobuf.dev/installation/). You will need this for release builds -- `make build-release`. With homebrew, installation is `brew install protobuf`. 6. Clone the Daft repo: `git clone git@github.com:Eventual-Inc/Daft.git` 7. Run `make .venv` from your newly cloned Daft repository to create a new virtual environment with all of Daft's development dependencies installed 8. Run `make hooks` to install pre-commit hooks: these will run tooling on every commit to ensure that your code meets Daft development standards ## Developing 1. `make build`: recompile your code after modifying any Rust code in `src/` 2. `DAFT_RUNNER=native make test`: run tests, you can set additional run parameters through `EXTRA_ARGS` 3. `DAFT_RUNNER=ray make test`: set the runner to the Ray runner and run tests 4. `make docs`: build docs 5. `make docs-serve`: build docs in development server 6. `make format`: format all Python and Rust code 7. `make lint`: lint all Python and Rust code 8. `make check-format`: check that all Python and Rust code is formatted, alias `make format-check` 9. `make precommit`: run all pre-commit hooks, must install pre-commit first(pip install pre-commit) 10. `make build-release`: perform a full release build of Daft 11. `make build-whl`: recompile your code after modifying any Rust code in `src/` for development, only generate `whl` file without installation 12. `make clean`: clean all build artifacts, including the python virtual environment. You can skip cleaning the virtual environment by setting `SKIP_VENV=true` ### Note about Developing `daft-dashboard` !!! note If you just want to *use* the Daft Dashboard (not modify it), see the [Daft Dashboard](../observability/dashboard.md) user guide. The instructions below are only needed for contributing to the dashboard's frontend code. If you wish to enable, or work on the daft-dashboard functionality, it requires [Node.js](https://nodejs.org/) (LTS) and npm. Install Node.js, then run `npm install` and `npm run build` in the `src/daft-dashboard/frontend` directory. Next (_make sure Daft is installed_), you can launch the dashboard using the `daft dashboard` command, for example: ```bash # You can learn more about this command by `daft dashboard -h` daft dashboard start -v -a 127.0.0.1 -p 3238 ``` Before executing a specific Daft job, enable reporting query execution data to the dashboard by setting the `DAFT_DASHBOARD_URL` environment variable, for example: ```bash export DAFT_DASHBOARD_URL="http://127.0.0.1:3238" ``` Next, you can access and view the dashboard through a web browser, for example, via address `http://127.0.0.1:3238`. ## Developing with Ray Running a development version of Daft on a local Ray cluster is as simple as including `daft.set_runner_ray()` in your Python script and then building and executing it as usual. To use a remote Ray cluster, run the following steps on the same operating system version as your Ray nodes, in order to ensure that your binaries are executable on Ray. 1. `mkdir wd`: this is the working directory, it will hold all the files to be submitted to Ray for a job 2. `ln -s daft wd/daft`: create a symbolic link from the Python module to the working directory 3. `make build-release`: an optimized build to ensure that the module is small enough to be successfully uploaded to Ray. Run this after modifying any Rust code in `src/` 4. `ray job submit --working-dir wd --address "http://:8265" -- python script.py`: submit `wd/script.py` to be run on Ray ## Debugging The debugging feature uses a special VSCode launch configuration to start the Python debugger with a script at `tools/attach_debugger.py`, which takes the target script's name as input. This script finds the process ID, updates the launch.json file, compiles the target script, and runs it. It then attaches a Rust debugger to the Python debugger, allowing both to work together. Breakpoints in Python code hit the Python debugger, while breakpoints in Rust code hit the Rust debugger. ### Preparation - **CodeLLDB Extension for Visual Studio Code**: This extension is useful for debugging Rust code invoked from Python. - **Setting Up the Virtual Environment Interpreter** (Ctrl+Shift+P -> Python: Select Interpreter -> .venv) - **Debug Settings in launch.json** This file is usually found in the `.vscode` folder of your project root. See the [official VSCode documentation](https://code.visualstudio.com/docs/editor/debugging#_launch-configurations) for more information about the launch.json file.
launch.json ```json { "configurations": [ { "name": "Debug Rust/Python", "type": "debugpy", "request": "launch", "program": "${workspaceFolder}/tools/attach_debugger.py", "args": [ "${file}" ], "console": "internalConsole", "serverReadyAction": { "pattern": "pID = ([0-9]+)", "action": "startDebugging", "name": "Rust LLDB" } }, { "name": "Rust LLDB", "pid": "0", "type": "lldb", "request": "attach", "program": "${command:python.interpreterPath}", "stopOnEntry": false, "sourceLanguages": [ "rust" ], "presentation": { "hidden": true } } ] } ```
### Running the debugger 1. Create a Python script containing Daft code. Ensure that your virtual environment is set up correctly. 2. Set breakpoints in any `.rs` or `.py` file. 3. In the `Run and Debug` panel on the left, select `Debug Rust/Python` from the drop-down menu on top and click the `Start Debugging` button. This will start a debugging session using the file that is currently opened in the VSCode editor. At this point, your debugger should stop on breakpoints in any .rs file located within the codebase. > **Note**: > On some systems, the LLDB debugger will not attach unless [ptrace protection](https://linux-audit.com/protect-ptrace-processes-kernel-yama-ptrace_scope) is disabled. > To disable, run the following command: > > ```shell > echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope > ``` ## Testing We run test suites across Python and Rust. Python tests focus on high-level DataFrame and Expression functionality, while Rust tests validate individual kernel implementations at a lower level. ### Python tests Our python tests are located in the `tests` directory, you can run all the tests at once with `make test`. To run specific tests, set the runner for the tests in the environment and then run the tests directly using [pytest](https://docs.pytest.org/en/stable/how-to/usage.html) or `make test EXTRA_ARGS="..."` as follows: ```bash # Using pytest DAFT_RUNNER=native pytest tests/dataframe/test_limit_offset.py::test_limit # Using make test DAFT_RUNNER=native make test EXTRA_ARGS="tests/dataframe/test_limit_offset.py::test_limit" ``` To enable debug logs from tests, set the `--log-cli-level` option, as well as disable capturing. ```bash # Using pytest DAFT_RUNNER=native pytest tests/dataframe/test_limit_offset.py::test_limit -s --log-cli-level=DEBUG # Using make test DAFT_RUNNER=native make test EXTRA_ARGS="tests/dataframe/test_limit_offset.py::test_limit -s --log-cli-level=DEBUG" ``` ### Rust tests Our rust tests are distributed across crates, you can run all tests with `cargo test --no-default-features --workspace`. To run rust tests that call into Python, the `--features python` flag and libpython3.\*.so dynamic libraries are required. Please ensure that these are installed, here's a table of common locations on different os: | Operating System | Package Manager | Architecture | Library Path Pattern | | -------------------- | --------------- | ------------- | -------------------------------------------------------------------------- | | **Ubuntu/Debian** | apt | x86_64 | `/usr/lib/x86_64-linux-gnu/libpython3.x.so.1.0` | | | | Other | `/usr/lib/libpython3.x.so.1.0` | | **Red Hat/CentOS** | yum/dnf | x86_64 | `/usr/lib64/libpython3.x.so.1.0` | | **macOS (Homebrew)** | Homebrew | Intel | `/usr/local/opt/python@3.x/lib/libpython3.x.dylib` | | | | Apple Silicon | `/opt/homebrew/opt/python@3.x/lib/libpython3.x.dylib` | | **macOS (System)** | Installer | All | `/Library/Frameworks/Python.framework/Versions/3.x/lib/libpython3.x.dylib` | Tip: you can run the following python command to get the full path to the pylib ```sh python -c "import sysconfig; print(sysconfig.get_config_var('LIBDIR') + '/' + sysconfig.get_config_var('LDLIBRARY'))" ``` Set environment variables to locate the Python library: ```sh export PYO3_PYTHON=".venv/bin/python" export PYO3_PYTHON_PYLIB="/usr/lib/x86_64-linux-gnu/libpython3.11.so.1" export RUSTFLAGS="-C link-arg=-Wl,-rpath,${PYO3_PYTHON_PYLIB%/*} -C link-arg=-L${PYO3_PYTHON_PYLIB%/*} -C link-arg=-lpython3.11" ``` Execute the test after configuration: ```sh cargo test -p daft-dsl --features python -- expr::tests ``` ## Benchmarking Benchmark tests are located in `tests/benchmarks`. If you would like to run benchmarks, make sure to first do `make build-release` instead of `make build` in order to compile an optimized build of Daft. 1. `pytest tests/benchmarks/[test_file.py] -m benchmark`: Run all benchmarks in a file 2. `pytest tests/benchmarks/[test_file.py] -k [test_name] -m benchmark`: Run a specific benchmark in a file More information about writing and using benchmarks can be found on the [pytest-benchmark docs](https://pytest-benchmark.readthedocs.io/en/latest/). ## Adding new expressions Since new expressions are a very common feature request, we wanted to make it easy for new contributors to add these. Adding a new expression requires implementation in Rust and exposing it to Python. ### Step 1: Implement the function in Rust Add your function to the appropriate crate (`daft-functions-json`, `daft-functions-utf8`, etc.). For more advanced use cases, see existing implementations in [daft-functions-utf8](https://github.com/Eventual-Inc/Daft/blob/main/src/daft-functions-utf8/src/lib.rs) ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Step 2: Register the function Okay, now that we have the actual function implementation available, we're not quite done yet. We also need to register this to our `FUNCTION_REGISTRY` which is a global registry of all expressions/functions. Whatever crate/module you are in, there should be a `daft_dsl::functions::FunctionModule` implementation that registers all of the functions. So all you need to do is add your new struct into there. for the `utf8` functions, it's defined here `src/daft-functions-utf8/src/lib.rs` ```rs impl daft_dsl::functions::FunctionModule for Utf8Functions { fn register(parent: &mut daft_dsl::functions::FunctionRegistry) { // ... parent.add_fn(MyToUpperCase); // add this line here } } ``` ### Step 3: Add python bindings Create expression method in `daft/expressions/expressions.py` ```py # the method name should usually match that of which you defined in your `ScalarUDF` implementation def to_uppercase(self) -> Expression: # make sure to add a docstring with a runnable `doctest` example """Convert UTF-8 string to all upper. Returns: Expression: a String expression which is `self` uppercased Examples: >>> import daft >>> df = daft.from_pydict({"x": ["foo", "bar", "baz"]}) >>> df = df.select(df["x"].to_uppercase()) >>> df.show() ╭──────╮ │ x │ │ --- │ │ Utf8 │ ╞══════╡ │ FOO │ ├╌╌╌╌╌╌┤ │ BAR │ ├╌╌╌╌╌╌┤ │ BAZ │ ╰──────╯ (Showing first 3 of 3 rows) """ # Get the function from our global `FUNCTION_REGISTRY` f = native.get_function_from_registry("to_uppercase") return Expression._from_pyexpr(f(self._expr)) ``` For functions with additional arguments, you will need to convert those all to expressions before calling the function_registry function. ```py def extract_all(self, pattern: str | Expression, index: int = 0) -> Expression: pattern_expr = Expression._to_expression(pattern) idx = Expression._to_expression(index) f = native.get_function_from_registry("extract_all") # Pass scalar values as kwargs return Expression._from_pyexpr(f(self._expr, pattern_expr._expr, index=idx._expr)) ``` Add Series method in `daft/series.py`: For series, It just delegates out to the expression implementation, so we can just call the helper method `_eval_expressions` ```py def to_uppercase(self) -> Series: return self._eval_expressions("to_uppercase") ``` and for functions with additional arguments: ```py def extract_all(self, pattern: Series, index: int = 0) -> Series: # Pass scalar values as kwargs return self._eval_expressions("extract_all", pattern, index=index) ``` #### Docstring Template We follow [Google style](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) python docstrings. ```py def method(args) -> return: """Summary of method Args: arg1: description Returns: return1: description Raises: title (optional) description Warning: title (optional) description Note: title (optional) description Examples: (make sure this is plural!) >>> code example (this needs to be a runnable `doctest` example) output Tip: title (optional) description """ ``` ### Step 4: Write tests For testing, you can add a new file, or update an existing one in `tests/expressions/` We have a fixture `test_expression` that will do most of the heavy lifting and ensure that the apis are consistent across expr, series, and sql. here's an example of testing the `extract` function using the `test_expression` fixture ```py def test_extract(test_expression): test_data = ["123-456", "789-012", "345-678"] regex = r"(\d)(\d*)" expected = ["123", "789", "345"] test_expression( data=test_data, expected=expected, name="regexp_extract", sql_name="regexp_extract", # if this is not provided, it will be the same as `name` args=[regex], kwargs={} ) ``` ## Pull Requests ### Best practices for pull requests For the best chance of having your pull request accepted, please follow these guidelines: - **Include unit tests** for all changes and new features. Pull requests without tests will not be merged. - **Keep changes focused.** Aim to solve one problem per pull request and avoid unrelated changes. - **Review before submitting.** Whenever possible, ask another contributor to review your code first or perform a thorough self-review. Ask yourself: _Is it clear why these changes are being made? Are they easy to understand?_ - **Use Conventional Commit messages** for pull request titles. For example: - New feature: `feat: adding API` - Bug fix: `fix: issue with API` - Documentation: `docs: adding API documentation` - **Test error cases.** Ensure your tests cover failure scenarios and provide clear, user-friendly error messages. ### Review process 1. **Draft vs. Open status** - Leave your pull request in Draft status if it is still a work in progress. - Mark it as Open once it is ready for review. 2. **Checks on GitHub** - Ensure all automated checks have passed. PRs with failing checks will not be prioritized for review. 3. **Reviewer assignment** - Contributors with write access: add a reviewer to the Assignee field. - Other contributors: reviewers will be assigned to your PR shortly. 4. **During review** - Assignees will review your PR and provide feedback if changes are needed. - Address review comments promptly to keep momentum. PRs without author engagement for more than a week may lose priority. - Iterate until assignees approve your PR. 5. **Merging** - Once the build is passing and approvals are in place, committers will merge the PR. --- ### Custom Code Cls (`docs/custom-code/cls.md`) ## Stateful Class UDFs with `@daft.cls` When your UDF requires expensive initialization such as loading a machine learning model, establishing database connections, or pre-computing lookup tables use `@daft.cls` to amortize the cost across multiple rows. The class is initialized once per worker, and the same instance processes all rows on that worker. ### Quick Example ```python import daft @daft.cls class TextClassifier: def __init__(self, model_path: str): # This expensive initialization happens once per worker self.model = load_model(model_path) def __call__(self, text: str) -> str: return self.model.predict(text) # Create an instance with initialization arguments classifier = TextClassifier("path/to/model.pkl") df = daft.from_pydict({"text": ["hello world", "goodbye world"]}) # Use the instance directly as a Daft function df = df.select(classifier(df["text"])) ``` ### How It Works 1. **Lazy Initialization**: When you create an instance like `classifier = TextClassifier("path/to/model.pkl")`, the `__init__` method is **not called immediately**. Instead, Daft saves the initialization arguments. 2. **Worker Initialization**: During query execution, Daft calls `__init__` on each instance with the saved arguments. Instances are reused for multiple rows. 3. **Method Calls**: Methods can be called with either: - **Expressions** (like `df["text"]`) - returns an Expression for DataFrame operations - **Scalars** (like `"hello"`) - executes immediately, initializing a local instance if needed ## Method Variants Similarly to `daft.func`, Daft supports the same variants for `daft.method` to optimize for different use cases: - **Row-wise** (default): Regular Python functions process one row at a time - **Async row-wise**: Async Python functions process rows concurrently - **Generator**: Generator functions produce multiple output rows per input row - **Batch** (`@daft.method.batch`): Process entire batches of data with `daft.Series` for high performance ```python @daft.cls class Something: def __call__(self, x: float) -> float: ... def my_method(self, x: float) -> float: ... async def async_method(self, x: float) -> float: ... @daft.method.batch(return_dtype=DataType.float64()) def my_batch_method(self, s: Series) -> Series: ... @daft.method.batch(return_dtype=DataType.float64()) async def async_batch_method(self, s: Series) -> Series: ... ``` Daft automatically detects which variant to use for regular functions based on your function signature. For batch functions, you must use the `@daft.method.batch` decorator. ### Resource Control Control computational resources, concurrency, and error handling with decorator parameters: ```python @daft.cls( cpus=2, # Request 2 CPUs per instance (placement hint) gpus=1, # Request 1 GPU per instance max_concurrency=4, # Cap concurrent instances at 4 use_process=True, # Run each instance in its own process max_retries=3, # Retry failing calls up to 3 times with backoff on_error="log", # On final failure, log and emit None instead of raising ) class ImageClassifier: def __init__(self, model_name: str): import torch self.model = torch.load(model_name).cuda() def __call__(self, image_path: str) -> str: image = load_image(image_path) return self.model(image) classifier = ImageClassifier("resnet50.pth") df = daft.from_pydict({"images": ["img1.jpg", "img2.jpg"]}) df = df.select(classifier(df["images"])) ``` **Parameters:** - `cpus`: CPUs per instance — placement hint used by the scheduler (fractional values allowed). - `gpus`: GPUs per instance (default: 0). Fractional values up to 1.0 are allowed; values above 1.0 must be integers. - `use_process`: Whether to run each instance in a separate process for isolation. - `max_concurrency`: Maximum number of concurrent instances across all workers. - `max_retries`: Number of retry attempts for failing calls (exponential backoff starting at 100 ms, ±25% jitter, capped at 60 s). - `on_error`: `"raise"` (default), `"log"`, or `"ignore"`. Controls behavior once retries are exhausted. - `name_override`: Display name for the UDF in plans and progress bars. ### Using `@daft.method` By default, all methods in a `@daft.cls` class can be used as Daft functions. Use the `@daft.method` decorator to override default behavior: ```python import daft from daft import DataType from typing import Iterator @daft.cls class TextProcessor: def __init__(self, prefix: str): self.prefix = prefix # No decorator needed - works with default inference def __call__(self, text: str) -> str: return f"{self.prefix}{text}" # Override return type @daft.method(return_dtype=DataType.list(DataType.string())) def split_words(self, text: str): return text.split() # Unnest struct fields @daft.method( return_dtype=DataType.struct({ "word_count": DataType.int64(), "char_count": DataType.int64() }), unnest=True ) def analyze(self, text: str): words = text.split() return { "word_count": len(words), "char_count": len(text) } processor = TextProcessor(">> ") df = daft.from_pydict({"text": ["hello world", "foo bar"]}) df = df.select( processor(df["text"]).alias("prefixed"), # Using __call__ processor.split_words(df["text"]).alias("words"), processor.analyze(df["text"]) # Expands into word_count and char_count columns ) ``` ### Method Variants Like `@daft.func`, methods support multiple execution patterns: #### Async Methods ```python import aiohttp @daft.cls class APIClient: def __init__(self, api_key: str): self.api_key = api_key async def fetch_data(self, url: str) -> str: async with aiohttp.ClientSession() as session: headers = {"Authorization": f"Bearer {self.api_key}"} async with session.get(url, headers=headers) as response: return await response.text() client = APIClient("my-secret-key") df = daft.from_pydict({"urls": ["https://api.example.com/1", "https://api.example.com/2"]}) df = df.select(client.fetch_data(df["urls"])) ``` When `max_concurrency` is set on a class with async methods, it controls the number of concurrent coroutines rather than the number of actor pool processes: ```python @daft.cls(max_concurrency=10) class APIClient: def __init__(self, api_key: str): self.api_key = api_key async def fetch_data(self, url: str) -> str: async with aiohttp.ClientSession() as session: headers = {"Authorization": f"Bearer {self.api_key}"} async with session.get(url, headers=headers) as response: return await response.text() ``` #### Generator Methods ```python from typing import Iterator @daft.cls class TokenGenerator: def __init__(self, tokenizer_name: str): from transformers import AutoTokenizer self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name) def tokenize(self, text: str) -> Iterator[str]: tokens = self.tokenizer.tokenize(text) for token in tokens: yield token tokenizer = TokenGenerator("bert-base-uncased") df = daft.from_pydict({"text": ["Hello world", "Daft is great"]}) # Each row produces multiple tokens df = df.select("text", tokenizer.tokenize(df["text"]).alias("token")) ``` ### Multiple Instances You can create multiple instances of the same class with different configurations: ```python @daft.cls class Normalizer: def __init__(self, mean: float, std: float): self.mean = mean self.std = std def normalize(self, value: float) -> float: return (value - self.mean) / self.std normalizer_a = Normalizer(mean=10.0, std=2.0) normalizer_b = Normalizer(mean=50.0, std=5.0) df = daft.from_pydict({ "metric_a": [8, 10, 12], "metric_b": [45, 50, 55] }) df = df.select( normalizer_a.normalize(df["metric_a"]).alias("norm_a"), normalizer_b.normalize(df["metric_b"]).alias("norm_b") ) ``` ### Eager Execution Call methods with scalar arguments to execute immediately: ```python @daft.cls class Calculator: def __init__(self, multiplier: int): self.multiplier = multiplier def __call__(self, x: int) -> int: return x * self.multiplier calc = Calculator(10) # Lazy execution - returns Expression expr = calc(df["value"]) # Eager execution - initializes instance and returns result result = calc(5) # Returns 50 ``` ### Best Practices 1. **Costly Initialization**: Use `@daft.cls` when some an expensive initialization step can be reused across multiple rows (e.g., loading models, establishing connections). The initialization cost is amortized across all rows processed by each worker. 2. **Simple Functions**: Use `@daft.func` for operations that don't require expensive setup. 3. **Resource Management**: Request GPUs only when needed with the `gpus` parameter 4. **Concurrency**: Set `max_concurrency` to limit the number of concurrent instances. 5. **Process Isolation**: Use `use_process=True` to run each instance in a separate process. This is useful for isolating instances when they are not thread-safe or to improve performance by avoiding GIL contention. ### Batch Methods with `@daft.method.batch` Similar to `@daft.func.batch`, use `@daft.method.batch` for batch processing in Daft classes: ```python import daft from daft import DataType, Series @daft.cls class Model: def __init__(self, model_name: str): self.model = load_model(model_name) @daft.method.batch(return_dtype=DataType.int64()) def predict(self, x: Series) -> Series: predictions = self.model.predict(x.to_arrow().to_numpy()) return predictions model = Model("resnet50") df = daft.from_pydict({"x": [1, 2, 3]}) df = df.select(model.predict(df["x"])) ``` ### Batch Sizing You can configure the maximum number of rows in each batch by setting the `batch_size` parameter: ```python @daft.func.batch(return_dtype=DataType.int64(), batch_size=1024) def batch_func_with_batch_size(x: Series) -> Series: assert len(x) <= 1024 return x ``` Considerations for tuning batch size: - **Avoiding out-of-memory errors**: Large batches can exhaust memory, especially with large rows (e.g., images, embeddings) or when using GPUs. - **Improving performance**: The batch size for your function depends on your workload. Experiment to find the optimal size. - **GPU utilization**: Match your batch size to your model's preferred inference batch size for best GPU performance. --- ### Custom Code Func (`docs/custom-code/func.md`) # Stateless UDFs with `@daft.func` When Daft's built-in functions aren't sufficient for your needs, the `@daft.func` and `@daft.cls` decorators let you run your own Python code over each row of data. Simply decorate a Python function or class, and it becomes usable in Daft DataFrame operations. ## Quick Example ```python import daft @daft.func def add_and_format(a: int, b: int) -> str: return f"Sum: {a + b}" df = daft.from_pydict({"x": [1, 2, 3], "y": [4, 5, 6]}) df = df.select(add_and_format(df["x"], df["y"])) df.show() ``` ``` ╭───────────╮ │ x │ │ --- │ │ Utf8 │ ╞═══════════╡ │ Sum: 5 │ ├╌╌╌╌╌╌╌╌╌╌╌┤ │ Sum: 7 │ ├╌╌╌╌╌╌╌╌╌╌╌┤ │ Sum: 9 │ ╰───────────╯ ``` ## Function Variants Daft supports multiple function variants to optimize for different use cases: - **Row-wise** (default): Regular Python functions process one row at a time - **Async row-wise**: Async Python functions process rows concurrently - **Generator**: Generator functions produce multiple output rows per input row - **Batch** (`@daft.func.batch`): Process entire batches of data with `daft.Series` for high performance Daft automatically detects which variant to use for regular functions based on your function signature. For batch functions, you must use the `@daft.func.batch` decorator. ### Row-wise Functions Row-wise functions are the default variant. They process one row at a time and return one value per row. ```python import daft @daft.func def multiply(a: int, b: int) -> int: return a * b df = daft.from_pydict({"x": [1, 2, 3], "y": [10, 20, 30]}) df = df.select(multiply(df["x"], df["y"])) df.show() ``` ``` ╭───────╮ │ x │ │ --- │ │ Int64 │ ╞═══════╡ │ 10 │ ├╌╌╌╌╌╌╌┤ │ 40 │ ├╌╌╌╌╌╌╌┤ │ 90 │ ╰───────╯ ``` #### Type Inference Daft automatically infers the return type from your function's type hint: ```python @daft.func def tokenize(text: str) -> list[int]: vocab = {char: i for i, char in enumerate(set(text))} return [vocab[char] for char in text] df = daft.from_pydict({"text": ["hello", "world"]}) df = df.select(tokenize(df["text"])) # The return type is automatically inferred as List[Int64] print(df.schema()) ``` If you need to override the inferred type, use the `return_dtype` parameter: ```python @daft.func(return_dtype=daft.DataType.int32()) def add(a: int, b: int) -> int: return a + b ``` #### Mixing Expressions and Literals You can mix DataFrame expressions with literal values: ```python @daft.func def add_constant(value: int, constant: int) -> int: return value + constant df = daft.from_pydict({"x": [1, 2, 3]}) df = df.select(add_constant(df["x"], 100)) # constant is a literal df.show() ``` ``` ╭───────╮ │ x │ │ --- │ │ Int64 │ ╞═══════╡ │ 101 │ ├╌╌╌╌╌╌╌┤ │ 102 │ ├╌╌╌╌╌╌╌┤ │ 103 │ ╰───────╯ ``` #### Keyword Arguments Functions with default arguments work as expected: ```python @daft.func def format_number(value: int, prefix: str = "$", suffix: str = "") -> str: return f"{prefix}{value}{suffix}" df = daft.from_pydict({"amount": [10, 20, 30]}) # Use defaults df.select(format_number(df["amount"])).show() # Override with literals df.select(format_number(df["amount"], prefix="€", suffix=" EUR")).show() # Override with expressions df.select(format_number(df["amount"], suffix=df["amount"].cast(daft.DataType.string()))).show() ``` #### Eager Evaluation When called without any expressions, functions execute immediately: ```python @daft.func def add(a: int, b: int) -> int: return a + b # This executes immediately and returns 8 result = add(3, 5) print(result) # 8 # This returns a Daft Expression expr = add(df["x"], df["y"]) ``` ### Async Row-wise Functions Decorate async functions to enable concurrent execution across rows: ```python import daft import asyncio import aiohttp @daft.func async def fetch_url(url: str) -> str: async with aiohttp.ClientSession() as session: async with session.get(url) as response: return await response.text() df = daft.from_pydict({ "urls": [ "https://api.example.com/1", "https://api.example.com/2", "https://api.example.com/3", ] }) # Requests are made concurrently df = df.select(fetch_url(df["urls"])) ``` Use `max_concurrency` to limit the number of concurrent coroutines, for example to rate-limit API calls: ```python @daft.func(max_concurrency=10) async def fetch_url(url: str) -> str: async with aiohttp.ClientSession() as session: async with session.get(url) as response: return await response.text() ``` ### Generator Functions Generator functions use `yield` to produce multiple output rows per input row. Other columns in the DataFrame are automatically broadcast to match the number of generated values. You may only use one generator function per DataFrame operation. ```python import daft from typing import Iterator @daft.func def repeat_value(value: str, count: int) -> Iterator[str]: for _ in range(count): yield value df = daft.from_pydict({ "id": [1, 2, 3], "word": ["hello", "world", "daft"], "times": [2, 3, 1] }) df = df.select("id", repeat_value(df["word"], df["times"])) df.show() ``` ``` ╭───────┬───────╮ │ id ┆ word │ │ --- ┆ --- │ │ Int64 ┆ Utf8 │ ╞═══════╪═══════╡ │ 1 ┆ hello │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 1 ┆ hello │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 2 ┆ world │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 2 ┆ world │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 2 ┆ world │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 3 ┆ daft │ ╰───────┴───────╯ ``` Notice how the `id` column values are repeated to match the number of generated values. #### Type Hints for Generators Use `Iterator[T]` or `Generator[T, None, None]` type hints to indicate the yielded type: ```python from typing import Iterator @daft.func def split_text(text: str) -> Iterator[str]: for word in text.split(): yield word ``` Alternatively, specify the return type explicitly: ```python @daft.func(return_dtype=daft.DataType.string()) def split_text(text: str): for word in text.split(): yield word ``` #### Handling Empty Generators If a generator yields no values for a particular input row, a null value is inserted: ```python @daft.func def yield_if_positive(value: int) -> Iterator[int]: if value > 0: yield value df = daft.from_pydict({"x": [-1, 0, 5, 10]}) df = df.select(yield_if_positive(df["x"])) df.show() ``` ``` ╭───────╮ │ x │ │ --- │ │ Int64 │ ╞═══════╡ │ None │ ├╌╌╌╌╌╌╌┤ │ None │ ├╌╌╌╌╌╌╌┤ │ 5 │ ├╌╌╌╌╌╌╌┤ │ 10 │ ╰───────╯ ``` ## Batch UDFs with `@daft.func.batch` For performance-critical operations, batch UDFs process entire batches of data at once using `daft.Series` objects instead of individual values. This allows you to leverage optimized libraries like PyArrow or NumPy for efficient vectorized operations. ### When to Use Batch UDFs Use batch UDFs when: - **Performance is critical**: Vectorized operations may be significantly faster than row-wise processing - **Working with optimized libraries**: You want to use PyArrow compute functions, NumPy operations, or other libraries that support vectorized operations on batch data - **Running batch inference**: You are running a model that supports batch inference ### Basic Batch UDF Batch UDFs receive `daft.Series` objects as arguments and return a `daft.Series`, `list`, `numpy.ndarray`, or `pyarrow.Array`: ```python import daft from daft import DataType, Series @daft.func.batch(return_dtype=DataType.int64()) def add_series(a: Series, b: Series) -> Series: import pyarrow.compute as pc # Convert to PyArrow for efficient computation a_arrow = a.to_arrow() b_arrow = b.to_arrow() result = pc.add(a_arrow, b_arrow) return result df = daft.from_pydict({"x": [1, 2, 3], "y": [4, 5, 6]}) df = df.select(add_series(df["x"], df["y"])) df.show() ``` ``` ╭───────╮ │ x │ │ --- │ │ Int64 │ ╞═══════╡ │ 5 │ ├╌╌╌╌╌╌╌┤ │ 7 │ ├╌╌╌╌╌╌╌┤ │ 9 │ ╰───────╯ ``` ### Mixing Series and Scalar Arguments Batch UDFs can accept both Series and scalar arguments. Scalar arguments are passed through without modification: ```python @daft.func.batch(return_dtype=DataType.int64()) def add_constant(a: Series, constant: int) -> Series: import pyarrow.compute as pc a_arrow = a.to_arrow() result = pc.add(a_arrow, constant) return result df = daft.from_pydict({"x": [1, 2, 3]}) df = df.select(add_constant(df["x"], 100)) df.show() ``` ``` ╭───────╮ │ x │ │ --- │ │ Int64 │ ╞═══════╡ │ 101 │ ├╌╌╌╌╌╌╌┤ │ 102 │ ├╌╌╌╌╌╌╌┤ │ 103 │ ╰───────╯ ``` ### Eager Evaluation Like regular functions, batch UDFs execute immediately when called with scalars: ```python @daft.func.batch(return_dtype=DataType.int64()) def multiply(a: Series, b: Series) -> Series: import pyarrow.compute as pc a_arrow = a.to_arrow() b_arrow = b.to_arrow() result = pc.multiply(a_arrow, b_arrow) return result # Lazy execution - returns Expression expr = multiply(df["x"], df["y"]) # Eager execution - computes immediately result = multiply(5, 10) # Returns 50 ``` ## Resources, Concurrency, and Error Handling `@daft.func` and `@daft.func.batch` accept a common set of keyword arguments for concurrency control, scheduling, and error handling: ```python @daft.func( cpus=2, # Each invocation needs 2 CPUs gpus=1, # Each invocation needs 1 GPU use_process=True, # Run the function in a subprocess instead of a thread max_retries=3, # Retry a failing invocation up to 3 times with backoff on_error="log", # On final failure, log the exception and emit None ) def call_flaky_api(url: str) -> str: import requests return requests.get(url).text ``` ### `cpus` and `gpus` `cpus` and `gpus` are used for concurrency control and scheduling — not placement. Daft uses them to decide how many invocations of your function can run in parallel on a given machine. If you annotate a function with `cpus=2` and it runs on a machine with 4 CPUs, Daft runs at most 2 invocations in parallel on that machine. `gpus` works the same way: `@daft.func(gpus=1)` on a machine with 2 GPUs means at most 2 concurrent invocations. Both accept fractional values (e.g. `cpus=0.5`, `gpus=0.5`). `gpus` values above 1.0 must be integers. ### `max_concurrency` `max_concurrency` controls the maximum number of concurrent invocations of an async `@daft.func` or `@daft.func.batch`. It does not apply to synchronous (non-async) UDFs — setting it on a sync function raises. ```python @daft.func(max_concurrency=10) async def fetch_url(url: str) -> str: async with aiohttp.ClientSession() as session: async with session.get(url) as response: return await response.text() ``` ### `use_process` Runs the function in a subprocess instead of a thread on the main process. Use this when your function is not thread-safe or holds the GIL heavily. The default (`None`) lets Daft pick at runtime based on observed performance. ### `max_retries` and `on_error` Control what happens when a function invocation raises an exception. - `max_retries=N` retries failing invocations up to `N` times with exponential backoff starting at 100 ms, doubling each attempt, capped at 60 s, with ±25% jitter. If the raised exception is a `daft.ai.utils.RetryAfterError`, the specified retry-after delay is honored instead of the default backoff. - `on_error` decides what to do after retries are exhausted: - `"raise"` (default) — fail the query. - `"log"` — log the exception and emit `None` for that invocation. - `"ignore"` — silently emit `None` for that invocation. ## Advanced Features ### Unnesting Struct Returns When your function returns a struct (dictionary), you can use `unnest=True` to automatically expand the struct fields into separate columns: ```python import daft @daft.func( return_dtype=daft.DataType.struct({ "first": daft.DataType.string(), "last": daft.DataType.string(), "age": daft.DataType.int64() }), unnest=True ) def parse_person(full_name: str, age: int): parts = full_name.split() return {"first": parts[0], "last": parts[1], "age": age} df = daft.from_pydict({ "name": ["Alice Smith", "Bob Jones"], "age": [30, 25] }) df = df.select(parse_person(df["name"], df["age"])) df.show() ``` ``` ╭───────┬───────┬───────╮ │ first ┆ last ┆ age │ │ --- ┆ --- ┆ --- │ │ Utf8 ┆ Utf8 ┆ Int64 │ ╞═══════╪═══════╪═══════╡ │ Alice ┆ Smith ┆ 30 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ Bob ┆ Jones ┆ 25 │ ╰───────┴───────┴───────╯ ``` Without `unnest=True`, you would get a single column containing struct values. ### Combining Generators with Unnest You can combine generator functions with `unnest=True` to yield multiple structs that get expanded into columns: ```python from typing import Iterator @daft.func( return_dtype=daft.DataType.struct({ "index": daft.DataType.int64(), "char": daft.DataType.string() }), unnest=True ) def enumerate_chars(text: str) -> Iterator[dict]: for i, char in enumerate(text): yield {"index": i, "char": char} df = daft.from_pydict({"word": ["hi", "bye"]}) df = df.select(enumerate_chars(df["word"])) df.show() ``` ``` ╭───────┬──────╮ │ index ┆ char │ │ --- ┆ --- │ │ Int64 ┆ Utf8 │ ╞═══════╪══════╡ │ 0 ┆ h │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌┤ │ 1 ┆ i │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌┤ │ 0 ┆ b │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌┤ │ 1 ┆ y │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌┤ │ 2 ┆ e │ ╰───────┴──────╯ ``` --- ### Custom Code Gpu (`docs/custom-code/gpu.md`) # Working with GPUs For GPU work, always reach for `@daft.cls`: it initializes your model once in `__init__` and reuses it across rows, so the expensive model load is amortized over the whole query. Request a GPU with the `gpus` parameter on the class decorator. ## Basic pattern ```python import daft from daft import DataType, Series @daft.cls(gpus=1) class Embedder: def __init__(self, model_name: str): from sentence_transformers import SentenceTransformer self.model = SentenceTransformer(model_name).cuda() @daft.method.batch(return_dtype=DataType.list(DataType.float32()), batch_size=64) def encode(self, text: Series) -> list[list[float]]: return self.model.encode(text.to_pylist()).tolist() embedder = Embedder("all-MiniLM-L6-v2") df = daft.from_pydict({"text": ["hello", "world", "daft"]}) df = df.select(embedder.encode(df["text"])) ``` The class is initialized once per Python worker; the model stays resident on the GPU for the whole query. ## Packing multiple invocations per GPU `gpus` accepts fractional values up to 1.0. Use this when a single invocation cannot saturate the GPU — for example, a small model with meaningful preprocessing overhead. ```python @daft.cls(gpus=0.5, max_concurrency=2) class SmallModel: def __init__(self, name: str): import torch self.model = torch.load(name).cuda() @daft.method.batch(return_dtype=DataType.float64(), batch_size=16) def infer(self, x: Series) -> Series: return self.model(x.to_arrow().to_numpy()) ``` With `gpus=0.5` and `max_concurrency=2`, two instances share one physical GPU. Daft isolates `CUDA_VISIBLE_DEVICES` per process, so device ordinals stay consistent inside each instance. Values above 1.0 must be integers (e.g. `gpus=2` for model parallelism across two GPUs). ## Examples by model family ### Sentence embeddings (Sentence Transformers) ```python @daft.cls(gpus=1) class SentenceEmbedder: def __init__(self, model_name: str = "BAAI/bge-large-en-v1.5"): from sentence_transformers import SentenceTransformer self.model = SentenceTransformer(model_name, device="cuda") @daft.method.batch(return_dtype=DataType.list(DataType.float32()), batch_size=128) def embed(self, text: Series) -> list[list[float]]: return self.model.encode(text.to_pylist(), batch_size=128).tolist() ``` ### Image classification / CLIP (Transformers) ```python @daft.cls(gpus=1) class CLIPScorer: def __init__(self, model_name: str = "openai/clip-vit-base-patch32"): import torch from transformers import CLIPModel, CLIPProcessor self.model = CLIPModel.from_pretrained(model_name).to("cuda").eval() self.processor = CLIPProcessor.from_pretrained(model_name) self.torch = torch @daft.method.batch(return_dtype=DataType.list(DataType.float32()), batch_size=32) def embed_images(self, image_bytes: Series) -> list[list[float]]: from io import BytesIO from PIL import Image images = [Image.open(BytesIO(b)).convert("RGB") for b in image_bytes.to_pylist()] inputs = self.processor(images=images, return_tensors="pt").to("cuda") with self.torch.no_grad(): features = self.model.get_image_features(**inputs) return features.cpu().tolist() ``` ### Text generation (vLLM) For production-grade LLM inference, Daft ships a built-in vLLM integration — see [AI Functions › Prompt](../ai-functions/prompt.md). For smaller custom generation tasks: ```python @daft.cls(gpus=1) class Generator: def __init__(self, model_name: str = "Qwen/Qwen2.5-1.5B-Instruct"): import torch from transformers import AutoModelForCausalLM, AutoTokenizer self.tokenizer = AutoTokenizer.from_pretrained(model_name) self.model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype=torch.bfloat16, device_map="cuda" ).eval() self.torch = torch @daft.method(return_dtype=DataType.string()) def generate(self, prompt: str) -> str: inputs = self.tokenizer(prompt, return_tensors="pt").to("cuda") with self.torch.no_grad(): output = self.model.generate(**inputs, max_new_tokens=128) return self.tokenizer.decode(output[0], skip_special_tokens=True) ``` ### Whisper (audio transcription) ```python @daft.cls(gpus=1) class Whisper: def __init__(self, model_name: str = "openai/whisper-large-v3"): import torch from transformers import pipeline self.pipe = pipeline( "automatic-speech-recognition", model=model_name, torch_dtype=torch.float16, device="cuda", ) @daft.method(return_dtype=DataType.string()) def transcribe(self, audio_path: str) -> str: return self.pipe(audio_path)["text"] ``` ## Choosing a batch size GPU throughput is batch-size sensitive. A few rules of thumb: - Start by matching the batch size to the model's preferred inference batch size. - Halve it if you hit out-of-memory errors — large rows (images, embeddings) eat VRAM fast. - Increase it gradually while watching GPU utilization: if you are under 80%, the GPU is waiting on you. See [Classes & Methods › Batch Sizing](cls.md#batch-sizing) for additional considerations. ## Retries and error handling GPU inference can fail for reasons unrelated to your code — OOM, transient driver errors, preempted instances. Use `max_retries` and `on_error` to keep long queries going: ```python @daft.cls(gpus=1, max_retries=2, on_error="log") class Model: ... ``` --- ### Custom Code Migration (`docs/custom-code/migration.md`) # Migrating to Daft's new UDF API Daft now offers a new UDF API via the `@daft.func` and `@daft.cls` decorators, replacing the legacy `@daft.udf` decorator, which was removed in Daft 0.8.0. The new API is more powerful and Pythonic. This guide will walk you through the steps to migrate your existing UDFs to the new API. ## Function UDF Python functions decorated with `@daft.udf` can be easily converted into a batch function using the `@daft.func.batch` decorator. ```python hl_lines="7" # ====== Legacy API ====== @daft.udf(...) def my_func(a: daft.Series, b: daft.Series): ... # ====== New API ====== @daft.func.batch(...) def my_func(a: daft.Series, b: daft.Series): ... ``` Then, they can be used exactly the same as before. Just like with legacy UDFs, inputs to batch functions are `daft.Series` objects, and the same return types are supported as well: `daft.Series`, `list`, `numpy.ndarray`, or `pyarrow.Array`. Most decorator parameters also have equivalent parameters in the legacy UDF decorator. See the [Decorator Parameters](#decorator-parameters) section for more details. ## Class UDF ### Creation Python classes decorated with `@daft.udf` can be easily converted into a class UDF with a batch method using the `@daft.cls` and `@daft.method.batch` decorators. ```python hl_lines="11 16" # ====== Legacy API ====== @daft.udf(...) class MyClass: def __init__(self): ... def __call__(self, value: daft.Series): ... # ====== New API ====== @daft.cls(...) class MyClass: def __init__(self): ... @daft.method.batch(...) def __call__(self, value: daft.Series): ... ``` ### Usage Unlike legacy class UDFs, which can be called directly as a function, for new class UDFs, you must first create an instance of the class and then call the method on the instance. ```python hl_lines="7 9" # ====== Legacy API ====== df.select( MyClass(df["value"]) ) # ====== New API ====== my_instance = MyClass() df.select( my_instance(df["value"]) ) ``` Instead of using `.with_init_args(...)` to specify the arguments to the `__init__` method, you can set those arguments in the construction of the class instance instead. ```python hl_lines="8 10" # ====== Legacy API ====== MyClassWithArgs = MyClass.with_init_args(arg1=1, arg2=2) df.select( MyClassWithArgs(df["value"]) ) # ====== New API ====== my_instance_with_args = MyClass(arg1=1, arg2=2) df.select( my_instance_with_args(df["value"]) ) ``` !!! note Class Methods Whereas legacy class UDFs require the implementation of the `__call__` method, new class UDFs allow you to implement other methods in the class and use them as UDFs. For example, here is a class with a `generate` method that can be used as a UDF: ```python @daft.cls(...) class MyClass: def __init__(self): ... def generate(self, value: daft.Series): ... ``` Then, you can use the `generate` method as follows: ```python my_instance = MyClass() df.select( my_instance.generate(df["value"]) ) ``` ## Decorator Parameters The following parameters stay the same between the legacy and new APIs: - **return_dtype** - **batch_size** - **use_process** - **ray_options** For these parameters, here's what you can do with the new API: - **concurrency**: The new API offers a `max_concurrency` parameter instead, which guarantees that at most `max_concurrency` instances of the UDF will be running at any given time, instead of exactly `concurrency` instances. - **num_cpus**: The new API offers a `cpus` parameter on `@daft.func`, `@daft.func.batch`, and `@daft.cls` with the same placement semantics. Fractional values (e.g. `0.5`) are supported. - **num_gpus**: The new API offers a `gpus` parameter on `@daft.func`, `@daft.func.batch`, and `@daft.cls` with the same placement semantics. Fractional values up to 1.0 are supported. - **memory_bytes**: The new API currently has no replacement for `memory_bytes`. `ray_options={"memory": ...}` is explicitly rejected (see [#6711](https://github.com/Eventual-Inc/Daft/issues/6711)). If you were using `memory_bytes` primarily to limit concurrency, use `max_concurrency` instead. ### New parameters (no legacy equivalent) The new API adds two parameters for controlling error handling that had no equivalent in `@daft.udf`: - **max_retries**: Retry failing calls with exponential backoff (100 ms → 60 s, ±25% jitter). Also honors `daft.ai.utils.RetryAfterError` for rate-limit-aware retries. - **on_error**: `"raise"` (default), `"log"`, or `"ignore"`. Controls behavior once retries are exhausted — `"log"` and `"ignore"` emit `None` for the failing row so the query keeps running. Both are available on `@daft.func`, `@daft.func.batch`, and `@daft.cls`. (`@daft.method` / `@daft.method.batch` accept the kwargs but currently ignore them — set the class-level value on `@daft.cls` instead.) See the [Resources, Concurrency, and Error Handling](func.md#resources-concurrency-and-error-handling) section for details. ## New Features Here are some features that are available in the new API that may help simplify your code and improve performance during your migration. See the main [Function UDF](./func.md) and [Class UDF](./cls.md) pages for a detailed description of the new API. ### Row-wise Functions If you find that your UDF is simply iterating over the rows of the input data and computing a result for each row without any vectorized or batch operations, consider implementing it as a row-wise function using the `@daft.func` decorator. Row-wise functions receive a single row of input data at a time, and return a single value for that row. Daft will automatically handle batching and conversion between Daft and Python types under the hood. Example: ```python # ====== Legacy Batch API ====== @daft.udf(return_dtype=daft.DataType.int64()) def my_sum(a: daft.Series, b: daft.Series): result = [] for a_val, b_val in zip(a, b): result.append(a_val + b_val) return result # ====== New Row-wise API ====== @daft.func def my_sum(a: int, b: int) -> int: return a + b ``` ### Return Type Inference With row-wise functions, Daft will also automatically infer the return type of the function based on the Python type annotations. For example, in the above example, by specifying `-> int`, Daft will automatically infer the return dtype to be `daft.DataType.int64()`. The `return_dtype` parameter is still supported, but it is not required. See the [Type Conversions](../api/datatypes/type_conversions.md#python-to-daft) page for a mapping from Python types to Daft types. ### Async Functions The new UDF API supports async Python functions natively for both row-wise and batch functions. Simply specify the `async` keyword in front of the function definition and use it like a regular Daft function. Daft will handle the asynchronous execution of the function under the hood, so if you are calling to async functions from within your UDF, you can now just `await` them directly. Example: ```python @daft.func async def my_api_call(prompt: str) -> str: response = await make_api_call(prompt) return response.text ``` ## Known Limitations - The new API does not yet expose a `memory_bytes` parameter, and `ray_options={"memory": ...}` is explicitly rejected ([#6711](https://github.com/Eventual-Inc/Daft/issues/6711)). If you were using `memory_bytes` primarily to bound concurrency, prefer `max_concurrency`. True memory-based placement on Ray is not currently supported by the new API; follow [#6711](https://github.com/Eventual-Inc/Daft/issues/6711) for updates. If you have any questions or feedback about the new UDF API, please submit an [issue on GitHub](https://github.com/Eventual-Inc/Daft/issues) or reach out to us on [Slack](https://daft.ai/slack). --- ### Custom Code Udaf (`docs/custom-code/udaf.md`) # Aggregate UDFs with `@daft.udaf` When Daft's built-in aggregation functions (sum, mean, count, etc.) aren't sufficient, `@daft.udaf` lets you define custom aggregations in Python. UDAFs work with `groupby().agg()` and global `agg()`, and support Daft's three-stage aggregation pipeline for efficient distributed execution. ## Quick Example ```python import daft from daft import DataType, Series @daft.udaf(return_dtype=DataType.float64(), state=DataType.float64()) class MySum: def aggregate(self, values: Series) -> float: return sum(values.to_pylist()) def combine(self, states: Series) -> float: return sum(states.to_pylist()) def finalize(self, state: float) -> float: return state my_sum = MySum() df = daft.from_pydict({"cat": ["a", "a", "b", "b"], "val": [1.0, 2.0, 3.0, 4.0]}) df.groupby("cat").agg(my_sum(daft.col("val")).alias("total")).show() ``` ``` ╭──────┬─────────╮ │ cat ┆ total │ │ --- ┆ --- │ │ Utf8 ┆ Float64 │ ╞══════╪═════════╡ │ a ┆ 3.0 │ ├╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ b ┆ 7.0 │ ╰──────┴─────────╯ ``` ## How It Works A UDAF class defines a three-stage aggregation pipeline: ```text Aggregation: aggregate(inputs) -> partial state Combination: combine(states) -> merged state (associative & commutative) Finalization: finalize(state) -> final output ``` 1. **`aggregate(*inputs: Series) -> value | dict`** — Aggregation stage. Receives input columns as `Series` objects, returns a partial state value. 2. **`combine(states: Series | dict[str, Series]) -> value | dict`** — Combination stage. Merges multiple partial states into one. Must be commutative and associative. 3. **`finalize(state: value | dict) -> value`** — Finalization stage. Converts the final merged state into the output value. Intermediate state is typed: the `state` parameter declares one data type per state component. The framework carries state between stages using these types, which lets Arrow and the query planner reason about intermediate results. Daft's planner automatically decomposes UDAFs into aggregation and finalization stages so partial aggregation happens close to the data. ## Single-State UDAF For simple accumulators, pass a single `DataType` as `state`: ```python @daft.udaf(return_dtype=DataType.float64(), state=DataType.float64()) class MySum: def aggregate(self, values: Series) -> float: return sum(values.to_pylist()) def combine(self, states: Series) -> float: return sum(states.to_pylist()) def finalize(self, state: float) -> float: return state ``` - `combine` receives a `Series` of partial state values - `finalize` receives a single state value ## Multi-State UDAF For aggregations that need to track multiple fields (e.g., both a sum and a count for computing a mean), pass a dict of `{name: DataType}` as `state`: ```python @daft.udaf( return_dtype=DataType.float64(), state={"sum": DataType.float64(), "count": DataType.int64()}, ) class MyMean: def aggregate(self, values: Series) -> dict: vals = values.to_pylist() return {"sum": float(sum(vals)), "count": len(vals)} def combine(self, states: dict[str, Series]) -> dict: return { "sum": float(sum(states["sum"].to_pylist())), "count": int(sum(states["count"].to_pylist())), } def finalize(self, state: dict) -> float: return state["sum"] / state["count"] ``` - `aggregate` returns a dict with one key per state field - `combine` receives a dict mapping field names to `Series` of partial values - `finalize` receives a dict mapping field names to single values ## Parameterized UDAF UDAFs can accept constructor arguments via `__init__`: ```python @daft.udaf(return_dtype=DataType.float64(), state=DataType.float64()) class BoundedSum: def __init__(self, max_val: float): self.max_val = max_val def aggregate(self, values: Series) -> float: return float(sum(min(v, self.max_val) for v in values.to_pylist())) def combine(self, states: Series) -> float: return sum(states.to_pylist()) def finalize(self, state: float) -> float: return state bounded_sum = BoundedSum(max_val=10.0) df.groupby("cat").agg(bounded_sum(daft.col("val")).alias("total")) ``` ## Multi-Input UDAF UDAFs can consume multiple input columns: ```python @daft.udaf(return_dtype=DataType.float64(), state=DataType.float64()) class WeightedSum: def aggregate(self, values: Series, weights: Series) -> float: v = values.to_pylist() w = weights.to_pylist() return float(sum(a * b for a, b in zip(v, w))) def combine(self, states: Series) -> float: return sum(states.to_pylist()) def finalize(self, state: float) -> float: return state ws = WeightedSum() df.groupby("cat").agg(ws(daft.col("val"), daft.col("weight")).alias("weighted_total")) ``` ## Global Aggregation UDAFs work without `groupby` for whole-table aggregation: ```python my_sum = MySum() df = daft.from_pydict({"val": [1.0, 2.0, 3.0, 4.0]}) df.agg(my_sum(daft.col("val")).alias("total")).show() ``` ``` ╭─────────╮ │ total │ │ --- │ │ Float64 │ ╞═════════╡ │ 10.0 │ ╰─────────╯ ``` ## Multiple UDAFs in One Aggregation You can use multiple UDAFs (or the same UDAF on different columns) in a single `.agg()` call: ```python my_sum = MySum() my_mean = MyMean() df.groupby("cat").agg( my_sum(daft.col("val")).alias("total"), my_mean(daft.col("val")).alias("avg"), ).show() ``` --- ### Custom Code Udfs (`docs/custom-code/udfs.md`) # Legacy User-Defined Functions (UDFs) !!! danger "Removed in 0.8.0" The legacy `@daft.udf` API was deprecated in Daft 0.7.0 and **removed in Daft 0.8.0**. Use [`@daft.func`](func.md) and [`@daft.cls`](cls.md) instead, and see the [migration guide](migration.md) for how to update your code. --- ### Datasets Common Crawl (`docs/datasets/common-crawl.md`) # How to use Common Crawl with Daft [Common Crawl](https://commoncrawl.org/get-started) is one of the most important open web datasets, containing more than 250 billion web pages that span 18 years of crawls. Since 2020, it has become a critical source of training data for generative AI, with the vast majority of data used to train models like GPT-3 coming from Common Crawl. [Mozilla Foundation's research](https://www.mozillafoundation.org/en/research/library/generative-ai-training-data/common-crawl/) noted that "Generative AI in its current form would probably not be possible without Common Crawl". Daft provides a simple, performant, and responsible way to access Common Crawl data. !!! warning "Warning" These APIs are in beta and may be subject to change as the Common Crawl dataset continues to be developed. ## Access Methods There are 3 main methods for accessing Common Crawl data: | Method | URL Scheme | Best For | Credentials Required | Warnings | |--------|------------|----------|----------------------|----------| | AWS S3 | ``s3://commoncrawl/...`` | Inside AWS (us-east-1) | Yes | Data transfer fees apply if outside of AWS us-east-1. | | HuggingFace Buckets | ``hf://buckets/commoncrawl/commoncrawl/...`` | Cross-region, outside AWS, cheapest | No (Optional, preferred) | Only 2026+ crawls are available | | HTTPS | ``https://data.commoncrawl.org/...`` | Fallback when no credentials | No | Slowest option | The access method chosen depends on the arguments passed to `daft.datasets.common_crawl` * If `in_aws=True` or `source="s3"`, Daft will read from AWS S3. * If `source="hf"` or `source=None` and the `crawl` ID is 2026 or later, Daft will read from the official HuggingFace bucket. * Otherwise, Daft will read from the HTTPS endpoint. !!! warning "Warning" `in_aws` is deprecated and will be removed in `v0.9.0`. Use `source="s3"` instead. ## Accessing Common Crawl from AWS Common Crawl data is hosted by the [Amazon Web Services' Open Data Sets Sponsorships program](https://aws.amazon.com/opendata/) which makes it freely accessible. However, access does require AWS authentication when downloading Common Crawl data from S3 directly. (_Outside of AWS, you can access Common Crawl without an AWS account_). **NOTE**: When using `daft.datasets.common_crawl`, you _must_ provide `in_aws=True` when accessing data within the AWS Cloud! All Common Crawl data is stored in the `us-east-1` region. It is **highly recommended** to access the data from an AWS service in the same region. From the [Common Crawl website](https://commoncrawl.org/get-started): > The connection to S3 should be faster and you avoid the minimal fees for inter-region data transfer (you have to send requests which are charged as outgoing traffic). !!! warning Using S3 from outside AWS incurs data transfer egress fees. Consider using the HuggingFace or HTTP sources instead. ### Authentication option 1: AWS Configurations If your environment has AWS credentials configured, Daft will automatically detect and use them. The default order of precedence is: 1. AWS environment variables (e.g. `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`) 2. Shared AWS CLI or SSO credentials (e.g. `~/.aws/config`, `~/.aws/credentials`) 3. IAM roles for EC2, ECS, EKS, or other AWS services ### Authentication option 2: Configure IOConfig ```python import daft from daft.io import IOConfig, S3Config io_config = IOConfig( s3=S3Config( key_id="your_access_key", access_key="your_secret_key", session_token="your_session_token", region_name="us-east-1", # Access Common Crawl data where it's located. ) ) # Use io_config when reading from the Common Crawl dataset daft.datasets.common_crawl("CC-MAIN-2025-33", io_config=io_config, in_aws=True) # NOTE: When using `daft.datasets.common_crawl`, you _must_ provide `in_aws=True` when accessing data within the AWS Cloud! ``` ## Accessing Common Crawl from HuggingFace If you are running _outside_ of AWS, we recommend using the [Hugging Face Buckets](#access-from-hugging-face-buckets) interface as it is the most accessible option. Daft will read from the official [HuggingFace Common Crawl bucket](https://huggingface.co/buckets/commoncrawl/commoncrawl) if the crawl is available or if `source="hf"` is set. Currently, only crawls from 2026 onwards are available. ```python import daft daft.datasets.common_crawl("CC-MAIN-2026-25", source="hf") ``` You can also access WARC files directly via `daft.read_warc`: ```python import daft # Read a single WARC file from HF Buckets daft.read_warc("hf://buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17/segments/1775805908305.14/warc/CC-MAIN-20260410081153-20260410111153-00000.warc.gz") # Read WARC files matching a glob pattern daft.read_warc("hf://buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17/segments/*/warc/*.warc.gz") ``` ### Authentication For the public Common Crawl data, no authentication is needed; however, this may lead to more aggressive rate limiting. For best access, pass a token via ``IOConfig``: ```python from daft.io import IOConfig daft.read_warc( "hf://buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2025-33/segments/...", io_config=IOConfig(hf=daft.io.HuggingFaceConfig(token="hf_xxxxxxxxxxxxxxxxxxxx")) ) ``` You can also authenticate using ``hf auth login``; the token is picked up automatically. ## Accessing Common Crawl from HTTP As a fallback, we also support reading from the HTTP endpoint. This is the slowest option and is not recommended for production use, but is useful for testing and development. ```python import daft # Use HTTPS as a fallback daft.datasets.common_crawl("CC-MAIN-2025-33", source="http") ``` ## Quickstart The simplest way to get started with Common Crawl is to load a small sample of data: ```python import daft daft.datasets.common_crawl("CC-MAIN-2025-33", num_files=1).show() ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Basic usage ### Loading different content types (WARC, WET, WAT) Common Crawl provides three types of content: **Raw [Web ARChive (WARC)]() files (default)** - Full HTTP responses with headers and content: ```python # Raw WARC data (default) daft.datasets.common_crawl("CC-MAIN-2025-33", content="raw") # or equivalently daft.datasets.common_crawl("CC-MAIN-2025-33", content="warc") ``` **Extracted text, aka WET files** - Plain text content extracted from web pages: ```python # Extracted text content daft.datasets.common_crawl("CC-MAIN-2025-33", content="text") # or equivalently daft.datasets.common_crawl("CC-MAIN-2025-33", content="wet") ``` **Metadata, aka WAT files** - Information about crawled pages without content: ```python # Metadata only daft.datasets.common_crawl("CC-MAIN-2025-33", content="metadata") # or equivalently daft.datasets.common_crawl("CC-MAIN-2025-33", content="wat") ``` ### Loading a subset of data For quick testing and development, it's helpful to limit the number of crawl files accessed: ```python # Process only 1 crawl file for testing daft.datasets.common_crawl("CC-MAIN-2025-33", num_files=1) ``` ### Working with specific segments Each crawl is split into 100 segments. You can target a specific segment: ```python daft.datasets.common_crawl("CC-MAIN-2025-33", segment="1754151279521.11") ``` ## Data schema Daft's Common Crawl dataset includes these key columns: | Column | Type | Description | | ------------------------------ | --------- | -------------------------------------------------- | | `WARC-Record-ID` | String | Unique identifier for each WARC record | | `WARC-Target-URI` | String | The URL that was crawled | | `WARC-Type` | String | Type of record (response, request, warcinfo, etc.) | | `WARC-Date` | Timestamp | When the page was crawled | | `WARC-Identified-Payload-Type` | String | MIME type of the content | | `warc_content` | Binary | The actual content (HTML, text, etc.) | | `warc_headers` | String | All WARC record headers as JSON | For more details on the WARC file format, check out the [WARC specification](https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.0/). ## Examples ### Analyzing content types Find the most common MIME types in a crawl: ```python ( daft.datasets.common_crawl("CC-MAIN-2025-33", num_files=1) .select(daft.col("WARC-Identified-Payload-Type")) .groupby("WARC-Identified-Payload-Type") .agg(daft.col("WARC-Identified-Payload-Type").count().alias("count")) .sort("count", desc=True) .show() ) ``` ```{title="Output"} ╭──────────────────────────────┬────────╮ │ WARC-Identified-Payload-Type ┆ count │ │ --- ┆ --- │ │ Utf8 ┆ UInt64 │ ╞══════════════════════════════╪════════╡ │ text/html ┆ 21907 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ application/xhtml+xml ┆ 2063 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ application/pdf ┆ 143 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ application/atom+xml ┆ 28 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ text/plain ┆ 23 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ application/rss+xml ┆ 14 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ application/xml ┆ 14 │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ text/calendar ┆ 7 │ ╰──────────────────────────────┴────────╯ ``` ### Extracting text for language models Content in Common Crawl WARC files are UTF-8 encoded. Use Daft's [try_decode][daft.functions.try_decode] function to extract clean text content for training: ```python from daft.functions import try_decode ( daft.datasets.common_crawl("CC-MAIN-2025-33", content="text", num_files=1) .with_column("text_content", try_decode(daft.col("warc_content"), charset="utf-8")) .where(daft.col("text_content").not_null()) .select("WARC-Target-URI", "text_content") .limit(3) .show() ) ``` ```{title="Output"} ╭────────────────────────────────┬──────────────────────────────────────────────────╮ │ WARC-Target-URI ┆ text_content │ │ --- ┆ --- │ │ Utf8 ┆ Utf8 │ ╞════════════════════════════════╪══════════════════════════════════════════════════╡ │ None ┆ Software-Info: ia-web-commons… │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ http://010ganji.com/html/ying… ┆ ETF选择困难？易方达基金划分四大类助您轻松投资！_ │ │ ┆ 首页… │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ http://01dom.ru/sale/prodleni… ┆ Скидка до 23% на керамические… │ ╰────────────────────────────────┴──────────────────────────────────────────────────╯ ``` ## Next steps - If you want a runnable example of using this dataset and embedding text using Qwen3, take a look at our [Getting Started with Common Crawl in Daft](../examples/common-crawl-daft-tutorial.md) tutorial. - When using datasets like Common Crawl for pre-training, content deduplication is essential for model performance. Check out our [MinHash deduplication example](../examples/minhash-dedupe.md) to see how this can be done in Daft. - See the [Common Crawl Dataset API reference](../api/datasets.md#common-crawl) for complete parameter documentation --- --- METRICS --- - Files Extracted: 99 - Estimated Token Budget: ~125750 tokens - Canonical Reference: https://codewiki.google/github.com/Eventual-Inc/Daft