{"owner":"Eventual-Inc","repo":"Daft","hasSkills":true,"totalSkillsCount":14,"totalTokensCount":13090,"categories":["marketplace","plugin-manifest","anthropic-skill","root-instruction","claude-rule"],"hasMcp":false,"mcpConfig":null,"found":[".claude-plugin/marketplace.json",".claude-plugin/plugin.json",".claude/skills/daft-distributed-scaling/SKILL.md",".claude/skills/daft-docs-navigation/SKILL.md",".claude/skills/daft-udf-tuning/SKILL.md","AGENTS.md","CLAUDE.md","docs/extensions/authoring.md","docs/extensions/community.md","docs/extensions/overview.md","docs/extensions/projects.md","docs/plugins/README.md","docs/skills.md","tools/observability/opentelemetry/README.md"],"skills":{".claude-plugin/marketplace.json":"{\n  \"name\": \"Eventual-Inc\",\n  \"owner\": {\n    \"name\": \"Eventual, Inc.\",\n    \"email\": \"yksu@eventualcomputing.com\"\n  },\n  \"plugins\": [\n    {\n      \"name\": \"daft\",\n      \"source\": \"./\",\n      \"description\": \"Skills for working with Daft: UDF tuning, distributed scaling, and docs navigation\",\n      \"version\": \"0.1.0\"\n    }\n  ]\n}\n",".claude-plugin/plugin.json":"{\n  \"name\": \"daft\",\n  \"description\": \"Skills for working with Daft: UDF tuning, distributed scaling, and docs navigation\",\n  \"version\": \"0.1.0\",\n  \"author\": {\n    \"name\": \"Eventual, Inc.\",\n    \"email\": \"yksu@eventualcomputing.com\"\n  },\n  \"repository\": \"https://github.com/Eventual-Inc/Daft\",\n  \"license\": \"Apache-2.0\",\n  \"keywords\": [\n    \"daft\",\n    \"dataframe\",\n    \"udf\",\n    \"distributed\",\n    \"ray\"\n  ]\n}\n",".claude/skills/daft-distributed-scaling/SKILL.md":"---\nname: \"daft-distributed-scaling\"\ndescription: \"Scale Daft workflows to distributed Ray clusters. Invoke when optimizing performance or handling large data.\"\n---\n\n# Daft Distributed Scaling\n\nScale single-node workflows to distributed execution.\n\n## Core Strategies\n\n| Strategy | API | Use Case | Pros/Cons |\n|---|---|---|---|\n| **Shuffle** | `repartition(N)` | Light data (e.g. file paths), Joins | **Global balance**. High memory usage (materializes data). |\n| **Streaming** | `into_batches(N)` | Heavy data (images, tensors) | **Low memory** (streaming). High scheduling overhead if batches too small. |\n\n## Quick Recipes\n\n### 1. Light Data: Repartitioning\nBest for distributing file paths before heavy reads.\n\n```python\n# Create enough partitions to saturate workers\ndf = daft.read_parquet(\"s3://metadata\").repartition(100)\ndf = df.with_column(\"data\", read_heavy_data(df[\"path\"]))\n```\n\n### 2. Heavy Data: Streaming Batches\nBest for processing large partitions without OOM.\n\n```python\n# Stream 1GB partition in 64-row chunks to control memory\ndf = df.read_parquet(\"heavy_data\").into_batches(64)\ndf = df.with_column(\"embed\", model.predict(df[\"img\"]))\n```\n\n## Advanced Tuning\n\nTarget: Keep all actors busy without OOM or scheduling bottlenecks.\n\n### Formula 1: Repartitioning (Light Data / Paths)\nCalculate the **Max Partition Count** to ensure each task has enough data to feed local actors.\n\n1.  **Min Rows Per Partition** = `Batch Size * (Total Concurrency / Nodes)`\n2.  **Max Partitions** = `Total Rows / Min Rows Per Partition`\n\n**Example**:\n- 1M rows, 4 nodes, 16 total concurrency, Batch Size 64.\n- **Min Rows**: `64 * (16/4) = 256`.\n- **Max Partitions**: `1,000,000 / 256 ≈ 3906`.\n- *Recommendation*: Use ~1000 partitions to run multiple batches per task.\n\n```python\ndf = df.repartition(1000) # Balanced fan-out\n```\n\n### Formula 2: Streaming (Heavy Data / Images)\nAvoid creating tiny partitions. Use `into_batches` to stream data within larger partitions.\n\n**Strategy**: Keep partitions large (e.g. 1GB+), use `into_batches(Batch Size)` to control memory.\n\n```python\n# Stream batches to control memory usage per actor\ndf = df.into_batches(64).with_column(\"preds\", model(max_concurrency=16).predict(df[\"img\"]))\n```\n",".claude/skills/daft-docs-navigation/SKILL.md":"---\nname: \"daft-docs-navigation\"\ndescription: \"Navigate Daft documentation. Invoke when user asks general questions about APIs, concepts, or examples, or wants to search the docs.\"\n---\n\n# Daft Docs Navigation\n\nNavigate Daft's documentation for APIs, concepts, and examples.\n\n## Documentation Structure\n\n1.  **Live Site**: [`https://docs.daft.ai`](https://docs.daft.ai) (Primary source, searchable).\n2.  **Source Repo**: `docs/` directory in the repository.\n    -   If `docs/` is missing or empty, clone the repo: `git clone https://github.com/Eventual-Inc/Daft.git`\n\n## Key Locations in `docs/`\n\n-   **API Reference**: `api/` (e.g., `api/io.md` for reading/writing).\n-   **Optimization**: `optimization/` (e.g., `optimization/partitioning.md`).\n-   **Distributed**: `distributed/` (e.g., `distributed/ray.md`).\n-   **UDFs**: `custom-code/` (e.g., `func.md`, `cls.md`).\n-   **Connectors**: `connectors/` (e.g., S3, Lance).\n-   **Table of Contents**: `docs/SUMMARY.md`.\n\n## Usage\n\n**Search for API Usage:**\n```bash\ngrep_search(pattern='from_glob', path='docs/')\n```\n\n**Browse Topics:**\n```bash\nread(file_path='docs/SUMMARY.md')\n```\n",".claude/skills/daft-udf-tuning/SKILL.md":"---\nname: \"daft-udf-tuning\"\ndescription: \"Optimize Daft UDF performance. Invoke when user needs GPU inference, encounters slow UDFs, or asks about async/batch processing.\"\n---\n\n# Daft UDF Tuning\n\nOptimize User-Defined Functions for performance.\n\n## UDF Types\n\n| Type | Decorator | Use Case |\n|---|---|---|\n| **Stateless** | `@daft.func` | Simple transforms. Use `async` for I/O-bound tasks. |\n| **Stateful** | `@daft.cls` | Expensive init (e.g., loading models). Supports `gpus=N`. |\n| **Batch** | `@daft.func.batch` | Vectorized CPU/GPU ops (NumPy/PyTorch). Faster. |\n\n## Quick Recipes\n\n### 1. Async I/O (Web APIs)\n\n```python\n@daft.func\nasync def fetch(url: str):\n    async with aiohttp.ClientSession() as s:\n        return await s.get(url).text()\n```\n\n### 2. GPU Batch Inference (PyTorch/Models)\n\n```python\n@daft.cls(gpus=1)\nclass Classifier:\n    def __init__(self):\n        self.model = load_model().cuda() # Run once per worker\n\n    @daft.method.batch(batch_size=32)\n    def predict(self, images):\n        return self.model(images.to_pylist())\n\n# Run with concurrency\ndf.with_column(\"preds\", Classifier(max_concurrency=4).predict(df[\"img\"]))\n```\n\n## Tuning Keys\n\n-   **`max_concurrency`**: Total parallel UDF instances.\n-   **`gpus=N`**: GPU request per instance.\n-   **`batch_size`**: Rows per call. Too small = overhead; too big = OOM.\n-   **`into_batches(N)`**: Pre-slice partitions if memory is tight.\n","AGENTS.md":"# Resources\n\n- https://docs.daft.ai for the user-facing API docs\n- CONTRIBUTING.md for detailed development process\n- https://github.com/Eventual-Inc/Daft for issues, discussions, and PRs\n\n# Dev Workflow\n\n1. [Once] Set up Python environment and install dependencies: `make .venv`\n2. [Optional] Activate .venv: `source .venv/bin/activate`. Not necessary with Makefile commands.\n3. If Rust code is modified, rebuild: `make build`\n4. Run tests. See [Testing Details](#testing-details).\n\n# Testing Details\n\n- `make test` runs tests in `tests/` directory. Uses `pytest` under the hood.\n  - Must set `DAFT_RUNNER` environment variable to `ray` or `native` to run the tests with the corresponding runner.\n    - Start with `DAFT_RUNNER=native` unless testing Ray or distributed code.\n  - `make test EXTRA_ARGS=\"...\"` passes additional arguments to `pytest`.\n    - `make test EXTRA_ARGS=\"-v tests/dataframe/test_select.py\"` runs the test in the given file.\n    - `make test EXTRA_ARGS=\"-v tests/dataframe/test_select.py::test_select_dataframe\"` runs the given test method.\n  - Default `integration`, `benchmark`, and `hypothesis` tests are disabled. Best to run on CI.\n- `make doctests` runs doctests in `daft/` directory. Tests docstrings in Daft APIs.\n\n# PR Conventions\n\n- Titles: Conventional Commits format; enforced by `.github/workflows/pr-labeller.yml`.\n- Descriptions: follow `.github/pull_request_template.md`.\n","CLAUDE.md":"AGENTS.md","docs/extensions/authoring.md":"# Authoring Guide\n\n!!! warning \"Experimental\"\n\n    Native extensions are experimental and may change in future releases.\n\n> Please see the [prompt](#prompt) if you want help generating an extension.\n> Looking for the broader extension model? See the [Extensions overview](overview.md).\n> Looking for extensions you can install today? See [Community Extensions](community.md).\n\nThis document is a guide for authoring Daft native extensions in Rust.\nDaft supports native Rust extensions by leveraging a stable C ABI based on the\n[Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html).\nExtensions are **not coupled** to any particular Arrow library version. The ABI boundary uses\nplain C structs (`ArrowSchema`, `ArrowArray`) so your extension can use any arrow-rs version\n(or even a different Arrow implementation entirely). Today we support authoring native\nscalar functions (UDFs) and aggregate functions (UDAFs).\n\n## Example\n\nThis example shows the end-result of our 'hello' extension with a native 'greet' scalar function.\n\n```python\nimport daft\n\n# Step 1. Import your extension module\nimport hello\n\n# Step 2. Load the extension into the current daft session\ndaft.load_extension(hello)\n\n# Step 3. Use in your dataframe!\ndf = daft.from_pydict({\"name\": [\"John\", \"Paul\"]})\ndf = df.select(hello.greet(df[\"name\"]))\ndf.show()\n\n\"\"\"\n╭──────────────╮\n│ greet        │\n│ ---          │\n│ String       │\n╞══════════════╡\n│ Hello, John! │\n├╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤\n│ Hello, Paul! │\n╰──────────────╯\n\"\"\"\n```\n\n## Tutorial\n\n### 1. Setup\n\nThis section walks through project setup from scratch; You will need a Rust toolchain and Python 3.10+.\n\n!!! note \"Note\"\n\n    We are actively working on a cloneable template.\n\n\n```bash\n# Install Rust (if not already installed)\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n\n# Install uv (if not already installed)\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n```\n\nNow you can create the project scaffolding.\n\n```bash\n# Create an empty directory\nmkdir hello && cd hello\n\n# Setup rust project\ncargo init --lib\n\n# Setup python project (for function signatures)\nuv init\n```\n\nThe crate must compile as a `cdylib` so it can be loaded at runtime via `dlopen`.\n\n```bash\ncat Cargo.toml\n```\n\n```toml\n[workspace]\n\n[package]\nname = \"hello\"\nedition = \"2024\"\nversion = \"0.1.0\"\n\n[lib]\nname = \"hello\"\ncrate-type = [\"cdylib\"]\n\n[dependencies]\ndaft-ext = { version = <version>, features = [\"arrow-58\"] }\narrow = { version = \"58\", features = [\"ffi\"] }\n```\n\n!!! tip \"Arrow version freedom\"\n\n    The `daft-ext` ABI uses C Data Interface types — your extension is **not** pinned to\n    Daft's arrow-rs version. Enable a feature flag on `daft-ext` matching your arrow-rs\n    version (`arrow-56`, `arrow-57`, or `arrow-58`) to get safe `.into()` conversions\n    between arrow-rs FFI types and the ABI types. For unsupported versions, use the\n    `from_owned`/`into_owned`/`from_raw`/`as_raw` escape hatches on `ArrowArray`\n    and `ArrowSchema`.\n\nThen update the pyproject to use `setuptools-rust` as the build system.\n\n```bash\ncat pyproject.toml\n```\n\n```toml\n[build-system]\nrequires = [\"setuptools\", \"setuptools-rust\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[project]\nname = \"hello\"\nversion = \"0.1.0\"\nrequires-python = \">=3.10\"\ndependencies = [\"daft\"]\n\n[project.optional-dependencies]\ntest = [\"pytest\"]\n```\n\n\nYou will need to create a `setup.py` file with the following contents.\n\n!!! note \"Note\"\n\n    The `RustExtension` entry tells `setuptools-rust` to compile the cdylib and place it inside the Python package directory.\n    We use the `Binding::NoBinding` because Daft extensions export a raw C symbol, not PyO3 bindings. The target name `hello.libhello`\n    places the compiled `.so` inside the `hello/` package directory so the Session's `load_extension` can find it.\n\n```bash\ncat setup.py\n```\n\n```python\nfrom setuptools import find_packages, setup\nfrom setuptools_rust import Binding, RustExtension\n\nsetup(\n    packages=find_packages(),\n    rust_extensions=[\n        RustExtension(\n            \"hello.libhello\",   # <python_package>.<lib_name>\n            path=\"Cargo.toml\",\n            binding=Binding.NoBinding,\n            strip=True,\n        )\n    ],\n)\n```\n\n### 2. Hello, World!\n\nAn extension has two parts: a **module** (the entry point) and one or more **functions** (scalar or aggregate).\n\n```bash\ncat src/lib.rs\n```\n\n```rust\nuse std::{ffi::CStr, sync::Arc};\n\nuse arrow::{\n    array::{Array, builder::StringBuilder, cast::AsArray},\n    datatypes::{DataType, Field},\n};\nuse daft_ext::prelude::*;\n\n// ── Module ──────────────────────────────────────────────────────────\n\n// #[daft_extension] generates the `daft_module_magic` C symbol that Daft's runtime looks for\n// when loading the shared library. It converts HelloExtension → hello_extension for the module name.\n#[daft_extension]\nstruct HelloExtension;\n\nimpl DaftExtension for HelloExtension {\n    /// This is the extension install hook for defining functions in the session.\n    /// Called once when the extension is loaded into a session. Register each function here.\n    fn install(session: &mut dyn DaftSession) {\n        session.define_function(Arc::new(Greet));\n    }\n}\n\n// ── Function ────────────────────────────────────────────────────────\n\n/// The function type which is registered as an Arc<dyn DaftScalarFunction>.\nstruct Greet;\n\nimpl DaftScalarFunction for Greet {\n    /// Function name used to look it up from Python.\n    /// Must be a `&CStr` (use the `c\"...\"` literal).\n    fn name(&self) -> &CStr {\n        c\"greet\"\n    }\n\n    /// Type checking.\n    /// Receives input fields as C Data Interface `ArrowSchema` types.\n    /// Use `.as_raw()` / `.into()` to convert between arrow-rs and ABI types.\n    fn return_field(&self, args: &[ArrowSchema]) -> DaftResult<ArrowSchema> {\n        if args.len() != 1 {\n            return Err(DaftError::TypeError(format!(\n                \"greet: expected 1 argument, got {}\",\n                args.len()\n            )));\n        }\n        let field = Field::try_from(&args[0])?;\n        let dt = field.data_type();\n        if *dt != DataType::Utf8 && *dt != DataType::LargeUtf8 {\n            return Err(DaftError::TypeError(format!(\n                \"greet: expected string argument, got {:?}\",\n                dt\n            )));\n        }\n        Ok(ArrowSchema::try_from(&Field::new(\n            \"greet\",\n            DataType::Utf8,\n            true,\n        ))?)\n    }\n\n    /// Evaluation. Receives columns as C Data Interface `ArrowData` types.\n    /// Use `.into()` to convert to/from arrow-rs FFI types.\n    /// All data flows through Arrow arrays — no per-row Python overhead.\n    fn call(&self, args: Vec<ArrowData>) -> DaftResult<ArrowData> {\n        let data = args.into_iter().next().ok_or_else(|| DaftError::TypeError(\"greet: expected 1 argument, got 0\".into()))?;\n        let ffi_array: arrow::ffi::FFI_ArrowArray = data.array.into();\n        let ffi_schema: arrow::ffi::FFI_ArrowSchema = data.schema.into();\n        let arrow_data = unsafe { arrow::ffi::from_ffi(ffi_array, &ffi_schema) }?;\n        let input = arrow::array::make_array(arrow_data);\n        let names = input.as_string::<i64>();\n        let mut builder = StringBuilder::with_capacity(names.len(), names.len() * 16);\n        for i in 0..names.len() {\n            if names.is_null(i) {\n                builder.append_null();\n            } else {\n                builder.append_value(format!(\"Hello, {}!\", names.value(i)));\n            }\n        }\n        let output = builder.finish();\n        let (out_arr, out_sch) = arrow::ffi::to_ffi(&output.to_data())?;\n        Ok(ArrowData {\n            array: out_arr.into(),\n            schema: out_sch.into(),\n        })\n    }\n}\n```\n\n!!! tip \"ABI pattern\"\n\n    The `DaftScalarFunction` trait uses C Data Interface types (`ArrowSchema`, `ArrowData`)\n    at the ABI boundary. Enable a `daft-ext` feature flag (`arrow-56`, `arrow-57`, or\n    `arrow-58`) matching your arrow-rs version to get `.into()` conversions. Use `.as_raw()`\n    for zero-copy borrows. This decoupling means your extension is not tied to Daft's\n    arrow-rs version.\n\n!!! tip \"String types\"\n\n    Daft uses `LargeUtf8` (i64 offsets) for strings internally. When downcasting string arrays,\n    use `as_string::<i64>()` — using `i32` will panic at runtime. Similarly, when checking types\n    in `return_field`, accept `DataType::LargeUtf8`.\n\n!!! tip \"Naming\"\n\n    Function names are global within a session. Use a prefix (e.g., `myext_greet`) to avoid\n    collisions when your extension defines many functions or might be loaded alongside others.\n\n!!! tip \"Multiple functions\"\n\n    Register as many functions as you need in `install()` — each is an independent struct\n    implementing `DaftScalarFunction`.\n\n!!! tip \"Errors\"\n\n    Return `Err(DaftError::TypeError(...))` for schema violations in `return_field`,\n    and `Err(DaftError::RuntimeError(...))` for execution failures in `call`.\n\n### Aggregate Functions\n\nExtensions can also register **aggregate functions** (UDAFs) alongside scalar functions.\nAn aggregate function follows a three-stage pipeline:\n\n1. **Aggregation** (`aggregate`) — process input arrays into partial state\n2. **Combination** (`combine`) — merge partial states\n3. **Finalization** (`finalize`) — produce the final output from merged state\n\nState is exchanged as `Vec<ArrowData>` — each element is one state field. The FFI layer\npacks these into a Struct array transparently.\n\n```rust\nuse std::ffi::CStr;\nuse daft_ext::prelude::*;\n\nstruct MySum;\n\nimpl DaftAggregateFunction for MySum {\n    fn name(&self) -> &CStr {\n        c\"my_sum\"\n    }\n\n    /// Declare the output type given the input field schemas.\n    fn return_field(&self, _args: &[ArrowSchema]) -> DaftResult<ArrowSchema> {\n        Ok(ArrowSchema::try_from(&Field::new(\"my_sum\", DataType::Int64, true))?)\n    }\n\n    /// Declare intermediate state fields.\n    /// Each state field becomes a child of the Struct exchanged at the FFI boundary.\n    fn state_fields(&self, _args: &[ArrowSchema]) -> DaftResult<Vec<ArrowSchema>> {\n        Ok(vec![\n            ArrowSchema::try_from(&Field::new(\"sum\", DataType::Int64, true))?,\n            ArrowSchema::try_from(&Field::new(\"count\", DataType::Int64, false))?,\n        ])\n    }\n\n    /// Process input arrays into partial state.\n    /// Returns one ArrowData per state field (single-row each).\n    fn aggregate(&self, inputs: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>> {\n        // inputs[0] is the column to sum\n        // compute sum and count, return as two single-element arrays\n        todo!()\n    }\n\n    /// Merge multiple partial states into one.\n    /// states[i] is the i-th state field with multiple rows (one per partial).\n    fn combine(&self, states: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>> {\n        // states[0] = sums array, states[1] = counts array\n        // sum them up, return single-row arrays\n        todo!()\n    }\n\n    /// Produce the final scalar result from merged state.\n    fn finalize(&self, states: Vec<ArrowData>) -> DaftResult<ArrowData> {\n        // states[0] = sum (single value), states[1] = count (single value)\n        // return the final sum\n        todo!()\n    }\n}\n```\n\nRegister aggregate functions in `install()` using `define_aggregate_function`:\n\n```rust\nimpl DaftExtension for MyExtension {\n    fn install(session: &mut dyn DaftSession) {\n        session.define_function(Arc::new(MyScalarFn));\n        session.define_aggregate_function(Arc::new(MySum));\n    }\n}\n```\n\nNow we define the python symbols for use in the Expression DSL; we link to rust via `daft.get_function` for scalars and `daft.get_aggregate_function` for aggregates.\n\n!!! note \"Note\"\n\n    Python sources let you write python function signatures and any argument pre-processing before linking\n    to the function symbol. The `daft.get_function` method is calling `get_function` on the active session\n    to resolve the given name and arguments to some defined function in the session. Notice how we don't\n    technically need python to resolve functions in SQL (hence no PyO3) but these python functions give us\n    nice pythonic functions with autocomplete and doc comments when using the Expression DSL.\n\n```bash\ncat hello/__init__.py\n```\n\n```python\nfrom __future__ import annotations\nfrom typing import TYPE_CHECKING\n\nimport daft\n\nif TYPE_CHECKING:\n    from daft.expressions import Expression\n\ndef greet(name: Expression) -> Expression:\n    \"\"\"Greet someone by name.\"\"\"\n    return daft.get_function(\"greet\", name)\n\ndef string_count(name: Expression) -> Expression:\n    \"\"\"Count non-null strings.\"\"\"\n    return daft.get_aggregate_function(\"string_count\", name)\n```\n\n`daft.get_function` looks up a scalar function registered with the current session by the name returned from `DaftScalarFunction::name()`.\n`daft.get_aggregate_function` does the same for aggregate functions registered via `DaftAggregateFunction::name()`.\n\nAdd an empty `hello/py.typed` marker if you want type-checker support.\n\n### 3. Build, Install, Test\n\n```bash\n# Compile the Rust cdylib and install the package in editable mode\nuv pip install -e .\n```\n\nHere are some sanity check tests. Notice how we use a scoped session rather than\nthe global active session. How you choose to load extensions is up to you, and this\ntutorial has covered both.\n\n!!! tip \"Session isolation\"\n\n    Extensions are loaded once into the process and the session serves as a scoping mechanism\n    for name resolution; calling `load_extension` multiple times will only `dlopen` once for\n    this process. Functions are only available in sessions where the extension is loaded.\n    Use the `with sess:` context manager to scope queries to a specific session.\n\n```bash\ncat tests/test_hello.py\n```\n\n```python\nimport daft\nimport hello\nfrom daft import col\nfrom daft.session import Session\nfrom hello import greet\n\ndef test_greet():\n    sess = Session()\n    sess.load_extension(hello)\n\n    df = daft.from_pydict({\"name\": [\"John\", \"Paul\"]})\n\n    with sess:\n        result = df.select(greet(col(\"name\"))).collect().to_pydict()\n\n    values = result[\"greet\"]\n    assert values[0] == \"Hello, John!\"\n    assert values[1] == \"Hello, Paul!\"\n\ndef test_greet_null():\n    sess = Session()\n    sess.load_extension(hello)\n\n    df = daft.from_pydict({\"name\": [\"George\", \"Ringo\", None]})\n\n    with sess:\n        result = df.select(greet(col(\"name\"))).collect().to_pydict()\n\n    values = result[\"greet\"]\n    assert values[0] == \"Hello, George!\"\n    assert values[1] == \"Hello, Ringo!\"\n    assert values[2] is None\n```\n\nNow run the tests!\n\n```bash\npytest -v tests/\n```\n\n## Prompt\n\nYou can paste this whole document and prompt into Claude Code to scaffold a Daft extension for you.\n\n````markdown\nCreate a Daft native extension called `<extension_name>` with the following functions:\n\n<describe each function: name, scalar or aggregate, arguments with types, return type, and behavior>\n\nFollow the Daft extension authoring guide at docs/extensions/authoring.md. Here is a summary of the key conventions:\n\n## Project structure\n\n```\n<extension_name>/\n  Cargo.toml           # [lib] crate-type = [\"cdylib\"], depends on daft-ext (with arrow feature), arrow\n  pyproject.toml       # build-system: setuptools + setuptools-rust\n  setup.py             # RustExtension(\"<pkg>.lib<pkg>\", binding=NoBinding, strip=True)\n  <extension_name>/\n    __init__.py        # Python wrappers using daft.get_function(\"name\", *args)\n    py.typed           # empty PEP 561 marker\n  src/\n    lib.rs             # #[daft_extension] struct + DaftScalarFunction / DaftAggregateFunction impls\n  tests/\n    test_<name>.py     # pytest tests using Session fixture\n  .gitignore           # /target, *.so, *.dylib, *.dll, *.egg-info, __pycache__, dist/\n```\n\n## Rust conventions\n\n- Use `daft_ext::prelude::*` for all imports (provides `ArrowSchema`, `ArrowData`, errors, traits).\n- Add `daft-ext` with a feature flag matching your arrow version (`arrow-56`, `arrow-57`, or `arrow-58`) for `.into()` conversions.\n- Import `arrow::array::Array` for `len()`/`is_null()` and `arrow::array::cast::AsArray` for downcasting.\n- Daft uses `LargeUtf8` (i64 offsets) for strings — downcast with `as_string::<i64>()`, never `i32`.\n- Apply `#[daft_extension]` to a struct implementing `DaftExtension`.\n- Register scalar functions via `session.define_function(Arc::new(MyFn))`.\n- Register aggregate functions via `session.define_aggregate_function(Arc::new(MyAgg))`.\n- Each scalar function is a struct implementing `DaftScalarFunction` with:\n  - `name(&self) -> &CStr` — use `c\"<extension_name>_<fn_name>\"` prefix to avoid collisions.\n  - `return_field(&self, args: &[ArrowSchema]) -> DaftResult<ArrowSchema>` — use `.as_raw()` to\n    borrow as arrow-rs `FFI_ArrowSchema` for type checking, then `.into()` to return output.\n  - `call(&self, args: &[ArrowData]) -> DaftResult<ArrowData>` — use `ArrowData::take_arg` then\n    `.into()` to convert to arrow-rs FFI types, compute, then `.into()` to return the result.\n- Each aggregate function is a struct implementing `DaftAggregateFunction` with:\n  - `name`, `return_field` — same as scalar functions.\n  - `state_fields(&self, args: &[ArrowSchema]) -> DaftResult<Vec<ArrowSchema>>` — intermediate state schema.\n  - `aggregate(&self, inputs: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>>` — partial aggregation.\n  - `combine(&self, states: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>>` — merge partial states.\n  - `finalize(&self, states: Vec<ArrowData>) -> DaftResult<ArrowData>` — produce final result.\n\n## Python conventions\n\n- Each scalar function wrapper calls `daft.get_function(\"<extension_name>_<fn_name>\", *args)`.\n- Each aggregate function wrapper calls `daft.get_aggregate_function(\"<extension_name>_<fn_name>\", *args)`.\n- Use `TYPE_CHECKING` guard for `Expression` import.\n- Add type hints and a docstring to each wrapper.\n````\n","docs/extensions/community.md":"# Community Extensions\n\n!!! tip \"Want to build your own extension?\"\n\n    Start with the [Extensions overview](overview.md), the [Python UDF docs](../custom-code/func.md),\n    or the [Native Extension Authoring Guide](authoring.md).\n\nCommunity extensions are reusable packages that add functionality to Daft\noutside of the core repository. They may be pure-Python UDF-based extensions,\nnative ABI extensions, custom DataSource/DataSink connectors, or higher-level\ndomain libraries that combine Daft expressions, UDFs, file types, model calls,\nand distributed execution patterns.\n\nProjects listed here are maintained independently of Daft's release cadence.\nEach project installs and versions independently of Daft.\n\n| Name | Kind | Repository | Description |\n|---|---|---|---|\n| [daft-h3](#daft-h3) | Native ABI | [gweaverbiodev/daft-h3](https://github.com/gweaverbiodev/daft-h3) | Native [H3](https://h3geo.org/) geospatial indexing functions. |\n| [daft-lance](#daft-lance) | Python UDF-based | [daft-engine/daft-lance](https://github.com/daft-engine/daft-lance) | Lance-specific distributed operations for compaction, scalar indexing, column merging, and REST catalog operations. |\n| [daft-html](#daft-html) | Native ABI | [daft-engine/daft-html](https://github.com/daft-engine/daft-html) | Native HTML document processing functions exposed as Daft expressions. |\n| [daft-geo](#daft-geo) | Native ABI / Datatypes | [daft-engine/daft-geo](https://github.com/daft-engine/daft-geo) | Geospatial prototype showing native functions and extension-backed datatypes. |\n| [daft-qdrant](#daft-qdrant) | Python UDF-based | [qdrant-labs/daft-qdrant](https://github.com/qdrant-labs/daft-qdrant) | Write vector embeddings and their payloads into [Qdrant](https://qdrant.tech/) collections. |\n| [daft-doris](#daft-doris) | Custom DataSource/DataSink | [jiangxt2/daft-doris](https://github.com/jiangxt2/daft-doris) | Apache Doris reads and HTTP Stream Load writes for Daft. |\n\nTo propose a new extension for this list, open a PR against this page.\n\n## daft-h3\n\n`daft-h3` adds H3 geospatial indexing functions such as latitude/longitude to\ncell conversion, cell traversal, grid distance, parent resolution, and string\nconversion.\n\n```bash\npip install daft-h3\n```\n\n```python\nimport daft\nimport daft_h3\nfrom daft import col\n\ndaft.load_extension(daft_h3)\n\ndf = daft.from_pydict({\"lat\": [37.7749], \"lng\": [-122.4194]})\ndf = df.select(\n    daft_h3.h3_latlng_to_cell(col(\"lat\"), col(\"lng\"), 7).alias(\"cell\"),\n)\ndf = df.select(daft_h3.h3_cell_to_str(col(\"cell\")).alias(\"hex\"))\ndf.show()\n```\n\nSee the [daft-h3 README](https://github.com/gweaverbiodev/daft-h3#readme) for\nthe full list of functions and behavior details.\n\n## daft-lance\n\n`daft-lance` extends Daft with Lance-specific distributed maintenance and data\nmanagement operations. It is a Python UDF-based extension: internally, it uses\nDaft's Python UDF and class-UDF APIs to distribute Lance tasks across Daft\nqueries, while users interact with simple Python functions.\n\n```bash\npip install daft-lance\n```\n\n```python\nfrom daft_lance import compact_files, create_scalar_index\n\ncompact_files(\"s3://bucket/my_dataset\")\n\ncreate_scalar_index(\n    \"s3://bucket/my_dataset\",\n    column=\"name\",\n    index_type=\"INVERTED\",\n)\n```\n\nSee the [daft-lance README](https://github.com/daft-engine/daft-lance#readme)\nfor additional operations such as column merging and REST catalog writes.\n\n## daft-html\n\n`daft-html` adds native HTML processing functions such as `html_to_text`,\n`html_extract_links`, `html_extract_tables`, and CSS-selector extraction.\n\n```bash\npip install daft-html\n```\n\n```python\nimport daft\nimport daft_html\nfrom daft import col\nfrom daft_html import html_to_text\n\ndaft.load_extension(daft_html)\n\ndf = daft.from_pydict({\"html\": [\"<html><body><h1>Hello</h1></body></html>\"]})\ndf = df.select(html_to_text(col(\"html\")).alias(\"text\"))\ndf.show()\n```\n\nSee the [daft-html README](https://github.com/daft-engine/daft-html#readme)\nfor the full list of document and CSS-selector functions.\n\n## daft-geo\n\n`daft-geo` is a geospatial prototype showing native Daft functions and\nextension-backed datatypes. It defines `Point2D` and `Point3D` as\n`DataType.extension(...)` values and provides point construction, accessors,\nEuclidean distance, and haversine distance.\n\n```bash\npip install git+https://github.com/daft-engine/daft-geo.git\n```\n\n!!! note\n\n    `daft-geo` is not yet published to PyPI. The command above installs the\n    latest commit from the repository's default branch, so behavior may change\n    between installs.\n\n```python\nimport daft\nimport daft_geo\nfrom daft import col\n\ndaft.load_extension(daft_geo)\n\ndf = daft.from_pydict({\"x\": [0.0], \"y\": [1.0]})\ndf = df.select(daft_geo.point2d(col(\"x\"), col(\"y\")).alias(\"point\"))\ndf = df.select(daft_geo.x(col(\"point\")).alias(\"x\"))\ndf.show()\n```\n\nSee the [daft-geo README](https://github.com/daft-engine/daft-geo#readme) for\ncurrent capabilities and caveats.\n\n## daft-qdrant\n\n`daft-qdrant` adds a `write_qdrant` DataFrame method for writing vector\nembeddings and their payloads into [Qdrant](https://qdrant.tech/) collections.\n\n```bash\npip install daft-qdrant\n```\n\n```python\nimport daft\nimport daft_qdrant\n\ndf = daft.from_pydict({\n    \"id\": [1, 2, 3],\n    \"vector\": [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]],\n    \"label\": [\"cat\", \"dog\", \"bird\"],\n})\n\ndf.write_qdrant(\"my-collection\", url=\"http://localhost:6333\")\n```\n\nSee the [daft-qdrant README](https://github.com/qdrant-labs/daft-qdrant#readme) for embedding pipeline examples and additional options.\n\n## daft-doris\n\n`daft-doris` is an independently maintained Python custom connector for\nApache Doris. It supports reads from Doris physical tables and batch writes\nthrough HTTP Stream Load. The read transport must be selected explicitly;\nFlight SQL is available as an opt-in experimental transport.\n\n```bash\npip install \"daft-doris[doris]\"\n```\n\nFor Flight SQL read support, install the optional Flight extra:\n\n```bash\npip install \"daft-doris[doris-flight]\"\n```\n\nRead a Doris table:\n\n```python\nfrom daft_doris import read_doris\n\ndf = read_doris(\n    host=\"doris-fe.example\",\n    database=\"analytics\",\n    table=\"events\",\n    transport=\"mysql\",\n)\n```\n\nWrite a Daft DataFrame through Stream Load:\n\n```python\nimport daft\n\nfrom daft_doris import DorisConnection, DorisTable, SecretRef, write_doris\n\ndf = daft.from_pydict(\n    {\n        \"event_id\": [1, 2],\n        \"score\": [95.0, 88.0],\n    }\n)\nresult = write_doris(\n    df,\n    connection=DorisConnection(\n        host=\"doris-fe.example.com\",\n        username=\"daft_writer\",\n        password=SecretRef.env(\"DORIS_PASSWORD\"),\n        http_port=8030,\n        redirect_hosts=(\"doris-be.example.com\",),\n        redirect_ports=(8040,),\n        redirect_policy=\"public\",\n    ),\n    table=DorisTable(database=\"analytics\", name=\"events\"),\n    operation=\"load\",\n)\n```\n\nThe connector is currently in Alpha and is maintained independently of Daft's\nrelease cadence. The current release supports Python 3.12–3.13 and Daft\n0.7.23; its Daft dependency is bounded to `>=0.7.23,<0.7.24`, so installing it\nalongside a newer Daft release may cause the resolver to install Daft 0.7.23.\nSee the [`daft-doris` repository](https://github.com/jiangxt2/daft-doris#readme)\nfor supported scope, compatibility information, and limitations.\n","docs/extensions/overview.md":"# Extensions\n\nDaft extensions are reusable libraries that add domain-specific functionality\non top of Daft without requiring every specialized function, datatype,\nintegration, or workflow to live directly in Daft core.\n\nThere are two broad ways to build extensions:\n\n- **Python UDF-based extensions**: Python packages built on Daft's\n  [custom-code APIs](../custom-code/func.md), including\n  [`@daft.func`](../api/udf.md), [`@daft.func.batch`](../api/udf.md),\n  [`@daft.cls`](../api/udf.md), and [`@daft.method.batch`](../api/udf.md).\n- **Native ABI extensions**: shared-library-backed packages that use Daft's\n  Arrow C Data Interface ABI for lower-level, vectorized native execution.\n\nBoth models can expose clean Python APIs that feel like ordinary Daft\nexpression functions. The implementation details can stay hidden behind\nfunctions, classes, and expressions.\n\n## Python UDF-Based Extensions\n\nThe fastest way to build a Daft extension is often pure Python. Using\n[`@daft.func`](../api/udf.md), [`@daft.func.batch`](../api/udf.md),\n[`@daft.cls`](../api/udf.md), and [`@daft.method.batch`](../api/udf.md),\ncontributors can package reusable Python logic that runs inside Daft's\ndistributed execution engine.\n\nThis is ideal for libraries that orchestrate existing Python ecosystems,\nexternal services, ML models, GPUs, data maintenance tasks, or domain-specific\nworkflows. These extensions do not require a native shared library or the\nArrow C ABI. They are normal Python packages that expose higher-level Daft\nAPIs.\n\n[`daft-lance`](https://github.com/daft-engine/daft-lance) is an example of\nthis model. It extends Daft with Lance-specific distributed operations such as\nfile compaction, scalar indexing, column merging, and REST catalog operations.\nInternally, it uses Daft's Python UDF and class-UDF APIs to distribute Lance\ntasks across a Daft query, while users interact with functions like\n`compact_files`, `create_scalar_index`, and `merge_columns_df`.\n\nThis pattern also exists inside Daft itself. The\n[`daft.functions.ai`](../api/ai.md) module exposes high-level functions like\n[`prompt`](../api/functions/prompt.md),\n[`embed_text`](../api/functions/embed_text.md),\n[`embed_image`](../api/functions/embed_image.md),\n[`classify_text`](../api/functions/classify_text.md), and\n[`classify_image`](../api/functions/classify_image.md). To users, these look\nlike normal Daft expression functions. Under the hood, they use Daft's UDF and\nclass-UDF machinery, including batching, concurrency controls, retries, and\n[GPU resource hints](../custom-code/gpu.md).\n\nThe file APIs follow a similar product pattern. [`daft.File`](../api/datatypes/file_types.md)\nvalues and helpers like [`file`](../api/functions/file.md),\n[`audio_file`](../api/functions/audio_file.md),\n[`video_file`](../api/functions/video_file.md),\n[`file_path`](../api/functions/file_path.md), and\n[`file_size`](../api/functions/file_size.md) give users expression-level\nbuilding blocks for working with files. Those file objects can then flow into\nPython UDFs, model pipelines, and domain-specific libraries without users\nneeding to think about execution details.\n\n## Native ABI Extensions\n\nFor contributors who need lower-level, vectorized performance, Daft also\nsupports native extensions through a C ABI based on the\n[Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html).\n\nNative ABI extensions can add high-performance scalar functions, aggregate\nfunctions, Python expression wrappers, and extension-backed datatypes. They\nship as pip-installable Python packages that bundle a native shared library.\nUsers import the package, load it into a Daft [`Session`](../api/sessions.md),\nand call ordinary Python wrappers in their DataFrame expressions.\n\nRust currently has the most ergonomic SDK through\n[`daft-ext`](https://github.com/Eventual-Inc/Daft/tree/main/src/daft-ext),\nwhile C++ is demonstrated through the raw ABI. Other systems languages are\npossible if they can produce a shared library, export the expected C ABI, and\nread/write Arrow C Data Interface arrays.\n\nDaft's own examples include:\n\n- [`examples/hello`](https://github.com/Eventual-Inc/Daft/tree/main/examples/hello):\n  a minimal Rust native extension that registers a `greet` scalar function.\n- [`examples/dvector`](https://github.com/Eventual-Inc/Daft/tree/main/examples/dvector):\n  a pgvector-style native extension for vector distance functions such as\n  `l2_distance`, `cosine_distance`, `inner_product`, and `jaccard_distance`.\n- [`examples/hello_cpp`](https://github.com/Eventual-Inc/Daft/tree/main/examples/hello_cpp):\n  a pure C++ native extension using Apache Arrow C++ and the raw Daft C ABI.\n\n## What Can Contributors Build?\n\nContributors can build:\n\n- Pure-Python UDF extension libraries using [`@daft.func`](../api/udf.md),\n  [`@daft.func.batch`](../api/udf.md), [`@daft.cls`](../api/udf.md), and\n  [`@daft.method.batch`](../api/udf.md)\n- Distributed task libraries like\n  [`daft-lance`](https://github.com/daft-engine/daft-lance)\n- GPU-backed model inference extensions\n- AI, file, media, and multimodal processing libraries that hide UDFs behind\n  expression APIs\n- Native scalar functions\n- Native aggregate functions / UDAFs\n- Python expression wrappers\n- Extension-backed logical datatypes such as\n  [`DataType.extension(...)`](../api/datatypes/all_datatypes.md)\n\n## Where to Start\n\n- Browse installable projects in [Community Extensions](community.md).\n- Learn how to build a native extension in the [Authoring Guide](authoring.md).\n- Browse projects using Daft in [Built on Daft](projects.md).\n- Learn more about Daft's [Python UDFs](../custom-code/func.md).\n","docs/extensions/projects.md":"# Built on Daft\n\nThis page highlights projects that depend on or build on top of\n[Daft](https://github.com/Eventual-Inc/Daft).\n\nThese projects are different from [Community Extensions](community.md). An\nextension directly adds reusable Daft APIs, functions, datatypes, or distributed\noperators. A project built on Daft may instead use Daft as a query engine,\nDataFrame runtime, compute backend, or pipeline framework.\n\nTo propose a project for this page, open a PR with a short description and a\nlink to the relevant Daft usage.\n\n## Projects Building on Daft\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [DeltaCat](https://github.com/ray-project/deltacat) | Portable multimodal lakehouse on Ray, Arrow, and Daft. | Daft is a core dependency: `dc.read()` returns a Daft DataFrame by default, and Daft is a first-class compute backend for reading and writing lake data. |\n| [SmooSense](https://github.com/SmooSenseAI/smoosense) | Web-based tool for interactively exploring large-scale multimodal tabular data. | Accepts Daft DataFrames as input in the Jupyter widget (`Sense(daft_df)`) and converts them for display alongside Pandas DataFrames. |\n| [eai-taxonomy](https://github.com/Essential-AI/eai-taxonomy) | Annotation tools and dataset filters from the Essential-Web project. | Uses Daft in annotation pipelines to read Parquet annotation files and compute inter-annotator agreement. |\n| [GraphReduce](https://github.com/wesmadrigal/GraphReduce) | Feature engineering library for large graphs of tabular data. | Supports Daft as a compute backend via `pip install \"graphreduce[daft]\"`. |\n| [Archetype](https://github.com/VangelisTech/archetype) | DataFrame-first, append-only ECS runtime for simulations and AI agents. | Uses Daft DataFrames as the native data structure for world state and processor logic. |\n| [hypergraph](https://github.com/gilad-rubin/hypergraph) | Python workflow orchestration framework for DAG pipelines and agentic workflows. | Provides a `DaftRunner` that compiles a hypergraph DAG into a Daft query plan using Daft UDFs. |\n| [Sashimi4Talent](https://github.com/colin-ho/Sashimi4Talent) | Data pipeline and web app for discovering and ranking GitHub contributors. | Uses Daft as the core processing engine for repository search, commits, contributor analysis, and dataset merges. |\n| [daft-image-playground](https://github.com/peckjon/daft-image-playground) | Flask-based intelligent image search engine. | Uses Daft to discover images and run native image decoding, resizing, and encoding. |\n| [SMiR](https://github.com/togethercomputer/SMiR) | Synthetic data pipeline for multi-image reasoning. | Uses Daft with the Ray runner to read Parquet from S3, download images, and run GPU-accelerated embedding UDFs. |\n| [Derezz](https://github.com/rchowell/Derezz) | CLI tool for video indexing and semantic search. | Uses Daft to read and write an S3 Tables-backed video frame index and query it. |\n| [teraflopai-daft](https://github.com/teraflop-ai/teraflopai-daft) | Daft plugin library for Teraflop AI NLP services. | Exposes custom Daft expressions such as `segment_text` and `search_text` that call the Teraflop AI provider API. |\n| [daft-sql-adapter](https://github.com/RaghwendraSingh/daft-sql-adapter) | CLI adapter that runs Spark SQL queries against Databricks Unity Catalog tables through Daft. | Uses Daft as the execution engine for SQL queries and table writes. |\n| [zarr-daft-datasource](https://github.com/liaoruoxue/zarr-daft-datasource) | Prototype compute-storage-separation architecture using Zarr, LanceDB, and Daft. | Implements a custom Daft `DataSource` that reads Zarr arrays into lazy Daft DataFrames. |\n| [pdf-document-processing-daft](https://github.com/doronkatz/pdf-document-processing-daft) | Demonstration pipeline for scalable PDF document processing. | Uses Daft DataFrames to parallelize PDF parsing, text chunking, embedding generation, and Parquet writes. |\n| [daft-ai-driving](https://github.com/legendPerceptor/daft-ai-driving) | Project for processing the KITTI autonomous driving dataset. | Uses Daft DataFrames for data loading, filtering, aggregation, and Parquet export. |\n| [sigil](https://github.com/junaidrahim/sigil) | CLI tool for collecting and analyzing AI coding session data. | Uses Daft to write local Parquet session data and query it with filtering and aggregation. |\n| [embed_qwen_k8s](https://github.com/jaychia/embed_qwen_k8s) | Kubernetes pipeline for Common Crawl text processing and embeddings. | Uses Daft to read WARC/Parquet data, apply UDFs for chunking and embedding, and write results to S3. |\n\n## Usage Patterns and Examples\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [daft-examples](https://github.com/Eventual-Inc/daft-examples) | Official Daft examples repository. | Shows idiomatic Daft usage across `daft.read_*`, UDFs, AI functions, multimodal data, SQL analytics, vector search, and integrations. |\n| [workflow-eventual-inc-daft-distributed-udf-processing](https://github.com/leeroopedia/workflow-eventual-inc-daft-distributed-udf-processing) | Workflow demonstrating custom Python UDFs for ML inference and GPU transforms. | Uses `@daft.func` and `@daft.cls` with lazy and distributed execution. |\n| [workflow-eventual-inc-daft-sql-query-analytics](https://github.com/leeroopedia/workflow-eventual-inc-daft-sql-query-analytics) | Workflow for SQL analytics with Daft sessions and catalogs. | Creates sessions, attaches catalogs, registers tables, executes SQL, and exports results. |\n| [workflow-eventual-inc-daft-data-lakehouse-etl](https://github.com/leeroopedia/workflow-eventual-inc-daft-data-lakehouse-etl) | ETL pipeline for Iceberg, Delta Lake, and Hudi lakehouse formats. | Uses Daft lakehouse readers, DataFrame transformations, and write APIs. |\n| [workflow-eventual-inc-daft-multimodal-ai-batch-inference](https://github.com/leeroopedia/workflow-eventual-inc-daft-multimodal-ai-batch-inference) | Workflow for AI inference over multimodal datasets. | Applies Daft AI functions and embedding workflows to Hugging Face, Parquet, and CSV datasets. |\n\n## Daft as a Chosen Backend\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [cuallee](https://github.com/canimus/cuallee) | DataFrame-agnostic data quality check library. | Implements a Daft validation backend and test suite for quality checks against Daft DataFrames. |\n| [Atlan application-sdk](https://github.com/atlanhq/application-sdk) | SDK for building data catalog integration applications. | Uses Daft to lazily read and filter JSON table metadata files during incremental sync. |\n| [narwhals-daft](https://github.com/narwhals-dev/narwhals-daft) | Narwhals plugin for Daft. | Wraps Daft DataFrames so the Narwhals expression API can execute lazily on Daft. |\n| [jett](https://github.com/ddeutils/jett) | ETL template framework supporting multiple DataFrame engines. | Lists Daft as a supported engine backend for YAML-defined ETL pipelines. |\n\n## Ecosystem and Content\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [AWS MCP servers](https://github.com/awslabs/mcp) | Collection of open-source MCP servers for AWS services. | The S3 Tables MCP server uses Daft as the SQL query engine for read-only queries against AWS S3 Tables. |\n| [Fabric_Notebooks_Demo](https://github.com/djouallah/Fabric_Notebooks_Demo) | Microsoft Fabric notebook demos. | Includes ETL notebooks using Daft to load and transform CSVs from Azure storage and write Delta Lake tables. |\n| [lakevision](https://github.com/lakevision-project/lakevision) | Web tool for exploring Apache Iceberg lakehouses. | Uses Daft to query and display sample data from Iceberg tables. |\n| [general-demos](https://github.com/jaehyeon-kim/general-demos) | Data engineering demo projects. | Includes a `daft-quickstart` folder demonstrating Daft with Apache Iceberg. |\n| [DaftHudi](https://github.com/dipankarmazumdar/DaftHudi) | Demo dashboard application using Apache Hudi, Daft, and Streamlit. | Uses Daft to read Apache Hudi tables and power analytical queries. |\n| [Engineering Lakehouses with Open Table Formats](https://github.com/PacktPublishing/Engineering-Lakehouses-with-Open-Table-Formats) | Code samples for a lakehouse engineering book. | Includes a notebook demonstrating reading and writing Apache Iceberg tables with Daft. |\n| [databricks-demos](https://github.com/david-hurley/databricks-demos) | Databricks demo notebooks. | Includes a notebook using Daft with Unity Catalog to write data to Delta tables. |\n| [knee-deep-in-the-lake](https://github.com/Cumulocity-IoT/knee-deep-in-the-lake) | Hands-on training repository for IoT data lake technologies. | Uses Daft in notebooks alongside PyArrow and Pandas for querying Parquet files with SQL. |\n","docs/plugins/README.md":"# Daft MkDocs Plugins\n\nCustom MkDocs plugins for the Daft documentation site.\n\n## nav-hide-children\n\nA plugin that allows you to conditionally hide navigation children based on configured paths.\n\n### Configuration\n\nAdd to your `mkdocs.yml`:\n\n```yaml\nplugins:\n  - nav-hide-children:\n      hide_children:\n        - [\"Python API\", \"Functions\"]  # Hide children of Python API > Functions\n        - [\"SQL Reference\"]            # Hide children of SQL Reference\n```\n\n### Usage\n\nThe plugin automatically injects an attribute `hide_children` into [navigation objects](https://www.mkdocs.org/dev-guide/themes/#navigation-objects) based on the config paths, which can be used in Jinja2 templates to conditionally hide navigation elements.\n","docs/skills.md":"# Daft Skills\n\nDaft ships with built-in skills that help AI coding agents work effectively with Daft.\n\n## Available skills\n\n### [daft-udf-tuning](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-udf-tuning/SKILL.md)\n\nOptimize User-Defined Functions for performance.\n\n**Triggers:** GPU inference setup, slow UDFs, async or batch processing questions.\n\n**What it helps with:**\n\n- Choosing between `@daft.func` (stateless), `@daft.cls` (stateful), and `@daft.func.batch` (vectorized)\n- Async I/O patterns for web APIs\n- GPU batch inference with PyTorch/model loading via `@daft.cls(gpus=N)`\n- Tuning `max_concurrency`, `batch_size`, `gpus`, and `into_batches(N)`\n\n### [daft-distributed-scaling](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-distributed-scaling/SKILL.md)\n\nScale single-node workflows to distributed execution.\n\n**Triggers:** Performance optimization, handling large datasets, distributed execution.\n\n**What it helps with:**\n\n- Choosing between shuffle (`repartition(N)`) and streaming (`into_batches(N)`) strategies\n- Calculating optimal partition counts for your cluster\n- Avoiding OOM errors with heavy data (images, tensors)\n- Advanced tuning formulas for repartitioning and streaming\n\n### [daft-docs-navigation](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-docs-navigation/SKILL.md)\n\nNavigate Daft's documentation for APIs, concepts, and examples.\n\n**Triggers:** General questions about Daft APIs, concepts, or searching the docs.\n\n**What it helps with:**\n\n- Navigating key doc locations: API reference, optimization, distributed, UDFs, connectors\n- Searching for API usage in the `docs/` directory\n- Browsing the docs structure via `docs/SUMMARY.md`\n\n## Installation\n\n### Already cloned Daft (automatic)\n\nIf you're working inside a Daft clone, the skills are already available. No setup needed - Claude Code picks them up from `.claude/skills/`.\n\n### Install with `npx skills add`\n\nTo use these Daft skills outside the repo (across many agents including Claude Code, Cursor, Codex, Gemini CLI, Copilot, and more):\n\n```bash\n# Install all skills\nnpx skills add Eventual-Inc/Daft\n\n# Install a specific skill\nnpx skills add Eventual-Inc/Daft --skill daft-udf-tuning\n```\n\nFollow the prompts to choose skills, agents, scope (project or global), and whether to symlink or copy.\n\n### Claude Code plugin\n\nThese Daft skills are also available as a Claude Code plugin.\n\n```bash\nclaude plugin marketplace add Eventual-Inc/Daft\nclaude plugin install daft@Eventual-Inc\n```\n","tools/observability/opentelemetry/README.md":"# OpenTelemetry\n\nThis directory contains the code for running the OpenTelemetry collector and exporter.\n\nThe docker compose file located in this directory will start the following services:\n\n1. [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) - an agent for collecting and exporting opentelemetry data\n2. [Jaeger](https://www.jaegertracing.io/) - an open source distributed tracing system\n3. [Prometheus](https://prometheus.io/) - an open source metrics monitoring system\n\n## Running the collector\n\nTo start the services, navigate to this directory and run the following command:\n\n```bash\ndocker compose up\n```\n\n## Executing a job with opentelemetry enabled\n\nThe opentelemetry collector exposes an OTLP gRPC endpoint on port 4317.\nTo execute a job with opentelemetry enabled, set the `DAFT_DEV_OTEL_EXPORTER_OTLP_ENDPOINT` environment variable to `grpc://localhost:4317` and run the job.\n\nExample:\n```bash\nDAFT_DEV_OTEL_EXPORTER_OTLP_ENDPOINT=grpc://localhost:4317 python tools/observability/opentelemetry/example.py\n```\n\n## Viewing the metrics\n\n### Traces (Jaeger)\n\nTo view the traces on the Jaeger UI, navigate to `http://localhost:16686/` and select the `daft` service from the dropdown menu.\n\n### Metrics (Prometheus)\n\nTo view the metrics on the Prometheus UI, navigate to `http://localhost:9090/`.\n\nTo query for all metrics provided by Daft, try the following [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query:\n\n```promql\n{service_name='daft'}\n```\n"},"files":{".claude-plugin/marketplace.json":"{\n  \"name\": \"Eventual-Inc\",\n  \"owner\": {\n    \"name\": \"Eventual, Inc.\",\n    \"email\": \"yksu@eventualcomputing.com\"\n  },\n  \"plugins\": [\n    {\n      \"name\": \"daft\",\n      \"source\": \"./\",\n      \"description\": \"Skills for working with Daft: UDF tuning, distributed scaling, and docs navigation\",\n      \"version\": \"0.1.0\"\n    }\n  ]\n}\n",".claude-plugin/plugin.json":"{\n  \"name\": \"daft\",\n  \"description\": \"Skills for working with Daft: UDF tuning, distributed scaling, and docs navigation\",\n  \"version\": \"0.1.0\",\n  \"author\": {\n    \"name\": \"Eventual, Inc.\",\n    \"email\": \"yksu@eventualcomputing.com\"\n  },\n  \"repository\": \"https://github.com/Eventual-Inc/Daft\",\n  \"license\": \"Apache-2.0\",\n  \"keywords\": [\n    \"daft\",\n    \"dataframe\",\n    \"udf\",\n    \"distributed\",\n    \"ray\"\n  ]\n}\n",".claude/skills/daft-distributed-scaling/SKILL.md":"---\nname: \"daft-distributed-scaling\"\ndescription: \"Scale Daft workflows to distributed Ray clusters. Invoke when optimizing performance or handling large data.\"\n---\n\n# Daft Distributed Scaling\n\nScale single-node workflows to distributed execution.\n\n## Core Strategies\n\n| Strategy | API | Use Case | Pros/Cons |\n|---|---|---|---|\n| **Shuffle** | `repartition(N)` | Light data (e.g. file paths), Joins | **Global balance**. High memory usage (materializes data). |\n| **Streaming** | `into_batches(N)` | Heavy data (images, tensors) | **Low memory** (streaming). High scheduling overhead if batches too small. |\n\n## Quick Recipes\n\n### 1. Light Data: Repartitioning\nBest for distributing file paths before heavy reads.\n\n```python\n# Create enough partitions to saturate workers\ndf = daft.read_parquet(\"s3://metadata\").repartition(100)\ndf = df.with_column(\"data\", read_heavy_data(df[\"path\"]))\n```\n\n### 2. Heavy Data: Streaming Batches\nBest for processing large partitions without OOM.\n\n```python\n# Stream 1GB partition in 64-row chunks to control memory\ndf = df.read_parquet(\"heavy_data\").into_batches(64)\ndf = df.with_column(\"embed\", model.predict(df[\"img\"]))\n```\n\n## Advanced Tuning\n\nTarget: Keep all actors busy without OOM or scheduling bottlenecks.\n\n### Formula 1: Repartitioning (Light Data / Paths)\nCalculate the **Max Partition Count** to ensure each task has enough data to feed local actors.\n\n1.  **Min Rows Per Partition** = `Batch Size * (Total Concurrency / Nodes)`\n2.  **Max Partitions** = `Total Rows / Min Rows Per Partition`\n\n**Example**:\n- 1M rows, 4 nodes, 16 total concurrency, Batch Size 64.\n- **Min Rows**: `64 * (16/4) = 256`.\n- **Max Partitions**: `1,000,000 / 256 ≈ 3906`.\n- *Recommendation*: Use ~1000 partitions to run multiple batches per task.\n\n```python\ndf = df.repartition(1000) # Balanced fan-out\n```\n\n### Formula 2: Streaming (Heavy Data / Images)\nAvoid creating tiny partitions. Use `into_batches` to stream data within larger partitions.\n\n**Strategy**: Keep partitions large (e.g. 1GB+), use `into_batches(Batch Size)` to control memory.\n\n```python\n# Stream batches to control memory usage per actor\ndf = df.into_batches(64).with_column(\"preds\", model(max_concurrency=16).predict(df[\"img\"]))\n```\n",".claude/skills/daft-docs-navigation/SKILL.md":"---\nname: \"daft-docs-navigation\"\ndescription: \"Navigate Daft documentation. Invoke when user asks general questions about APIs, concepts, or examples, or wants to search the docs.\"\n---\n\n# Daft Docs Navigation\n\nNavigate Daft's documentation for APIs, concepts, and examples.\n\n## Documentation Structure\n\n1.  **Live Site**: [`https://docs.daft.ai`](https://docs.daft.ai) (Primary source, searchable).\n2.  **Source Repo**: `docs/` directory in the repository.\n    -   If `docs/` is missing or empty, clone the repo: `git clone https://github.com/Eventual-Inc/Daft.git`\n\n## Key Locations in `docs/`\n\n-   **API Reference**: `api/` (e.g., `api/io.md` for reading/writing).\n-   **Optimization**: `optimization/` (e.g., `optimization/partitioning.md`).\n-   **Distributed**: `distributed/` (e.g., `distributed/ray.md`).\n-   **UDFs**: `custom-code/` (e.g., `func.md`, `cls.md`).\n-   **Connectors**: `connectors/` (e.g., S3, Lance).\n-   **Table of Contents**: `docs/SUMMARY.md`.\n\n## Usage\n\n**Search for API Usage:**\n```bash\ngrep_search(pattern='from_glob', path='docs/')\n```\n\n**Browse Topics:**\n```bash\nread(file_path='docs/SUMMARY.md')\n```\n",".claude/skills/daft-udf-tuning/SKILL.md":"---\nname: \"daft-udf-tuning\"\ndescription: \"Optimize Daft UDF performance. Invoke when user needs GPU inference, encounters slow UDFs, or asks about async/batch processing.\"\n---\n\n# Daft UDF Tuning\n\nOptimize User-Defined Functions for performance.\n\n## UDF Types\n\n| Type | Decorator | Use Case |\n|---|---|---|\n| **Stateless** | `@daft.func` | Simple transforms. Use `async` for I/O-bound tasks. |\n| **Stateful** | `@daft.cls` | Expensive init (e.g., loading models). Supports `gpus=N`. |\n| **Batch** | `@daft.func.batch` | Vectorized CPU/GPU ops (NumPy/PyTorch). Faster. |\n\n## Quick Recipes\n\n### 1. Async I/O (Web APIs)\n\n```python\n@daft.func\nasync def fetch(url: str):\n    async with aiohttp.ClientSession() as s:\n        return await s.get(url).text()\n```\n\n### 2. GPU Batch Inference (PyTorch/Models)\n\n```python\n@daft.cls(gpus=1)\nclass Classifier:\n    def __init__(self):\n        self.model = load_model().cuda() # Run once per worker\n\n    @daft.method.batch(batch_size=32)\n    def predict(self, images):\n        return self.model(images.to_pylist())\n\n# Run with concurrency\ndf.with_column(\"preds\", Classifier(max_concurrency=4).predict(df[\"img\"]))\n```\n\n## Tuning Keys\n\n-   **`max_concurrency`**: Total parallel UDF instances.\n-   **`gpus=N`**: GPU request per instance.\n-   **`batch_size`**: Rows per call. Too small = overhead; too big = OOM.\n-   **`into_batches(N)`**: Pre-slice partitions if memory is tight.\n","AGENTS.md":"# Resources\n\n- https://docs.daft.ai for the user-facing API docs\n- CONTRIBUTING.md for detailed development process\n- https://github.com/Eventual-Inc/Daft for issues, discussions, and PRs\n\n# Dev Workflow\n\n1. [Once] Set up Python environment and install dependencies: `make .venv`\n2. [Optional] Activate .venv: `source .venv/bin/activate`. Not necessary with Makefile commands.\n3. If Rust code is modified, rebuild: `make build`\n4. Run tests. See [Testing Details](#testing-details).\n\n# Testing Details\n\n- `make test` runs tests in `tests/` directory. Uses `pytest` under the hood.\n  - Must set `DAFT_RUNNER` environment variable to `ray` or `native` to run the tests with the corresponding runner.\n    - Start with `DAFT_RUNNER=native` unless testing Ray or distributed code.\n  - `make test EXTRA_ARGS=\"...\"` passes additional arguments to `pytest`.\n    - `make test EXTRA_ARGS=\"-v tests/dataframe/test_select.py\"` runs the test in the given file.\n    - `make test EXTRA_ARGS=\"-v tests/dataframe/test_select.py::test_select_dataframe\"` runs the given test method.\n  - Default `integration`, `benchmark`, and `hypothesis` tests are disabled. Best to run on CI.\n- `make doctests` runs doctests in `daft/` directory. Tests docstrings in Daft APIs.\n\n# PR Conventions\n\n- Titles: Conventional Commits format; enforced by `.github/workflows/pr-labeller.yml`.\n- Descriptions: follow `.github/pull_request_template.md`.\n","CLAUDE.md":"AGENTS.md","docs/extensions/authoring.md":"# Authoring Guide\n\n!!! warning \"Experimental\"\n\n    Native extensions are experimental and may change in future releases.\n\n> Please see the [prompt](#prompt) if you want help generating an extension.\n> Looking for the broader extension model? See the [Extensions overview](overview.md).\n> Looking for extensions you can install today? See [Community Extensions](community.md).\n\nThis document is a guide for authoring Daft native extensions in Rust.\nDaft supports native Rust extensions by leveraging a stable C ABI based on the\n[Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html).\nExtensions are **not coupled** to any particular Arrow library version. The ABI boundary uses\nplain C structs (`ArrowSchema`, `ArrowArray`) so your extension can use any arrow-rs version\n(or even a different Arrow implementation entirely). Today we support authoring native\nscalar functions (UDFs) and aggregate functions (UDAFs).\n\n## Example\n\nThis example shows the end-result of our 'hello' extension with a native 'greet' scalar function.\n\n```python\nimport daft\n\n# Step 1. Import your extension module\nimport hello\n\n# Step 2. Load the extension into the current daft session\ndaft.load_extension(hello)\n\n# Step 3. Use in your dataframe!\ndf = daft.from_pydict({\"name\": [\"John\", \"Paul\"]})\ndf = df.select(hello.greet(df[\"name\"]))\ndf.show()\n\n\"\"\"\n╭──────────────╮\n│ greet        │\n│ ---          │\n│ String       │\n╞══════════════╡\n│ Hello, John! │\n├╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤\n│ Hello, Paul! │\n╰──────────────╯\n\"\"\"\n```\n\n## Tutorial\n\n### 1. Setup\n\nThis section walks through project setup from scratch; You will need a Rust toolchain and Python 3.10+.\n\n!!! note \"Note\"\n\n    We are actively working on a cloneable template.\n\n\n```bash\n# Install Rust (if not already installed)\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n\n# Install uv (if not already installed)\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n```\n\nNow you can create the project scaffolding.\n\n```bash\n# Create an empty directory\nmkdir hello && cd hello\n\n# Setup rust project\ncargo init --lib\n\n# Setup python project (for function signatures)\nuv init\n```\n\nThe crate must compile as a `cdylib` so it can be loaded at runtime via `dlopen`.\n\n```bash\ncat Cargo.toml\n```\n\n```toml\n[workspace]\n\n[package]\nname = \"hello\"\nedition = \"2024\"\nversion = \"0.1.0\"\n\n[lib]\nname = \"hello\"\ncrate-type = [\"cdylib\"]\n\n[dependencies]\ndaft-ext = { version = <version>, features = [\"arrow-58\"] }\narrow = { version = \"58\", features = [\"ffi\"] }\n```\n\n!!! tip \"Arrow version freedom\"\n\n    The `daft-ext` ABI uses C Data Interface types — your extension is **not** pinned to\n    Daft's arrow-rs version. Enable a feature flag on `daft-ext` matching your arrow-rs\n    version (`arrow-56`, `arrow-57`, or `arrow-58`) to get safe `.into()` conversions\n    between arrow-rs FFI types and the ABI types. For unsupported versions, use the\n    `from_owned`/`into_owned`/`from_raw`/`as_raw` escape hatches on `ArrowArray`\n    and `ArrowSchema`.\n\nThen update the pyproject to use `setuptools-rust` as the build system.\n\n```bash\ncat pyproject.toml\n```\n\n```toml\n[build-system]\nrequires = [\"setuptools\", \"setuptools-rust\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[project]\nname = \"hello\"\nversion = \"0.1.0\"\nrequires-python = \">=3.10\"\ndependencies = [\"daft\"]\n\n[project.optional-dependencies]\ntest = [\"pytest\"]\n```\n\n\nYou will need to create a `setup.py` file with the following contents.\n\n!!! note \"Note\"\n\n    The `RustExtension` entry tells `setuptools-rust` to compile the cdylib and place it inside the Python package directory.\n    We use the `Binding::NoBinding` because Daft extensions export a raw C symbol, not PyO3 bindings. The target name `hello.libhello`\n    places the compiled `.so` inside the `hello/` package directory so the Session's `load_extension` can find it.\n\n```bash\ncat setup.py\n```\n\n```python\nfrom setuptools import find_packages, setup\nfrom setuptools_rust import Binding, RustExtension\n\nsetup(\n    packages=find_packages(),\n    rust_extensions=[\n        RustExtension(\n            \"hello.libhello\",   # <python_package>.<lib_name>\n            path=\"Cargo.toml\",\n            binding=Binding.NoBinding,\n            strip=True,\n        )\n    ],\n)\n```\n\n### 2. Hello, World!\n\nAn extension has two parts: a **module** (the entry point) and one or more **functions** (scalar or aggregate).\n\n```bash\ncat src/lib.rs\n```\n\n```rust\nuse std::{ffi::CStr, sync::Arc};\n\nuse arrow::{\n    array::{Array, builder::StringBuilder, cast::AsArray},\n    datatypes::{DataType, Field},\n};\nuse daft_ext::prelude::*;\n\n// ── Module ──────────────────────────────────────────────────────────\n\n// #[daft_extension] generates the `daft_module_magic` C symbol that Daft's runtime looks for\n// when loading the shared library. It converts HelloExtension → hello_extension for the module name.\n#[daft_extension]\nstruct HelloExtension;\n\nimpl DaftExtension for HelloExtension {\n    /// This is the extension install hook for defining functions in the session.\n    /// Called once when the extension is loaded into a session. Register each function here.\n    fn install(session: &mut dyn DaftSession) {\n        session.define_function(Arc::new(Greet));\n    }\n}\n\n// ── Function ────────────────────────────────────────────────────────\n\n/// The function type which is registered as an Arc<dyn DaftScalarFunction>.\nstruct Greet;\n\nimpl DaftScalarFunction for Greet {\n    /// Function name used to look it up from Python.\n    /// Must be a `&CStr` (use the `c\"...\"` literal).\n    fn name(&self) -> &CStr {\n        c\"greet\"\n    }\n\n    /// Type checking.\n    /// Receives input fields as C Data Interface `ArrowSchema` types.\n    /// Use `.as_raw()` / `.into()` to convert between arrow-rs and ABI types.\n    fn return_field(&self, args: &[ArrowSchema]) -> DaftResult<ArrowSchema> {\n        if args.len() != 1 {\n            return Err(DaftError::TypeError(format!(\n                \"greet: expected 1 argument, got {}\",\n                args.len()\n            )));\n        }\n        let field = Field::try_from(&args[0])?;\n        let dt = field.data_type();\n        if *dt != DataType::Utf8 && *dt != DataType::LargeUtf8 {\n            return Err(DaftError::TypeError(format!(\n                \"greet: expected string argument, got {:?}\",\n                dt\n            )));\n        }\n        Ok(ArrowSchema::try_from(&Field::new(\n            \"greet\",\n            DataType::Utf8,\n            true,\n        ))?)\n    }\n\n    /// Evaluation. Receives columns as C Data Interface `ArrowData` types.\n    /// Use `.into()` to convert to/from arrow-rs FFI types.\n    /// All data flows through Arrow arrays — no per-row Python overhead.\n    fn call(&self, args: Vec<ArrowData>) -> DaftResult<ArrowData> {\n        let data = args.into_iter().next().ok_or_else(|| DaftError::TypeError(\"greet: expected 1 argument, got 0\".into()))?;\n        let ffi_array: arrow::ffi::FFI_ArrowArray = data.array.into();\n        let ffi_schema: arrow::ffi::FFI_ArrowSchema = data.schema.into();\n        let arrow_data = unsafe { arrow::ffi::from_ffi(ffi_array, &ffi_schema) }?;\n        let input = arrow::array::make_array(arrow_data);\n        let names = input.as_string::<i64>();\n        let mut builder = StringBuilder::with_capacity(names.len(), names.len() * 16);\n        for i in 0..names.len() {\n            if names.is_null(i) {\n                builder.append_null();\n            } else {\n                builder.append_value(format!(\"Hello, {}!\", names.value(i)));\n            }\n        }\n        let output = builder.finish();\n        let (out_arr, out_sch) = arrow::ffi::to_ffi(&output.to_data())?;\n        Ok(ArrowData {\n            array: out_arr.into(),\n            schema: out_sch.into(),\n        })\n    }\n}\n```\n\n!!! tip \"ABI pattern\"\n\n    The `DaftScalarFunction` trait uses C Data Interface types (`ArrowSchema`, `ArrowData`)\n    at the ABI boundary. Enable a `daft-ext` feature flag (`arrow-56`, `arrow-57`, or\n    `arrow-58`) matching your arrow-rs version to get `.into()` conversions. Use `.as_raw()`\n    for zero-copy borrows. This decoupling means your extension is not tied to Daft's\n    arrow-rs version.\n\n!!! tip \"String types\"\n\n    Daft uses `LargeUtf8` (i64 offsets) for strings internally. When downcasting string arrays,\n    use `as_string::<i64>()` — using `i32` will panic at runtime. Similarly, when checking types\n    in `return_field`, accept `DataType::LargeUtf8`.\n\n!!! tip \"Naming\"\n\n    Function names are global within a session. Use a prefix (e.g., `myext_greet`) to avoid\n    collisions when your extension defines many functions or might be loaded alongside others.\n\n!!! tip \"Multiple functions\"\n\n    Register as many functions as you need in `install()` — each is an independent struct\n    implementing `DaftScalarFunction`.\n\n!!! tip \"Errors\"\n\n    Return `Err(DaftError::TypeError(...))` for schema violations in `return_field`,\n    and `Err(DaftError::RuntimeError(...))` for execution failures in `call`.\n\n### Aggregate Functions\n\nExtensions can also register **aggregate functions** (UDAFs) alongside scalar functions.\nAn aggregate function follows a three-stage pipeline:\n\n1. **Aggregation** (`aggregate`) — process input arrays into partial state\n2. **Combination** (`combine`) — merge partial states\n3. **Finalization** (`finalize`) — produce the final output from merged state\n\nState is exchanged as `Vec<ArrowData>` — each element is one state field. The FFI layer\npacks these into a Struct array transparently.\n\n```rust\nuse std::ffi::CStr;\nuse daft_ext::prelude::*;\n\nstruct MySum;\n\nimpl DaftAggregateFunction for MySum {\n    fn name(&self) -> &CStr {\n        c\"my_sum\"\n    }\n\n    /// Declare the output type given the input field schemas.\n    fn return_field(&self, _args: &[ArrowSchema]) -> DaftResult<ArrowSchema> {\n        Ok(ArrowSchema::try_from(&Field::new(\"my_sum\", DataType::Int64, true))?)\n    }\n\n    /// Declare intermediate state fields.\n    /// Each state field becomes a child of the Struct exchanged at the FFI boundary.\n    fn state_fields(&self, _args: &[ArrowSchema]) -> DaftResult<Vec<ArrowSchema>> {\n        Ok(vec![\n            ArrowSchema::try_from(&Field::new(\"sum\", DataType::Int64, true))?,\n            ArrowSchema::try_from(&Field::new(\"count\", DataType::Int64, false))?,\n        ])\n    }\n\n    /// Process input arrays into partial state.\n    /// Returns one ArrowData per state field (single-row each).\n    fn aggregate(&self, inputs: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>> {\n        // inputs[0] is the column to sum\n        // compute sum and count, return as two single-element arrays\n        todo!()\n    }\n\n    /// Merge multiple partial states into one.\n    /// states[i] is the i-th state field with multiple rows (one per partial).\n    fn combine(&self, states: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>> {\n        // states[0] = sums array, states[1] = counts array\n        // sum them up, return single-row arrays\n        todo!()\n    }\n\n    /// Produce the final scalar result from merged state.\n    fn finalize(&self, states: Vec<ArrowData>) -> DaftResult<ArrowData> {\n        // states[0] = sum (single value), states[1] = count (single value)\n        // return the final sum\n        todo!()\n    }\n}\n```\n\nRegister aggregate functions in `install()` using `define_aggregate_function`:\n\n```rust\nimpl DaftExtension for MyExtension {\n    fn install(session: &mut dyn DaftSession) {\n        session.define_function(Arc::new(MyScalarFn));\n        session.define_aggregate_function(Arc::new(MySum));\n    }\n}\n```\n\nNow we define the python symbols for use in the Expression DSL; we link to rust via `daft.get_function` for scalars and `daft.get_aggregate_function` for aggregates.\n\n!!! note \"Note\"\n\n    Python sources let you write python function signatures and any argument pre-processing before linking\n    to the function symbol. The `daft.get_function` method is calling `get_function` on the active session\n    to resolve the given name and arguments to some defined function in the session. Notice how we don't\n    technically need python to resolve functions in SQL (hence no PyO3) but these python functions give us\n    nice pythonic functions with autocomplete and doc comments when using the Expression DSL.\n\n```bash\ncat hello/__init__.py\n```\n\n```python\nfrom __future__ import annotations\nfrom typing import TYPE_CHECKING\n\nimport daft\n\nif TYPE_CHECKING:\n    from daft.expressions import Expression\n\ndef greet(name: Expression) -> Expression:\n    \"\"\"Greet someone by name.\"\"\"\n    return daft.get_function(\"greet\", name)\n\ndef string_count(name: Expression) -> Expression:\n    \"\"\"Count non-null strings.\"\"\"\n    return daft.get_aggregate_function(\"string_count\", name)\n```\n\n`daft.get_function` looks up a scalar function registered with the current session by the name returned from `DaftScalarFunction::name()`.\n`daft.get_aggregate_function` does the same for aggregate functions registered via `DaftAggregateFunction::name()`.\n\nAdd an empty `hello/py.typed` marker if you want type-checker support.\n\n### 3. Build, Install, Test\n\n```bash\n# Compile the Rust cdylib and install the package in editable mode\nuv pip install -e .\n```\n\nHere are some sanity check tests. Notice how we use a scoped session rather than\nthe global active session. How you choose to load extensions is up to you, and this\ntutorial has covered both.\n\n!!! tip \"Session isolation\"\n\n    Extensions are loaded once into the process and the session serves as a scoping mechanism\n    for name resolution; calling `load_extension` multiple times will only `dlopen` once for\n    this process. Functions are only available in sessions where the extension is loaded.\n    Use the `with sess:` context manager to scope queries to a specific session.\n\n```bash\ncat tests/test_hello.py\n```\n\n```python\nimport daft\nimport hello\nfrom daft import col\nfrom daft.session import Session\nfrom hello import greet\n\ndef test_greet():\n    sess = Session()\n    sess.load_extension(hello)\n\n    df = daft.from_pydict({\"name\": [\"John\", \"Paul\"]})\n\n    with sess:\n        result = df.select(greet(col(\"name\"))).collect().to_pydict()\n\n    values = result[\"greet\"]\n    assert values[0] == \"Hello, John!\"\n    assert values[1] == \"Hello, Paul!\"\n\ndef test_greet_null():\n    sess = Session()\n    sess.load_extension(hello)\n\n    df = daft.from_pydict({\"name\": [\"George\", \"Ringo\", None]})\n\n    with sess:\n        result = df.select(greet(col(\"name\"))).collect().to_pydict()\n\n    values = result[\"greet\"]\n    assert values[0] == \"Hello, George!\"\n    assert values[1] == \"Hello, Ringo!\"\n    assert values[2] is None\n```\n\nNow run the tests!\n\n```bash\npytest -v tests/\n```\n\n## Prompt\n\nYou can paste this whole document and prompt into Claude Code to scaffold a Daft extension for you.\n\n````markdown\nCreate a Daft native extension called `<extension_name>` with the following functions:\n\n<describe each function: name, scalar or aggregate, arguments with types, return type, and behavior>\n\nFollow the Daft extension authoring guide at docs/extensions/authoring.md. Here is a summary of the key conventions:\n\n## Project structure\n\n```\n<extension_name>/\n  Cargo.toml           # [lib] crate-type = [\"cdylib\"], depends on daft-ext (with arrow feature), arrow\n  pyproject.toml       # build-system: setuptools + setuptools-rust\n  setup.py             # RustExtension(\"<pkg>.lib<pkg>\", binding=NoBinding, strip=True)\n  <extension_name>/\n    __init__.py        # Python wrappers using daft.get_function(\"name\", *args)\n    py.typed           # empty PEP 561 marker\n  src/\n    lib.rs             # #[daft_extension] struct + DaftScalarFunction / DaftAggregateFunction impls\n  tests/\n    test_<name>.py     # pytest tests using Session fixture\n  .gitignore           # /target, *.so, *.dylib, *.dll, *.egg-info, __pycache__, dist/\n```\n\n## Rust conventions\n\n- Use `daft_ext::prelude::*` for all imports (provides `ArrowSchema`, `ArrowData`, errors, traits).\n- Add `daft-ext` with a feature flag matching your arrow version (`arrow-56`, `arrow-57`, or `arrow-58`) for `.into()` conversions.\n- Import `arrow::array::Array` for `len()`/`is_null()` and `arrow::array::cast::AsArray` for downcasting.\n- Daft uses `LargeUtf8` (i64 offsets) for strings — downcast with `as_string::<i64>()`, never `i32`.\n- Apply `#[daft_extension]` to a struct implementing `DaftExtension`.\n- Register scalar functions via `session.define_function(Arc::new(MyFn))`.\n- Register aggregate functions via `session.define_aggregate_function(Arc::new(MyAgg))`.\n- Each scalar function is a struct implementing `DaftScalarFunction` with:\n  - `name(&self) -> &CStr` — use `c\"<extension_name>_<fn_name>\"` prefix to avoid collisions.\n  - `return_field(&self, args: &[ArrowSchema]) -> DaftResult<ArrowSchema>` — use `.as_raw()` to\n    borrow as arrow-rs `FFI_ArrowSchema` for type checking, then `.into()` to return output.\n  - `call(&self, args: &[ArrowData]) -> DaftResult<ArrowData>` — use `ArrowData::take_arg` then\n    `.into()` to convert to arrow-rs FFI types, compute, then `.into()` to return the result.\n- Each aggregate function is a struct implementing `DaftAggregateFunction` with:\n  - `name`, `return_field` — same as scalar functions.\n  - `state_fields(&self, args: &[ArrowSchema]) -> DaftResult<Vec<ArrowSchema>>` — intermediate state schema.\n  - `aggregate(&self, inputs: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>>` — partial aggregation.\n  - `combine(&self, states: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>>` — merge partial states.\n  - `finalize(&self, states: Vec<ArrowData>) -> DaftResult<ArrowData>` — produce final result.\n\n## Python conventions\n\n- Each scalar function wrapper calls `daft.get_function(\"<extension_name>_<fn_name>\", *args)`.\n- Each aggregate function wrapper calls `daft.get_aggregate_function(\"<extension_name>_<fn_name>\", *args)`.\n- Use `TYPE_CHECKING` guard for `Expression` import.\n- Add type hints and a docstring to each wrapper.\n````\n","docs/extensions/community.md":"# Community Extensions\n\n!!! tip \"Want to build your own extension?\"\n\n    Start with the [Extensions overview](overview.md), the [Python UDF docs](../custom-code/func.md),\n    or the [Native Extension Authoring Guide](authoring.md).\n\nCommunity extensions are reusable packages that add functionality to Daft\noutside of the core repository. They may be pure-Python UDF-based extensions,\nnative ABI extensions, custom DataSource/DataSink connectors, or higher-level\ndomain libraries that combine Daft expressions, UDFs, file types, model calls,\nand distributed execution patterns.\n\nProjects listed here are maintained independently of Daft's release cadence.\nEach project installs and versions independently of Daft.\n\n| Name | Kind | Repository | Description |\n|---|---|---|---|\n| [daft-h3](#daft-h3) | Native ABI | [gweaverbiodev/daft-h3](https://github.com/gweaverbiodev/daft-h3) | Native [H3](https://h3geo.org/) geospatial indexing functions. |\n| [daft-lance](#daft-lance) | Python UDF-based | [daft-engine/daft-lance](https://github.com/daft-engine/daft-lance) | Lance-specific distributed operations for compaction, scalar indexing, column merging, and REST catalog operations. |\n| [daft-html](#daft-html) | Native ABI | [daft-engine/daft-html](https://github.com/daft-engine/daft-html) | Native HTML document processing functions exposed as Daft expressions. |\n| [daft-geo](#daft-geo) | Native ABI / Datatypes | [daft-engine/daft-geo](https://github.com/daft-engine/daft-geo) | Geospatial prototype showing native functions and extension-backed datatypes. |\n| [daft-qdrant](#daft-qdrant) | Python UDF-based | [qdrant-labs/daft-qdrant](https://github.com/qdrant-labs/daft-qdrant) | Write vector embeddings and their payloads into [Qdrant](https://qdrant.tech/) collections. |\n| [daft-doris](#daft-doris) | Custom DataSource/DataSink | [jiangxt2/daft-doris](https://github.com/jiangxt2/daft-doris) | Apache Doris reads and HTTP Stream Load writes for Daft. |\n\nTo propose a new extension for this list, open a PR against this page.\n\n## daft-h3\n\n`daft-h3` adds H3 geospatial indexing functions such as latitude/longitude to\ncell conversion, cell traversal, grid distance, parent resolution, and string\nconversion.\n\n```bash\npip install daft-h3\n```\n\n```python\nimport daft\nimport daft_h3\nfrom daft import col\n\ndaft.load_extension(daft_h3)\n\ndf = daft.from_pydict({\"lat\": [37.7749], \"lng\": [-122.4194]})\ndf = df.select(\n    daft_h3.h3_latlng_to_cell(col(\"lat\"), col(\"lng\"), 7).alias(\"cell\"),\n)\ndf = df.select(daft_h3.h3_cell_to_str(col(\"cell\")).alias(\"hex\"))\ndf.show()\n```\n\nSee the [daft-h3 README](https://github.com/gweaverbiodev/daft-h3#readme) for\nthe full list of functions and behavior details.\n\n## daft-lance\n\n`daft-lance` extends Daft with Lance-specific distributed maintenance and data\nmanagement operations. It is a Python UDF-based extension: internally, it uses\nDaft's Python UDF and class-UDF APIs to distribute Lance tasks across Daft\nqueries, while users interact with simple Python functions.\n\n```bash\npip install daft-lance\n```\n\n```python\nfrom daft_lance import compact_files, create_scalar_index\n\ncompact_files(\"s3://bucket/my_dataset\")\n\ncreate_scalar_index(\n    \"s3://bucket/my_dataset\",\n    column=\"name\",\n    index_type=\"INVERTED\",\n)\n```\n\nSee the [daft-lance README](https://github.com/daft-engine/daft-lance#readme)\nfor additional operations such as column merging and REST catalog writes.\n\n## daft-html\n\n`daft-html` adds native HTML processing functions such as `html_to_text`,\n`html_extract_links`, `html_extract_tables`, and CSS-selector extraction.\n\n```bash\npip install daft-html\n```\n\n```python\nimport daft\nimport daft_html\nfrom daft import col\nfrom daft_html import html_to_text\n\ndaft.load_extension(daft_html)\n\ndf = daft.from_pydict({\"html\": [\"<html><body><h1>Hello</h1></body></html>\"]})\ndf = df.select(html_to_text(col(\"html\")).alias(\"text\"))\ndf.show()\n```\n\nSee the [daft-html README](https://github.com/daft-engine/daft-html#readme)\nfor the full list of document and CSS-selector functions.\n\n## daft-geo\n\n`daft-geo` is a geospatial prototype showing native Daft functions and\nextension-backed datatypes. It defines `Point2D` and `Point3D` as\n`DataType.extension(...)` values and provides point construction, accessors,\nEuclidean distance, and haversine distance.\n\n```bash\npip install git+https://github.com/daft-engine/daft-geo.git\n```\n\n!!! note\n\n    `daft-geo` is not yet published to PyPI. The command above installs the\n    latest commit from the repository's default branch, so behavior may change\n    between installs.\n\n```python\nimport daft\nimport daft_geo\nfrom daft import col\n\ndaft.load_extension(daft_geo)\n\ndf = daft.from_pydict({\"x\": [0.0], \"y\": [1.0]})\ndf = df.select(daft_geo.point2d(col(\"x\"), col(\"y\")).alias(\"point\"))\ndf = df.select(daft_geo.x(col(\"point\")).alias(\"x\"))\ndf.show()\n```\n\nSee the [daft-geo README](https://github.com/daft-engine/daft-geo#readme) for\ncurrent capabilities and caveats.\n\n## daft-qdrant\n\n`daft-qdrant` adds a `write_qdrant` DataFrame method for writing vector\nembeddings and their payloads into [Qdrant](https://qdrant.tech/) collections.\n\n```bash\npip install daft-qdrant\n```\n\n```python\nimport daft\nimport daft_qdrant\n\ndf = daft.from_pydict({\n    \"id\": [1, 2, 3],\n    \"vector\": [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]],\n    \"label\": [\"cat\", \"dog\", \"bird\"],\n})\n\ndf.write_qdrant(\"my-collection\", url=\"http://localhost:6333\")\n```\n\nSee the [daft-qdrant README](https://github.com/qdrant-labs/daft-qdrant#readme) for embedding pipeline examples and additional options.\n\n## daft-doris\n\n`daft-doris` is an independently maintained Python custom connector for\nApache Doris. It supports reads from Doris physical tables and batch writes\nthrough HTTP Stream Load. The read transport must be selected explicitly;\nFlight SQL is available as an opt-in experimental transport.\n\n```bash\npip install \"daft-doris[doris]\"\n```\n\nFor Flight SQL read support, install the optional Flight extra:\n\n```bash\npip install \"daft-doris[doris-flight]\"\n```\n\nRead a Doris table:\n\n```python\nfrom daft_doris import read_doris\n\ndf = read_doris(\n    host=\"doris-fe.example\",\n    database=\"analytics\",\n    table=\"events\",\n    transport=\"mysql\",\n)\n```\n\nWrite a Daft DataFrame through Stream Load:\n\n```python\nimport daft\n\nfrom daft_doris import DorisConnection, DorisTable, SecretRef, write_doris\n\ndf = daft.from_pydict(\n    {\n        \"event_id\": [1, 2],\n        \"score\": [95.0, 88.0],\n    }\n)\nresult = write_doris(\n    df,\n    connection=DorisConnection(\n        host=\"doris-fe.example.com\",\n        username=\"daft_writer\",\n        password=SecretRef.env(\"DORIS_PASSWORD\"),\n        http_port=8030,\n        redirect_hosts=(\"doris-be.example.com\",),\n        redirect_ports=(8040,),\n        redirect_policy=\"public\",\n    ),\n    table=DorisTable(database=\"analytics\", name=\"events\"),\n    operation=\"load\",\n)\n```\n\nThe connector is currently in Alpha and is maintained independently of Daft's\nrelease cadence. The current release supports Python 3.12–3.13 and Daft\n0.7.23; its Daft dependency is bounded to `>=0.7.23,<0.7.24`, so installing it\nalongside a newer Daft release may cause the resolver to install Daft 0.7.23.\nSee the [`daft-doris` repository](https://github.com/jiangxt2/daft-doris#readme)\nfor supported scope, compatibility information, and limitations.\n","docs/extensions/overview.md":"# Extensions\n\nDaft extensions are reusable libraries that add domain-specific functionality\non top of Daft without requiring every specialized function, datatype,\nintegration, or workflow to live directly in Daft core.\n\nThere are two broad ways to build extensions:\n\n- **Python UDF-based extensions**: Python packages built on Daft's\n  [custom-code APIs](../custom-code/func.md), including\n  [`@daft.func`](../api/udf.md), [`@daft.func.batch`](../api/udf.md),\n  [`@daft.cls`](../api/udf.md), and [`@daft.method.batch`](../api/udf.md).\n- **Native ABI extensions**: shared-library-backed packages that use Daft's\n  Arrow C Data Interface ABI for lower-level, vectorized native execution.\n\nBoth models can expose clean Python APIs that feel like ordinary Daft\nexpression functions. The implementation details can stay hidden behind\nfunctions, classes, and expressions.\n\n## Python UDF-Based Extensions\n\nThe fastest way to build a Daft extension is often pure Python. Using\n[`@daft.func`](../api/udf.md), [`@daft.func.batch`](../api/udf.md),\n[`@daft.cls`](../api/udf.md), and [`@daft.method.batch`](../api/udf.md),\ncontributors can package reusable Python logic that runs inside Daft's\ndistributed execution engine.\n\nThis is ideal for libraries that orchestrate existing Python ecosystems,\nexternal services, ML models, GPUs, data maintenance tasks, or domain-specific\nworkflows. These extensions do not require a native shared library or the\nArrow C ABI. They are normal Python packages that expose higher-level Daft\nAPIs.\n\n[`daft-lance`](https://github.com/daft-engine/daft-lance) is an example of\nthis model. It extends Daft with Lance-specific distributed operations such as\nfile compaction, scalar indexing, column merging, and REST catalog operations.\nInternally, it uses Daft's Python UDF and class-UDF APIs to distribute Lance\ntasks across a Daft query, while users interact with functions like\n`compact_files`, `create_scalar_index`, and `merge_columns_df`.\n\nThis pattern also exists inside Daft itself. The\n[`daft.functions.ai`](../api/ai.md) module exposes high-level functions like\n[`prompt`](../api/functions/prompt.md),\n[`embed_text`](../api/functions/embed_text.md),\n[`embed_image`](../api/functions/embed_image.md),\n[`classify_text`](../api/functions/classify_text.md), and\n[`classify_image`](../api/functions/classify_image.md). To users, these look\nlike normal Daft expression functions. Under the hood, they use Daft's UDF and\nclass-UDF machinery, including batching, concurrency controls, retries, and\n[GPU resource hints](../custom-code/gpu.md).\n\nThe file APIs follow a similar product pattern. [`daft.File`](../api/datatypes/file_types.md)\nvalues and helpers like [`file`](../api/functions/file.md),\n[`audio_file`](../api/functions/audio_file.md),\n[`video_file`](../api/functions/video_file.md),\n[`file_path`](../api/functions/file_path.md), and\n[`file_size`](../api/functions/file_size.md) give users expression-level\nbuilding blocks for working with files. Those file objects can then flow into\nPython UDFs, model pipelines, and domain-specific libraries without users\nneeding to think about execution details.\n\n## Native ABI Extensions\n\nFor contributors who need lower-level, vectorized performance, Daft also\nsupports native extensions through a C ABI based on the\n[Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html).\n\nNative ABI extensions can add high-performance scalar functions, aggregate\nfunctions, Python expression wrappers, and extension-backed datatypes. They\nship as pip-installable Python packages that bundle a native shared library.\nUsers import the package, load it into a Daft [`Session`](../api/sessions.md),\nand call ordinary Python wrappers in their DataFrame expressions.\n\nRust currently has the most ergonomic SDK through\n[`daft-ext`](https://github.com/Eventual-Inc/Daft/tree/main/src/daft-ext),\nwhile C++ is demonstrated through the raw ABI. Other systems languages are\npossible if they can produce a shared library, export the expected C ABI, and\nread/write Arrow C Data Interface arrays.\n\nDaft's own examples include:\n\n- [`examples/hello`](https://github.com/Eventual-Inc/Daft/tree/main/examples/hello):\n  a minimal Rust native extension that registers a `greet` scalar function.\n- [`examples/dvector`](https://github.com/Eventual-Inc/Daft/tree/main/examples/dvector):\n  a pgvector-style native extension for vector distance functions such as\n  `l2_distance`, `cosine_distance`, `inner_product`, and `jaccard_distance`.\n- [`examples/hello_cpp`](https://github.com/Eventual-Inc/Daft/tree/main/examples/hello_cpp):\n  a pure C++ native extension using Apache Arrow C++ and the raw Daft C ABI.\n\n## What Can Contributors Build?\n\nContributors can build:\n\n- Pure-Python UDF extension libraries using [`@daft.func`](../api/udf.md),\n  [`@daft.func.batch`](../api/udf.md), [`@daft.cls`](../api/udf.md), and\n  [`@daft.method.batch`](../api/udf.md)\n- Distributed task libraries like\n  [`daft-lance`](https://github.com/daft-engine/daft-lance)\n- GPU-backed model inference extensions\n- AI, file, media, and multimodal processing libraries that hide UDFs behind\n  expression APIs\n- Native scalar functions\n- Native aggregate functions / UDAFs\n- Python expression wrappers\n- Extension-backed logical datatypes such as\n  [`DataType.extension(...)`](../api/datatypes/all_datatypes.md)\n\n## Where to Start\n\n- Browse installable projects in [Community Extensions](community.md).\n- Learn how to build a native extension in the [Authoring Guide](authoring.md).\n- Browse projects using Daft in [Built on Daft](projects.md).\n- Learn more about Daft's [Python UDFs](../custom-code/func.md).\n","docs/extensions/projects.md":"# Built on Daft\n\nThis page highlights projects that depend on or build on top of\n[Daft](https://github.com/Eventual-Inc/Daft).\n\nThese projects are different from [Community Extensions](community.md). An\nextension directly adds reusable Daft APIs, functions, datatypes, or distributed\noperators. A project built on Daft may instead use Daft as a query engine,\nDataFrame runtime, compute backend, or pipeline framework.\n\nTo propose a project for this page, open a PR with a short description and a\nlink to the relevant Daft usage.\n\n## Projects Building on Daft\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [DeltaCat](https://github.com/ray-project/deltacat) | Portable multimodal lakehouse on Ray, Arrow, and Daft. | Daft is a core dependency: `dc.read()` returns a Daft DataFrame by default, and Daft is a first-class compute backend for reading and writing lake data. |\n| [SmooSense](https://github.com/SmooSenseAI/smoosense) | Web-based tool for interactively exploring large-scale multimodal tabular data. | Accepts Daft DataFrames as input in the Jupyter widget (`Sense(daft_df)`) and converts them for display alongside Pandas DataFrames. |\n| [eai-taxonomy](https://github.com/Essential-AI/eai-taxonomy) | Annotation tools and dataset filters from the Essential-Web project. | Uses Daft in annotation pipelines to read Parquet annotation files and compute inter-annotator agreement. |\n| [GraphReduce](https://github.com/wesmadrigal/GraphReduce) | Feature engineering library for large graphs of tabular data. | Supports Daft as a compute backend via `pip install \"graphreduce[daft]\"`. |\n| [Archetype](https://github.com/VangelisTech/archetype) | DataFrame-first, append-only ECS runtime for simulations and AI agents. | Uses Daft DataFrames as the native data structure for world state and processor logic. |\n| [hypergraph](https://github.com/gilad-rubin/hypergraph) | Python workflow orchestration framework for DAG pipelines and agentic workflows. | Provides a `DaftRunner` that compiles a hypergraph DAG into a Daft query plan using Daft UDFs. |\n| [Sashimi4Talent](https://github.com/colin-ho/Sashimi4Talent) | Data pipeline and web app for discovering and ranking GitHub contributors. | Uses Daft as the core processing engine for repository search, commits, contributor analysis, and dataset merges. |\n| [daft-image-playground](https://github.com/peckjon/daft-image-playground) | Flask-based intelligent image search engine. | Uses Daft to discover images and run native image decoding, resizing, and encoding. |\n| [SMiR](https://github.com/togethercomputer/SMiR) | Synthetic data pipeline for multi-image reasoning. | Uses Daft with the Ray runner to read Parquet from S3, download images, and run GPU-accelerated embedding UDFs. |\n| [Derezz](https://github.com/rchowell/Derezz) | CLI tool for video indexing and semantic search. | Uses Daft to read and write an S3 Tables-backed video frame index and query it. |\n| [teraflopai-daft](https://github.com/teraflop-ai/teraflopai-daft) | Daft plugin library for Teraflop AI NLP services. | Exposes custom Daft expressions such as `segment_text` and `search_text` that call the Teraflop AI provider API. |\n| [daft-sql-adapter](https://github.com/RaghwendraSingh/daft-sql-adapter) | CLI adapter that runs Spark SQL queries against Databricks Unity Catalog tables through Daft. | Uses Daft as the execution engine for SQL queries and table writes. |\n| [zarr-daft-datasource](https://github.com/liaoruoxue/zarr-daft-datasource) | Prototype compute-storage-separation architecture using Zarr, LanceDB, and Daft. | Implements a custom Daft `DataSource` that reads Zarr arrays into lazy Daft DataFrames. |\n| [pdf-document-processing-daft](https://github.com/doronkatz/pdf-document-processing-daft) | Demonstration pipeline for scalable PDF document processing. | Uses Daft DataFrames to parallelize PDF parsing, text chunking, embedding generation, and Parquet writes. |\n| [daft-ai-driving](https://github.com/legendPerceptor/daft-ai-driving) | Project for processing the KITTI autonomous driving dataset. | Uses Daft DataFrames for data loading, filtering, aggregation, and Parquet export. |\n| [sigil](https://github.com/junaidrahim/sigil) | CLI tool for collecting and analyzing AI coding session data. | Uses Daft to write local Parquet session data and query it with filtering and aggregation. |\n| [embed_qwen_k8s](https://github.com/jaychia/embed_qwen_k8s) | Kubernetes pipeline for Common Crawl text processing and embeddings. | Uses Daft to read WARC/Parquet data, apply UDFs for chunking and embedding, and write results to S3. |\n\n## Usage Patterns and Examples\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [daft-examples](https://github.com/Eventual-Inc/daft-examples) | Official Daft examples repository. | Shows idiomatic Daft usage across `daft.read_*`, UDFs, AI functions, multimodal data, SQL analytics, vector search, and integrations. |\n| [workflow-eventual-inc-daft-distributed-udf-processing](https://github.com/leeroopedia/workflow-eventual-inc-daft-distributed-udf-processing) | Workflow demonstrating custom Python UDFs for ML inference and GPU transforms. | Uses `@daft.func` and `@daft.cls` with lazy and distributed execution. |\n| [workflow-eventual-inc-daft-sql-query-analytics](https://github.com/leeroopedia/workflow-eventual-inc-daft-sql-query-analytics) | Workflow for SQL analytics with Daft sessions and catalogs. | Creates sessions, attaches catalogs, registers tables, executes SQL, and exports results. |\n| [workflow-eventual-inc-daft-data-lakehouse-etl](https://github.com/leeroopedia/workflow-eventual-inc-daft-data-lakehouse-etl) | ETL pipeline for Iceberg, Delta Lake, and Hudi lakehouse formats. | Uses Daft lakehouse readers, DataFrame transformations, and write APIs. |\n| [workflow-eventual-inc-daft-multimodal-ai-batch-inference](https://github.com/leeroopedia/workflow-eventual-inc-daft-multimodal-ai-batch-inference) | Workflow for AI inference over multimodal datasets. | Applies Daft AI functions and embedding workflows to Hugging Face, Parquet, and CSV datasets. |\n\n## Daft as a Chosen Backend\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [cuallee](https://github.com/canimus/cuallee) | DataFrame-agnostic data quality check library. | Implements a Daft validation backend and test suite for quality checks against Daft DataFrames. |\n| [Atlan application-sdk](https://github.com/atlanhq/application-sdk) | SDK for building data catalog integration applications. | Uses Daft to lazily read and filter JSON table metadata files during incremental sync. |\n| [narwhals-daft](https://github.com/narwhals-dev/narwhals-daft) | Narwhals plugin for Daft. | Wraps Daft DataFrames so the Narwhals expression API can execute lazily on Daft. |\n| [jett](https://github.com/ddeutils/jett) | ETL template framework supporting multiple DataFrame engines. | Lists Daft as a supported engine backend for YAML-defined ETL pipelines. |\n\n## Ecosystem and Content\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [AWS MCP servers](https://github.com/awslabs/mcp) | Collection of open-source MCP servers for AWS services. | The S3 Tables MCP server uses Daft as the SQL query engine for read-only queries against AWS S3 Tables. |\n| [Fabric_Notebooks_Demo](https://github.com/djouallah/Fabric_Notebooks_Demo) | Microsoft Fabric notebook demos. | Includes ETL notebooks using Daft to load and transform CSVs from Azure storage and write Delta Lake tables. |\n| [lakevision](https://github.com/lakevision-project/lakevision) | Web tool for exploring Apache Iceberg lakehouses. | Uses Daft to query and display sample data from Iceberg tables. |\n| [general-demos](https://github.com/jaehyeon-kim/general-demos) | Data engineering demo projects. | Includes a `daft-quickstart` folder demonstrating Daft with Apache Iceberg. |\n| [DaftHudi](https://github.com/dipankarmazumdar/DaftHudi) | Demo dashboard application using Apache Hudi, Daft, and Streamlit. | Uses Daft to read Apache Hudi tables and power analytical queries. |\n| [Engineering Lakehouses with Open Table Formats](https://github.com/PacktPublishing/Engineering-Lakehouses-with-Open-Table-Formats) | Code samples for a lakehouse engineering book. | Includes a notebook demonstrating reading and writing Apache Iceberg tables with Daft. |\n| [databricks-demos](https://github.com/david-hurley/databricks-demos) | Databricks demo notebooks. | Includes a notebook using Daft with Unity Catalog to write data to Delta tables. |\n| [knee-deep-in-the-lake](https://github.com/Cumulocity-IoT/knee-deep-in-the-lake) | Hands-on training repository for IoT data lake technologies. | Uses Daft in notebooks alongside PyArrow and Pandas for querying Parquet files with SQL. |\n","docs/plugins/README.md":"# Daft MkDocs Plugins\n\nCustom MkDocs plugins for the Daft documentation site.\n\n## nav-hide-children\n\nA plugin that allows you to conditionally hide navigation children based on configured paths.\n\n### Configuration\n\nAdd to your `mkdocs.yml`:\n\n```yaml\nplugins:\n  - nav-hide-children:\n      hide_children:\n        - [\"Python API\", \"Functions\"]  # Hide children of Python API > Functions\n        - [\"SQL Reference\"]            # Hide children of SQL Reference\n```\n\n### Usage\n\nThe plugin automatically injects an attribute `hide_children` into [navigation objects](https://www.mkdocs.org/dev-guide/themes/#navigation-objects) based on the config paths, which can be used in Jinja2 templates to conditionally hide navigation elements.\n","docs/skills.md":"# Daft Skills\n\nDaft ships with built-in skills that help AI coding agents work effectively with Daft.\n\n## Available skills\n\n### [daft-udf-tuning](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-udf-tuning/SKILL.md)\n\nOptimize User-Defined Functions for performance.\n\n**Triggers:** GPU inference setup, slow UDFs, async or batch processing questions.\n\n**What it helps with:**\n\n- Choosing between `@daft.func` (stateless), `@daft.cls` (stateful), and `@daft.func.batch` (vectorized)\n- Async I/O patterns for web APIs\n- GPU batch inference with PyTorch/model loading via `@daft.cls(gpus=N)`\n- Tuning `max_concurrency`, `batch_size`, `gpus`, and `into_batches(N)`\n\n### [daft-distributed-scaling](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-distributed-scaling/SKILL.md)\n\nScale single-node workflows to distributed execution.\n\n**Triggers:** Performance optimization, handling large datasets, distributed execution.\n\n**What it helps with:**\n\n- Choosing between shuffle (`repartition(N)`) and streaming (`into_batches(N)`) strategies\n- Calculating optimal partition counts for your cluster\n- Avoiding OOM errors with heavy data (images, tensors)\n- Advanced tuning formulas for repartitioning and streaming\n\n### [daft-docs-navigation](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-docs-navigation/SKILL.md)\n\nNavigate Daft's documentation for APIs, concepts, and examples.\n\n**Triggers:** General questions about Daft APIs, concepts, or searching the docs.\n\n**What it helps with:**\n\n- Navigating key doc locations: API reference, optimization, distributed, UDFs, connectors\n- Searching for API usage in the `docs/` directory\n- Browsing the docs structure via `docs/SUMMARY.md`\n\n## Installation\n\n### Already cloned Daft (automatic)\n\nIf you're working inside a Daft clone, the skills are already available. No setup needed - Claude Code picks them up from `.claude/skills/`.\n\n### Install with `npx skills add`\n\nTo use these Daft skills outside the repo (across many agents including Claude Code, Cursor, Codex, Gemini CLI, Copilot, and more):\n\n```bash\n# Install all skills\nnpx skills add Eventual-Inc/Daft\n\n# Install a specific skill\nnpx skills add Eventual-Inc/Daft --skill daft-udf-tuning\n```\n\nFollow the prompts to choose skills, agents, scope (project or global), and whether to symlink or copy.\n\n### Claude Code plugin\n\nThese Daft skills are also available as a Claude Code plugin.\n\n```bash\nclaude plugin marketplace add Eventual-Inc/Daft\nclaude plugin install daft@Eventual-Inc\n```\n","tools/observability/opentelemetry/README.md":"# OpenTelemetry\n\nThis directory contains the code for running the OpenTelemetry collector and exporter.\n\nThe docker compose file located in this directory will start the following services:\n\n1. [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) - an agent for collecting and exporting opentelemetry data\n2. [Jaeger](https://www.jaegertracing.io/) - an open source distributed tracing system\n3. [Prometheus](https://prometheus.io/) - an open source metrics monitoring system\n\n## Running the collector\n\nTo start the services, navigate to this directory and run the following command:\n\n```bash\ndocker compose up\n```\n\n## Executing a job with opentelemetry enabled\n\nThe opentelemetry collector exposes an OTLP gRPC endpoint on port 4317.\nTo execute a job with opentelemetry enabled, set the `DAFT_DEV_OTEL_EXPORTER_OTLP_ENDPOINT` environment variable to `grpc://localhost:4317` and run the job.\n\nExample:\n```bash\nDAFT_DEV_OTEL_EXPORTER_OTLP_ENDPOINT=grpc://localhost:4317 python tools/observability/opentelemetry/example.py\n```\n\n## Viewing the metrics\n\n### Traces (Jaeger)\n\nTo view the traces on the Jaeger UI, navigate to `http://localhost:16686/` and select the `daft` service from the dropdown menu.\n\n### Metrics (Prometheus)\n\nTo view the metrics on the Prometheus UI, navigate to `http://localhost:9090/`.\n\nTo query for all metrics provided by Daft, try the following [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query:\n\n```promql\n{service_name='daft'}\n```\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/Eventual-Inc/Daft/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# Resources\n\n- https://docs.daft.ai for the user-facing API docs\n- CONTRIBUTING.md for detailed development process\n- https://github.com/Eventual-Inc/Daft for issues, discussions, and PRs\n\n# Dev Workflow\n\n1. [Once] Set up Python environment and install dependencies: `make .venv`\n2. [Optional] Activate .venv: `source .venv/bin/activate`. Not necessary with Makefile commands.\n3. If Rust code is modified, rebuild: `make build`\n4. Run tests. See [Testing Details](#testing-details).\n\n# Testing Details\n\n- `make test` runs tests in `tests/` directory. Uses `pytest` under the hood.\n  - Must set `DAFT_RUNNER` environment variable to `ray` or `native` to run the tests with the corresponding runner.\n    - Start with `DAFT_RUNNER=native` unless testing Ray or distributed code.\n  - `make test EXTRA_ARGS=\"...\"` passes additional arguments to `pytest`.\n    - `make test EXTRA_ARGS=\"-v tests/dataframe/test_select.py\"` runs the test in the given file.\n    - `make test EXTRA_ARGS=\"-v tests/dataframe/test_select.py::test_select_dataframe\"` runs the given test method.\n  - Default `integration`, `benchmark`, and `hypothesis` tests are disabled. Best to run on CI.\n- `make doctests` runs doctests in `daft/` directory. Tests docstrings in Daft APIs.\n\n# PR Conventions\n\n- Titles: Conventional Commits format; enforced by `.github/workflows/pr-labeller.yml`.\n- Descriptions: follow `.github/pull_request_template.md`.\n","isInternal":false,"tokens":357,"sizeBytes":1411},{"name":"skills.md","path":"docs/skills.md","rawUrl":"https://raw.githubusercontent.com/Eventual-Inc/Daft/HEAD/docs/skills.md","title":"docs - Workspace Skills & Capabilities Index","category":"root-instruction","format":"markdown","content":"# Daft Skills\n\nDaft ships with built-in skills that help AI coding agents work effectively with Daft.\n\n## Available skills\n\n### [daft-udf-tuning](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-udf-tuning/SKILL.md)\n\nOptimize User-Defined Functions for performance.\n\n**Triggers:** GPU inference setup, slow UDFs, async or batch processing questions.\n\n**What it helps with:**\n\n- Choosing between `@daft.func` (stateless), `@daft.cls` (stateful), and `@daft.func.batch` (vectorized)\n- Async I/O patterns for web APIs\n- GPU batch inference with PyTorch/model loading via `@daft.cls(gpus=N)`\n- Tuning `max_concurrency`, `batch_size`, `gpus`, and `into_batches(N)`\n\n### [daft-distributed-scaling](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-distributed-scaling/SKILL.md)\n\nScale single-node workflows to distributed execution.\n\n**Triggers:** Performance optimization, handling large datasets, distributed execution.\n\n**What it helps with:**\n\n- Choosing between shuffle (`repartition(N)`) and streaming (`into_batches(N)`) strategies\n- Calculating optimal partition counts for your cluster\n- Avoiding OOM errors with heavy data (images, tensors)\n- Advanced tuning formulas for repartitioning and streaming\n\n### [daft-docs-navigation](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-docs-navigation/SKILL.md)\n\nNavigate Daft's documentation for APIs, concepts, and examples.\n\n**Triggers:** General questions about Daft APIs, concepts, or searching the docs.\n\n**What it helps with:**\n\n- Navigating key doc locations: API reference, optimization, distributed, UDFs, connectors\n- Searching for API usage in the `docs/` directory\n- Browsing the docs structure via `docs/SUMMARY.md`\n\n## Installation\n\n### Already cloned Daft (automatic)\n\nIf you're working inside a Daft clone, the skills are already available. No setup needed - Claude Code picks them up from `.claude/skills/`.\n\n### Install with `npx skills add`\n\nTo use these Daft skills outside the repo (across many agents including Claude Code, Cursor, Codex, Gemini CLI, Copilot, and more):\n\n```bash\n# Install all skills\nnpx skills add Eventual-Inc/Daft\n\n# Install a specific skill\nnpx skills add Eventual-Inc/Daft --skill daft-udf-tuning\n```\n\nFollow the prompts to choose skills, agents, scope (project or global), and whether to symlink or copy.\n\n### Claude Code plugin\n\nThese Daft skills are also available as a Claude Code plugin.\n\n```bash\nclaude plugin marketplace add Eventual-Inc/Daft\nclaude plugin install daft@Eventual-Inc\n```\n","isInternal":false,"tokens":644,"sizeBytes":2538},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/Eventual-Inc/Daft/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"AGENTS.md","isInternal":false,"tokens":3,"sizeBytes":9},{"name":"SKILL.md","path":".claude/skills/daft-distributed-scaling/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Eventual-Inc/Daft/HEAD/.claude/skills/daft-distributed-scaling/SKILL.md","title":"daft-distributed-scaling","category":"anthropic-skill","format":"markdown","content":"---\nname: \"daft-distributed-scaling\"\ndescription: \"Scale Daft workflows to distributed Ray clusters. Invoke when optimizing performance or handling large data.\"\n---\n\n# Daft Distributed Scaling\n\nScale single-node workflows to distributed execution.\n\n## Core Strategies\n\n| Strategy | API | Use Case | Pros/Cons |\n|---|---|---|---|\n| **Shuffle** | `repartition(N)` | Light data (e.g. file paths), Joins | **Global balance**. High memory usage (materializes data). |\n| **Streaming** | `into_batches(N)` | Heavy data (images, tensors) | **Low memory** (streaming). High scheduling overhead if batches too small. |\n\n## Quick Recipes\n\n### 1. Light Data: Repartitioning\nBest for distributing file paths before heavy reads.\n\n```python\n# Create enough partitions to saturate workers\ndf = daft.read_parquet(\"s3://metadata\").repartition(100)\ndf = df.with_column(\"data\", read_heavy_data(df[\"path\"]))\n```\n\n### 2. Heavy Data: Streaming Batches\nBest for processing large partitions without OOM.\n\n```python\n# Stream 1GB partition in 64-row chunks to control memory\ndf = df.read_parquet(\"heavy_data\").into_batches(64)\ndf = df.with_column(\"embed\", model.predict(df[\"img\"]))\n```\n\n## Advanced Tuning\n\nTarget: Keep all actors busy without OOM or scheduling bottlenecks.\n\n### Formula 1: Repartitioning (Light Data / Paths)\nCalculate the **Max Partition Count** to ensure each task has enough data to feed local actors.\n\n1.  **Min Rows Per Partition** = `Batch Size * (Total Concurrency / Nodes)`\n2.  **Max Partitions** = `Total Rows / Min Rows Per Partition`\n\n**Example**:\n- 1M rows, 4 nodes, 16 total concurrency, Batch Size 64.\n- **Min Rows**: `64 * (16/4) = 256`.\n- **Max Partitions**: `1,000,000 / 256 ≈ 3906`.\n- *Recommendation*: Use ~1000 partitions to run multiple batches per task.\n\n```python\ndf = df.repartition(1000) # Balanced fan-out\n```\n\n### Formula 2: Streaming (Heavy Data / Images)\nAvoid creating tiny partitions. Use `into_batches` to stream data within larger partitions.\n\n**Strategy**: Keep partitions large (e.g. 1GB+), use `into_batches(Batch Size)` to control memory.\n\n```python\n# Stream batches to control memory usage per actor\ndf = df.into_batches(64).with_column(\"preds\", model(max_concurrency=16).predict(df[\"img\"]))\n```\n","frontmatter":{"name":"daft-distributed-scaling","description":"Scale Daft workflows to distributed Ray clusters. Invoke when optimizing performance or handling large data."},"isInternal":false,"tokens":576,"sizeBytes":2227},{"name":"SKILL.md","path":".claude/skills/daft-docs-navigation/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Eventual-Inc/Daft/HEAD/.claude/skills/daft-docs-navigation/SKILL.md","title":"daft-docs-navigation","category":"anthropic-skill","format":"markdown","content":"---\nname: \"daft-docs-navigation\"\ndescription: \"Navigate Daft documentation. Invoke when user asks general questions about APIs, concepts, or examples, or wants to search the docs.\"\n---\n\n# Daft Docs Navigation\n\nNavigate Daft's documentation for APIs, concepts, and examples.\n\n## Documentation Structure\n\n1.  **Live Site**: [`https://docs.daft.ai`](https://docs.daft.ai) (Primary source, searchable).\n2.  **Source Repo**: `docs/` directory in the repository.\n    -   If `docs/` is missing or empty, clone the repo: `git clone https://github.com/Eventual-Inc/Daft.git`\n\n## Key Locations in `docs/`\n\n-   **API Reference**: `api/` (e.g., `api/io.md` for reading/writing).\n-   **Optimization**: `optimization/` (e.g., `optimization/partitioning.md`).\n-   **Distributed**: `distributed/` (e.g., `distributed/ray.md`).\n-   **UDFs**: `custom-code/` (e.g., `func.md`, `cls.md`).\n-   **Connectors**: `connectors/` (e.g., S3, Lance).\n-   **Table of Contents**: `docs/SUMMARY.md`.\n\n## Usage\n\n**Search for API Usage:**\n```bash\ngrep_search(pattern='from_glob', path='docs/')\n```\n\n**Browse Topics:**\n```bash\nread(file_path='docs/SUMMARY.md')\n```\n","frontmatter":{"name":"daft-docs-navigation","description":"Navigate Daft documentation. Invoke when user asks general questions about APIs, concepts, or examples, or wants to search the docs."},"isInternal":false,"tokens":322,"sizeBytes":1130},{"name":"SKILL.md","path":".claude/skills/daft-udf-tuning/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Eventual-Inc/Daft/HEAD/.claude/skills/daft-udf-tuning/SKILL.md","title":"daft-udf-tuning","category":"anthropic-skill","format":"markdown","content":"---\nname: \"daft-udf-tuning\"\ndescription: \"Optimize Daft UDF performance. Invoke when user needs GPU inference, encounters slow UDFs, or asks about async/batch processing.\"\n---\n\n# Daft UDF Tuning\n\nOptimize User-Defined Functions for performance.\n\n## UDF Types\n\n| Type | Decorator | Use Case |\n|---|---|---|\n| **Stateless** | `@daft.func` | Simple transforms. Use `async` for I/O-bound tasks. |\n| **Stateful** | `@daft.cls` | Expensive init (e.g., loading models). Supports `gpus=N`. |\n| **Batch** | `@daft.func.batch` | Vectorized CPU/GPU ops (NumPy/PyTorch). Faster. |\n\n## Quick Recipes\n\n### 1. Async I/O (Web APIs)\n\n```python\n@daft.func\nasync def fetch(url: str):\n    async with aiohttp.ClientSession() as s:\n        return await s.get(url).text()\n```\n\n### 2. GPU Batch Inference (PyTorch/Models)\n\n```python\n@daft.cls(gpus=1)\nclass Classifier:\n    def __init__(self):\n        self.model = load_model().cuda() # Run once per worker\n\n    @daft.method.batch(batch_size=32)\n    def predict(self, images):\n        return self.model(images.to_pylist())\n\n# Run with concurrency\ndf.with_column(\"preds\", Classifier(max_concurrency=4).predict(df[\"img\"]))\n```\n\n## Tuning Keys\n\n-   **`max_concurrency`**: Total parallel UDF instances.\n-   **`gpus=N`**: GPU request per instance.\n-   **`batch_size`**: Rows per call. Too small = overhead; too big = OOM.\n-   **`into_batches(N)`**: Pre-slice partitions if memory is tight.\n","frontmatter":{"name":"daft-udf-tuning","description":"Optimize Daft UDF performance. Invoke when user needs GPU inference, encounters slow UDFs, or asks about async/batch processing."},"isInternal":false,"tokens":402,"sizeBytes":1410},{"name":"authoring.md","path":"docs/extensions/authoring.md","rawUrl":"https://raw.githubusercontent.com/Eventual-Inc/Daft/HEAD/docs/extensions/authoring.md","title":"Extensions Skill","category":"anthropic-skill","format":"markdown","content":"# Authoring Guide\n\n!!! warning \"Experimental\"\n\n    Native extensions are experimental and may change in future releases.\n\n> Please see the [prompt](#prompt) if you want help generating an extension.\n> Looking for the broader extension model? See the [Extensions overview](overview.md).\n> Looking for extensions you can install today? See [Community Extensions](community.md).\n\nThis document is a guide for authoring Daft native extensions in Rust.\nDaft supports native Rust extensions by leveraging a stable C ABI based on the\n[Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html).\nExtensions are **not coupled** to any particular Arrow library version. The ABI boundary uses\nplain C structs (`ArrowSchema`, `ArrowArray`) so your extension can use any arrow-rs version\n(or even a different Arrow implementation entirely). Today we support authoring native\nscalar functions (UDFs) and aggregate functions (UDAFs).\n\n## Example\n\nThis example shows the end-result of our 'hello' extension with a native 'greet' scalar function.\n\n```python\nimport daft\n\n# Step 1. Import your extension module\nimport hello\n\n# Step 2. Load the extension into the current daft session\ndaft.load_extension(hello)\n\n# Step 3. Use in your dataframe!\ndf = daft.from_pydict({\"name\": [\"John\", \"Paul\"]})\ndf = df.select(hello.greet(df[\"name\"]))\ndf.show()\n\n\"\"\"\n╭──────────────╮\n│ greet        │\n│ ---          │\n│ String       │\n╞══════════════╡\n│ Hello, John! │\n├╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤\n│ Hello, Paul! │\n╰──────────────╯\n\"\"\"\n```\n\n## Tutorial\n\n### 1. Setup\n\nThis section walks through project setup from scratch; You will need a Rust toolchain and Python 3.10+.\n\n!!! note \"Note\"\n\n    We are actively working on a cloneable template.\n\n\n```bash\n# Install Rust (if not already installed)\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n\n# Install uv (if not already installed)\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n```\n\nNow you can create the project scaffolding.\n\n```bash\n# Create an empty directory\nmkdir hello && cd hello\n\n# Setup rust project\ncargo init --lib\n\n# Setup python project (for function signatures)\nuv init\n```\n\nThe crate must compile as a `cdylib` so it can be loaded at runtime via `dlopen`.\n\n```bash\ncat Cargo.toml\n```\n\n```toml\n[workspace]\n\n[package]\nname = \"hello\"\nedition = \"2024\"\nversion = \"0.1.0\"\n\n[lib]\nname = \"hello\"\ncrate-type = [\"cdylib\"]\n\n[dependencies]\ndaft-ext = { version = <version>, features = [\"arrow-58\"] }\narrow = { version = \"58\", features = [\"ffi\"] }\n```\n\n!!! tip \"Arrow version freedom\"\n\n    The `daft-ext` ABI uses C Data Interface types — your extension is **not** pinned to\n    Daft's arrow-rs version. Enable a feature flag on `daft-ext` matching your arrow-rs\n    version (`arrow-56`, `arrow-57`, or `arrow-58`) to get safe `.into()` conversions\n    between arrow-rs FFI types and the ABI types. For unsupported versions, use the\n    `from_owned`/`into_owned`/`from_raw`/`as_raw` escape hatches on `ArrowArray`\n    and `ArrowSchema`.\n\nThen update the pyproject to use `setuptools-rust` as the build system.\n\n```bash\ncat pyproject.toml\n```\n\n```toml\n[build-system]\nrequires = [\"setuptools\", \"setuptools-rust\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[project]\nname = \"hello\"\nversion = \"0.1.0\"\nrequires-python = \">=3.10\"\ndependencies = [\"daft\"]\n\n[project.optional-dependencies]\ntest = [\"pytest\"]\n```\n\n\nYou will need to create a `setup.py` file with the following contents.\n\n!!! note \"Note\"\n\n    The `RustExtension` entry tells `setuptools-rust` to compile the cdylib and place it inside the Python package directory.\n    We use the `Binding::NoBinding` because Daft extensions export a raw C symbol, not PyO3 bindings. The target name `hello.libhello`\n    places the compiled `.so` inside the `hello/` package directory so the Session's `load_extension` can find it.\n\n```bash\ncat setup.py\n```\n\n```python\nfrom setuptools import find_packages, setup\nfrom setuptools_rust import Binding, RustExtension\n\nsetup(\n    packages=find_packages(),\n    rust_extensions=[\n        RustExtension(\n            \"hello.libhello\",   # <python_package>.<lib_name>\n            path=\"Cargo.toml\",\n            binding=Binding.NoBinding,\n            strip=True,\n        )\n    ],\n)\n```\n\n### 2. Hello, World!\n\nAn extension has two parts: a **module** (the entry point) and one or more **functions** (scalar or aggregate).\n\n```bash\ncat src/lib.rs\n```\n\n```rust\nuse std::{ffi::CStr, sync::Arc};\n\nuse arrow::{\n    array::{Array, builder::StringBuilder, cast::AsArray},\n    datatypes::{DataType, Field},\n};\nuse daft_ext::prelude::*;\n\n// ── Module ──────────────────────────────────────────────────────────\n\n// #[daft_extension] generates the `daft_module_magic` C symbol that Daft's runtime looks for\n// when loading the shared library. It converts HelloExtension → hello_extension for the module name.\n#[daft_extension]\nstruct HelloExtension;\n\nimpl DaftExtension for HelloExtension {\n    /// This is the extension install hook for defining functions in the session.\n    /// Called once when the extension is loaded into a session. Register each function here.\n    fn install(session: &mut dyn DaftSession) {\n        session.define_function(Arc::new(Greet));\n    }\n}\n\n// ── Function ────────────────────────────────────────────────────────\n\n/// The function type which is registered as an Arc<dyn DaftScalarFunction>.\nstruct Greet;\n\nimpl DaftScalarFunction for Greet {\n    /// Function name used to look it up from Python.\n    /// Must be a `&CStr` (use the `c\"...\"` literal).\n    fn name(&self) -> &CStr {\n        c\"greet\"\n    }\n\n    /// Type checking.\n    /// Receives input fields as C Data Interface `ArrowSchema` types.\n    /// Use `.as_raw()` / `.into()` to convert between arrow-rs and ABI types.\n    fn return_field(&self, args: &[ArrowSchema]) -> DaftResult<ArrowSchema> {\n        if args.len() != 1 {\n            return Err(DaftError::TypeError(format!(\n                \"greet: expected 1 argument, got {}\",\n                args.len()\n            )));\n        }\n        let field = Field::try_from(&args[0])?;\n        let dt = field.data_type();\n        if *dt != DataType::Utf8 && *dt != DataType::LargeUtf8 {\n            return Err(DaftError::TypeError(format!(\n                \"greet: expected string argument, got {:?}\",\n                dt\n            )));\n        }\n        Ok(ArrowSchema::try_from(&Field::new(\n            \"greet\",\n            DataType::Utf8,\n            true,\n        ))?)\n    }\n\n    /// Evaluation. Receives columns as C Data Interface `ArrowData` types.\n    /// Use `.into()` to convert to/from arrow-rs FFI types.\n    /// All data flows through Arrow arrays — no per-row Python overhead.\n    fn call(&self, args: Vec<ArrowData>) -> DaftResult<ArrowData> {\n        let data = args.into_iter().next().ok_or_else(|| DaftError::TypeError(\"greet: expected 1 argument, got 0\".into()))?;\n        let ffi_array: arrow::ffi::FFI_ArrowArray = data.array.into();\n        let ffi_schema: arrow::ffi::FFI_ArrowSchema = data.schema.into();\n        let arrow_data = unsafe { arrow::ffi::from_ffi(ffi_array, &ffi_schema) }?;\n        let input = arrow::array::make_array(arrow_data);\n        let names = input.as_string::<i64>();\n        let mut builder = StringBuilder::with_capacity(names.len(), names.len() * 16);\n        for i in 0..names.len() {\n            if names.is_null(i) {\n                builder.append_null();\n            } else {\n                builder.append_value(format!(\"Hello, {}!\", names.value(i)));\n            }\n        }\n        let output = builder.finish();\n        let (out_arr, out_sch) = arrow::ffi::to_ffi(&output.to_data())?;\n        Ok(ArrowData {\n            array: out_arr.into(),\n            schema: out_sch.into(),\n        })\n    }\n}\n```\n\n!!! tip \"ABI pattern\"\n\n    The `DaftScalarFunction` trait uses C Data Interface types (`ArrowSchema`, `ArrowData`)\n    at the ABI boundary. Enable a `daft-ext` feature flag (`arrow-56`, `arrow-57`, or\n    `arrow-58`) matching your arrow-rs version to get `.into()` conversions. Use `.as_raw()`\n    for zero-copy borrows. This decoupling means your extension is not tied to Daft's\n    arrow-rs version.\n\n!!! tip \"String types\"\n\n    Daft uses `LargeUtf8` (i64 offsets) for strings internally. When downcasting string arrays,\n    use `as_string::<i64>()` — using `i32` will panic at runtime. Similarly, when checking types\n    in `return_field`, accept `DataType::LargeUtf8`.\n\n!!! tip \"Naming\"\n\n    Function names are global within a session. Use a prefix (e.g., `myext_greet`) to avoid\n    collisions when your extension defines many functions or might be loaded alongside others.\n\n!!! tip \"Multiple functions\"\n\n    Register as many functions as you need in `install()` — each is an independent struct\n    implementing `DaftScalarFunction`.\n\n!!! tip \"Errors\"\n\n    Return `Err(DaftError::TypeError(...))` for schema violations in `return_field`,\n    and `Err(DaftError::RuntimeError(...))` for execution failures in `call`.\n\n### Aggregate Functions\n\nExtensions can also register **aggregate functions** (UDAFs) alongside scalar functions.\nAn aggregate function follows a three-stage pipeline:\n\n1. **Aggregation** (`aggregate`) — process input arrays into partial state\n2. **Combination** (`combine`) — merge partial states\n3. **Finalization** (`finalize`) — produce the final output from merged state\n\nState is exchanged as `Vec<ArrowData>` — each element is one state field. The FFI layer\npacks these into a Struct array transparently.\n\n```rust\nuse std::ffi::CStr;\nuse daft_ext::prelude::*;\n\nstruct MySum;\n\nimpl DaftAggregateFunction for MySum {\n    fn name(&self) -> &CStr {\n        c\"my_sum\"\n    }\n\n    /// Declare the output type given the input field schemas.\n    fn return_field(&self, _args: &[ArrowSchema]) -> DaftResult<ArrowSchema> {\n        Ok(ArrowSchema::try_from(&Field::new(\"my_sum\", DataType::Int64, true))?)\n    }\n\n    /// Declare intermediate state fields.\n    /// Each state field becomes a child of the Struct exchanged at the FFI boundary.\n    fn state_fields(&self, _args: &[ArrowSchema]) -> DaftResult<Vec<ArrowSchema>> {\n        Ok(vec![\n            ArrowSchema::try_from(&Field::new(\"sum\", DataType::Int64, true))?,\n            ArrowSchema::try_from(&Field::new(\"count\", DataType::Int64, false))?,\n        ])\n    }\n\n    /// Process input arrays into partial state.\n    /// Returns one ArrowData per state field (single-row each).\n    fn aggregate(&self, inputs: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>> {\n        // inputs[0] is the column to sum\n        // compute sum and count, return as two single-element arrays\n        todo!()\n    }\n\n    /// Merge multiple partial states into one.\n    /// states[i] is the i-th state field with multiple rows (one per partial).\n    fn combine(&self, states: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>> {\n        // states[0] = sums array, states[1] = counts array\n        // sum them up, return single-row arrays\n        todo!()\n    }\n\n    /// Produce the final scalar result from merged state.\n    fn finalize(&self, states: Vec<ArrowData>) -> DaftResult<ArrowData> {\n        // states[0] = sum (single value), states[1] = count (single value)\n        // return the final sum\n        todo!()\n    }\n}\n```\n\nRegister aggregate functions in `install()` using `define_aggregate_function`:\n\n```rust\nimpl DaftExtension for MyExtension {\n    fn install(session: &mut dyn DaftSession) {\n        session.define_function(Arc::new(MyScalarFn));\n        session.define_aggregate_function(Arc::new(MySum));\n    }\n}\n```\n\nNow we define the python symbols for use in the Expression DSL; we link to rust via `daft.get_function` for scalars and `daft.get_aggregate_function` for aggregates.\n\n!!! note \"Note\"\n\n    Python sources let you write python function signatures and any argument pre-processing before linking\n    to the function symbol. The `daft.get_function` method is calling `get_function` on the active session\n    to resolve the given name and arguments to some defined function in the session. Notice how we don't\n    technically need python to resolve functions in SQL (hence no PyO3) but these python functions give us\n    nice pythonic functions with autocomplete and doc comments when using the Expression DSL.\n\n```bash\ncat hello/__init__.py\n```\n\n```python\nfrom __future__ import annotations\nfrom typing import TYPE_CHECKING\n\nimport daft\n\nif TYPE_CHECKING:\n    from daft.expressions import Expression\n\ndef greet(name: Expression) -> Expression:\n    \"\"\"Greet someone by name.\"\"\"\n    return daft.get_function(\"greet\", name)\n\ndef string_count(name: Expression) -> Expression:\n    \"\"\"Count non-null strings.\"\"\"\n    return daft.get_aggregate_function(\"string_count\", name)\n```\n\n`daft.get_function` looks up a scalar function registered with the current session by the name returned from `DaftScalarFunction::name()`.\n`daft.get_aggregate_function` does the same for aggregate functions registered via `DaftAggregateFunction::name()`.\n\nAdd an empty `hello/py.typed` marker if you want type-checker support.\n\n### 3. Build, Install, Test\n\n```bash\n# Compile the Rust cdylib and install the package in editable mode\nuv pip install -e .\n```\n\nHere are some sanity check tests. Notice how we use a scoped session rather than\nthe global active session. How you choose to load extensions is up to you, and this\ntutorial has covered both.\n\n!!! tip \"Session isolation\"\n\n    Extensions are loaded once into the process and the session serves as a scoping mechanism\n    for name resolution; calling `load_extension` multiple times will only `dlopen` once for\n    this process. Functions are only available in sessions where the extension is loaded.\n    Use the `with sess:` context manager to scope queries to a specific session.\n\n```bash\ncat tests/test_hello.py\n```\n\n```python\nimport daft\nimport hello\nfrom daft import col\nfrom daft.session import Session\nfrom hello import greet\n\ndef test_greet():\n    sess = Session()\n    sess.load_extension(hello)\n\n    df = daft.from_pydict({\"name\": [\"John\", \"Paul\"]})\n\n    with sess:\n        result = df.select(greet(col(\"name\"))).collect().to_pydict()\n\n    values = result[\"greet\"]\n    assert values[0] == \"Hello, John!\"\n    assert values[1] == \"Hello, Paul!\"\n\ndef test_greet_null():\n    sess = Session()\n    sess.load_extension(hello)\n\n    df = daft.from_pydict({\"name\": [\"George\", \"Ringo\", None]})\n\n    with sess:\n        result = df.select(greet(col(\"name\"))).collect().to_pydict()\n\n    values = result[\"greet\"]\n    assert values[0] == \"Hello, George!\"\n    assert values[1] == \"Hello, Ringo!\"\n    assert values[2] is None\n```\n\nNow run the tests!\n\n```bash\npytest -v tests/\n```\n\n## Prompt\n\nYou can paste this whole document and prompt into Claude Code to scaffold a Daft extension for you.\n\n````markdown\nCreate a Daft native extension called `<extension_name>` with the following functions:\n\n<describe each function: name, scalar or aggregate, arguments with types, return type, and behavior>\n\nFollow the Daft extension authoring guide at docs/extensions/authoring.md. Here is a summary of the key conventions:\n\n## Project structure\n\n```\n<extension_name>/\n  Cargo.toml           # [lib] crate-type = [\"cdylib\"], depends on daft-ext (with arrow feature), arrow\n  pyproject.toml       # build-system: setuptools + setuptools-rust\n  setup.py             # RustExtension(\"<pkg>.lib<pkg>\", binding=NoBinding, strip=True)\n  <extension_name>/\n    __init__.py        # Python wrappers using daft.get_function(\"name\", *args)\n    py.typed           # empty PEP 561 marker\n  src/\n    lib.rs             # #[daft_extension] struct + DaftScalarFunction / DaftAggregateFunction impls\n  tests/\n    test_<name>.py     # pytest tests using Session fixture\n  .gitignore           # /target, *.so, *.dylib, *.dll, *.egg-info, __pycache__, dist/\n```\n\n## Rust conventions\n\n- Use `daft_ext::prelude::*` for all imports (provides `ArrowSchema`, `ArrowData`, errors, traits).\n- Add `daft-ext` with a feature flag matching your arrow version (`arrow-56`, `arrow-57`, or `arrow-58`) for `.into()` conversions.\n- Import `arrow::array::Array` for `len()`/`is_null()` and `arrow::array::cast::AsArray` for downcasting.\n- Daft uses `LargeUtf8` (i64 offsets) for strings — downcast with `as_string::<i64>()`, never `i32`.\n- Apply `#[daft_extension]` to a struct implementing `DaftExtension`.\n- Register scalar functions via `session.define_function(Arc::new(MyFn))`.\n- Register aggregate functions via `session.define_aggregate_function(Arc::new(MyAgg))`.\n- Each scalar function is a struct implementing `DaftScalarFunction` with:\n  - `name(&self) -> &CStr` — use `c\"<extension_name>_<fn_name>\"` prefix to avoid collisions.\n  - `return_field(&self, args: &[ArrowSchema]) -> DaftResult<ArrowSchema>` — use `.as_raw()` to\n    borrow as arrow-rs `FFI_ArrowSchema` for type checking, then `.into()` to return output.\n  - `call(&self, args: &[ArrowData]) -> DaftResult<ArrowData>` — use `ArrowData::take_arg` then\n    `.into()` to convert to arrow-rs FFI types, compute, then `.into()` to return the result.\n- Each aggregate function is a struct implementing `DaftAggregateFunction` with:\n  - `name`, `return_field` — same as scalar functions.\n  - `state_fields(&self, args: &[ArrowSchema]) -> DaftResult<Vec<ArrowSchema>>` — intermediate state schema.\n  - `aggregate(&self, inputs: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>>` — partial aggregation.\n  - `combine(&self, states: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>>` — merge partial states.\n  - `finalize(&self, states: Vec<ArrowData>) -> DaftResult<ArrowData>` — produce final result.\n\n## Python conventions\n\n- Each scalar function wrapper calls `daft.get_function(\"<extension_name>_<fn_name>\", *args)`.\n- Each aggregate function wrapper calls `daft.get_aggregate_function(\"<extension_name>_<fn_name>\", *args)`.\n- Use `TYPE_CHECKING` guard for `Expression` import.\n- Add type hints and a docstring to each wrapper.\n````\n","isInternal":false,"tokens":4488,"sizeBytes":18270},{"name":"community.md","path":"docs/extensions/community.md","rawUrl":"https://raw.githubusercontent.com/Eventual-Inc/Daft/HEAD/docs/extensions/community.md","title":"Extensions Skill","category":"anthropic-skill","format":"markdown","content":"# Community Extensions\n\n!!! tip \"Want to build your own extension?\"\n\n    Start with the [Extensions overview](overview.md), the [Python UDF docs](../custom-code/func.md),\n    or the [Native Extension Authoring Guide](authoring.md).\n\nCommunity extensions are reusable packages that add functionality to Daft\noutside of the core repository. They may be pure-Python UDF-based extensions,\nnative ABI extensions, custom DataSource/DataSink connectors, or higher-level\ndomain libraries that combine Daft expressions, UDFs, file types, model calls,\nand distributed execution patterns.\n\nProjects listed here are maintained independently of Daft's release cadence.\nEach project installs and versions independently of Daft.\n\n| Name | Kind | Repository | Description |\n|---|---|---|---|\n| [daft-h3](#daft-h3) | Native ABI | [gweaverbiodev/daft-h3](https://github.com/gweaverbiodev/daft-h3) | Native [H3](https://h3geo.org/) geospatial indexing functions. |\n| [daft-lance](#daft-lance) | Python UDF-based | [daft-engine/daft-lance](https://github.com/daft-engine/daft-lance) | Lance-specific distributed operations for compaction, scalar indexing, column merging, and REST catalog operations. |\n| [daft-html](#daft-html) | Native ABI | [daft-engine/daft-html](https://github.com/daft-engine/daft-html) | Native HTML document processing functions exposed as Daft expressions. |\n| [daft-geo](#daft-geo) | Native ABI / Datatypes | [daft-engine/daft-geo](https://github.com/daft-engine/daft-geo) | Geospatial prototype showing native functions and extension-backed datatypes. |\n| [daft-qdrant](#daft-qdrant) | Python UDF-based | [qdrant-labs/daft-qdrant](https://github.com/qdrant-labs/daft-qdrant) | Write vector embeddings and their payloads into [Qdrant](https://qdrant.tech/) collections. |\n| [daft-doris](#daft-doris) | Custom DataSource/DataSink | [jiangxt2/daft-doris](https://github.com/jiangxt2/daft-doris) | Apache Doris reads and HTTP Stream Load writes for Daft. |\n\nTo propose a new extension for this list, open a PR against this page.\n\n## daft-h3\n\n`daft-h3` adds H3 geospatial indexing functions such as latitude/longitude to\ncell conversion, cell traversal, grid distance, parent resolution, and string\nconversion.\n\n```bash\npip install daft-h3\n```\n\n```python\nimport daft\nimport daft_h3\nfrom daft import col\n\ndaft.load_extension(daft_h3)\n\ndf = daft.from_pydict({\"lat\": [37.7749], \"lng\": [-122.4194]})\ndf = df.select(\n    daft_h3.h3_latlng_to_cell(col(\"lat\"), col(\"lng\"), 7).alias(\"cell\"),\n)\ndf = df.select(daft_h3.h3_cell_to_str(col(\"cell\")).alias(\"hex\"))\ndf.show()\n```\n\nSee the [daft-h3 README](https://github.com/gweaverbiodev/daft-h3#readme) for\nthe full list of functions and behavior details.\n\n## daft-lance\n\n`daft-lance` extends Daft with Lance-specific distributed maintenance and data\nmanagement operations. It is a Python UDF-based extension: internally, it uses\nDaft's Python UDF and class-UDF APIs to distribute Lance tasks across Daft\nqueries, while users interact with simple Python functions.\n\n```bash\npip install daft-lance\n```\n\n```python\nfrom daft_lance import compact_files, create_scalar_index\n\ncompact_files(\"s3://bucket/my_dataset\")\n\ncreate_scalar_index(\n    \"s3://bucket/my_dataset\",\n    column=\"name\",\n    index_type=\"INVERTED\",\n)\n```\n\nSee the [daft-lance README](https://github.com/daft-engine/daft-lance#readme)\nfor additional operations such as column merging and REST catalog writes.\n\n## daft-html\n\n`daft-html` adds native HTML processing functions such as `html_to_text`,\n`html_extract_links`, `html_extract_tables`, and CSS-selector extraction.\n\n```bash\npip install daft-html\n```\n\n```python\nimport daft\nimport daft_html\nfrom daft import col\nfrom daft_html import html_to_text\n\ndaft.load_extension(daft_html)\n\ndf = daft.from_pydict({\"html\": [\"<html><body><h1>Hello</h1></body></html>\"]})\ndf = df.select(html_to_text(col(\"html\")).alias(\"text\"))\ndf.show()\n```\n\nSee the [daft-html README](https://github.com/daft-engine/daft-html#readme)\nfor the full list of document and CSS-selector functions.\n\n## daft-geo\n\n`daft-geo` is a geospatial prototype showing native Daft functions and\nextension-backed datatypes. It defines `Point2D` and `Point3D` as\n`DataType.extension(...)` values and provides point construction, accessors,\nEuclidean distance, and haversine distance.\n\n```bash\npip install git+https://github.com/daft-engine/daft-geo.git\n```\n\n!!! note\n\n    `daft-geo` is not yet published to PyPI. The command above installs the\n    latest commit from the repository's default branch, so behavior may change\n    between installs.\n\n```python\nimport daft\nimport daft_geo\nfrom daft import col\n\ndaft.load_extension(daft_geo)\n\ndf = daft.from_pydict({\"x\": [0.0], \"y\": [1.0]})\ndf = df.select(daft_geo.point2d(col(\"x\"), col(\"y\")).alias(\"point\"))\ndf = df.select(daft_geo.x(col(\"point\")).alias(\"x\"))\ndf.show()\n```\n\nSee the [daft-geo README](https://github.com/daft-engine/daft-geo#readme) for\ncurrent capabilities and caveats.\n\n## daft-qdrant\n\n`daft-qdrant` adds a `write_qdrant` DataFrame method for writing vector\nembeddings and their payloads into [Qdrant](https://qdrant.tech/) collections.\n\n```bash\npip install daft-qdrant\n```\n\n```python\nimport daft\nimport daft_qdrant\n\ndf = daft.from_pydict({\n    \"id\": [1, 2, 3],\n    \"vector\": [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]],\n    \"label\": [\"cat\", \"dog\", \"bird\"],\n})\n\ndf.write_qdrant(\"my-collection\", url=\"http://localhost:6333\")\n```\n\nSee the [daft-qdrant README](https://github.com/qdrant-labs/daft-qdrant#readme) for embedding pipeline examples and additional options.\n\n## daft-doris\n\n`daft-doris` is an independently maintained Python custom connector for\nApache Doris. It supports reads from Doris physical tables and batch writes\nthrough HTTP Stream Load. The read transport must be selected explicitly;\nFlight SQL is available as an opt-in experimental transport.\n\n```bash\npip install \"daft-doris[doris]\"\n```\n\nFor Flight SQL read support, install the optional Flight extra:\n\n```bash\npip install \"daft-doris[doris-flight]\"\n```\n\nRead a Doris table:\n\n```python\nfrom daft_doris import read_doris\n\ndf = read_doris(\n    host=\"doris-fe.example\",\n    database=\"analytics\",\n    table=\"events\",\n    transport=\"mysql\",\n)\n```\n\nWrite a Daft DataFrame through Stream Load:\n\n```python\nimport daft\n\nfrom daft_doris import DorisConnection, DorisTable, SecretRef, write_doris\n\ndf = daft.from_pydict(\n    {\n        \"event_id\": [1, 2],\n        \"score\": [95.0, 88.0],\n    }\n)\nresult = write_doris(\n    df,\n    connection=DorisConnection(\n        host=\"doris-fe.example.com\",\n        username=\"daft_writer\",\n        password=SecretRef.env(\"DORIS_PASSWORD\"),\n        http_port=8030,\n        redirect_hosts=(\"doris-be.example.com\",),\n        redirect_ports=(8040,),\n        redirect_policy=\"public\",\n    ),\n    table=DorisTable(database=\"analytics\", name=\"events\"),\n    operation=\"load\",\n)\n```\n\nThe connector is currently in Alpha and is maintained independently of Daft's\nrelease cadence. The current release supports Python 3.12–3.13 and Daft\n0.7.23; its Daft dependency is bounded to `>=0.7.23,<0.7.24`, so installing it\nalongside a newer Daft release may cause the resolver to install Daft 0.7.23.\nSee the [`daft-doris` repository](https://github.com/jiangxt2/daft-doris#readme)\nfor supported scope, compatibility information, and limitations.\n","isInternal":false,"tokens":2026,"sizeBytes":7303},{"name":"overview.md","path":"docs/extensions/overview.md","rawUrl":"https://raw.githubusercontent.com/Eventual-Inc/Daft/HEAD/docs/extensions/overview.md","title":"Extensions Skill","category":"anthropic-skill","format":"markdown","content":"# Extensions\n\nDaft extensions are reusable libraries that add domain-specific functionality\non top of Daft without requiring every specialized function, datatype,\nintegration, or workflow to live directly in Daft core.\n\nThere are two broad ways to build extensions:\n\n- **Python UDF-based extensions**: Python packages built on Daft's\n  [custom-code APIs](../custom-code/func.md), including\n  [`@daft.func`](../api/udf.md), [`@daft.func.batch`](../api/udf.md),\n  [`@daft.cls`](../api/udf.md), and [`@daft.method.batch`](../api/udf.md).\n- **Native ABI extensions**: shared-library-backed packages that use Daft's\n  Arrow C Data Interface ABI for lower-level, vectorized native execution.\n\nBoth models can expose clean Python APIs that feel like ordinary Daft\nexpression functions. The implementation details can stay hidden behind\nfunctions, classes, and expressions.\n\n## Python UDF-Based Extensions\n\nThe fastest way to build a Daft extension is often pure Python. Using\n[`@daft.func`](../api/udf.md), [`@daft.func.batch`](../api/udf.md),\n[`@daft.cls`](../api/udf.md), and [`@daft.method.batch`](../api/udf.md),\ncontributors can package reusable Python logic that runs inside Daft's\ndistributed execution engine.\n\nThis is ideal for libraries that orchestrate existing Python ecosystems,\nexternal services, ML models, GPUs, data maintenance tasks, or domain-specific\nworkflows. These extensions do not require a native shared library or the\nArrow C ABI. They are normal Python packages that expose higher-level Daft\nAPIs.\n\n[`daft-lance`](https://github.com/daft-engine/daft-lance) is an example of\nthis model. It extends Daft with Lance-specific distributed operations such as\nfile compaction, scalar indexing, column merging, and REST catalog operations.\nInternally, it uses Daft's Python UDF and class-UDF APIs to distribute Lance\ntasks across a Daft query, while users interact with functions like\n`compact_files`, `create_scalar_index`, and `merge_columns_df`.\n\nThis pattern also exists inside Daft itself. The\n[`daft.functions.ai`](../api/ai.md) module exposes high-level functions like\n[`prompt`](../api/functions/prompt.md),\n[`embed_text`](../api/functions/embed_text.md),\n[`embed_image`](../api/functions/embed_image.md),\n[`classify_text`](../api/functions/classify_text.md), and\n[`classify_image`](../api/functions/classify_image.md). To users, these look\nlike normal Daft expression functions. Under the hood, they use Daft's UDF and\nclass-UDF machinery, including batching, concurrency controls, retries, and\n[GPU resource hints](../custom-code/gpu.md).\n\nThe file APIs follow a similar product pattern. [`daft.File`](../api/datatypes/file_types.md)\nvalues and helpers like [`file`](../api/functions/file.md),\n[`audio_file`](../api/functions/audio_file.md),\n[`video_file`](../api/functions/video_file.md),\n[`file_path`](../api/functions/file_path.md), and\n[`file_size`](../api/functions/file_size.md) give users expression-level\nbuilding blocks for working with files. Those file objects can then flow into\nPython UDFs, model pipelines, and domain-specific libraries without users\nneeding to think about execution details.\n\n## Native ABI Extensions\n\nFor contributors who need lower-level, vectorized performance, Daft also\nsupports native extensions through a C ABI based on the\n[Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html).\n\nNative ABI extensions can add high-performance scalar functions, aggregate\nfunctions, Python expression wrappers, and extension-backed datatypes. They\nship as pip-installable Python packages that bundle a native shared library.\nUsers import the package, load it into a Daft [`Session`](../api/sessions.md),\nand call ordinary Python wrappers in their DataFrame expressions.\n\nRust currently has the most ergonomic SDK through\n[`daft-ext`](https://github.com/Eventual-Inc/Daft/tree/main/src/daft-ext),\nwhile C++ is demonstrated through the raw ABI. Other systems languages are\npossible if they can produce a shared library, export the expected C ABI, and\nread/write Arrow C Data Interface arrays.\n\nDaft's own examples include:\n\n- [`examples/hello`](https://github.com/Eventual-Inc/Daft/tree/main/examples/hello):\n  a minimal Rust native extension that registers a `greet` scalar function.\n- [`examples/dvector`](https://github.com/Eventual-Inc/Daft/tree/main/examples/dvector):\n  a pgvector-style native extension for vector distance functions such as\n  `l2_distance`, `cosine_distance`, `inner_product`, and `jaccard_distance`.\n- [`examples/hello_cpp`](https://github.com/Eventual-Inc/Daft/tree/main/examples/hello_cpp):\n  a pure C++ native extension using Apache Arrow C++ and the raw Daft C ABI.\n\n## What Can Contributors Build?\n\nContributors can build:\n\n- Pure-Python UDF extension libraries using [`@daft.func`](../api/udf.md),\n  [`@daft.func.batch`](../api/udf.md), [`@daft.cls`](../api/udf.md), and\n  [`@daft.method.batch`](../api/udf.md)\n- Distributed task libraries like\n  [`daft-lance`](https://github.com/daft-engine/daft-lance)\n- GPU-backed model inference extensions\n- AI, file, media, and multimodal processing libraries that hide UDFs behind\n  expression APIs\n- Native scalar functions\n- Native aggregate functions / UDAFs\n- Python expression wrappers\n- Extension-backed logical datatypes such as\n  [`DataType.extension(...)`](../api/datatypes/all_datatypes.md)\n\n## Where to Start\n\n- Browse installable projects in [Community Extensions](community.md).\n- Learn how to build a native extension in the [Authoring Guide](authoring.md).\n- Browse projects using Daft in [Built on Daft](projects.md).\n- Learn more about Daft's [Python UDFs](../custom-code/func.md).\n","isInternal":false,"tokens":1327,"sizeBytes":5643},{"name":"projects.md","path":"docs/extensions/projects.md","rawUrl":"https://raw.githubusercontent.com/Eventual-Inc/Daft/HEAD/docs/extensions/projects.md","title":"Extensions Skill","category":"anthropic-skill","format":"markdown","content":"# Built on Daft\n\nThis page highlights projects that depend on or build on top of\n[Daft](https://github.com/Eventual-Inc/Daft).\n\nThese projects are different from [Community Extensions](community.md). An\nextension directly adds reusable Daft APIs, functions, datatypes, or distributed\noperators. A project built on Daft may instead use Daft as a query engine,\nDataFrame runtime, compute backend, or pipeline framework.\n\nTo propose a project for this page, open a PR with a short description and a\nlink to the relevant Daft usage.\n\n## Projects Building on Daft\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [DeltaCat](https://github.com/ray-project/deltacat) | Portable multimodal lakehouse on Ray, Arrow, and Daft. | Daft is a core dependency: `dc.read()` returns a Daft DataFrame by default, and Daft is a first-class compute backend for reading and writing lake data. |\n| [SmooSense](https://github.com/SmooSenseAI/smoosense) | Web-based tool for interactively exploring large-scale multimodal tabular data. | Accepts Daft DataFrames as input in the Jupyter widget (`Sense(daft_df)`) and converts them for display alongside Pandas DataFrames. |\n| [eai-taxonomy](https://github.com/Essential-AI/eai-taxonomy) | Annotation tools and dataset filters from the Essential-Web project. | Uses Daft in annotation pipelines to read Parquet annotation files and compute inter-annotator agreement. |\n| [GraphReduce](https://github.com/wesmadrigal/GraphReduce) | Feature engineering library for large graphs of tabular data. | Supports Daft as a compute backend via `pip install \"graphreduce[daft]\"`. |\n| [Archetype](https://github.com/VangelisTech/archetype) | DataFrame-first, append-only ECS runtime for simulations and AI agents. | Uses Daft DataFrames as the native data structure for world state and processor logic. |\n| [hypergraph](https://github.com/gilad-rubin/hypergraph) | Python workflow orchestration framework for DAG pipelines and agentic workflows. | Provides a `DaftRunner` that compiles a hypergraph DAG into a Daft query plan using Daft UDFs. |\n| [Sashimi4Talent](https://github.com/colin-ho/Sashimi4Talent) | Data pipeline and web app for discovering and ranking GitHub contributors. | Uses Daft as the core processing engine for repository search, commits, contributor analysis, and dataset merges. |\n| [daft-image-playground](https://github.com/peckjon/daft-image-playground) | Flask-based intelligent image search engine. | Uses Daft to discover images and run native image decoding, resizing, and encoding. |\n| [SMiR](https://github.com/togethercomputer/SMiR) | Synthetic data pipeline for multi-image reasoning. | Uses Daft with the Ray runner to read Parquet from S3, download images, and run GPU-accelerated embedding UDFs. |\n| [Derezz](https://github.com/rchowell/Derezz) | CLI tool for video indexing and semantic search. | Uses Daft to read and write an S3 Tables-backed video frame index and query it. |\n| [teraflopai-daft](https://github.com/teraflop-ai/teraflopai-daft) | Daft plugin library for Teraflop AI NLP services. | Exposes custom Daft expressions such as `segment_text` and `search_text` that call the Teraflop AI provider API. |\n| [daft-sql-adapter](https://github.com/RaghwendraSingh/daft-sql-adapter) | CLI adapter that runs Spark SQL queries against Databricks Unity Catalog tables through Daft. | Uses Daft as the execution engine for SQL queries and table writes. |\n| [zarr-daft-datasource](https://github.com/liaoruoxue/zarr-daft-datasource) | Prototype compute-storage-separation architecture using Zarr, LanceDB, and Daft. | Implements a custom Daft `DataSource` that reads Zarr arrays into lazy Daft DataFrames. |\n| [pdf-document-processing-daft](https://github.com/doronkatz/pdf-document-processing-daft) | Demonstration pipeline for scalable PDF document processing. | Uses Daft DataFrames to parallelize PDF parsing, text chunking, embedding generation, and Parquet writes. |\n| [daft-ai-driving](https://github.com/legendPerceptor/daft-ai-driving) | Project for processing the KITTI autonomous driving dataset. | Uses Daft DataFrames for data loading, filtering, aggregation, and Parquet export. |\n| [sigil](https://github.com/junaidrahim/sigil) | CLI tool for collecting and analyzing AI coding session data. | Uses Daft to write local Parquet session data and query it with filtering and aggregation. |\n| [embed_qwen_k8s](https://github.com/jaychia/embed_qwen_k8s) | Kubernetes pipeline for Common Crawl text processing and embeddings. | Uses Daft to read WARC/Parquet data, apply UDFs for chunking and embedding, and write results to S3. |\n\n## Usage Patterns and Examples\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [daft-examples](https://github.com/Eventual-Inc/daft-examples) | Official Daft examples repository. | Shows idiomatic Daft usage across `daft.read_*`, UDFs, AI functions, multimodal data, SQL analytics, vector search, and integrations. |\n| [workflow-eventual-inc-daft-distributed-udf-processing](https://github.com/leeroopedia/workflow-eventual-inc-daft-distributed-udf-processing) | Workflow demonstrating custom Python UDFs for ML inference and GPU transforms. | Uses `@daft.func` and `@daft.cls` with lazy and distributed execution. |\n| [workflow-eventual-inc-daft-sql-query-analytics](https://github.com/leeroopedia/workflow-eventual-inc-daft-sql-query-analytics) | Workflow for SQL analytics with Daft sessions and catalogs. | Creates sessions, attaches catalogs, registers tables, executes SQL, and exports results. |\n| [workflow-eventual-inc-daft-data-lakehouse-etl](https://github.com/leeroopedia/workflow-eventual-inc-daft-data-lakehouse-etl) | ETL pipeline for Iceberg, Delta Lake, and Hudi lakehouse formats. | Uses Daft lakehouse readers, DataFrame transformations, and write APIs. |\n| [workflow-eventual-inc-daft-multimodal-ai-batch-inference](https://github.com/leeroopedia/workflow-eventual-inc-daft-multimodal-ai-batch-inference) | Workflow for AI inference over multimodal datasets. | Applies Daft AI functions and embedding workflows to Hugging Face, Parquet, and CSV datasets. |\n\n## Daft as a Chosen Backend\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [cuallee](https://github.com/canimus/cuallee) | DataFrame-agnostic data quality check library. | Implements a Daft validation backend and test suite for quality checks against Daft DataFrames. |\n| [Atlan application-sdk](https://github.com/atlanhq/application-sdk) | SDK for building data catalog integration applications. | Uses Daft to lazily read and filter JSON table metadata files during incremental sync. |\n| [narwhals-daft](https://github.com/narwhals-dev/narwhals-daft) | Narwhals plugin for Daft. | Wraps Daft DataFrames so the Narwhals expression API can execute lazily on Daft. |\n| [jett](https://github.com/ddeutils/jett) | ETL template framework supporting multiple DataFrame engines. | Lists Daft as a supported engine backend for YAML-defined ETL pipelines. |\n\n## Ecosystem and Content\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [AWS MCP servers](https://github.com/awslabs/mcp) | Collection of open-source MCP servers for AWS services. | The S3 Tables MCP server uses Daft as the SQL query engine for read-only queries against AWS S3 Tables. |\n| [Fabric_Notebooks_Demo](https://github.com/djouallah/Fabric_Notebooks_Demo) | Microsoft Fabric notebook demos. | Includes ETL notebooks using Daft to load and transform CSVs from Azure storage and write Delta Lake tables. |\n| [lakevision](https://github.com/lakevision-project/lakevision) | Web tool for exploring Apache Iceberg lakehouses. | Uses Daft to query and display sample data from Iceberg tables. |\n| [general-demos](https://github.com/jaehyeon-kim/general-demos) | Data engineering demo projects. | Includes a `daft-quickstart` folder demonstrating Daft with Apache Iceberg. |\n| [DaftHudi](https://github.com/dipankarmazumdar/DaftHudi) | Demo dashboard application using Apache Hudi, Daft, and Streamlit. | Uses Daft to read Apache Hudi tables and power analytical queries. |\n| [Engineering Lakehouses with Open Table Formats](https://github.com/PacktPublishing/Engineering-Lakehouses-with-Open-Table-Formats) | Code samples for a lakehouse engineering book. | Includes a notebook demonstrating reading and writing Apache Iceberg tables with Daft. |\n| [databricks-demos](https://github.com/david-hurley/databricks-demos) | Databricks demo notebooks. | Includes a notebook using Daft with Unity Catalog to write data to Delta tables. |\n| [knee-deep-in-the-lake](https://github.com/Cumulocity-IoT/knee-deep-in-the-lake) | Hands-on training repository for IoT data lake technologies. | Uses Daft in notebooks alongside PyArrow and Pandas for querying Parquet files with SQL. |\n","isInternal":false,"tokens":2190,"sizeBytes":8754},{"name":"marketplace.json","path":".claude-plugin/marketplace.json","rawUrl":"https://raw.githubusercontent.com/Eventual-Inc/Daft/HEAD/.claude-plugin/marketplace.json","title":"Eventual-Inc","category":"marketplace","format":"json","content":"{\n  \"name\": \"Eventual-Inc\",\n  \"owner\": {\n    \"name\": \"Eventual, Inc.\",\n    \"email\": \"yksu@eventualcomputing.com\"\n  },\n  \"plugins\": [\n    {\n      \"name\": \"daft\",\n      \"source\": \"./\",\n      \"description\": \"Skills for working with Daft: UDF tuning, distributed scaling, and docs navigation\",\n      \"version\": \"0.1.0\"\n    }\n  ]\n}\n","frontmatter":{"name":"Eventual-Inc","owner":{"name":"Eventual, Inc.","email":"yksu@eventualcomputing.com"},"plugins":[{"name":"daft","source":"./","description":"Skills for working with Daft: UDF tuning, distributed scaling, and docs navigation","version":"0.1.0"}]},"isInternal":false,"tokens":103,"sizeBytes":327},{"name":"plugin.json","path":".claude-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/Eventual-Inc/Daft/HEAD/.claude-plugin/plugin.json","title":"daft","category":"plugin-manifest","format":"json","content":"{\n  \"name\": \"daft\",\n  \"description\": \"Skills for working with Daft: UDF tuning, distributed scaling, and docs navigation\",\n  \"version\": \"0.1.0\",\n  \"author\": {\n    \"name\": \"Eventual, Inc.\",\n    \"email\": \"yksu@eventualcomputing.com\"\n  },\n  \"repository\": \"https://github.com/Eventual-Inc/Daft\",\n  \"license\": \"Apache-2.0\",\n  \"keywords\": [\n    \"daft\",\n    \"dataframe\",\n    \"udf\",\n    \"distributed\",\n    \"ray\"\n  ]\n}\n","frontmatter":{"name":"daft","description":"Skills for working with Daft: UDF tuning, distributed scaling, and docs navigation","version":"0.1.0","author":{"name":"Eventual, Inc.","email":"yksu@eventualcomputing.com"},"repository":"https://github.com/Eventual-Inc/Daft","license":"Apache-2.0","keywords":["daft","dataframe","udf","distributed","ray"]},"isInternal":false,"tokens":135,"sizeBytes":410},{"name":"README.md","path":"docs/plugins/README.md","rawUrl":"https://raw.githubusercontent.com/Eventual-Inc/Daft/HEAD/docs/plugins/README.md","title":"plugins Documentation","category":"plugin-manifest","format":"markdown","content":"# Daft MkDocs Plugins\n\nCustom MkDocs plugins for the Daft documentation site.\n\n## nav-hide-children\n\nA plugin that allows you to conditionally hide navigation children based on configured paths.\n\n### Configuration\n\nAdd to your `mkdocs.yml`:\n\n```yaml\nplugins:\n  - nav-hide-children:\n      hide_children:\n        - [\"Python API\", \"Functions\"]  # Hide children of Python API > Functions\n        - [\"SQL Reference\"]            # Hide children of SQL Reference\n```\n\n### Usage\n\nThe plugin automatically injects an attribute `hide_children` into [navigation objects](https://www.mkdocs.org/dev-guide/themes/#navigation-objects) based on the config paths, which can be used in Jinja2 templates to conditionally hide navigation elements.\n","isInternal":false,"tokens":158,"sizeBytes":729},{"name":"README.md","path":"tools/observability/opentelemetry/README.md","rawUrl":"https://raw.githubusercontent.com/Eventual-Inc/Daft/HEAD/tools/observability/opentelemetry/README.md","title":"opentelemetry Documentation","category":"plugin-manifest","format":"markdown","content":"# OpenTelemetry\n\nThis directory contains the code for running the OpenTelemetry collector and exporter.\n\nThe docker compose file located in this directory will start the following services:\n\n1. [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) - an agent for collecting and exporting opentelemetry data\n2. [Jaeger](https://www.jaegertracing.io/) - an open source distributed tracing system\n3. [Prometheus](https://prometheus.io/) - an open source metrics monitoring system\n\n## Running the collector\n\nTo start the services, navigate to this directory and run the following command:\n\n```bash\ndocker compose up\n```\n\n## Executing a job with opentelemetry enabled\n\nThe opentelemetry collector exposes an OTLP gRPC endpoint on port 4317.\nTo execute a job with opentelemetry enabled, set the `DAFT_DEV_OTEL_EXPORTER_OTLP_ENDPOINT` environment variable to `grpc://localhost:4317` and run the job.\n\nExample:\n```bash\nDAFT_DEV_OTEL_EXPORTER_OTLP_ENDPOINT=grpc://localhost:4317 python tools/observability/opentelemetry/example.py\n```\n\n## Viewing the metrics\n\n### Traces (Jaeger)\n\nTo view the traces on the Jaeger UI, navigate to `http://localhost:16686/` and select the `daft` service from the dropdown menu.\n\n### Metrics (Prometheus)\n\nTo view the metrics on the Prometheus UI, navigate to `http://localhost:9090/`.\n\nTo query for all metrics provided by Daft, try the following [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query:\n\n```promql\n{service_name='daft'}\n```\n","isInternal":false,"tokens":359,"sizeBytes":1500}],"systemPromptSnippet":"<yakaai_skills repo=\"Eventual-Inc/Daft\">\n<!-- File: AGENTS.md (Tokens: ~357 | Category: root-instruction) -->\n# Resources\n\n- https://docs.daft.ai for the user-facing API docs\n- CONTRIBUTING.md for detailed development process\n- https://github.com/Eventual-Inc/Daft for issues, discussions, and PRs\n\n# Dev Workflow\n\n1. [Once] Set up Python environment and install dependencies: `make .venv`\n2. [Optional] Activate .venv: `source .venv/bin/activate`. Not necessary with Makefile commands.\n3. If Rust code is modified, rebuild: `make build`\n4. Run tests. See [Testing Details](#testing-details).\n\n# Testing Details\n\n- `make test` runs tests in `tests/` directory. Uses `pytest` under the hood.\n  - Must set `DAFT_RUNNER` environment variable to `ray` or `native` to run the tests with the corresponding runner.\n    - Start with `DAFT_RUNNER=native` unless testing Ray or distributed code.\n  - `make test EXTRA_ARGS=\"...\"` passes additional arguments to `pytest`.\n    - `make test EXTRA_ARGS=\"-v tests/dataframe/test_select.py\"` runs the test in the given file.\n    - `make test EXTRA_ARGS=\"-v tests/dataframe/test_select.py::test_select_dataframe\"` runs the given test method.\n  - Default `integration`, `benchmark`, and `hypothesis` tests are disabled. Best to run on CI.\n- `make doctests` runs doctests in `daft/` directory. Tests docstrings in Daft APIs.\n\n# PR Conventions\n\n- Titles: Conventional Commits format; enforced by `.github/workflows/pr-labeller.yml`.\n- Descriptions: follow `.github/pull_request_template.md`.\n\n<!-- File: docs/skills.md (Tokens: ~644 | Category: root-instruction) -->\n# Daft Skills\n\nDaft ships with built-in skills that help AI coding agents work effectively with Daft.\n\n## Available skills\n\n### [daft-udf-tuning](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-udf-tuning/SKILL.md)\n\nOptimize User-Defined Functions for performance.\n\n**Triggers:** GPU inference setup, slow UDFs, async or batch processing questions.\n\n**What it helps with:**\n\n- Choosing between `@daft.func` (stateless), `@daft.cls` (stateful), and `@daft.func.batch` (vectorized)\n- Async I/O patterns for web APIs\n- GPU batch inference with PyTorch/model loading via `@daft.cls(gpus=N)`\n- Tuning `max_concurrency`, `batch_size`, `gpus`, and `into_batches(N)`\n\n### [daft-distributed-scaling](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-distributed-scaling/SKILL.md)\n\nScale single-node workflows to distributed execution.\n\n**Triggers:** Performance optimization, handling large datasets, distributed execution.\n\n**What it helps with:**\n\n- Choosing between shuffle (`repartition(N)`) and streaming (`into_batches(N)`) strategies\n- Calculating optimal partition counts for your cluster\n- Avoiding OOM errors with heavy data (images, tensors)\n- Advanced tuning formulas for repartitioning and streaming\n\n### [daft-docs-navigation](https://github.com/Eventual-Inc/Daft/blob/main/.claude/skills/daft-docs-navigation/SKILL.md)\n\nNavigate Daft's documentation for APIs, concepts, and examples.\n\n**Triggers:** General questions about Daft APIs, concepts, or searching the docs.\n\n**What it helps with:**\n\n- Navigating key doc locations: API reference, optimization, distributed, UDFs, connectors\n- Searching for API usage in the `docs/` directory\n- Browsing the docs structure via `docs/SUMMARY.md`\n\n## Installation\n\n### Already cloned Daft (automatic)\n\nIf you're working inside a Daft clone, the skills are already available. No setup needed - Claude Code picks them up from `.claude/skills/`.\n\n### Install with `npx skills add`\n\nTo use these Daft skills outside the repo (across many agents including Claude Code, Cursor, Codex, Gemini CLI, Copilot, and more):\n\n```bash\n# Install all skills\nnpx skills add Eventual-Inc/Daft\n\n# Install a specific skill\nnpx skills add Eventual-Inc/Daft --skill daft-udf-tuning\n```\n\nFollow the prompts to choose skills, agents, scope (project or global), and whether to symlink or copy.\n\n### Claude Code plugin\n\nThese Daft skills are also available as a Claude Code plugin.\n\n```bash\nclaude plugin marketplace add Eventual-Inc/Daft\nclaude plugin install daft@Eventual-Inc\n```\n\n<!-- File: CLAUDE.md (Tokens: ~3 | Category: claude-rule) -->\nAGENTS.md\n\n<!-- File: .claude/skills/daft-distributed-scaling/SKILL.md (Tokens: ~576 | Category: anthropic-skill) -->\n---\nname: \"daft-distributed-scaling\"\ndescription: \"Scale Daft workflows to distributed Ray clusters. Invoke when optimizing performance or handling large data.\"\n---\n\n# Daft Distributed Scaling\n\nScale single-node workflows to distributed execution.\n\n## Core Strategies\n\n| Strategy | API | Use Case | Pros/Cons |\n|---|---|---|---|\n| **Shuffle** | `repartition(N)` | Light data (e.g. file paths), Joins | **Global balance**. High memory usage (materializes data). |\n| **Streaming** | `into_batches(N)` | Heavy data (images, tensors) | **Low memory** (streaming). High scheduling overhead if batches too small. |\n\n## Quick Recipes\n\n### 1. Light Data: Repartitioning\nBest for distributing file paths before heavy reads.\n\n```python\n# Create enough partitions to saturate workers\ndf = daft.read_parquet(\"s3://metadata\").repartition(100)\ndf = df.with_column(\"data\", read_heavy_data(df[\"path\"]))\n```\n\n### 2. Heavy Data: Streaming Batches\nBest for processing large partitions without OOM.\n\n```python\n# Stream 1GB partition in 64-row chunks to control memory\ndf = df.read_parquet(\"heavy_data\").into_batches(64)\ndf = df.with_column(\"embed\", model.predict(df[\"img\"]))\n```\n\n## Advanced Tuning\n\nTarget: Keep all actors busy without OOM or scheduling bottlenecks.\n\n### Formula 1: Repartitioning (Light Data / Paths)\nCalculate the **Max Partition Count** to ensure each task has enough data to feed local actors.\n\n1.  **Min Rows Per Partition** = `Batch Size * (Total Concurrency / Nodes)`\n2.  **Max Partitions** = `Total Rows / Min Rows Per Partition`\n\n**Example**:\n- 1M rows, 4 nodes, 16 total concurrency, Batch Size 64.\n- **Min Rows**: `64 * (16/4) = 256`.\n- **Max Partitions**: `1,000,000 / 256 ≈ 3906`.\n- *Recommendation*: Use ~1000 partitions to run multiple batches per task.\n\n```python\ndf = df.repartition(1000) # Balanced fan-out\n```\n\n### Formula 2: Streaming (Heavy Data / Images)\nAvoid creating tiny partitions. Use `into_batches` to stream data within larger partitions.\n\n**Strategy**: Keep partitions large (e.g. 1GB+), use `into_batches(Batch Size)` to control memory.\n\n```python\n# Stream batches to control memory usage per actor\ndf = df.into_batches(64).with_column(\"preds\", model(max_concurrency=16).predict(df[\"img\"]))\n```\n\n<!-- File: .claude/skills/daft-docs-navigation/SKILL.md (Tokens: ~322 | Category: anthropic-skill) -->\n---\nname: \"daft-docs-navigation\"\ndescription: \"Navigate Daft documentation. Invoke when user asks general questions about APIs, concepts, or examples, or wants to search the docs.\"\n---\n\n# Daft Docs Navigation\n\nNavigate Daft's documentation for APIs, concepts, and examples.\n\n## Documentation Structure\n\n1.  **Live Site**: [`https://docs.daft.ai`](https://docs.daft.ai) (Primary source, searchable).\n2.  **Source Repo**: `docs/` directory in the repository.\n    -   If `docs/` is missing or empty, clone the repo: `git clone https://github.com/Eventual-Inc/Daft.git`\n\n## Key Locations in `docs/`\n\n-   **API Reference**: `api/` (e.g., `api/io.md` for reading/writing).\n-   **Optimization**: `optimization/` (e.g., `optimization/partitioning.md`).\n-   **Distributed**: `distributed/` (e.g., `distributed/ray.md`).\n-   **UDFs**: `custom-code/` (e.g., `func.md`, `cls.md`).\n-   **Connectors**: `connectors/` (e.g., S3, Lance).\n-   **Table of Contents**: `docs/SUMMARY.md`.\n\n## Usage\n\n**Search for API Usage:**\n```bash\ngrep_search(pattern='from_glob', path='docs/')\n```\n\n**Browse Topics:**\n```bash\nread(file_path='docs/SUMMARY.md')\n```\n\n<!-- File: .claude/skills/daft-udf-tuning/SKILL.md (Tokens: ~402 | Category: anthropic-skill) -->\n---\nname: \"daft-udf-tuning\"\ndescription: \"Optimize Daft UDF performance. Invoke when user needs GPU inference, encounters slow UDFs, or asks about async/batch processing.\"\n---\n\n# Daft UDF Tuning\n\nOptimize User-Defined Functions for performance.\n\n## UDF Types\n\n| Type | Decorator | Use Case |\n|---|---|---|\n| **Stateless** | `@daft.func` | Simple transforms. Use `async` for I/O-bound tasks. |\n| **Stateful** | `@daft.cls` | Expensive init (e.g., loading models). Supports `gpus=N`. |\n| **Batch** | `@daft.func.batch` | Vectorized CPU/GPU ops (NumPy/PyTorch). Faster. |\n\n## Quick Recipes\n\n### 1. Async I/O (Web APIs)\n\n```python\n@daft.func\nasync def fetch(url: str):\n    async with aiohttp.ClientSession() as s:\n        return await s.get(url).text()\n```\n\n### 2. GPU Batch Inference (PyTorch/Models)\n\n```python\n@daft.cls(gpus=1)\nclass Classifier:\n    def __init__(self):\n        self.model = load_model().cuda() # Run once per worker\n\n    @daft.method.batch(batch_size=32)\n    def predict(self, images):\n        return self.model(images.to_pylist())\n\n# Run with concurrency\ndf.with_column(\"preds\", Classifier(max_concurrency=4).predict(df[\"img\"]))\n```\n\n## Tuning Keys\n\n-   **`max_concurrency`**: Total parallel UDF instances.\n-   **`gpus=N`**: GPU request per instance.\n-   **`batch_size`**: Rows per call. Too small = overhead; too big = OOM.\n-   **`into_batches(N)`**: Pre-slice partitions if memory is tight.\n\n<!-- File: docs/extensions/authoring.md (Tokens: ~4488 | Category: anthropic-skill) -->\n# Authoring Guide\n\n!!! warning \"Experimental\"\n\n    Native extensions are experimental and may change in future releases.\n\n> Please see the [prompt](#prompt) if you want help generating an extension.\n> Looking for the broader extension model? See the [Extensions overview](overview.md).\n> Looking for extensions you can install today? See [Community Extensions](community.md).\n\nThis document is a guide for authoring Daft native extensions in Rust.\nDaft supports native Rust extensions by leveraging a stable C ABI based on the\n[Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html).\nExtensions are **not coupled** to any particular Arrow library version. The ABI boundary uses\nplain C structs (`ArrowSchema`, `ArrowArray`) so your extension can use any arrow-rs version\n(or even a different Arrow implementation entirely). Today we support authoring native\nscalar functions (UDFs) and aggregate functions (UDAFs).\n\n## Example\n\nThis example shows the end-result of our 'hello' extension with a native 'greet' scalar function.\n\n```python\nimport daft\n\n# Step 1. Import your extension module\nimport hello\n\n# Step 2. Load the extension into the current daft session\ndaft.load_extension(hello)\n\n# Step 3. Use in your dataframe!\ndf = daft.from_pydict({\"name\": [\"John\", \"Paul\"]})\ndf = df.select(hello.greet(df[\"name\"]))\ndf.show()\n\n\"\"\"\n╭──────────────╮\n│ greet        │\n│ ---          │\n│ String       │\n╞══════════════╡\n│ Hello, John! │\n├╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤\n│ Hello, Paul! │\n╰──────────────╯\n\"\"\"\n```\n\n## Tutorial\n\n### 1. Setup\n\nThis section walks through project setup from scratch; You will need a Rust toolchain and Python 3.10+.\n\n!!! note \"Note\"\n\n    We are actively working on a cloneable template.\n\n\n```bash\n# Install Rust (if not already installed)\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n\n# Install uv (if not already installed)\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n```\n\nNow you can create the project scaffolding.\n\n```bash\n# Create an empty directory\nmkdir hello && cd hello\n\n# Setup rust project\ncargo init --lib\n\n# Setup python project (for function signatures)\nuv init\n```\n\nThe crate must compile as a `cdylib` so it can be loaded at runtime via `dlopen`.\n\n```bash\ncat Cargo.toml\n```\n\n```toml\n[workspace]\n\n[package]\nname = \"hello\"\nedition = \"2024\"\nversion = \"0.1.0\"\n\n[lib]\nname = \"hello\"\ncrate-type = [\"cdylib\"]\n\n[dependencies]\ndaft-ext = { version = <version>, features = [\"arrow-58\"] }\narrow = { version = \"58\", features = [\"ffi\"] }\n```\n\n!!! tip \"Arrow version freedom\"\n\n    The `daft-ext` ABI uses C Data Interface types — your extension is **not** pinned to\n    Daft's arrow-rs version. Enable a feature flag on `daft-ext` matching your arrow-rs\n    version (`arrow-56`, `arrow-57`, or `arrow-58`) to get safe `.into()` conversions\n    between arrow-rs FFI types and the ABI types. For unsupported versions, use the\n    `from_owned`/`into_owned`/`from_raw`/`as_raw` escape hatches on `ArrowArray`\n    and `ArrowSchema`.\n\nThen update the pyproject to use `setuptools-rust` as the build system.\n\n```bash\ncat pyproject.toml\n```\n\n```toml\n[build-system]\nrequires = [\"setuptools\", \"setuptools-rust\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[project]\nname = \"hello\"\nversion = \"0.1.0\"\nrequires-python = \">=3.10\"\ndependencies = [\"daft\"]\n\n[project.optional-dependencies]\ntest = [\"pytest\"]\n```\n\n\nYou will need to create a `setup.py` file with the following contents.\n\n!!! note \"Note\"\n\n    The `RustExtension` entry tells `setuptools-rust` to compile the cdylib and place it inside the Python package directory.\n    We use the `Binding::NoBinding` because Daft extensions export a raw C symbol, not PyO3 bindings. The target name `hello.libhello`\n    places the compiled `.so` inside the `hello/` package directory so the Session's `load_extension` can find it.\n\n```bash\ncat setup.py\n```\n\n```python\nfrom setuptools import find_packages, setup\nfrom setuptools_rust import Binding, RustExtension\n\nsetup(\n    packages=find_packages(),\n    rust_extensions=[\n        RustExtension(\n            \"hello.libhello\",   # <python_package>.<lib_name>\n            path=\"Cargo.toml\",\n            binding=Binding.NoBinding,\n            strip=True,\n        )\n    ],\n)\n```\n\n### 2. Hello, World!\n\nAn extension has two parts: a **module** (the entry point) and one or more **functions** (scalar or aggregate).\n\n```bash\ncat src/lib.rs\n```\n\n```rust\nuse std::{ffi::CStr, sync::Arc};\n\nuse arrow::{\n    array::{Array, builder::StringBuilder, cast::AsArray},\n    datatypes::{DataType, Field},\n};\nuse daft_ext::prelude::*;\n\n// ── Module ──────────────────────────────────────────────────────────\n\n// #[daft_extension] generates the `daft_module_magic` C symbol that Daft's runtime looks for\n// when loading the shared library. It converts HelloExtension → hello_extension for the module name.\n#[daft_extension]\nstruct HelloExtension;\n\nimpl DaftExtension for HelloExtension {\n    /// This is the extension install hook for defining functions in the session.\n    /// Called once when the extension is loaded into a session. Register each function here.\n    fn install(session: &mut dyn DaftSession) {\n        session.define_function(Arc::new(Greet));\n    }\n}\n\n// ── Function ────────────────────────────────────────────────────────\n\n/// The function type which is registered as an Arc<dyn DaftScalarFunction>.\nstruct Greet;\n\nimpl DaftScalarFunction for Greet {\n    /// Function name used to look it up from Python.\n    /// Must be a `&CStr` (use the `c\"...\"` literal).\n    fn name(&self) -> &CStr {\n        c\"greet\"\n    }\n\n    /// Type checking.\n    /// Receives input fields as C Data Interface `ArrowSchema` types.\n    /// Use `.as_raw()` / `.into()` to convert between arrow-rs and ABI types.\n    fn return_field(&self, args: &[ArrowSchema]) -> DaftResult<ArrowSchema> {\n        if args.len() != 1 {\n            return Err(DaftError::TypeError(format!(\n                \"greet: expected 1 argument, got {}\",\n                args.len()\n            )));\n        }\n        let field = Field::try_from(&args[0])?;\n        let dt = field.data_type();\n        if *dt != DataType::Utf8 && *dt != DataType::LargeUtf8 {\n            return Err(DaftError::TypeError(format!(\n                \"greet: expected string argument, got {:?}\",\n                dt\n            )));\n        }\n        Ok(ArrowSchema::try_from(&Field::new(\n            \"greet\",\n            DataType::Utf8,\n            true,\n        ))?)\n    }\n\n    /// Evaluation. Receives columns as C Data Interface `ArrowData` types.\n    /// Use `.into()` to convert to/from arrow-rs FFI types.\n    /// All data flows through Arrow arrays — no per-row Python overhead.\n    fn call(&self, args: Vec<ArrowData>) -> DaftResult<ArrowData> {\n        let data = args.into_iter().next().ok_or_else(|| DaftError::TypeError(\"greet: expected 1 argument, got 0\".into()))?;\n        let ffi_array: arrow::ffi::FFI_ArrowArray = data.array.into();\n        let ffi_schema: arrow::ffi::FFI_ArrowSchema = data.schema.into();\n        let arrow_data = unsafe { arrow::ffi::from_ffi(ffi_array, &ffi_schema) }?;\n        let input = arrow::array::make_array(arrow_data);\n        let names = input.as_string::<i64>();\n        let mut builder = StringBuilder::with_capacity(names.len(), names.len() * 16);\n        for i in 0..names.len() {\n            if names.is_null(i) {\n                builder.append_null();\n            } else {\n                builder.append_value(format!(\"Hello, {}!\", names.value(i)));\n            }\n        }\n        let output = builder.finish();\n        let (out_arr, out_sch) = arrow::ffi::to_ffi(&output.to_data())?;\n        Ok(ArrowData {\n            array: out_arr.into(),\n            schema: out_sch.into(),\n        })\n    }\n}\n```\n\n!!! tip \"ABI pattern\"\n\n    The `DaftScalarFunction` trait uses C Data Interface types (`ArrowSchema`, `ArrowData`)\n    at the ABI boundary. Enable a `daft-ext` feature flag (`arrow-56`, `arrow-57`, or\n    `arrow-58`) matching your arrow-rs version to get `.into()` conversions. Use `.as_raw()`\n    for zero-copy borrows. This decoupling means your extension is not tied to Daft's\n    arrow-rs version.\n\n!!! tip \"String types\"\n\n    Daft uses `LargeUtf8` (i64 offsets) for strings internally. When downcasting string arrays,\n    use `as_string::<i64>()` — using `i32` will panic at runtime. Similarly, when checking types\n    in `return_field`, accept `DataType::LargeUtf8`.\n\n!!! tip \"Naming\"\n\n    Function names are global within a session. Use a prefix (e.g., `myext_greet`) to avoid\n    collisions when your extension defines many functions or might be loaded alongside others.\n\n!!! tip \"Multiple functions\"\n\n    Register as many functions as you need in `install()` — each is an independent struct\n    implementing `DaftScalarFunction`.\n\n!!! tip \"Errors\"\n\n    Return `Err(DaftError::TypeError(...))` for schema violations in `return_field`,\n    and `Err(DaftError::RuntimeError(...))` for execution failures in `call`.\n\n### Aggregate Functions\n\nExtensions can also register **aggregate functions** (UDAFs) alongside scalar functions.\nAn aggregate function follows a three-stage pipeline:\n\n1. **Aggregation** (`aggregate`) — process input arrays into partial state\n2. **Combination** (`combine`) — merge partial states\n3. **Finalization** (`finalize`) — produce the final output from merged state\n\nState is exchanged as `Vec<ArrowData>` — each element is one state field. The FFI layer\npacks these into a Struct array transparently.\n\n```rust\nuse std::ffi::CStr;\nuse daft_ext::prelude::*;\n\nstruct MySum;\n\nimpl DaftAggregateFunction for MySum {\n    fn name(&self) -> &CStr {\n        c\"my_sum\"\n    }\n\n    /// Declare the output type given the input field schemas.\n    fn return_field(&self, _args: &[ArrowSchema]) -> DaftResult<ArrowSchema> {\n        Ok(ArrowSchema::try_from(&Field::new(\"my_sum\", DataType::Int64, true))?)\n    }\n\n    /// Declare intermediate state fields.\n    /// Each state field becomes a child of the Struct exchanged at the FFI boundary.\n    fn state_fields(&self, _args: &[ArrowSchema]) -> DaftResult<Vec<ArrowSchema>> {\n        Ok(vec![\n            ArrowSchema::try_from(&Field::new(\"sum\", DataType::Int64, true))?,\n            ArrowSchema::try_from(&Field::new(\"count\", DataType::Int64, false))?,\n        ])\n    }\n\n    /// Process input arrays into partial state.\n    /// Returns one ArrowData per state field (single-row each).\n    fn aggregate(&self, inputs: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>> {\n        // inputs[0] is the column to sum\n        // compute sum and count, return as two single-element arrays\n        todo!()\n    }\n\n    /// Merge multiple partial states into one.\n    /// states[i] is the i-th state field with multiple rows (one per partial).\n    fn combine(&self, states: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>> {\n        // states[0] = sums array, states[1] = counts array\n        // sum them up, return single-row arrays\n        todo!()\n    }\n\n    /// Produce the final scalar result from merged state.\n    fn finalize(&self, states: Vec<ArrowData>) -> DaftResult<ArrowData> {\n        // states[0] = sum (single value), states[1] = count (single value)\n        // return the final sum\n        todo!()\n    }\n}\n```\n\nRegister aggregate functions in `install()` using `define_aggregate_function`:\n\n```rust\nimpl DaftExtension for MyExtension {\n    fn install(session: &mut dyn DaftSession) {\n        session.define_function(Arc::new(MyScalarFn));\n        session.define_aggregate_function(Arc::new(MySum));\n    }\n}\n```\n\nNow we define the python symbols for use in the Expression DSL; we link to rust via `daft.get_function` for scalars and `daft.get_aggregate_function` for aggregates.\n\n!!! note \"Note\"\n\n    Python sources let you write python function signatures and any argument pre-processing before linking\n    to the function symbol. The `daft.get_function` method is calling `get_function` on the active session\n    to resolve the given name and arguments to some defined function in the session. Notice how we don't\n    technically need python to resolve functions in SQL (hence no PyO3) but these python functions give us\n    nice pythonic functions with autocomplete and doc comments when using the Expression DSL.\n\n```bash\ncat hello/__init__.py\n```\n\n```python\nfrom __future__ import annotations\nfrom typing import TYPE_CHECKING\n\nimport daft\n\nif TYPE_CHECKING:\n    from daft.expressions import Expression\n\ndef greet(name: Expression) -> Expression:\n    \"\"\"Greet someone by name.\"\"\"\n    return daft.get_function(\"greet\", name)\n\ndef string_count(name: Expression) -> Expression:\n    \"\"\"Count non-null strings.\"\"\"\n    return daft.get_aggregate_function(\"string_count\", name)\n```\n\n`daft.get_function` looks up a scalar function registered with the current session by the name returned from `DaftScalarFunction::name()`.\n`daft.get_aggregate_function` does the same for aggregate functions registered via `DaftAggregateFunction::name()`.\n\nAdd an empty `hello/py.typed` marker if you want type-checker support.\n\n### 3. Build, Install, Test\n\n```bash\n# Compile the Rust cdylib and install the package in editable mode\nuv pip install -e .\n```\n\nHere are some sanity check tests. Notice how we use a scoped session rather than\nthe global active session. How you choose to load extensions is up to you, and this\ntutorial has covered both.\n\n!!! tip \"Session isolation\"\n\n    Extensions are loaded once into the process and the session serves as a scoping mechanism\n    for name resolution; calling `load_extension` multiple times will only `dlopen` once for\n    this process. Functions are only available in sessions where the extension is loaded.\n    Use the `with sess:` context manager to scope queries to a specific session.\n\n```bash\ncat tests/test_hello.py\n```\n\n```python\nimport daft\nimport hello\nfrom daft import col\nfrom daft.session import Session\nfrom hello import greet\n\ndef test_greet():\n    sess = Session()\n    sess.load_extension(hello)\n\n    df = daft.from_pydict({\"name\": [\"John\", \"Paul\"]})\n\n    with sess:\n        result = df.select(greet(col(\"name\"))).collect().to_pydict()\n\n    values = result[\"greet\"]\n    assert values[0] == \"Hello, John!\"\n    assert values[1] == \"Hello, Paul!\"\n\ndef test_greet_null():\n    sess = Session()\n    sess.load_extension(hello)\n\n    df = daft.from_pydict({\"name\": [\"George\", \"Ringo\", None]})\n\n    with sess:\n        result = df.select(greet(col(\"name\"))).collect().to_pydict()\n\n    values = result[\"greet\"]\n    assert values[0] == \"Hello, George!\"\n    assert values[1] == \"Hello, Ringo!\"\n    assert values[2] is None\n```\n\nNow run the tests!\n\n```bash\npytest -v tests/\n```\n\n## Prompt\n\nYou can paste this whole document and prompt into Claude Code to scaffold a Daft extension for you.\n\n````markdown\nCreate a Daft native extension called `<extension_name>` with the following functions:\n\n<describe each function: name, scalar or aggregate, arguments with types, return type, and behavior>\n\nFollow the Daft extension authoring guide at docs/extensions/authoring.md. Here is a summary of the key conventions:\n\n## Project structure\n\n```\n<extension_name>/\n  Cargo.toml           # [lib] crate-type = [\"cdylib\"], depends on daft-ext (with arrow feature), arrow\n  pyproject.toml       # build-system: setuptools + setuptools-rust\n  setup.py             # RustExtension(\"<pkg>.lib<pkg>\", binding=NoBinding, strip=True)\n  <extension_name>/\n    __init__.py        # Python wrappers using daft.get_function(\"name\", *args)\n    py.typed           # empty PEP 561 marker\n  src/\n    lib.rs             # #[daft_extension] struct + DaftScalarFunction / DaftAggregateFunction impls\n  tests/\n    test_<name>.py     # pytest tests using Session fixture\n  .gitignore           # /target, *.so, *.dylib, *.dll, *.egg-info, __pycache__, dist/\n```\n\n## Rust conventions\n\n- Use `daft_ext::prelude::*` for all imports (provides `ArrowSchema`, `ArrowData`, errors, traits).\n- Add `daft-ext` with a feature flag matching your arrow version (`arrow-56`, `arrow-57`, or `arrow-58`) for `.into()` conversions.\n- Import `arrow::array::Array` for `len()`/`is_null()` and `arrow::array::cast::AsArray` for downcasting.\n- Daft uses `LargeUtf8` (i64 offsets) for strings — downcast with `as_string::<i64>()`, never `i32`.\n- Apply `#[daft_extension]` to a struct implementing `DaftExtension`.\n- Register scalar functions via `session.define_function(Arc::new(MyFn))`.\n- Register aggregate functions via `session.define_aggregate_function(Arc::new(MyAgg))`.\n- Each scalar function is a struct implementing `DaftScalarFunction` with:\n  - `name(&self) -> &CStr` — use `c\"<extension_name>_<fn_name>\"` prefix to avoid collisions.\n  - `return_field(&self, args: &[ArrowSchema]) -> DaftResult<ArrowSchema>` — use `.as_raw()` to\n    borrow as arrow-rs `FFI_ArrowSchema` for type checking, then `.into()` to return output.\n  - `call(&self, args: &[ArrowData]) -> DaftResult<ArrowData>` — use `ArrowData::take_arg` then\n    `.into()` to convert to arrow-rs FFI types, compute, then `.into()` to return the result.\n- Each aggregate function is a struct implementing `DaftAggregateFunction` with:\n  - `name`, `return_field` — same as scalar functions.\n  - `state_fields(&self, args: &[ArrowSchema]) -> DaftResult<Vec<ArrowSchema>>` — intermediate state schema.\n  - `aggregate(&self, inputs: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>>` — partial aggregation.\n  - `combine(&self, states: Vec<ArrowData>) -> DaftResult<Vec<ArrowData>>` — merge partial states.\n  - `finalize(&self, states: Vec<ArrowData>) -> DaftResult<ArrowData>` — produce final result.\n\n## Python conventions\n\n- Each scalar function wrapper calls `daft.get_function(\"<extension_name>_<fn_name>\", *args)`.\n- Each aggregate function wrapper calls `daft.get_aggregate_function(\"<extension_name>_<fn_name>\", *args)`.\n- Use `TYPE_CHECKING` guard for `Expression` import.\n- Add type hints and a docstring to each wrapper.\n````\n\n<!-- File: docs/extensions/community.md (Tokens: ~2026 | Category: anthropic-skill) -->\n# Community Extensions\n\n!!! tip \"Want to build your own extension?\"\n\n    Start with the [Extensions overview](overview.md), the [Python UDF docs](../custom-code/func.md),\n    or the [Native Extension Authoring Guide](authoring.md).\n\nCommunity extensions are reusable packages that add functionality to Daft\noutside of the core repository. They may be pure-Python UDF-based extensions,\nnative ABI extensions, custom DataSource/DataSink connectors, or higher-level\ndomain libraries that combine Daft expressions, UDFs, file types, model calls,\nand distributed execution patterns.\n\nProjects listed here are maintained independently of Daft's release cadence.\nEach project installs and versions independently of Daft.\n\n| Name | Kind | Repository | Description |\n|---|---|---|---|\n| [daft-h3](#daft-h3) | Native ABI | [gweaverbiodev/daft-h3](https://github.com/gweaverbiodev/daft-h3) | Native [H3](https://h3geo.org/) geospatial indexing functions. |\n| [daft-lance](#daft-lance) | Python UDF-based | [daft-engine/daft-lance](https://github.com/daft-engine/daft-lance) | Lance-specific distributed operations for compaction, scalar indexing, column merging, and REST catalog operations. |\n| [daft-html](#daft-html) | Native ABI | [daft-engine/daft-html](https://github.com/daft-engine/daft-html) | Native HTML document processing functions exposed as Daft expressions. |\n| [daft-geo](#daft-geo) | Native ABI / Datatypes | [daft-engine/daft-geo](https://github.com/daft-engine/daft-geo) | Geospatial prototype showing native functions and extension-backed datatypes. |\n| [daft-qdrant](#daft-qdrant) | Python UDF-based | [qdrant-labs/daft-qdrant](https://github.com/qdrant-labs/daft-qdrant) | Write vector embeddings and their payloads into [Qdrant](https://qdrant.tech/) collections. |\n| [daft-doris](#daft-doris) | Custom DataSource/DataSink | [jiangxt2/daft-doris](https://github.com/jiangxt2/daft-doris) | Apache Doris reads and HTTP Stream Load writes for Daft. |\n\nTo propose a new extension for this list, open a PR against this page.\n\n## daft-h3\n\n`daft-h3` adds H3 geospatial indexing functions such as latitude/longitude to\ncell conversion, cell traversal, grid distance, parent resolution, and string\nconversion.\n\n```bash\npip install daft-h3\n```\n\n```python\nimport daft\nimport daft_h3\nfrom daft import col\n\ndaft.load_extension(daft_h3)\n\ndf = daft.from_pydict({\"lat\": [37.7749], \"lng\": [-122.4194]})\ndf = df.select(\n    daft_h3.h3_latlng_to_cell(col(\"lat\"), col(\"lng\"), 7).alias(\"cell\"),\n)\ndf = df.select(daft_h3.h3_cell_to_str(col(\"cell\")).alias(\"hex\"))\ndf.show()\n```\n\nSee the [daft-h3 README](https://github.com/gweaverbiodev/daft-h3#readme) for\nthe full list of functions and behavior details.\n\n## daft-lance\n\n`daft-lance` extends Daft with Lance-specific distributed maintenance and data\nmanagement operations. It is a Python UDF-based extension: internally, it uses\nDaft's Python UDF and class-UDF APIs to distribute Lance tasks across Daft\nqueries, while users interact with simple Python functions.\n\n```bash\npip install daft-lance\n```\n\n```python\nfrom daft_lance import compact_files, create_scalar_index\n\ncompact_files(\"s3://bucket/my_dataset\")\n\ncreate_scalar_index(\n    \"s3://bucket/my_dataset\",\n    column=\"name\",\n    index_type=\"INVERTED\",\n)\n```\n\nSee the [daft-lance README](https://github.com/daft-engine/daft-lance#readme)\nfor additional operations such as column merging and REST catalog writes.\n\n## daft-html\n\n`daft-html` adds native HTML processing functions such as `html_to_text`,\n`html_extract_links`, `html_extract_tables`, and CSS-selector extraction.\n\n```bash\npip install daft-html\n```\n\n```python\nimport daft\nimport daft_html\nfrom daft import col\nfrom daft_html import html_to_text\n\ndaft.load_extension(daft_html)\n\ndf = daft.from_pydict({\"html\": [\"<html><body><h1>Hello</h1></body></html>\"]})\ndf = df.select(html_to_text(col(\"html\")).alias(\"text\"))\ndf.show()\n```\n\nSee the [daft-html README](https://github.com/daft-engine/daft-html#readme)\nfor the full list of document and CSS-selector functions.\n\n## daft-geo\n\n`daft-geo` is a geospatial prototype showing native Daft functions and\nextension-backed datatypes. It defines `Point2D` and `Point3D` as\n`DataType.extension(...)` values and provides point construction, accessors,\nEuclidean distance, and haversine distance.\n\n```bash\npip install git+https://github.com/daft-engine/daft-geo.git\n```\n\n!!! note\n\n    `daft-geo` is not yet published to PyPI. The command above installs the\n    latest commit from the repository's default branch, so behavior may change\n    between installs.\n\n```python\nimport daft\nimport daft_geo\nfrom daft import col\n\ndaft.load_extension(daft_geo)\n\ndf = daft.from_pydict({\"x\": [0.0], \"y\": [1.0]})\ndf = df.select(daft_geo.point2d(col(\"x\"), col(\"y\")).alias(\"point\"))\ndf = df.select(daft_geo.x(col(\"point\")).alias(\"x\"))\ndf.show()\n```\n\nSee the [daft-geo README](https://github.com/daft-engine/daft-geo#readme) for\ncurrent capabilities and caveats.\n\n## daft-qdrant\n\n`daft-qdrant` adds a `write_qdrant` DataFrame method for writing vector\nembeddings and their payloads into [Qdrant](https://qdrant.tech/) collections.\n\n```bash\npip install daft-qdrant\n```\n\n```python\nimport daft\nimport daft_qdrant\n\ndf = daft.from_pydict({\n    \"id\": [1, 2, 3],\n    \"vector\": [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]],\n    \"label\": [\"cat\", \"dog\", \"bird\"],\n})\n\ndf.write_qdrant(\"my-collection\", url=\"http://localhost:6333\")\n```\n\nSee the [daft-qdrant README](https://github.com/qdrant-labs/daft-qdrant#readme) for embedding pipeline examples and additional options.\n\n## daft-doris\n\n`daft-doris` is an independently maintained Python custom connector for\nApache Doris. It supports reads from Doris physical tables and batch writes\nthrough HTTP Stream Load. The read transport must be selected explicitly;\nFlight SQL is available as an opt-in experimental transport.\n\n```bash\npip install \"daft-doris[doris]\"\n```\n\nFor Flight SQL read support, install the optional Flight extra:\n\n```bash\npip install \"daft-doris[doris-flight]\"\n```\n\nRead a Doris table:\n\n```python\nfrom daft_doris import read_doris\n\ndf = read_doris(\n    host=\"doris-fe.example\",\n    database=\"analytics\",\n    table=\"events\",\n    transport=\"mysql\",\n)\n```\n\nWrite a Daft DataFrame through Stream Load:\n\n```python\nimport daft\n\nfrom daft_doris import DorisConnection, DorisTable, SecretRef, write_doris\n\ndf = daft.from_pydict(\n    {\n        \"event_id\": [1, 2],\n        \"score\": [95.0, 88.0],\n    }\n)\nresult = write_doris(\n    df,\n    connection=DorisConnection(\n        host=\"doris-fe.example.com\",\n        username=\"daft_writer\",\n        password=SecretRef.env(\"DORIS_PASSWORD\"),\n        http_port=8030,\n        redirect_hosts=(\"doris-be.example.com\",),\n        redirect_ports=(8040,),\n        redirect_policy=\"public\",\n    ),\n    table=DorisTable(database=\"analytics\", name=\"events\"),\n    operation=\"load\",\n)\n```\n\nThe connector is currently in Alpha and is maintained independently of Daft's\nrelease cadence. The current release supports Python 3.12–3.13 and Daft\n0.7.23; its Daft dependency is bounded to `>=0.7.23,<0.7.24`, so installing it\nalongside a newer Daft release may cause the resolver to install Daft 0.7.23.\nSee the [`daft-doris` repository](https://github.com/jiangxt2/daft-doris#readme)\nfor supported scope, compatibility information, and limitations.\n\n<!-- File: docs/extensions/overview.md (Tokens: ~1327 | Category: anthropic-skill) -->\n# Extensions\n\nDaft extensions are reusable libraries that add domain-specific functionality\non top of Daft without requiring every specialized function, datatype,\nintegration, or workflow to live directly in Daft core.\n\nThere are two broad ways to build extensions:\n\n- **Python UDF-based extensions**: Python packages built on Daft's\n  [custom-code APIs](../custom-code/func.md), including\n  [`@daft.func`](../api/udf.md), [`@daft.func.batch`](../api/udf.md),\n  [`@daft.cls`](../api/udf.md), and [`@daft.method.batch`](../api/udf.md).\n- **Native ABI extensions**: shared-library-backed packages that use Daft's\n  Arrow C Data Interface ABI for lower-level, vectorized native execution.\n\nBoth models can expose clean Python APIs that feel like ordinary Daft\nexpression functions. The implementation details can stay hidden behind\nfunctions, classes, and expressions.\n\n## Python UDF-Based Extensions\n\nThe fastest way to build a Daft extension is often pure Python. Using\n[`@daft.func`](../api/udf.md), [`@daft.func.batch`](../api/udf.md),\n[`@daft.cls`](../api/udf.md), and [`@daft.method.batch`](../api/udf.md),\ncontributors can package reusable Python logic that runs inside Daft's\ndistributed execution engine.\n\nThis is ideal for libraries that orchestrate existing Python ecosystems,\nexternal services, ML models, GPUs, data maintenance tasks, or domain-specific\nworkflows. These extensions do not require a native shared library or the\nArrow C ABI. They are normal Python packages that expose higher-level Daft\nAPIs.\n\n[`daft-lance`](https://github.com/daft-engine/daft-lance) is an example of\nthis model. It extends Daft with Lance-specific distributed operations such as\nfile compaction, scalar indexing, column merging, and REST catalog operations.\nInternally, it uses Daft's Python UDF and class-UDF APIs to distribute Lance\ntasks across a Daft query, while users interact with functions like\n`compact_files`, `create_scalar_index`, and `merge_columns_df`.\n\nThis pattern also exists inside Daft itself. The\n[`daft.functions.ai`](../api/ai.md) module exposes high-level functions like\n[`prompt`](../api/functions/prompt.md),\n[`embed_text`](../api/functions/embed_text.md),\n[`embed_image`](../api/functions/embed_image.md),\n[`classify_text`](../api/functions/classify_text.md), and\n[`classify_image`](../api/functions/classify_image.md). To users, these look\nlike normal Daft expression functions. Under the hood, they use Daft's UDF and\nclass-UDF machinery, including batching, concurrency controls, retries, and\n[GPU resource hints](../custom-code/gpu.md).\n\nThe file APIs follow a similar product pattern. [`daft.File`](../api/datatypes/file_types.md)\nvalues and helpers like [`file`](../api/functions/file.md),\n[`audio_file`](../api/functions/audio_file.md),\n[`video_file`](../api/functions/video_file.md),\n[`file_path`](../api/functions/file_path.md), and\n[`file_size`](../api/functions/file_size.md) give users expression-level\nbuilding blocks for working with files. Those file objects can then flow into\nPython UDFs, model pipelines, and domain-specific libraries without users\nneeding to think about execution details.\n\n## Native ABI Extensions\n\nFor contributors who need lower-level, vectorized performance, Daft also\nsupports native extensions through a C ABI based on the\n[Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html).\n\nNative ABI extensions can add high-performance scalar functions, aggregate\nfunctions, Python expression wrappers, and extension-backed datatypes. They\nship as pip-installable Python packages that bundle a native shared library.\nUsers import the package, load it into a Daft [`Session`](../api/sessions.md),\nand call ordinary Python wrappers in their DataFrame expressions.\n\nRust currently has the most ergonomic SDK through\n[`daft-ext`](https://github.com/Eventual-Inc/Daft/tree/main/src/daft-ext),\nwhile C++ is demonstrated through the raw ABI. Other systems languages are\npossible if they can produce a shared library, export the expected C ABI, and\nread/write Arrow C Data Interface arrays.\n\nDaft's own examples include:\n\n- [`examples/hello`](https://github.com/Eventual-Inc/Daft/tree/main/examples/hello):\n  a minimal Rust native extension that registers a `greet` scalar function.\n- [`examples/dvector`](https://github.com/Eventual-Inc/Daft/tree/main/examples/dvector):\n  a pgvector-style native extension for vector distance functions such as\n  `l2_distance`, `cosine_distance`, `inner_product`, and `jaccard_distance`.\n- [`examples/hello_cpp`](https://github.com/Eventual-Inc/Daft/tree/main/examples/hello_cpp):\n  a pure C++ native extension using Apache Arrow C++ and the raw Daft C ABI.\n\n## What Can Contributors Build?\n\nContributors can build:\n\n- Pure-Python UDF extension libraries using [`@daft.func`](../api/udf.md),\n  [`@daft.func.batch`](../api/udf.md), [`@daft.cls`](../api/udf.md), and\n  [`@daft.method.batch`](../api/udf.md)\n- Distributed task libraries like\n  [`daft-lance`](https://github.com/daft-engine/daft-lance)\n- GPU-backed model inference extensions\n- AI, file, media, and multimodal processing libraries that hide UDFs behind\n  expression APIs\n- Native scalar functions\n- Native aggregate functions / UDAFs\n- Python expression wrappers\n- Extension-backed logical datatypes such as\n  [`DataType.extension(...)`](../api/datatypes/all_datatypes.md)\n\n## Where to Start\n\n- Browse installable projects in [Community Extensions](community.md).\n- Learn how to build a native extension in the [Authoring Guide](authoring.md).\n- Browse projects using Daft in [Built on Daft](projects.md).\n- Learn more about Daft's [Python UDFs](../custom-code/func.md).\n\n<!-- File: docs/extensions/projects.md (Tokens: ~2190 | Category: anthropic-skill) -->\n# Built on Daft\n\nThis page highlights projects that depend on or build on top of\n[Daft](https://github.com/Eventual-Inc/Daft).\n\nThese projects are different from [Community Extensions](community.md). An\nextension directly adds reusable Daft APIs, functions, datatypes, or distributed\noperators. A project built on Daft may instead use Daft as a query engine,\nDataFrame runtime, compute backend, or pipeline framework.\n\nTo propose a project for this page, open a PR with a short description and a\nlink to the relevant Daft usage.\n\n## Projects Building on Daft\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [DeltaCat](https://github.com/ray-project/deltacat) | Portable multimodal lakehouse on Ray, Arrow, and Daft. | Daft is a core dependency: `dc.read()` returns a Daft DataFrame by default, and Daft is a first-class compute backend for reading and writing lake data. |\n| [SmooSense](https://github.com/SmooSenseAI/smoosense) | Web-based tool for interactively exploring large-scale multimodal tabular data. | Accepts Daft DataFrames as input in the Jupyter widget (`Sense(daft_df)`) and converts them for display alongside Pandas DataFrames. |\n| [eai-taxonomy](https://github.com/Essential-AI/eai-taxonomy) | Annotation tools and dataset filters from the Essential-Web project. | Uses Daft in annotation pipelines to read Parquet annotation files and compute inter-annotator agreement. |\n| [GraphReduce](https://github.com/wesmadrigal/GraphReduce) | Feature engineering library for large graphs of tabular data. | Supports Daft as a compute backend via `pip install \"graphreduce[daft]\"`. |\n| [Archetype](https://github.com/VangelisTech/archetype) | DataFrame-first, append-only ECS runtime for simulations and AI agents. | Uses Daft DataFrames as the native data structure for world state and processor logic. |\n| [hypergraph](https://github.com/gilad-rubin/hypergraph) | Python workflow orchestration framework for DAG pipelines and agentic workflows. | Provides a `DaftRunner` that compiles a hypergraph DAG into a Daft query plan using Daft UDFs. |\n| [Sashimi4Talent](https://github.com/colin-ho/Sashimi4Talent) | Data pipeline and web app for discovering and ranking GitHub contributors. | Uses Daft as the core processing engine for repository search, commits, contributor analysis, and dataset merges. |\n| [daft-image-playground](https://github.com/peckjon/daft-image-playground) | Flask-based intelligent image search engine. | Uses Daft to discover images and run native image decoding, resizing, and encoding. |\n| [SMiR](https://github.com/togethercomputer/SMiR) | Synthetic data pipeline for multi-image reasoning. | Uses Daft with the Ray runner to read Parquet from S3, download images, and run GPU-accelerated embedding UDFs. |\n| [Derezz](https://github.com/rchowell/Derezz) | CLI tool for video indexing and semantic search. | Uses Daft to read and write an S3 Tables-backed video frame index and query it. |\n| [teraflopai-daft](https://github.com/teraflop-ai/teraflopai-daft) | Daft plugin library for Teraflop AI NLP services. | Exposes custom Daft expressions such as `segment_text` and `search_text` that call the Teraflop AI provider API. |\n| [daft-sql-adapter](https://github.com/RaghwendraSingh/daft-sql-adapter) | CLI adapter that runs Spark SQL queries against Databricks Unity Catalog tables through Daft. | Uses Daft as the execution engine for SQL queries and table writes. |\n| [zarr-daft-datasource](https://github.com/liaoruoxue/zarr-daft-datasource) | Prototype compute-storage-separation architecture using Zarr, LanceDB, and Daft. | Implements a custom Daft `DataSource` that reads Zarr arrays into lazy Daft DataFrames. |\n| [pdf-document-processing-daft](https://github.com/doronkatz/pdf-document-processing-daft) | Demonstration pipeline for scalable PDF document processing. | Uses Daft DataFrames to parallelize PDF parsing, text chunking, embedding generation, and Parquet writes. |\n| [daft-ai-driving](https://github.com/legendPerceptor/daft-ai-driving) | Project for processing the KITTI autonomous driving dataset. | Uses Daft DataFrames for data loading, filtering, aggregation, and Parquet export. |\n| [sigil](https://github.com/junaidrahim/sigil) | CLI tool for collecting and analyzing AI coding session data. | Uses Daft to write local Parquet session data and query it with filtering and aggregation. |\n| [embed_qwen_k8s](https://github.com/jaychia/embed_qwen_k8s) | Kubernetes pipeline for Common Crawl text processing and embeddings. | Uses Daft to read WARC/Parquet data, apply UDFs for chunking and embedding, and write results to S3. |\n\n## Usage Patterns and Examples\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [daft-examples](https://github.com/Eventual-Inc/daft-examples) | Official Daft examples repository. | Shows idiomatic Daft usage across `daft.read_*`, UDFs, AI functions, multimodal data, SQL analytics, vector search, and integrations. |\n| [workflow-eventual-inc-daft-distributed-udf-processing](https://github.com/leeroopedia/workflow-eventual-inc-daft-distributed-udf-processing) | Workflow demonstrating custom Python UDFs for ML inference and GPU transforms. | Uses `@daft.func` and `@daft.cls` with lazy and distributed execution. |\n| [workflow-eventual-inc-daft-sql-query-analytics](https://github.com/leeroopedia/workflow-eventual-inc-daft-sql-query-analytics) | Workflow for SQL analytics with Daft sessions and catalogs. | Creates sessions, attaches catalogs, registers tables, executes SQL, and exports results. |\n| [workflow-eventual-inc-daft-data-lakehouse-etl](https://github.com/leeroopedia/workflow-eventual-inc-daft-data-lakehouse-etl) | ETL pipeline for Iceberg, Delta Lake, and Hudi lakehouse formats. | Uses Daft lakehouse readers, DataFrame transformations, and write APIs. |\n| [workflow-eventual-inc-daft-multimodal-ai-batch-inference](https://github.com/leeroopedia/workflow-eventual-inc-daft-multimodal-ai-batch-inference) | Workflow for AI inference over multimodal datasets. | Applies Daft AI functions and embedding workflows to Hugging Face, Parquet, and CSV datasets. |\n\n## Daft as a Chosen Backend\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [cuallee](https://github.com/canimus/cuallee) | DataFrame-agnostic data quality check library. | Implements a Daft validation backend and test suite for quality checks against Daft DataFrames. |\n| [Atlan application-sdk](https://github.com/atlanhq/application-sdk) | SDK for building data catalog integration applications. | Uses Daft to lazily read and filter JSON table metadata files during incremental sync. |\n| [narwhals-daft](https://github.com/narwhals-dev/narwhals-daft) | Narwhals plugin for Daft. | Wraps Daft DataFrames so the Narwhals expression API can execute lazily on Daft. |\n| [jett](https://github.com/ddeutils/jett) | ETL template framework supporting multiple DataFrame engines. | Lists Daft as a supported engine backend for YAML-defined ETL pipelines. |\n\n## Ecosystem and Content\n\n| Project | Description | How It Uses Daft |\n|---|---|---|\n| [AWS MCP servers](https://github.com/awslabs/mcp) | Collection of open-source MCP servers for AWS services. | The S3 Tables MCP server uses Daft as the SQL query engine for read-only queries against AWS S3 Tables. |\n| [Fabric_Notebooks_Demo](https://github.com/djouallah/Fabric_Notebooks_Demo) | Microsoft Fabric notebook demos. | Includes ETL notebooks using Daft to load and transform CSVs from Azure storage and write Delta Lake tables. |\n| [lakevision](https://github.com/lakevision-project/lakevision) | Web tool for exploring Apache Iceberg lakehouses. | Uses Daft to query and display sample data from Iceberg tables. |\n| [general-demos](https://github.com/jaehyeon-kim/general-demos) | Data engineering demo projects. | Includes a `daft-quickstart` folder demonstrating Daft with Apache Iceberg. |\n| [DaftHudi](https://github.com/dipankarmazumdar/DaftHudi) | Demo dashboard application using Apache Hudi, Daft, and Streamlit. | Uses Daft to read Apache Hudi tables and power analytical queries. |\n| [Engineering Lakehouses with Open Table Formats](https://github.com/PacktPublishing/Engineering-Lakehouses-with-Open-Table-Formats) | Code samples for a lakehouse engineering book. | Includes a notebook demonstrating reading and writing Apache Iceberg tables with Daft. |\n| [databricks-demos](https://github.com/david-hurley/databricks-demos) | Databricks demo notebooks. | Includes a notebook using Daft with Unity Catalog to write data to Delta tables. |\n| [knee-deep-in-the-lake](https://github.com/Cumulocity-IoT/knee-deep-in-the-lake) | Hands-on training repository for IoT data lake technologies. | Uses Daft in notebooks alongside PyArrow and Pandas for querying Parquet files with SQL. |\n\n<!-- File: .claude-plugin/marketplace.json (Tokens: ~103 | Category: marketplace) -->\n{\n  \"name\": \"Eventual-Inc\",\n  \"owner\": {\n    \"name\": \"Eventual, Inc.\",\n    \"email\": \"yksu@eventualcomputing.com\"\n  },\n  \"plugins\": [\n    {\n      \"name\": \"daft\",\n      \"source\": \"./\",\n      \"description\": \"Skills for working with Daft: UDF tuning, distributed scaling, and docs navigation\",\n      \"version\": \"0.1.0\"\n    }\n  ]\n}\n\n<!-- File: .claude-plugin/plugin.json (Tokens: ~135 | Category: plugin-manifest) -->\n{\n  \"name\": \"daft\",\n  \"description\": \"Skills for working with Daft: UDF tuning, distributed scaling, and docs navigation\",\n  \"version\": \"0.1.0\",\n  \"author\": {\n    \"name\": \"Eventual, Inc.\",\n    \"email\": \"yksu@eventualcomputing.com\"\n  },\n  \"repository\": \"https://github.com/Eventual-Inc/Daft\",\n  \"license\": \"Apache-2.0\",\n  \"keywords\": [\n    \"daft\",\n    \"dataframe\",\n    \"udf\",\n    \"distributed\",\n    \"ray\"\n  ]\n}\n\n<!-- File: docs/plugins/README.md (Tokens: ~158 | Category: plugin-manifest) -->\n# Daft MkDocs Plugins\n\nCustom MkDocs plugins for the Daft documentation site.\n\n## nav-hide-children\n\nA plugin that allows you to conditionally hide navigation children based on configured paths.\n\n### Configuration\n\nAdd to your `mkdocs.yml`:\n\n```yaml\nplugins:\n  - nav-hide-children:\n      hide_children:\n        - [\"Python API\", \"Functions\"]  # Hide children of Python API > Functions\n        - [\"SQL Reference\"]            # Hide children of SQL Reference\n```\n\n### Usage\n\nThe plugin automatically injects an attribute `hide_children` into [navigation objects](https://www.mkdocs.org/dev-guide/themes/#navigation-objects) based on the config paths, which can be used in Jinja2 templates to conditionally hide navigation elements.\n\n<!-- File: tools/observability/opentelemetry/README.md (Tokens: ~359 | Category: plugin-manifest) -->\n# OpenTelemetry\n\nThis directory contains the code for running the OpenTelemetry collector and exporter.\n\nThe docker compose file located in this directory will start the following services:\n\n1. [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) - an agent for collecting and exporting opentelemetry data\n2. [Jaeger](https://www.jaegertracing.io/) - an open source distributed tracing system\n3. [Prometheus](https://prometheus.io/) - an open source metrics monitoring system\n\n## Running the collector\n\nTo start the services, navigate to this directory and run the following command:\n\n```bash\ndocker compose up\n```\n\n## Executing a job with opentelemetry enabled\n\nThe opentelemetry collector exposes an OTLP gRPC endpoint on port 4317.\nTo execute a job with opentelemetry enabled, set the `DAFT_DEV_OTEL_EXPORTER_OTLP_ENDPOINT` environment variable to `grpc://localhost:4317` and run the job.\n\nExample:\n```bash\nDAFT_DEV_OTEL_EXPORTER_OTLP_ENDPOINT=grpc://localhost:4317 python tools/observability/opentelemetry/example.py\n```\n\n## Viewing the metrics\n\n### Traces (Jaeger)\n\nTo view the traces on the Jaeger UI, navigate to `http://localhost:16686/` and select the `daft` service from the dropdown menu.\n\n### Metrics (Prometheus)\n\nTo view the metrics on the Prometheus UI, navigate to `http://localhost:9090/`.\n\nTo query for all metrics provided by Daft, try the following [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query:\n\n```promql\n{service_name='daft'}\n```\n</yakaai_skills>"}