## File: README.md

<h1 align="center">
    **data load tool (dlt) — the open-source Python library that automates all your tedious data loading tasks**
</h1>



<h3 align="center">

🚀 Join our thriving community of likeminded developers and build the future together!

</h3>




## Installation

dlt supports Python 3.10 through Python 3.14. Note that some optional extras are not yet available for Python 3.14, so support for this version is considered experimental.

```sh
pip install dlt
```

Add the extras you need for your sources and destinations, for example:

```sh
pip install "dlt[duckdb]"        # local DuckDB destination
pip install "dlt[bigquery]"      # or snowflake, postgres, redshift, databricks, athena, ...
pip install "dlt[s3]"            # or gs, az for cloud filesystems
pip install "dlt[sql_database]"  # read from any SQL database
pip install "dlt[hub]"           # data quality, transformations, and AI (see below)
```

Prefer [uv](https://docs.astral.sh/uv/)? `uv add "dlt[duckdb]"`.

## Quick Start

Describe an API declaratively and load it into DuckDB — dlt handles requests, pagination, schema inference, and typing for you:

```python
import dlt
from dlt.sources.rest_api import rest_api_source

# 1. Describe the API declaratively
source = rest_api_source({
    "client": {"base_url": "https://pokeapi.co/api/v2/"},
    "resources": [
        {"name": "pokemon", "endpoint": {"path": "pokemon", "params": {"limit": 1000}}},
    ],
})

# 2. Point a pipeline at any destination
pipeline = dlt.pipeline(
    pipeline_name="pokemon",
    destination="duckdb",
    dataset_name="pokemon_data",
)

# 3. Extract, normalize, and load
print(pipeline.run(source))

# 4. ...and read it straight back as a DataFrame
print(pipeline.dataset().pokemon.df())
```

...or load any Python iterable — a [resource](https://dlthub.com/docs/general-usage/resource) is just a generator, and dlt infers the schema, types the columns, and writes the table:

```python
import dlt

@dlt.resource(table_name="players", primary_key="id", write_disposition="merge")
def players():
    yield {"id": 1, "name": "Magnus", "rating": 2839}
    yield {"id": 2, "name": "Pragg", "rating": 2758}

dlt.pipeline(destination="duckdb", dataset_name="chess").run(players())
```

Check out a super simple demo in **[Colab](https://colab.research.google.com/drive/1NfSB1DpwbbHX9_t5vlalBTf13utwpMGx?usp=sharing)** or a more advanced [Hugging Face demo with Marimo notebooks](https://molab.marimo.io/github/marimo-team/gallery-examples/blob/main/notebooks/external/dlthub-huggingface.py).

## Why dlt

**dlt** loads data from messy, often unstructured sources into well-structured, typed datasets. It's a **library, not a platform** — you `pip install` it into your existing code and keep your workflow and the other tools you already use. No black boxes: clean Pythonic interfaces, human-readable file formats, schemas you can inspect, no hidden side effects.

dlt and its docs are **built from the ground up for LLMs and coding agents**. Pair the typed, declarative primitives below with [dlthub.com/context](https://dlthub.com/context) and the [LLM-native workflow](https://dlthub.com/docs/dlt-ecosystem/llm-tooling/llm-native-workflow) to go from prompt to working pipeline — across [5000+ sources](https://dlthub.com/workspace) — often in a single shot.

## Extract from any source

**REST APIs** — describe the endpoints declaratively; filter, map, and flatten records right at the source ([docs](https://dlthub.com/docs/tutorial/rest-api)):

```python
from dlt.sources.rest_api import rest_api_source

source = rest_api_source({
    "client": {
        "base_url": "https://api.example.com/v1",
        "paginator": {"type": "cursor", "cursor_path": "next_cursor"},
    },
    "resources": [
        {
            "name": "guests",
            "endpoint": {"path": "events/guests"},
            "processing_steps": [
                {"filter": lambda r: r["approval_status"] == "approved"},
                {"map": lambda r: {**r, "email": r["email"].lower()}},
            ],
        },
    ],
})
```

**SQL databases** — reflect tables and types straight from the database ([docs](https://dlthub.com/docs/tutorial/sql-database)):

```python
from dlt.sources.sql_database import sql_database

source = sql_database("mysql+pymysql://user:pass@host/db")
```

**Files in any bucket** — list, then parse CSV / JSONL / Parquet from local disk, S3, GCS, or Azure ([docs](https://dlthub.com/docs/tutorial/filesystem)):

```python
from dlt.sources.filesystem import filesystem, read_csv_duckdb

source = (
    filesystem(bucket_url="s3://my-bucket/data", file_glob="*.csv")
    | read_csv_duckdb()
).with_name("events")
```

**DataFrames & Arrow** — pandas, Polars, and Arrow tables load directly; Arrow-backed frames move with zero copies:

```python
import dlt
import pandas as pd

df = pd.DataFrame({"event": ["dlt summit", "DuckCon"], "signups": [1240, 860]})
dlt.pipeline(destination="duckdb", dataset_name="events").run(df, table_name="events")
```

See [many more sources](https://dlthub.com/docs/dlt-ecosystem/verified-sources) in the ecosystem.

## Load to 20+ destinations — swap one string

The same resource runs anywhere. Change the `destination` string and dlt takes care of credentials, DDL in the target dialect, staging, and schema drift:

```python
pipeline = dlt.pipeline(
    pipeline_name="luma",
    destination="duckdb",       # → snowflake, bigquery, postgres, redshift, databricks,
    dataset_name="luma_data",   #   athena, clickhouse, motherduck, filesystem (S3/GCS/Azure),
)                               #   iceberg, delta, ... and custom reverse-ETL destinations
pipeline.run(source)
```

dlt handles the parts you'd rather not:

- **Credentials** → `secrets.toml` / env vars, injected automatically
- **DDL** → `CREATE TABLE` in the target's dialect
- **Type mapping** → source types converted to the destination's types
- **Staging** → S3 / GCS for warehouses that need it
- **Schema drift** → `ALTER TABLE` on the fly

Browse all [supported destinations](https://dlthub.com/docs/dlt-ecosystem/destinations/), or build a [custom one](https://dlthub.com/docs/dlt-ecosystem/destinations/destination).

## Declare intent with decorators

Decorators let you declare *what* you want — incremental loading, merge strategies, schema contracts, column hints — instead of hand-rolling it. Every knob can be overridden at runtime ([docs](https://dlthub.com/docs/general-usage/resource)):

```python
import dlt

@dlt.resource(
    primary_key="id",
    write_disposition="merge",                       # upsert on the primary key
    columns={"email": {"x-annotation-pii": True}},   # type and annotate columns
    schema_contract={"columns": "freeze"},           # reject unexpected columns
)
def events(
    updated_at=dlt.sources.incremental("updated_at"),  # load only new/changed rows
):
    yield from fetch_events(since=updated_at.last_value)


@dlt.source
def luma(api_key: str = dlt.secrets.value):
    return events(), guests()   # group one or more resources behind shared config/auth
```

[**Schema contracts**](https://dlthub.com/docs/general-usage/schema-contracts) enforce the shape at the gate, with three modes — `evolve` (accept and adapt the schema), `freeze` (reject the record), and `discard` (drop the offending row/column) — applied independently to `tables`, `columns`, and `data_type`. You also get [schema inference](https://dlthub.com/docs/general-usage/schema), [normalization of nested data](https://dlthub.com/docs/general-usage/schema/#data-normalizer), [incremental loading](https://dlthub.com/docs/general-usage/incremental-loading), and [secrets & config injection](https://dlthub.com/docs/general-usage/credentials) out of the box.

## Read your data back: the Dataset API

A pipeline is durable. Reconnect to one by name with `dlt.attach` and read any table back in the shape that fits your tool ([docs](https://dlthub.com/docs/general-usage/dataset-access/)):

```python
import dlt

pipeline = dlt.attach(pipeline_name="luma", destination="duckdb", dataset_name="luma_data")

dataset = pipeline.dataset()
dataset.tables               # ['events', 'guests', ...]

guests = dataset.guests      # a lazy dlt.Relation
guests.df()                  # pandas DataFrame
guests.arrow()               # pyarrow.Table (zero-copy)
guests.to_ibis()             # ibis expression — lazy, composable
```

## Transform with Ibis — Python in, SQL out

Lift any loaded table into an [Ibis](https://ibis-project.org/) expression, compose group-bys, joins, and window functions in Python, and let dlt compile it to SQL in the destination's dialect. Nothing runs until you ask for the result:

```python
import ibis

guests = pipeline.dataset().guests.to_ibis()

guests_by_event = (
    guests
    .group_by("event_id")
    .aggregate(n_guests=ibis._.api_id.count())
)

guests_by_event.to_pyarrow()   # compiles to SQL and runs on the destination
```

dlt also supports [Python and SQL data access](https://dlthub.com/docs/general-usage/dataset-access/), [transformations](https://dlthub.com/docs/dlt-ecosystem/transformations), [pipeline inspection](https://dlthub.com/docs/general-usage/dashboard), and [visualizing data in Marimo notebooks](https://dlthub.com/docs/general-usage/dataset-access/marimo).

## Documentation

For detailed usage and configuration, please refer to the [official documentation](https://dlthub.com/docs).

## Examples

You can find examples for various use cases in the [examples](docs/examples) folder, or in the [code examples section](https://dlthub.com/docs/examples) of our docs page.

## Adding as dependency

`dlt` follows the semantic versioning with the [`MAJOR.MINOR.PATCH`](https://peps.python.org/pep-0440/#semantic-versioning) pattern.

* `major` means breaking changes and removed deprecations
* `minor` new features, sometimes automatic migrations
* `patch` bug fixes

We suggest that you allow only `patch` level updates automatically using the [Compatible Release Specifier](https://packaging.python.org/en/latest/specifications/version-specifiers/#compatible-release). For example **dlt~=1.23.0** allows only versions **>=1.23.0** and less than **<1.24.0**

Please also see our [release notes](https://github.com/dlt-hub/dlt/releases) for notable changes between versions.

## Get Involved

The dlt project is quickly growing, and we're excited to have you join our community! Here's how you can get involved:

- **Connect with the Community**: Join other dlt users and contributors on our [Slack](https://dlthub.com/community)
- **Report issues and suggest features**: Please use the [GitHub Issues](https://github.com/dlt-hub/dlt/issues) to report bugs or suggest new features. Before creating a new issue, make sure to search the tracker for possible duplicates and add a comment if you find one.
- **Track progress of our work and our plans**: Please check out our [public Github project](https://github.com/orgs/dlt-hub/projects/9)
- **Improve documentation**: Help us enhance the dlt documentation.

## Contribute code
Please read [CONTRIBUTING](CONTRIBUTING.md) before you make a PR.

- 📣 **New destinations are unlikely to be merged** due to high maintenance cost (but we are happy to improve SQLAlchemy destination to handle more dialects)
- Significant changes require tests and docs and in many cases writing tests will be more laborious than writing code
- Bugfixes and improvements are welcome! You'll get help with writing tests and docs + a decent review.

## Sponsors


  <a href="https://blacksmith.sh/?utm_source=dlt&utm_medium=readme&utm_campaign=sponsorship" target="_blank">
    
  </a>


[Blacksmith](https://blacksmith.sh/?utm_source=dlt&utm_medium=readme&utm_campaign=sponsorship) is a drop-in replacement for GitHub-hosted runners that speed up our CI/CD pipelines by 2x and up to 75% cheaper. We're grateful to Blacksmith for sponsoring us with free CI/CD minutes--which helps us keep builds fast and our costs lower.

## License

`dlt` is released under the [Apache 2.0 License](LICENSE.txt).

---

## File: deploy/dlt/README.md

Example `Dockerfile` that installs `dlt` package on an alpine linux image. For actual pipeline deployment please refer to [deploy a pipeline walkthrough](https://dlthub.com/docs/walkthroughs/deploy-a-pipeline/deploy-with-github-actions)

---

## File: dlt/destinations/impl/bigquery/README.md

# Loader account setup

1. Create a new services account, add private key to it and download the `services.json` file.
2. Make sure the newly created account has access to BigQuery API.
3. You must add the following roles to the account above: `BigQuery Data Editor`, `BigQuey Job User` and `BigQuery Read Session User` (storage API)
4. IAM to add roles is here https://console.cloud.google.com/iam-admin/iam?project=chat-analytics-rasa-ci

---

## File: dlt/destinations/impl/mssql/README.md

# loader account setup

1. Create new database `CREATE DATABASE dlt_data`
2. Create new user, set password `CREATE USER loader WITH PASSWORD = 'loader';`
3. Set as database owner (we could set lower permission) `ALTER DATABASE dlt_data OWNER TO loader`

---

## File: dlt/destinations/impl/postgres/README.md

# loader account setup

1. Create new database `CREATE DATABASE dlt_data`
2. Create new user, set password `CREATE USER loader WITH PASSWORD 'loader';`
3. Set as database owner (we could set lower permission) `ALTER DATABASE dlt_data OWNER TO loader`

---

## File: dlt/destinations/impl/redshift/README.md

# Public Access setup
There's *Modify publicly accessible settings* in Actions of each Redshift cluster. Assign your IP there.

# Runtime optimization

https://www.intermix.io/blog/top-14-performance-tuning-techniques-for-amazon-redshift/

1. we should use separate work queue for loader user
2. they suggest to not use dist keys
3. data must be inserted in order of sortkey

# loader account setup

1. Create new database `CREATE DATABASE dlt_ci`
2. Create new user, set password
3. Set as database owner (we could set lower permission) `ALTER DATABASE dlt_ci OWNER TO loader`

# Public access setup for Serverless
Follow https://docs.aws.amazon.com/redshift/latest/mgmt/serverless-connecting.html `Connecting from the public subnet to the Amazon Redshift Serverless endpoint using Network Load Balancer`

that will use terraform template to create load balancer endpoint and assign public IP. The cost of the load balancer is ~16$/month + cost of IP

It seems that port 5439 is closed to the VPC on which serverless redshift created itself. In the cluster panel: Data Access : VPC security group add Inbound Rule to allow 5439 port from any subnet 0.0.0.0/0

---

## File: dlt/destinations/impl/weaviate/README.md

## Running Weaviate locally

Start Weaviate with Docker Compose:
```sh
docker compose -f tests/load/weaviate/docker-compose.yml up -d
```

Stop and clean up:
```sh
docker compose -f tests/load/weaviate/docker-compose.yml down -v --remove-orphans
```

This starts Weaviate with the contextionary vectorizer (no external APIs required). Add to `config.toml`:
```toml
[destination.weaviate]
connection_type = "local"
vectorizer = "text2vec-contextionary"
module_config = {text2vec-contextionary = {vectorizeClassName = false, vectorizePropertyName = true}}
```

For more details, see [Weaviate Local Quickstart](https://weaviate.io/developers/weaviate/quickstart/local).

---

## File: dlt/helpers/marimo/README.md

# How to add a marimo widget

This guide shows how to add the hypothetical `pipeline_browser` widget.
A marimo widget **is** a marimo notebook. We'll call this the *widget notebook*. 

## 1. Create a new marimo notebook under `dlt/helpers`

Create the notebook with 

```shell
marimo edit dlt/helpers/_pipeline_browser.py
``` 

The widget notebook file name is prefixed with `_` for convention 
and to avoid name collisions.

## 2. Edit your widget notebook

Edit the `_pipeline_browser.py` notebook via the marimo GUI until you're satisfied. 

Tips:
- Use a [setup cell](https://docs.marimo.io/guides/reusing_functions/#1-create-a-setup-cell) for your import and constants.
- Set a cell names to be able to [run cells directly](https://docs.marimo.io/api/cell/) in a unit test. This is done by setting a function name instead of the anonymous `_`
- If you want your widget to take input arguments, define these variables inside the setup
cell and set them to `None`. In downstream cells, you can use `mo.stop(VAR is None)` to hide
cells if the argument is missing

The next snippet shows the setup cell of a widget that takes the argument `pipeline_name`.
The cell that depends on it to instantiate a pipeline via `dlt.attach(pipeline_name)` starts
with `mo.stop(pipeline_name is None)`

```python
# dlt/helpers/marimo/_pipeline_browser.py

with app.setup:
    from typing import Any, cast
    from itertools import chain

    import marimo as mo

    import dlt
    from dlt.common.utils import without_none

    pipeline_name = None


@app.cell
def _():
    mo.stop(pipeline_name is None)
    pipeline = dlt.attach(pipeline_name)
    return (pipeline,)

# ...
```

## 3. Register the widget

To make the widget publicly available via `dlt`, you need to modify `dlt/helpers/marimo/__init__.py`.

The following is required:
1. import the `app` variable from the widget notebook `.py` file and give it an alias
2. add this alias to the `__all__` clause
3. create a function that instantiates the widget from the `app` argument
   and has the input parameters
4. add an `if/else` condition inside `render()` to render the widget

The following snippet shows what's required to render the `_pipeline_browser.py` widget

```python
import marimo
import mowidgets

from dlt.helpers.marimo._pipeline_browser import app as pipeline_browser

# pre-existing function
def render(app: marimo.App, *args, **kwargs):
    if not isinstance(app, marimo.App):
        raise ValueError("app must be an instance of marimo.App")

    if ...:
        ...
    # add new condition
    elif app is pipeline_browser:
        return pipeline_browser_widget(app, *args, **kwargs)
    else:
        raise ValueError("app must be either load_package_viewer or schema_viewer")

# ...

def pipeline_browser_widget(app, pipeline_name, *args, **kwargs):
    return mowidgets.widgetize(
        app,
        data_access=True,
        # this must match the input name in the notebook
        inputs={"pipeline_name": pipeline_name}
    )

__all__ = (
    "render",
    ...,
    # add the aliased `app` variable; not the newly created `_widget` function
    "pipeline_browser",
)
```

# 4. Trying the widget

Open a new marimo notebook with `marimo edit dev.py`. To use the widget, import
`render` and the app object from the `dlt.helpers.marimo`. Calling `render(app)`
will return a widget object. Calling `await` on it will render it.

Example snippet

```python
# dev.py
@app.cell
def _():
    import marimo as mo
    from dlt.helpers.marimo import render, pipeline_browser
    return pipeline_browser, render


@app.cell
async def _(pipeline_browser, render):
    # to display directly
    await render(pipeline_browser)
    return

# call `await` on `render()` to display the widget directly
@app.cell
async def _(pipeline_browser, render):
    await render(pipeline_browser)
    return

# or assign a variable to display elsewhere and access the widget's data
@app.cell
async def _(pipeline_browser, render):
    w = render(pipeline_browser)
    await w
    return (w,)


@app.cell
def _(w):
    w.data
    return
```

Alternatively, you can skip the render function and call the `_widget()` function
on the app object directly

```python
@app.cell
def _():
    import marimo as mo
    from dlt.helpers.marimo import pipeline_browser_widget, pipeline_browser
    return pipeline_browser, pipeline_browser_widget


@app.cell
async def _(pipeline_browser, pipeline_browser_widget):
    # to display directly
    await pipeline_browser_widget(pipeline_browser)
    return
```



## 5. unit tests

Add tests for cells with complex operations. You can retrieve a cell by name and give it inputs.

Source code for the widget
```python
# dlt/helpers/marimo/_pipeline_browser.py
@app.cell
def selector(base_path):
    pipelines = [p.name for p in pathlib.Path(base_path).expanduser().iterdir()]
    select_pipeline = mo.ui.dropdown(pipelines, value="pokemon")
    select_pipeline
    return (select_pipeline,)
```

Unit test
```python
# tests/helpers/marimo/test_pipeline_browser.py
# import the named cell directly from the widget notebook
from dlt.helpers.marimo._pipeline_browser import selector

def test_cell_selector():
    base_path = ...

    # outputs is the HTML to be displayed
    # definitions is a dictionary of values returned
    outputs, definitions = selector.run(base_path=base_path)

    assert definitions["select_pipeline"] == ... 
```

## 6. Integration tests
You can run the full widget notebook to see if it can run "end-to-end"
for a set of definitions. These input definitions can override nodes of the
DAG, which allows to test various states of the application.

```python
# tests/helpers/marimo/test_pipeline_browser.py
import pytest
# import the aliased app
from dlt.helpers.marimo import pipeline_browser

@pytest.mark.parametrize("input1", [...])
@pytest.mark.parametrize("input2", [...])
def test_pipeline_browser(input1, input2) -> None:
    input_definitions = {
        "input1": input1,
        "input2": input2,
    }
    outputs, definitions = pipeline_browser.run(defs=input_definitions)

    assert ...
```

---

## File: docs/examples/archive/README.md

# Writing Source Extractors
`dlt` sources are iterators or lists and writing them does not require any additional knowledge beyond basic python. `dlt` sources are also pythonic in nature: they are simple, can be chained, pipelined and composed like any other python iterator or a sequence.


# Examples

1. `quickstart` loads a nested json document into `duckdb` and then queries it with built in `sql_client` demonstrating the parent-child table joins.

1. `sql_query` source and `read_table` example. This source iterates over any `SELECT` statement made against database system supported by `SqlAlchemy`. The example connects to Redshift and iterates a table containing Ethereum transactions. Shows the inferred schema (which nicely preserves typing). Mind that our source is a one-liner :)

1. `rasa` example and `rasa_tracker_store` source extracts rasa tracker store events to a set of inferred tables. It shows a few common patterns
- shows how to **pipeline resources**: it depends on a "head" resource that reads base data (ie. events from kafka/postgres/file). the dependent resource is called `transformer`
- it shows how to write **stream resource** which creates table schemas and sends data to those tables depending on the event type
- it stores `last_timestamp_value` in the state

1. `singer_tap`, `stdout` and `singer_tap_example` is fully functional wrapper for any singer/meltano source
- clones the desired tap, installs it and runs it in a virtual env
- passes the catalog and config files
- like rasa it is a **transformer** (on stdio pipe) and `stream` resource
- it stores singer state in `dlt` state

1. `singer_tap_jsonl_example` like the above but instead of process pipe it reads singer messages from file. it creates a huge hubspot schema.

1. `google_sheets` a source that returns values from specified sheet. The example takes a sheet, infers a schema, loads it to BigQuery/Redshift and displays inferred schema. it uses the `secrets.toml` to manage credentials and is an example of one-liner pipeline

1. `chess` an example of a pipeline project with its own config and credential files. it is also an example of how transformers are connected to resources and resource selection. **it should be run from examples/chess` folder**. It also shows: **how to use retry decorator** and **how to run resources/transformers in parallel with a decorator**

2. `chess/chess_dbt.py`: an example of a `dbt` transformations package working with a dataset loaded by `dlt`. The package is incrementally processing the loaded data following the new loaded packages stored in `_dlt_loads` table at the end of every pipeline run. Note the automatic usage of isolated virtual environment to run dbt and sharing of the credentials.

1. `run_dbt_jaffle` runs dbt's jaffle shop example taken directly from the github repo and queries the results with `sql_client`. `duckdb` database is used to load and transform the data. The database `write` access is passed from `dlt` to `dbt` and back.

Not yet ported:

1. `discord_iterator` an example that load example discord data (messages, channels) into warehouse from supplied files. Shows several auxiliary pipeline functions and an example of pipelining iterators (with `map` function). You can also see that produced schema is quite complicated due to several layers of nesting.

2. `ethereum` source shows that you can build highly scalable, parallel and robust sources as simple iterators.

---

## File: docs/examples/CONTRIBUTING.md

# How to contribute your example

Note: All paths in this guide are relative to the `dlt` repository directory.

## Add snippet

- Go to `docs/examples/`.
- Copy the template in `./_template/..`.
- Make sure the folder and your examples script have the same name
- Update the doc string which will compromise the generated markdown file, check the other examples how it is done
- If your example requires any secrets, add the vars to the example.secrects.toml but do not enter the values.
- Add your example code, make sure you have a `if __name__ = "__main__"` clause in which you run the example script, this will be used for testing
- You should add one or two assertions after running your example

## Testing
- You can test your example simply by running your example script from your example folder. On CI a test will be automatically generated.

## Checking your generated markdown

The command `npm start`  starts a local development server and opens up a browser window.

- To install npm read [README](../website/README.md).
- You should your example be automatically added to the examples section in the local version of the docs. Check the rendered output and see wether it looks the way you intended.

## Add ENV variables

If you use any secrets for the code snippets, e.g. Zendesk requires credentials. Please talk to us. We will add them to our google secrets vault.

## Add dependencies

If your example requires any additional dependency, then you can add it

- To `pyproject.toml` in the `[dependency-groups]` section in the `docs` group.
- Do not forget to update your `uv.lock` file with `uv sync` command and commit.