File: README.md Copy Chapter

**data load tool (dlt) — the open-source Python library that automates all your tedious data loading tasks**

🚀 Join our thriving community of likeminded developers and build the future together!

Installation Copy Chapter

dlt supports Python 3.10 through Python 3.14. Note that some optional extras are not yet available for Python 3.14, so support for this version is considered experimental.

pip install dlt

Add the extras you need for your sources and destinations, for example:

pip install "dlt[duckdb]"        # local DuckDB destination
pip install "dlt[bigquery]"      # or snowflake, postgres, redshift, databricks, athena, ...
pip install "dlt[s3]"            # or gs, az for cloud filesystems
pip install "dlt[sql_database]"  # read from any SQL database
pip install "dlt[hub]"           # data quality, transformations, and AI (see below)

Prefer uv? uv add "dlt[duckdb]".

Quick Start Copy Chapter

Describe an API declaratively and load it into DuckDB — dlt handles requests, pagination, schema inference, and typing for you:

import dlt
from dlt.sources.rest_api import rest_api_source

# 1. Describe the API declaratively
source = rest_api_source({
    "client": {"base_url": "https://pokeapi.co/api/v2/"},
    "resources": [
        {"name": "pokemon", "endpoint": {"path": "pokemon", "params": {"limit": 1000}}},
    ],
})

# 2. Point a pipeline at any destination
pipeline = dlt.pipeline(
    pipeline_name="pokemon",
    destination="duckdb",
    dataset_name="pokemon_data",
)

# 3. Extract, normalize, and load
print(pipeline.run(source))

# 4. ...and read it straight back as a DataFrame
print(pipeline.dataset().pokemon.df())

...or load any Python iterable — a resource is just a generator, and dlt infers the schema, types the columns, and writes the table:

import dlt

@dlt.resource(table_name="players", primary_key="id", write_disposition="merge")
def players():
    yield {"id": 1, "name": "Magnus", "rating": 2839}
    yield {"id": 2, "name": "Pragg", "rating": 2758}

dlt.pipeline(destination="duckdb", dataset_name="chess").run(players())

Check out a super simple demo in Colab or a more advanced Hugging Face demo with Marimo notebooks.

Why dlt Copy Chapter

dlt loads data from messy, often unstructured sources into well-structured, typed datasets. It's a library, not a platform — you pip install it into your existing code and keep your workflow and the other tools you already use. No black boxes: clean Pythonic interfaces, human-readable file formats, schemas you can inspect, no hidden side effects.

dlt and its docs are built from the ground up for LLMs and coding agents. Pair the typed, declarative primitives below with dlthub.com/context and the LLM-native workflow to go from prompt to working pipeline — across 5000+ sources — often in a single shot.

Extract from any source Copy Chapter

REST APIs — describe the endpoints declaratively; filter, map, and flatten records right at the source (docs):

from dlt.sources.rest_api import rest_api_source

source = rest_api_source({
    "client": {
        "base_url": "https://api.example.com/v1",
        "paginator": {"type": "cursor", "cursor_path": "next_cursor"},
    },
    "resources": [
        {
            "name": "guests",
            "endpoint": {"path": "events/guests"},
            "processing_steps": [
                {"filter": lambda r: r["approval_status"] == "approved"},
                {"map": lambda r: {**r, "email": r["email"].lower()}},
            ],
        },
    ],
})

SQL databases — reflect tables and types straight from the database (docs):

from dlt.sources.sql_database import sql_database

source = sql_database("mysql+pymysql://user:pass@host/db")

Files in any bucket — list, then parse CSV / JSONL / Parquet from local disk, S3, GCS, or Azure (docs):

from dlt.sources.filesystem import filesystem, read_csv_duckdb

source = (
    filesystem(bucket_url="s3://my-bucket/data", file_glob="*.csv")
    | read_csv_duckdb()
).with_name("events")

DataFrames & Arrow — pandas, Polars, and Arrow tables load directly; Arrow-backed frames move with zero copies:

import dlt
import pandas as pd

df = pd.DataFrame({"event": ["dlt summit", "DuckCon"], "signups": [1240, 860]})
dlt.pipeline(destination="duckdb", dataset_name="events").run(df, table_name="events")

See many more sources in the ecosystem.

Load to 20+ destinations — swap one string Copy Chapter

The same resource runs anywhere. Change the destination string and dlt takes care of credentials, DDL in the target dialect, staging, and schema drift:

pipeline = dlt.pipeline(
    pipeline_name="luma",
    destination="duckdb",       # → snowflake, bigquery, postgres, redshift, databricks,
    dataset_name="luma_data",   #   athena, clickhouse, motherduck, filesystem (S3/GCS/Azure),
)                               #   iceberg, delta, ... and custom reverse-ETL destinations
pipeline.run(source)

dlt handles the parts you'd rather not:

• Credentials → secrets.toml / env vars, injected automatically
• DDL → CREATE TABLE in the target's dialect
• Type mapping → source types converted to the destination's types
• Staging → S3 / GCS for warehouses that need it
• Schema drift → ALTER TABLE on the fly

Browse all supported destinations, or build a custom one.

Declare intent with decorators Copy Chapter

Decorators let you declare what you want — incremental loading, merge strategies, schema contracts, column hints — instead of hand-rolling it. Every knob can be overridden at runtime (docs):

import dlt

@dlt.resource(
    primary_key="id",
    write_disposition="merge",                       # upsert on the primary key
    columns={"email": {"x-annotation-pii": True}},   # type and annotate columns
    schema_contract={"columns": "freeze"},           # reject unexpected columns
)
def events(
    updated_at=dlt.sources.incremental("updated_at"),  # load only new/changed rows
):
    yield from fetch_events(since=updated_at.last_value)


@dlt.source
def luma(api_key: str = dlt.secrets.value):
    return events(), guests()   # group one or more resources behind shared config/auth

Schema contracts enforce the shape at the gate, with three modes — evolve (accept and adapt the schema), freeze (reject the record), and discard (drop the offending row/column) — applied independently to tables, columns, and data_type. You also get schema inference, normalization of nested data, incremental loading, and secrets & config injection out of the box.

Read your data back: the Dataset API Copy Chapter

A pipeline is durable. Reconnect to one by name with dlt.attach and read any table back in the shape that fits your tool (docs):

import dlt

pipeline = dlt.attach(pipeline_name="luma", destination="duckdb", dataset_name="luma_data")

dataset = pipeline.dataset()
dataset.tables               # ['events', 'guests', ...]

guests = dataset.guests      # a lazy dlt.Relation
guests.df()                  # pandas DataFrame
guests.arrow()               # pyarrow.Table (zero-copy)
guests.to_ibis()             # ibis expression — lazy, composable

Transform with Ibis — Python in, SQL out Copy Chapter

Lift any loaded table into an Ibis expression, compose group-bys, joins, and window functions in Python, and let dlt compile it to SQL in the destination's dialect. Nothing runs until you ask for the result:

import ibis

guests = pipeline.dataset().guests.to_ibis()

guests_by_event = (
    guests
    .group_by("event_id")
    .aggregate(n_guests=ibis._.api_id.count())
)

guests_by_event.to_pyarrow()   # compiles to SQL and runs on the destination

dlt also supports Python and SQL data access, transformations, pipeline inspection, and visualizing data in Marimo notebooks.

Documentation Copy Chapter

For detailed usage and configuration, please refer to the official documentation.

Examples Copy Chapter

You can find examples for various use cases in the examples folder, or in the code examples section of our docs page.

Adding as dependency Copy Chapter

dlt follows the semantic versioning with the MAJOR.MINOR.PATCH pattern.

• major means breaking changes and removed deprecations
• minor new features, sometimes automatic migrations
• patch bug fixes

We suggest that you allow only patch level updates automatically using the Compatible Release Specifier. For example dlt~=1.23.0 allows only versions >=1.23.0 and less than <1.24.0

Please also see our release notes for notable changes between versions.

Get Involved Copy Chapter

The dlt project is quickly growing, and we're excited to have you join our community! Here's how you can get involved:

• Connect with the Community: Join other dlt users and contributors on our Slack
• Report issues and suggest features: Please use the GitHub Issues to report bugs or suggest new features. Before creating a new issue, make sure to search the tracker for possible duplicates and add a comment if you find one.
• Track progress of our work and our plans: Please check out our public Github project
• Improve documentation: Help us enhance the dlt documentation.

Contribute code Copy Chapter

Please read CONTRIBUTING before you make a PR.

• 📣 New destinations are unlikely to be merged due to high maintenance cost (but we are happy to improve SQLAlchemy destination to handle more dialects)
• Significant changes require tests and docs and in many cases writing tests will be more laborious than writing code
• Bugfixes and improvements are welcome! You'll get help with writing tests and docs + a decent review.

Sponsors Copy Chapter

Blacksmith is a drop-in replacement for GitHub-hosted runners that speed up our CI/CD pipelines by 2x and up to 75% cheaper. We're grateful to Blacksmith for sponsoring us with free CI/CD minutes--which helps us keep builds fast and our costs lower.

License Copy Chapter

dlt is released under the Apache 2.0 License.

File: deploy/dlt/README.md Copy Chapter

Example Dockerfile that installs dlt package on an alpine linux image. For actual pipeline deployment please refer to deploy a pipeline walkthrough

File: dlt/destinations/impl/bigquery/README.md Copy Chapter

Loader account setup

1. Create a new services account, add private key to it and download the services.json file.
2. Make sure the newly created account has access to BigQuery API.
3. You must add the following roles to the account above: BigQuery Data Editor, BigQuey Job User and BigQuery Read Session User (storage API)
4. IAM to add roles is here https://console.cloud.google.com/iam-admin/iam?project=chat-analytics-rasa-ci

File: dlt/destinations/impl/mssql/README.md Copy Chapter

loader account setup

1. Create new database CREATE DATABASE dlt_data
2. Create new user, set password CREATE USER loader WITH PASSWORD = 'loader';
3. Set as database owner (we could set lower permission) ALTER DATABASE dlt_data OWNER TO loader

File: dlt/destinations/impl/postgres/README.md Copy Chapter

loader account setup

1. Create new database CREATE DATABASE dlt_data
2. Create new user, set password CREATE USER loader WITH PASSWORD 'loader';
3. Set as database owner (we could set lower permission) ALTER DATABASE dlt_data OWNER TO loader

File: dlt/destinations/impl/redshift/README.md Copy Chapter

Public Access setup

There's Modify publicly accessible settings in Actions of each Redshift cluster. Assign your IP there.

Runtime optimization

https://www.intermix.io/blog/top-14-performance-tuning-techniques-for-amazon-redshift/

1. we should use separate work queue for loader user
2. they suggest to not use dist keys
3. data must be inserted in order of sortkey

loader account setup

1. Create new database CREATE DATABASE dlt_ci
2. Create new user, set password
3. Set as database owner (we could set lower permission) ALTER DATABASE dlt_ci OWNER TO loader

Public access setup for Serverless

Follow https://docs.aws.amazon.com/redshift/latest/mgmt/serverless-connecting.html Connecting from the public subnet to the Amazon Redshift Serverless endpoint using Network Load Balancer

that will use terraform template to create load balancer endpoint and assign public IP. The cost of the load balancer is ~16$/month + cost of IP

It seems that port 5439 is closed to the VPC on which serverless redshift created itself. In the cluster panel: Data Access : VPC security group add Inbound Rule to allow 5439 port from any subnet 0.0.0.0/0

File: dlt/destinations/impl/weaviate/README.md Copy Chapter

Running Weaviate locally Copy Chapter

Start Weaviate with Docker Compose:

docker compose -f tests/load/weaviate/docker-compose.yml up -d

Stop and clean up:

docker compose -f tests/load/weaviate/docker-compose.yml down -v --remove-orphans

This starts Weaviate with the contextionary vectorizer (no external APIs required). Add to config.toml:

[destination.weaviate]
connection_type = "local"
vectorizer = "text2vec-contextionary"
module_config = {text2vec-contextionary = {vectorizeClassName = false, vectorizePropertyName = true}}

For more details, see Weaviate Local Quickstart.

File: dlt/helpers/marimo/README.md Copy Chapter

How to add a marimo widget

This guide shows how to add the hypothetical pipeline_browser widget.
A marimo widget is a marimo notebook. We'll call this the widget notebook.

1. Create a new marimo notebook under `dlt/helpers` Copy Chapter

Create the notebook with

marimo edit dlt/helpers/_pipeline_browser.py

The widget notebook file name is prefixed with _ for convention
and to avoid name collisions.

2. Edit your widget notebook Copy Chapter

Edit the _pipeline_browser.py notebook via the marimo GUI until you're satisfied.

Tips:

• Use a setup cell for your import and constants.
• Set a cell names to be able to run cells directly in a unit test. This is done by setting a function name instead of the anonymous _
• If you want your widget to take input arguments, define these variables inside the setup
  cell and set them to None. In downstream cells, you can use mo.stop(VAR is None) to hide
  cells if the argument is missing

The next snippet shows the setup cell of a widget that takes the argument pipeline_name.
The cell that depends on it to instantiate a pipeline via dlt.attach(pipeline_name) starts
with mo.stop(pipeline_name is None)

# dlt/helpers/marimo/_pipeline_browser.py

with app.setup:
    from typing import Any, cast
    from itertools import chain

    import marimo as mo

    import dlt
    from dlt.common.utils import without_none

    pipeline_name = None


@app.cell
def _():
    mo.stop(pipeline_name is None)
    pipeline = dlt.attach(pipeline_name)
    return (pipeline,)

# ...

3. Register the widget Copy Chapter

To make the widget publicly available via dlt, you need to modify dlt/helpers/marimo/__init__.py.

The following is required:

1. import the app variable from the widget notebook .py file and give it an alias
2. add this alias to the __all__ clause
3. create a function that instantiates the widget from the app argument
   and has the input parameters
4. add an if/else condition inside render() to render the widget

The following snippet shows what's required to render the _pipeline_browser.py widget

import marimo
import mowidgets

from dlt.helpers.marimo._pipeline_browser import app as pipeline_browser

# pre-existing function
def render(app: marimo.App, *args, **kwargs):
    if not isinstance(app, marimo.App):
        raise ValueError("app must be an instance of marimo.App")

    if ...:
        ...
    # add new condition
    elif app is pipeline_browser:
        return pipeline_browser_widget(app, *args, **kwargs)
    else:
        raise ValueError("app must be either load_package_viewer or schema_viewer")

# ...

def pipeline_browser_widget(app, pipeline_name, *args, **kwargs):
    return mowidgets.widgetize(
        app,
        data_access=True,
        # this must match the input name in the notebook
        inputs={"pipeline_name": pipeline_name}
    )

__all__ = (
    "render",
    ...,
    # add the aliased `app` variable; not the newly created `_widget` function
    "pipeline_browser",
)

4. Trying the widget

Open a new marimo notebook with marimo edit dev.py. To use the widget, import
render and the app object from the dlt.helpers.marimo. Calling render(app)
will return a widget object. Calling await on it will render it.

Example snippet

# dev.py
@app.cell
def _():
    import marimo as mo
    from dlt.helpers.marimo import render, pipeline_browser
    return pipeline_browser, render


@app.cell
async def _(pipeline_browser, render):
    # to display directly
    await render(pipeline_browser)
    return

# call `await` on `render()` to display the widget directly
@app.cell
async def _(pipeline_browser, render):
    await render(pipeline_browser)
    return

# or assign a variable to display elsewhere and access the widget's data
@app.cell
async def _(pipeline_browser, render):
    w = render(pipeline_browser)
    await w
    return (w,)


@app.cell
def _(w):
    w.data
    return

Alternatively, you can skip the render function and call the _widget() function
on the app object directly

@app.cell
def _():
    import marimo as mo
    from dlt.helpers.marimo import pipeline_browser_widget, pipeline_browser
    return pipeline_browser, pipeline_browser_widget


@app.cell
async def _(pipeline_browser, pipeline_browser_widget):
    # to display directly
    await pipeline_browser_widget(pipeline_browser)
    return

5. unit tests Copy Chapter

Add tests for cells with complex operations. You can retrieve a cell by name and give it inputs.

Source code for the widget

# dlt/helpers/marimo/_pipeline_browser.py
@app.cell
def selector(base_path):
    pipelines = [p.name for p in pathlib.Path(base_path).expanduser().iterdir()]
    select_pipeline = mo.ui.dropdown(pipelines, value="pokemon")
    select_pipeline
    return (select_pipeline,)

Unit test

# tests/helpers/marimo/test_pipeline_browser.py
# import the named cell directly from the widget notebook
from dlt.helpers.marimo._pipeline_browser import selector

def test_cell_selector():
    base_path = ...

    # outputs is the HTML to be displayed
    # definitions is a dictionary of values returned
    outputs, definitions = selector.run(base_path=base_path)

    assert definitions["select_pipeline"] == ...

6. Integration tests Copy Chapter

You can run the full widget notebook to see if it can run "end-to-end"
for a set of definitions. These input definitions can override nodes of the
DAG, which allows to test various states of the application.

# tests/helpers/marimo/test_pipeline_browser.py
import pytest
# import the aliased app
from dlt.helpers.marimo import pipeline_browser

@pytest.mark.parametrize("input1", [...])
@pytest.mark.parametrize("input2", [...])
def test_pipeline_browser(input1, input2) -> None:
    input_definitions = {
        "input1": input1,
        "input2": input2,
    }
    outputs, definitions = pipeline_browser.run(defs=input_definitions)

    assert ...

File: docs/examples/archive/README.md Copy Chapter

Writing Source Extractors

dlt sources are iterators or lists and writing them does not require any additional knowledge beyond basic python. dlt sources are also pythonic in nature: they are simple, can be chained, pipelined and composed like any other python iterator or a sequence.

Examples

1. quickstart loads a nested json document into duckdb and then queries it with built in sql_client demonstrating the parent-child table joins.

2. sql_query source and read_table example. This source iterates over any SELECT statement made against database system supported by SqlAlchemy. The example connects to Redshift and iterates a table containing Ethereum transactions. Shows the inferred schema (which nicely preserves typing). Mind that our source is a one-liner :)

3. rasa example and rasa_tracker_store source extracts rasa tracker store events to a set of inferred tables. It shows a few common patterns

• shows how to pipeline resources: it depends on a "head" resource that reads base data (ie. events from kafka/postgres/file). the dependent resource is called transformer
• it shows how to write stream resource which creates table schemas and sends data to those tables depending on the event type
• it stores last_timestamp_value in the state

1. singer_tap, stdout and singer_tap_example is fully functional wrapper for any singer/meltano source

• clones the desired tap, installs it and runs it in a virtual env
• passes the catalog and config files
• like rasa it is a transformer (on stdio pipe) and stream resource
• it stores singer state in dlt state

1. singer_tap_jsonl_example like the above but instead of process pipe it reads singer messages from file. it creates a huge hubspot schema.

2. google_sheets a source that returns values from specified sheet. The example takes a sheet, infers a schema, loads it to BigQuery/Redshift and displays inferred schema. it uses the secrets.toml to manage credentials and is an example of one-liner pipeline

3. chess an example of a pipeline project with its own config and credential files. it is also an example of how transformers are connected to resources and resource selection. it should be run from examples/chess` folder. It also shows: how to use retry decorator and how to run resources/transformers in parallel with a decorator

4. chess/chess_dbt.py: an example of a dbt transformations package working with a dataset loaded by dlt. The package is incrementally processing the loaded data following the new loaded packages stored in _dlt_loads table at the end of every pipeline run. Note the automatic usage of isolated virtual environment to run dbt and sharing of the credentials.

5. run_dbt_jaffle runs dbt's jaffle shop example taken directly from the github repo and queries the results with sql_client. duckdb database is used to load and transform the data. The database write access is passed from dlt to dbt and back.

Not yet ported:

1. discord_iterator an example that load example discord data (messages, channels) into warehouse from supplied files. Shows several auxiliary pipeline functions and an example of pipelining iterators (with map function). You can also see that produced schema is quite complicated due to several layers of nesting.

2. ethereum source shows that you can build highly scalable, parallel and robust sources as simple iterators.

File: docs/examples/CONTRIBUTING.md Copy Chapter

How to contribute your example

Note: All paths in this guide are relative to the dlt repository directory.

Add snippet Copy Chapter

• Go to docs/examples/.
• Copy the template in ./_template/...
• Make sure the folder and your examples script have the same name
• Update the doc string which will compromise the generated markdown file, check the other examples how it is done
• If your example requires any secrets, add the vars to the example.secrects.toml but do not enter the values.
• Add your example code, make sure you have a if __name__ = "__main__" clause in which you run the example script, this will be used for testing
• You should add one or two assertions after running your example

Testing Copy Chapter

• You can test your example simply by running your example script from your example folder. On CI a test will be automatically generated.

Checking your generated markdown Copy Chapter

The command npm start starts a local development server and opens up a browser window.

• To install npm read README.
• You should your example be automatically added to the examples section in the local version of the docs. Check the rendered output and see wether it looks the way you intended.

Add ENV variables Copy Chapter

If you use any secrets for the code snippets, e.g. Zendesk requires credentials. Please talk to us. We will add them to our google secrets vault.

Add dependencies Copy Chapter

If your example requires any additional dependency, then you can add it

• To pyproject.toml in the [dependency-groups] section in the docs group.
• Do not forget to update your uv.lock file with uv sync command and commit.