## 1. Project Overview & Quickstart (pgdogdev/pgdog) ## File: README.md [](https://github.com/levkk/pgdog/actions/workflows/ci.yml) PgDog is a proxy for scaling PostgreSQL. It supports connection pooling, load balancing queries and sharding entire databases. Written in Rust, PgDog is fast, secure and can manage thousands of connections on commodity hardware. ## Documentation 📘 PgDog documentation can be **[found here](https://docs.pgdog.dev/)**. Any questions? Chat with us on **[Discord](https://discord.com/invite/CcBZkjSJdd)**. ## Quick start ### Kubernetes Helm chart is **[here](https://github.com/pgdogdev/helm)**. To install it, run: ```bash helm repo add pgdogdev https://helm.pgdog.dev helm install pgdog pgdogdev/pgdog ``` ### AWS If you're using AWS RDS, you can deploy PgDog using one of two supported methods: 1. [Helm chart](https://github.com/pgdogdev/helm) with [EKS](https://aws.amazon.com/eks/), or a self-hosted Kubernetes cluster 2. [Terraform module](https://github.com/pgdogdev/pgdog-ecs-terraform) to deploy PgDog on [ECS](https://aws.amazon.com/ecs/) ### Try in Docker You can try PgDog quickly using Docker. Install [Docker Compose](https://docs.docker.com/compose/) and run: ``` docker-compose up ``` Once started, you can connect to PgDog with psql or any other PostgreSQL client: ``` PGPASSWORD=postgres psql -h 127.0.0.1 -p 6432 -U postgres ``` The demo comes with 3 shards and 2 sharded tables: ```sql INSERT INTO users (id, email) VALUES (1, 'admin@acme.com'); INSERT INTO payments (id, user_id, amount) VALUES (1, 1, 100.0); SELECT * FROM users WHERE id = 1; SELECT * FROM payments WHERE user_id = 1; ``` ## Features 📘 **[Configuration](https://docs.pgdog.dev/configuration/)** All PgDog features are configurable and can be turned on and off. PgDog requires 2 configuration files to operate: 1. `pgdog.toml`: hosts, sharding configuration, and other settings 2. `users.toml`: usernames and passwords ### Example Most options have reasonable defaults, so a basic configuration for a single user and database running on the same machine is pretty short: **`pgdog.toml`** ```toml [general] port = 6432 default_pool_size = 10 [[databases]] name = "pgdog" host = "127.0.0.1" ``` **`users.toml`** ```toml [[users]] name = "alice" database = "pgdog" password = "hunter2" ``` If a database in `pgdog.toml` doesn't have a user in `users.toml`, the connection pool for that database will not be created and users won't be able to connect. If you'd like to try it out locally, create the database and user like so: ```sql CREATE DATABASE pgdog; CREATE USER pgdog PASSWORD 'pgdog' LOGIN; ``` ### Transaction pooling 📘 **[Transactions](https://docs.pgdog.dev/features/transaction-mode)** Like PgBouncer, PgDog supports transaction (and session) pooling, allowing thousands of clients to use just a few PostgreSQL server connections. Unlike PgBouncer, PgDog can parse and handle `SET` statements and startup options, ensuring session state is set correctly when sharing server connections between clients with different parameters. PgDog also has more advanced connection recovery options, like automatic abandoned transaction rollbacks and connection re-synchronization to avoid churning server connections during an application crash. ### Load balancer 📘 **[Load balancer](https://docs.pgdog.dev/features/load-balancer/)** PgDog is an application layer (OSI Level 7) load balancer for PostgreSQL. It understands the Postgres protocol, can proxy multiple replicas (and primary) and distributes transactions evenly between databases. The load balancer supports 3 strategies: round robin, random and least active connections. **Example** The load balancer is enabled automatically when a database has more than one host: ```toml [[databases]] name = "prod" host = "10.0.0.1" role = "primary" [[databases]] name = "prod" host = "10.0.0.2" role = "replica" ``` #### Health checks 📘 **[Healthchecks](https://docs.pgdog.dev/features/load-balancer/healthchecks/)** PgDog maintains a real-time list of healthy hosts. When a database fails a health check, it's removed from the active rotation and queries are re-routed to other replicas. This works like an HTTP load balancer, except it's for your database. Health checks maximize database availability and protect against bad network connections, temporary hardware failures or misconfiguration. #### Single endpoint 📘 **[Single endpoint](https://docs.pgdog.dev/features/load-balancer/#single-endpoint)** PgDog uses [`pg_raw_parse`](https://github.com/pgdogdev/pg_raw_parse), which includes the PostgreSQL native parser. By parsing queries, PgDog can detect writes (e.g. `INSERT`, `UPDATE`, `CREATE TABLE`, etc.) and send them to the primary, leaving the replicas to serve reads (`SELECT`). This allows applications to connect to the same PgDog deployment for both reads and writes. ##### Transactions 📘 **[Load balancer & transactions](https://docs.pgdog.dev/features/load-balancer/transactions/)** Transactions can execute multiple statements, so in a primary & replica configuration, PgDog routes them to the primary. Clients can indicate a transaction is read-only, in which case PgDog will send it to a replica: ```sql BEGIN READ ONLY; -- This goes to a replica. SELECT * FROM users LIMIT 1; COMMIT; ``` #### Failover 📘 **[Failover](https://docs.pgdog.dev/features/load-balancer/replication-failover/)** PgDog monitors Postgres replication state and can automatically redirect writes to a different database if a replica is promoted. This doesn't replace tools like Patroni that actually orchestrate failovers. You can use PgDog alongside Patroni (or AWS RDS or other managed Postgres host), to gracefully failover live traffic. **Example** To enable failover, set all database `role` attributes to `auto` and enable replication monitoring (`lsn_check_delay` setting): ```toml [general] lsn_check_delay = 0 [[databases]] name = "prod" host = "10.0.0.1" role = "auto" [[databases]] name = "prod" host = "10.0.0.2" role = "auto" ``` ### Authentication 📘 **[Authentication](https://docs.pgdog.dev/features/authentication/)** PgDog supports five authentication methods: 1. Password-based 2. AWS RDS IAM 3. Azure Workload Identity 4. HashiCorp Vault dynamic credentials 5. HashiCorp Vault static role credentials #### Password-based authentication Password-based authentication allows for clients to authenticate to PgDog and for PgDog to authenticate to PostgreSQL. It currently supports the following password hashing algorithms: - SCRAM-SHA-256 - MD5 - Plain #### RDS IAM backend authentication PgDog can keep client-to-PgDog authentication unchanged while using AWS RDS IAM tokens for PgDog-to-PostgreSQL authentication on a per-user basis. **Example** ```toml [[users]] name = "alice" database = "pgdog" password = "client-password" server_auth = "rds_iam" # Optional; PgDog infers region from *.region.rds.amazonaws.com(.cn) hostnames when omitted. # server_iam_region = "us-east-1" ``` When any user has `server_auth = "rds_iam"`, the following settings must be configured as well: - `tls_verify` must **not** be `"disabled"`. - `passthrough_auth` must be `"disabled"`. #### Azure Workload Identity authentication PgDog can also use Azure Workload Identity for PgDog-to-PostgreSQL authentication, while keeping client-to-PgDog authentication unchanged. This is configured on a per-user basis, similarly to RDS IAM: **Example** ```toml [[users]] name = "alice" database = "pgdog" password = "client-password" server_auth = "azure_workload_identity" ``` When any user has `server_auth = "azure_workload_identity"`, the following settings must be configured as well: - `tls_verify` must **not** be `"disabled"`. - `passthrough_auth` must be `"disabled"`. #### HashiCorp Vault dynamic role authentication PgDog can fetch dynamic database credentials (username and password) from HashiCorp Vault's database secrets engine, while keeping client-to-PgDog authentication unchanged. Credentials are cached and rotated automatically after a configured percentage of the Vault lease has elapsed. **Example** In `users.toml`: ```toml [[users]] name = "alice" database = "pgdog" password = "client-password" server_auth = "vault_dynamic" server_vault_path = "database/creds/pgdog" # Refresh credentials after 80% of the lease has elapsed (default). # vault_refresh_percent = 80 ``` In `pgdog.toml`: ```toml [vault] url = "https://vault.internal:8200" auth_method = "kubernetes" # or "approle" kubernetes_role = "pgdog" ``` PgDog logs into Vault with Kubernetes auth (using the pod's service account JWT) or AppRole (`approle_role_id` plus `approle_secret_id_file` or the `VAULT_SECRET_ID` environment variable). When any user has `server_auth = "vault_dynamic"` or `"vault_static"`, the following settings must be configured as well: - `tls_verify` must **not** be `"disabled"`. - `passthrough_auth` must be `"disabled"`. #### HashiCorp Vault static role authentication Unlike dynamic credentials, a Vault static database role has a fixed username and only its password rotates, on a schedule Vault manages. PgDog supports two independent uses of a static role, they don't need to point at the same role, and each has its own username setting: - `vault_path`: verify the password a client sends to PgDog against Vault's current password for the role, instead of a statically configured password. - `server_auth = "vault_static"` with `server_vault_path`: use the role's Vault-managed password for PgDog-to-PostgreSQL connections. Unlike `vault_dynamic`, PgDog doesn't take the username from Vault, it connects as `server_user` or `name`, if `server_user` isn't set. **Example** In `users.toml`, for a client authenticating as `alice` (verified against a static role registered under that same name) while PgDog connects to Postgres as `pgdog_service` (a separate static role): ```toml [[users]] name = "alice" database = "pgdog" vault_path = "database/static-creds/alice" server_user = "pgdog_service" server_auth = "vault_static" server_vault_path = "database/static-creds/pgdog-service" ``` In `pgdog.toml`, the same `[vault]` section used for dynamic credentials applies. Both settings are optional and independent: set only `vault_path` to verify client passwords while keeping any other backend authentication method, or only `server_auth = "vault_static"` to use a static role for backend connections while clients authenticate with a regular password. ### Sharding 📘 **[Sharding](https://docs.pgdog.dev/features/sharding/)** PgDog is able to manage databases with multiple shards. By using the PostgreSQL parser, PgDog extracts sharding keys and determines the best routing strategy for each query. For cross-shard queries, PgDog assembles and transforms results in memory, sending all rows to the client as if they are coming from a single database. **Example** Configuring multiple hosts for the same database with different shard numbers (`shard` setting) enables sharding: ```toml [[databases]] name = "prod" host = "10.0.0.1" shard = 0 [[databases]] name = "prod" host = "10.0.0.2" shard = 1 ``` Note: read below for how to configure query routing. At least one sharded table is required for sharding to work as expected. #### Sharding functions 📘 **[Sharding functions](https://docs.pgdog.dev/features/sharding/sharding-functions/)** PgDog has two main sharding algorithms: 1. PostgreSQL partition functions (`HASH`, `LIST`, `RANGE`) 2. Using schemas ##### Partition-based sharding Partition-based sharding functions are taken directly from Postgres source code. This choice intentionally allows to shard data both with PgDog and with Postgres [foreign tables](https://www.postgresql.org/docs/current/sql-createforeigntable.html) and [`postgres_fdw`](https://www.postgresql.org/docs/current/postgres-fdw.html). **Examples** The `PARTITION BY HASH` algorithm is used by default when configuring sharded tables: ```toml [[sharded_tables]] database = "prod" column = "user_id" ``` List-based sharding (same as `PARTITION BY LIST` in Postgres) can be configured as follows: ```toml # Sharded table definition still required. [[sharded_tables]] database = "prod" column = "user_id" # Value-specific shard mappings. [[sharded_mapping]] database = "prod" column = "user_id" values = [1, 2, 3, 4] shard = 0 [[sharded_mapping]] database = "prod" column = "user_id" values = [5, 6, 7, 8] shard = 1 ``` For range-based sharding, replace the `values` setting with a range, for example: ```toml start = 0 # include end = 5 # exclusive ``` ##### Schema-based sharding 📘 **[Schema-based sharding](https://docs.pgdog.dev/configuration/pgdog.toml/sharded_schemas/)** Schema-based sharding works on the basis of PostgreSQL schemas. Tables under the same schema are placed on the same shard and all queries that refer to those tables are routed to that shard automatically. **Example** Configuring sharded schemas uses a different configuration from sharded tables: ```toml [[sharded_schemas]] database = "prod" name = "customer_a" shard = 0 [[sharded_schemas]] database = "prod" name = "customer_b" shard = 1 ``` Queries that refer tables in schema `customer_a` will be sent to shard 0. For example, a query that refers to a table by its fully-qualified name will be sent to one shard only: ```sql INSERT INTO customer_a.orders (id, user_id, amount) VALUES ($1, $2, $3); ``` Alternatively, the schema name can be specified in the `search_path` session variable: ```sql SET search_path TO public, customer_a; -- All subsequent queries will be sent to shard 0. SELECT * FROM orders LIMIT 1; ``` You can also set the `search_path` for the duration of a single transaction, using `SET LOCAL`, ensuring only that transaction is sent to the desired shard: ```sql -- The entire transaction will be sent to shard 1. BEGIN; SET LOCAL search_path TO public, customer_b; SELECT * FROM orders LIMIT 1; COMMIT; ``` #### Direct-to-shard queries 📘 **[Direct-to-shard queries](https://docs.pgdog.dev/features/sharding/query-routing/)** Queries that contain a sharding key are sent to one database only. This is the best case scenario for sharded databases, since the load is uniformly distributed across the cluster. **Example**: ```sql -- user_id is the sharding key. SELECT * FROM users WHERE user_id = $1; ``` #### Cross-shard queries - 📘 **[Cross-shard queries](https://docs.pgdog.dev/features/sharding/cross-shard-queries/)** - 📘 **[SELECT](https://docs.pgdog.dev/features/sharding/cross-shard-queries/select/)** - 📘 **[INSERT](https://docs.pgdog.dev/features/sharding/cross-shard-queries/insert/)** - 📘 **[UPDATE and DELETE](https://docs.pgdog.dev/features/sharding/cross-shard-queries/update/)** - 📘 **[DDL](https://docs.pgdog.dev/features/sharding/cross-shard-queries/ddl/)** Queries with multiple sharding keys or without one are sent to all databases and results are post-processed and assembled in memory. PgDog then sends the final result to the client. Currently, support for certain SQL features in cross-shard queries is limited. However, the list of supported ones keeps growing: | Feature | Supported | Notes | |-|-|-| | Aggregates | Partial | `count`, `min`, `max`, `stddev`, `variance`, `sum`, `avg` are supported. | | `ORDER BY` | Partial | Column in `ORDER BY` clause must be present in the result set. | | `GROUP BY` | Partial | Same as `ORDER BY`, referenced columns must be present in result set. | | Multi-tuple `INSERT` | Supported | PgDog generates one statement per tuple and executes them automatically. | | Sharding key `UPDATE` | Supported | PgDog generates a `SELECT`, `INSERT` and `DELETE` statements and execute them automatically. | | Subqueries | No | The same subquery is executed on all shards. | | CTEs | No | The same CTE is executed on all shards. | #### Using `COPY` 📘 **[Copy](https://docs.pgdog.dev/features/sharding/cross-shard-queries/copy/)** PgDog has a text, CSV & binary parser and can split rows sent via `COPY` command between all shards automatically. This allows clients to ingest data into sharded PostgreSQL without preprocessing **Example** ```sql COPY orders (id, user_id, amount) FROM STDIN CSV HEADER; ``` Columns must be specified in the `COPY` statement, so PgDog can infer the sharding key automatically, but are optional in the data file. #### Consistency (two-phase commit) 📘 **[Two-phase commit](https://docs.pgdog.dev/features/sharding/2pc/)** To make sure cross-shard writes are atomic, PgDog supports Postgres' [two-phase transactions](https://www.postgresql.org/docs/current/two-phase.html). When enabled, PgDog handles `COMMIT` statements sent by clients by executing the 2pc exchange on their behalf: ```sql PREPARE TRANSACTION '__pgdog_unique_id'; COMMIT PREPARED '__pgdog_unique_id'; ``` In case the client disconnects or Postgres crashes, PgDog will automatically rollback the transaction if it's in phase I and commit it if it's in phase II. #### Unique identifiers 📘 **[Unique IDs](https://docs.pgdog.dev/features/sharding/unique-ids/)** While applications can use `UUID` (v4 and now v7) to generate unique primary keys, PgDog supports creating unique `BIGINT` identifiers, without using a sequence: ```sql SELECT pgdog.unique_id(); ``` This uses a timestamp-based algorithm, can produce millions of unique numbers per second and doesn't require an expensive cross-shard index to guarantee uniqueness. #### Shard key updates PgDog supports changing the sharding key for a row online. Under the hood, it will execute 3 statements to make it happen: 1. `SELECT` to get the entire row from its original shard 2. `INSERT` to write the new, changed row to the new shard 3. `DELETE` to remove it from the old shard This happens automatically, and the client can retrieve the new row as normal: ```sql UPDATE orders SET user_id = 5 WHERE user_id = 1 RETURNING *; -- This will return the new row ``` Note: Only one row can be updated at a time and if a query attempts to update multiple, PgDog will abort the transaction. To enable shard key updates, add this to `pgdog.toml`: ```toml [rewrite] enabled = true shard_key = "rewrite" # options: ignore (possible data loss), error (block shard key update) ``` #### Multi-tuple inserts PgDog can handle multi-tuple `INSERT` queries by sending each tuple to the right shard, e.g.: ```sql INSERT INTO payments (id, user_id, amount) -- user_id is the sharding key VALUES (pgdog.unique_id(), 1, 25.00), -- Tuples go to different shards (pgdog.unique_id(), 5, 55.0); -- Each tuple gets a unique primary key because unique ID function is invoked twice ``` This happens automatically, if enabled: ```toml [rewrite] enabled = true split_inserts = "rewrite" # other options: ignore, error ``` #### Re-sharding - 📘 **[Re-sharding](https://docs.pgdog.dev/features/sharding/resharding/)** - 📘 **[Schema sync](https://docs.pgdog.dev/features/sharding/resharding/schema/)** - 📘 **[Data sync](https://docs.pgdog.dev/features/sharding/resharding/hash/)** PgDog understands the PostgreSQL logical replication protocol and can orchestrate data splits between databases, in the background and without downtime. This allows to shard existing databases and add more shards to existing clusters in production, without impacting database operations. The re-sharding process is done in 5 steps: 1. Create new empty cluster with the desired number of shards 2. Configure it in `pgdog.toml` and run `schema-sync` command to copy table schemas to the new databases 3. Run `data-sync` command to copy and re-shard table data with logical replication (tables are copied in parallel) 4. While keeping previous command running (it streams row updates in real-time), run `schema-sync --data-sync-complete` to create secondary indexes on the new databases (much faster to do this after data is copied) 5. Cutover traffic to new cluster with `MAINTENANCE ON`, `RELOAD`, `MAINTENANCE OFF` command sequence Cutover can be done atomically with multiple PgDog containers because `RELOAD` doesn't resume traffic, `MAINTENANCE OFF` does, so the config is the same in all containers before queries are resumed. No complex synchronization tooling like etcd or Zookeeper is required. ### Monitoring 📘 **[Metrics](https://docs.pgdog.dev/features/metrics/)** PgDog exposes both the standard PgBouncer-style admin database, an OpenMetrics endpoint and can push metrics to an OTEL endpoint. The admin database isn't 100% compatible, so we recommend you use either OpenMetrics or OTEL ingestion for monitoring. We include two examples: - [Datadog configuration and dashboard](examples/datadog) - [Graphana + Prometheus configuration and dashboard](examples/grafana_prometheus) ## Running PgDog locally Install the latest version of the Rust compiler from [rust-lang.org](https://rust-lang.org). Clone this repository and build the project in release mode: ```bash cargo build --release ``` It's important to use the release profile if you're deploying to production or want to run performance benchmarks. #### Try sharding Sharded database clusters are set in the config. For example, to set up a 2 shard cluster, you can: **`pgdog.toml`** ```toml [[databases]] name = "pgdog_sharded" host = "127.0.0.1" database_name = "shard_0" shard = 0 [[databases]] name = "pgdog_sharded" host = "127.0.0.1" database_name = "shard_1" shard = 1 [[sharded_tables]] database = "pgdog_sharded" column = "user_id" ``` Don't forget to configure a user: **`users.toml`** ```toml [[users]] database = "pgdog_sharded" name = "pgdog" password = "pgdog" ``` And finally, to make it work locally, create the required databases: ```sql CREATE DATABASE shard_0; CREATE DATABASE shard_1; GRANT ALL ON DATABASE shard_0 TO pgdog; GRANT ALL ON DATABASE shard_1 TO pgdog; ``` ### Start PgDog Running PgDog can be done with Cargo: ```bash cargo run --release ``` #### Command-line options PgDog supports several command-line options: - `-c, --config `: Path to the configuration file (default: `"pgdog.toml"`) - `-u, --users `: Path to the users.toml file (default: `"users.toml"`) - `-d, --database_url `: Connection URL(s). Can be specified multiple times to add multiple database connections. When provided, these URLs override database configurations from the config file. Example using database URLs directly: ```bash cargo run --release -- -d postgres://user:pass@localhost:5432/db1 -d postgres://user:pass@localhost:5433/db2 ``` You can connect to PgDog with `psql` or any other PostgreSQL client: ```bash psql postgres://pgdog:pgdog@127.0.0.1:6432/pgdog ``` ## 🚦 Status 🚦 PgDog is used in production and at scale. Most features are stable, while some are experimental. Check [documentation](https://docs.pgdog.dev/features/) for more details. New sharding features are added almost weekly. ## Performance 📘 **[Architecture & benchmarks](https://docs.pgdog.dev/architecture/)** PgDog is heavily optimized for performance. We use Rust, [Tokio](https://tokio.rs/), [bytes crate](https://docs.rs/bytes/latest/bytes/) to avoid unnecessary memory allocations, and profile for performance regressions on a regular basis. ## License PgDog is free and open source software, licensed under the AGPL v3. While often misunderstood, this license is very permissive and allows the following without any additional requirements from you or your organization: * Internal use * Private modifications for internal use without sharing any source code You can freely use PgDog to power your PostgreSQL databases without having to share any source code, including proprietary work product or any PgDog modifications you make. AGPL was written specifically for organizations that offer PgDog _as a public service_ (e.g. database cloud providers) and require those organizations to share any modifications they make to PgDog, including new features and bug fixes. ## Contributions Please read our [Contribution Guidelines](CONTRIBUTING.md). --- ## File: examples/pgbouncer_benchmark/README.md ### Setup 1. Make sure you don't have anything running on ports 6433 and 6432. 2. (Optional) Delete all previous images to make sure to pull latest from Hub: `docker rm -vf $(docker ps -aq)` and `docker rmi -f $(docker images -aq)` 3. `docker-compose up` ### Run the benchmark ```bash export PGHOST=127.0.0.1 export PGPASSWORD=postgres export PGUSER=postgres export PGDATABASE=postgres ``` ### PgBouncer ```bash export PGPORT=6432 pgbench -i pgbench -c 10 -t 100000 ``` ### PgDog ```bash export PGPORT=6433 pgbench -i pgbench -c 10 -t 100000 ``` --- ## File: examples/datadog/README.md # Datadog integration PgDog can export metrics to Datadog via two mechanisms: 1. OpenMetrics collector 2. OTLP (OTEL) ingestion endpoint Both export the same metrics, at a configurable interval of your choice. ## OpenMetrics PgDog exports a lot of metrics via an OpenMetrics endpoint. You can enable the endpoint by specifying the port number in `pgdog.toml`: ```toml # pgdog.toml [general] openmetrics_port = 9090 ``` The endpoint will run on `http://0.0.0.0:9090`. A sample config is included in [`openmetrics.d/conf.yaml`](openmetrics.d/conf.yaml). ## OLTP/OTEL PgDog can push OTEL metrics to a configurable endpoint. Datadog supports ingesting metrics this way and we support it natively: ```toml # pgdog.toml [otel] datadog_api_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" endpoint = "https://otlp.us5.datadoghq.com/v1/metrics" ``` By default, PgDog will push metrics to the endpoint every 10 seconds. ## Dashboard We have a pre-build Datadog dashboard you can import directly into your stack. See [dashboard.json](dashboard.json). --- ## File: pgdog-plugin/README.md # PgDog plugins [](https://docs.pgdog.dev/features/plugins/) [](https://docsrs.pgdog.dev/pgdog_plugin/index.html) [](https://crates.io/crates/pgdog-plugin) PgDog plugin system is based around shared libraries loaded at runtime. The plugins currently can only be written in Rust. This is because PgDog passes Rust-specific data types to plugin functions, and those cannot be easily made C ABI-compatible. This crate implements the bridge between PgDog and plugins, making sure data types can be safely passed through the FFI. Automatic checks include: - Rust compiler version check - `pgdog-plugin` crate version check This crate should be linked at compile time against your plugins. ## Writing plugins See [crate documentation](https://docsrs.pgdog.dev/pgdog_plugin/index.html) and [pgdog tutorial](https://docs.pgdog.dev/features/plugins/) for guides and examples. Example plugins are [available in GitHub](https://github.com/pgdogdev/pgdog/tree/main/plugins) as well. ## License This library is distributed under the MIT license. See [LICENSE](LICENSE) for details. --- ## File: plugins/README.md # PgDog plugins This directory contains plugins that ship with PgDog and are built by original author(s) or by the community. You can use these as-is or modify them to your needs. ## Plugins ### `pgdog-example-plugin` Example plugin that can be used as reference by the community. It currently records when a write was made to a table and, for the next 5 seconds after the write, redirects all `SELECT` queries that touch table to the primary. It's a simple workaround for Postgres replica lag, if you're using batch writes. ## 2. Official Technical Reference & Guides (pgdogdev/docs) ## File: README.md # PgDog Documentation This repository contains the documentation hosted on [https://docs.pgdog.dev](https://docs.pgdog.dev). ## Contributions Contributions are welcome. Please open a pull request / issue with requested changes. Once the PR is merged, you should see changes in production within a few minutes. --- ## File: docs/index.md --- icon: material/home --- # Introduction to PgDog PgDog is a connection pooler, load balancer and database sharder for PostgreSQL. Written in Rust, PgDog is fast, reliable and scales Postgres databases without requiring changes to your application. ## Getting started PgDog is an open source project. You can download its code from our [repository](https://github.com/pgdogdev/pgdog) in GitHub. If you're deploying PgDog to your cloud (or on prem), you can use our [Docker image](installation.md) (and Helm chart), the [compiled binaries](https://github.com/pgdogdev/pgdog/releases) (incl. a `.deb` package), or build it from source. ### Docker image Every commit in the `main` branch, and weekly tagged releases, have corresponding images in our [Docker](https://github.com/orgs/pgdogdev/packages/container/package/pgdog) repository, for example: ```bash docker run ghcr.io/pgdogdev/pgdog:v0.1.48 ``` You can read more about how to deploy PgDog [here](installation.md). ## Why PgDog PostgreSQL is a process-based, single-primary database. As such, it has hard limits on how many clients can connect, how many queries a single server can execute, and how much data it can write at any given time. PgDog helps you work around these limits. It's a single binary, deployed between your application and the database, that speaks both the protocol applications use to talk to Postgres and the replication protocol Postgres uses internally. This lets PgDog do its job transparently, without changes to Postgres or the applications that query it. ## Connection pooler Like PgBouncer or RDS Proxy, PgDog is a connection pooler: it multiplexes thousands of application connections over just a handful of Postgres server connections. This effectively removes the limit on how many clients a PostgreSQL database can serve at once. Unlike those proxies, PgDog handles features that usually force a pooler to pin or reset connections. It supports SET statements, LISTEN/NOTIFY, and advisory locks without breaking connection state, so your application keeps working as if it were talking to Postgres directly. PgDog is also multithreaded, so a single instance can serve many more clients while still relying on the same small number of Postgres connections. You can read more about how the connection pooler works [here](features/connection-pooler/index.md). ## Load balancer If your database has read replicas, PgDog can distribute read queries across them using one of several built-in load balancing [algorithms](features/load-balancer/index.md). This lets you scale reads simply by adding replicas, with no application changes and no extra infrastructure like HAProxy or Patroni. You can read more about how PgDog load balances queries [here](features/load-balancer/index.md). ## Sharding PostgreSQL Unlike NoSQL databases, PostgreSQL can serve INSERT, UPDATE, and DELETE queries from only one server. Once that server's capacity is exceeded, applications have to find new and creative ways to reduce their load on the database, such as batching requests or deferring workloads to run overnight. At the same time, database operators face rising operating costs from vacuums that fall behind schedule, table bloat and downtime. Incidents become frequent, and engineers end up more focused on keeping the database from breaking than on building new features. The solution to an overextended database is sharding: splitting all tables and indices equally between multiple machines. For example, if your primary database is 750 GB, splitting it into 3 shards will produce 3 databases of 250 GB each. As databases get smaller, vacuums start to catch up, indices fit into memory again, and queries run faster with reliable performance. As shards accumulate more data and grow, they can be split again, scaling PostgreSQL horizontally. Sharded databases can grow into petabytes (thousands of TB) while serving both OLTP and OLAP workloads. PgDog operates on the application layer of the stack: it speaks PostgreSQL and understands not only the queries sent by applications but also the logical replication protocol used by the server. This allows it to automatically route queries while moving data between machines to create more capacity. This documentation provides a detailed overview of all PgDog features, along with reference material for production operations. ## Read more {{ next_steps_links([ ("Installation", "/installation/", "Deploy PgDog with Helm on Kubernetes, run it on AWS ECS with Terraform, with Docker, with pre-built binaries, or by building from source."), ("Connection pooler", "/features/connection-pooler/", "Multiplex thousands of application connections over a small number of PostgreSQL server connections."), ("Load balancer", "/features/load-balancer/", "Distribute read queries across replicas and send write queries to the primary database."), ("Sharding", "/features/sharding/", "Scale PostgreSQL horizontally with query routing, data migration, and schema management."), ]) }}