Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/about.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,4 @@ PgDog source code, so the whole community can benefit from your knowledge and ex

## Project name

This project is dedicated to the bestest dog in the world who's been patiently sitting at my feet the entire time PgDog has been developed.
This project is dedicated to the best dog in the world who's been patiently sitting at my feet the entire time PgDog has been developed.
2 changes: 1 addition & 1 deletion docs/architecture/benchmarks.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Benchmarks

PgDog does its best to minimize its impact on database performance. Great care is taken to make sure as few operations are possible are performed
PgDog does its best to minimize its impact on database performance. Great care is taken to make sure as few operations as possible are performed
when passing data between clients and servers.

All benchmarks listed below were done on my local system (AMD Ryzen 7 5800X). Real world performance is impacted by factors like network speed, query complexity and especially by hardware used for running PgDog and PostgreSQL servers.
Expand Down
32 changes: 18 additions & 14 deletions docs/features/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,27 +3,31 @@
PgDog contains multiple foundational and unique features which make it a great choice
for modern PostgreSQL deployments.

Most features are configurable and can be toggled and tuned. Experimental features are marked
as such, and users are advised to test them before deploying to production. Most foundational features like
load balancing, healthchecks, and query routing have been battle-tested and work well in production.
Most features are configurable and can be toggled on/off and tuned to fit your environment. Experimental features are marked
as such, and users are advised to test them before deploying to production.

Foundational features like load balancing, health checks, and query routing have been battle-tested and work well in production.

## Summary

Short summary of currently implemented features.
Short summary of currently implemented features below.

| Feature | Description |
|---------|-------------|
| [Load balancer](load-balancer/index.md) | Distribute `SELECT` queries evenly between replicas. Separate reads from writes with a single endpoint. |
| [Health checks](load-balancer/healthchecks.md) | Check databases are up and running, and can serve queries. |
| [Transaction mode](transaction-mode.md) | Share PostgreSQL connections between thousands of clients, a necessary feature for production deployments. |
| [Hot reload](../configuration/index.md) | Update configuration at runtime without restarting PgDog. |
| [Sharding](sharding/index.md) | Automatic query routing and logical replication between data nodes to scale PostgreSQL horizontally. |
| [Prepared statements](prepared-statements.md) | Support for Postgres named prepared statements. |
| [Plugins](plugins/index.md) | Pluggable libraries to add functionality to PgDog at runtime. |
| [Authentication](authentication.md) | Support for various PostgreSQL authentication mechanisms, like SCRAM. |
| [Load balancer](load-balancer/index.md) | Distribute `SELECT` queries evenly between replicas. Separate reads from writes, allowing applications to connect to a single endpoint. |
| [Health checks](load-balancer/healthchecks.md) | Check databases are up and running. Broken databases are blocked from serving queries. |
| [Transaction mode](transaction-mode.md) | Multiplex PostgreSQL server connections between thousands of clients. |
| [Hot reload](../configuration/index.md) | Update configuration at runtime without restarting the proxy. |
| [Sharding](sharding/index.md) | Automatic query routing and data migration between nodes to scale PostgreSQL horizontally. Schema management, distributed transactions. |
| [Prepared statements](prepared-statements.md) | Support for Postgres named prepared statements in transaction mode. |
| [Plugins](plugins/index.md) | Pluggable libraries to add functionality to PgDog at runtime, without recompiling code. |
| [Authentication](authentication.md) | Support for various PostgreSQL user authentication mechanisms, like SCRAM. |
| [Session mode](session-mode.md) | Compatibility mode with direct PostgreSQL connections. |
| [Metrics](metrics.md) | Real time reporting, including Prometheus/OpenMetrics and admin database. |
| [Metrics](metrics.md) | Real time reporting, including Prometheus/OpenMetrics and an admin database. |
| [Mirroring](mirroring.md) | Copy queries from one database to another in the background. |
| [Pub/Sub](pub_sub.md) | Support for `LISTEN`/`NOTIFY` in transaction mode. |
| [Encryption](tls.md) | TLS encryption for client and server connections. |

## OS support
### OS support

PgDog doesn't use any OS-specific features and should run on all systems supported by the Rust compiler, e.g. Linux (x86 and ARM64), Mac OS, and Windows.
60 changes: 38 additions & 22 deletions docs/features/load-balancer/healthchecks.md
Original file line number Diff line number Diff line change
@@ -1,58 +1,74 @@
# Health checks

Databases proxied by PgDog are regularly checked with health checks. A health check is a simple query, e.g.,
`SELECT 1`, that ensures the database is reachable and able to process queries.

## How it works

If a database fails a health check, it's placed in a list of banned hosts. Banned databases are removed
from the load balancer and will not serve queries. This allows PgDog to reduce errors clients see
when a database fails, for example due to hardware issues.
Databases proxied by PgDog's load balancer are regularly checked with health checks. A health check is a simple query, e.g.,
`SELECT 1`, that ensures the database is reachable and able to handle requests. If a replica database fails a health check,
it's removed from the load balancer and prevented from serving additional queries for a configurable period of time.

<center>
<img src="/images/healtchecks.png" width="65%" alt="Healtchecks"/>
<img src="/images/healthchecks.png" width="65%" alt="Healthchecks"/>
</center>

### Checking connections
### Primary checks

While all databases receive health checks, only replicas can be removed from the load balancer. If the primary fails a health check, it will continue to serve writes. This is because the cluster doesn't have an alternative place to route these requests and attempting the primary again has a higher chance of success than blocking queries outright.

### Individual connections

In addition to checking databases, PgDog ensures that every connection in the pool is healthy on a regular basis. Before giving a connection to a client, PgDog will occasionally send the same simple query to the server, and if the query fails, ban the entire database from serving any more queries.
In addition to checking entire databases, the load balancer checks that every connection in the pool is healthy on a regular basis. Before giving a connection to a client, it will, from time to time, send a short query to the server, and if it fails, ban the entire database from serving any more requests.

To reduce the overhead of health checks, connection-specific checks are done infrequently, configurable via the `healtcheck_interval` setting:
To reduce the overhead of health checks, these connection-specific checks are done infrequently. This is configurable via the `healthcheck_interval` setting:

```toml
[general]
healthcheck_interval = 30_000 # Run a health check every 30 seconds
```

Health checks are **enabled** by default. The default setting value is `30_000` (30 seconds).
The default value for this setting is `30_000` (30 seconds).

### Configuring health checks

Health checks are **enabled** by default.

If you want, you can effectively disable them by setting both `healthcheck_interval` and `idle_healthcheck_interval` settings to a very high value, for example:

```toml
[general]
healthcheck_interval = 31557600000 # 1 year
idle_healthcheck_interval = 31557600000
```


### Database bans

### Triggering bans
A single health check failure will prevent the entire database from serving traffic. In our documentation (and the code), we refer to this as a "ban".

A single health check failure will prevent the entire database from serving traffic. This may seem aggressive at first, but it reduces the error rate dramatically in heavily used production deployments. PostgreSQL is very reliable, so even a single query failure may indicate an issue with hardware or network connectivity.
This may seem aggressive at first, but it reduces the error rate dramatically in heavily used production deployments. PostgreSQL is very reliable, so even a single failure often indicates an issue with the hardware or network connectivity.


#### Failsafe

To avoid health checks taking a database cluster offline, the load balancer has a built-in safety mechanism. If all replicas fail health checks, bans from all databases are removed and all databases are allowed to serve traffic again. This ensures that intermittent network failures don't impact database operations. Once the bans are removed, load balancing returns to its normal state.
To avoid health checks taking the whole database cluster offline, the load balancer has a built-in safety mechanism. If all replicas fail a health check, the bans from all databases in the cluster are removed.

This makes sure that intermittent network failures don't impact database operations. Once the bans are removed, load balancing returns to its normal state.

#### Ban expiration

Database bans have an expiration. Once the ban expires, the replica is unbanned and allowed to serve traffic again. This is done to maintain a healthy level of traffic across all databases and to allow for intermittent
issues, like network connectivity, to resolve themselves without manual intervention.
Database bans eventually expire and are removed automatically. Once this happens, the banned databases are allowed to serve traffic again. This is done to maintain a healthy level of traffic across all databases and to allow for intermittent issues, like network connectivity, to resolve themselves without manual intervention.

This behavior is controlled with the `ban_timeout` setting:
This behavior can be controlled with the `ban_timeout` setting:

```toml
[general]
ban_timeout = 300_000 # Expire bans automatically after 5 minutes
```

The default value is `300_000` (5 minutes).
The default value for this setting is `300_000` (5 minutes).

#### Health check timeout

### Health check timeout
By default, the load balancer gives the database **5 seconds** to answer a health check. If it doesn't receive a reply within that time frame, the database will be banned and removed from the load balancer.

By default, PgDog gives the database **5 seconds** to answer a health check. This is configurable with `healthcheck_timeout`. If PgDog doesn't receive a reply within that time frame, the database will be banned from serving traffic.
This is configurable with `healthcheck_timeout` setting:

```toml
[global]
Expand Down
67 changes: 36 additions & 31 deletions docs/features/load-balancer/index.md
Original file line number Diff line number Diff line change
@@ -1,43 +1,42 @@
---
next_steps:
- ["Health checks", "/features/healthchecks/", "Learn how PgDog ensures only healthy databases serve read queries."]
- ["Health checks", "/features/healthchecks/", "Learn how PgDog ensures only healthy databases are allowed to serve read queries."]
---

# Load balancer overview

PgDog operates at the application layer (OSI Level 7) and is capable of load balancing queries across
multiple PostgreSQL replicas.
multiple PostgreSQL replicas. This allows applications to connect to a single endpoint and spread traffic evenly between multiple databases.

## How it works

When a query is sent to PgDog, it inspects it using a SQL parser. If the query is a read and PgDog configuration contains multiple databases, it will send that query to one of the replicas, spreading the query load evenly between all instances in the cluster.
When a query is sent to PgDog, it inspects it using a SQL parser. If the query is a read and the configuration contains multiple databases, it will send that query to one of the replicas. This spreads the query load evenly between all database instances in the cluster.

If the configuration contains the primary as well, PgDog will separate writes from reads and send writes to the primary, without requiring any application changes.
If the config contains a primary, PgDog will split write queries from read queries and send writes to the primary, without requiring any application changes.

<center>
<img src="/images/replicas.png" width="60%" alt="Load balancer" />
</center>

### Strategies
### Algorithms

The PgDog load balancer is configurable and can route queries
using one of several strategies:
The load balancer is configurable and can route queries using one of the following strategies:

* Random (default)
* Least active connections
* Round robin

Choosing a strategy depends on your query workload and the size of replica databases. Each strategy has its pros and cons. If you're not sure, using the **random** strategy is usually good enough
Choosing the right strategy depends on your query workload and the size of replica databases. Each strategy has its pros and cons. If you're not sure, using the **random** strategy is usually good enough
for most deployments.


#### Random

Queries are sent to a database based using a random number generator modulus the number of replicas in the pool.
Queries are routed to a database based on a random number generator modulus the number of replicas in the pool.
This strategy is the simplest to understand and often effective at splitting traffic evenly across the cluster. It's unbiased
and assumes nothing about available resources or query performance.
and assumes nothing about available resources or individual query performance.

This strategy is used by **default**.
This algorithm is used by **default**.

##### Configuration

Expand All @@ -48,11 +47,10 @@ load_balancer_strategy = "random"

#### Least active connections

PgDog keeps track of how many active connections each database has and can route queries to databases
which are least busy executing requests. This allows to "bin pack" the cluster based on how seemingly active
(or inactive) the databases are.
PgDog keeps track of how many connections are active in each database and can route queries to databases
which are less busy. This allows to "bin pack" the cluster with workload.

This strategy is useful when all databases have identical resources and all queries have roughly the same
This algorithm is useful when all databases have identical resources and all queries have roughly the same
cost and runtime.

##### Configuration
Expand All @@ -64,11 +62,11 @@ load_balancer_strategy = "least_active_connections"

#### Round robin

This strategy is often used in HTTP load balancers, like nginx, to route requests to hosts in the
same order they appear in the configuration. Each database receives exactly one query before the next
This strategy is often used in HTTP load balancers (e.g., like nginx) to route requests to hosts in the
same order as they appear in the configuration. Each database receives exactly one query before the next
one is used.

This strategy makes the same assumptions as [least active connections](#least-active-connections), except it makes no attempt to bin pack the cluster with workload, and distributes queries evenly.
This algorithm makes the same assumptions as [least active connections](#least-active-connections), except it makes no attempt to bin pack the cluster and distributes queries evenly.

##### Configuration

Expand All @@ -79,15 +77,17 @@ load_balancer_strategy = "round_robin"

## Reads and writes

The load balancer can split reads (`SELECT` queries) from write queries. If PgDog detects that a query is _not_ a `SELECT`, like an `INSERT` or and `UPDATE`, that query will be sent to primary. This allows a PgDog deployment to proxy an entire PostgreSQL cluster without creating separate read and write endpoints.
The load balancer can split reads (`SELECT` queries) from write queries. If it detects that a query is _not_ a `SELECT`, like an `INSERT` or an `UPDATE`, that query will be sent to primary. This allows a deployment to proxy an entire PostgreSQL cluster without creating separate read and write endpoints.

This strategy is effective most of the time, and PgDog also handles several edge cases.
This strategy is effective most of the time and PgDog also handles several edge cases.

#### Select for update
### `SELECT FOR UPDATE`

The most common one is `SELECT [...] FOR UDPATE` which locks rows for exclusive access. Much like the name suggests, the most common use case for this is to update the row, which is a write operation. PgDog will detect this and send the query to the primary instead.
The most common edge case is `SELECT FOR UPDATE` which locks rows for exclusive access. Much like the name suggests, it's often used to update the selected rows, which is a write operation.

#### CTEs that write
The load balancer detects this and will send the query to a primary instead of a replica.

### CTEs

Some `SELECT` queries can trigger a write to the database from a CTE, for example:

Expand All @@ -98,28 +98,33 @@ WITH t AS (
SELECT * FROM users INNER JOIN t ON t.id = users.id
```

PgDog will check all CTEs and if any of them contain queries that could write, it will send the entire query to the primary.
The load balancer will check all CTEs and, if any of them contain queries that could write, it will route the entire query to a primary.

### Transactions

All explicit transactions are routed to the primary. An explicit transaction is started by using the `BEGIN` statement, e.g.:
All multi-statement transactions are routed to the primary. They are started by using the `BEGIN` command, e.g.:

```postgresql
BEGIN;
INSERT INTO users (email, created_at) VALUES ($1, NOW()) RETURNING *;
COMMIT;
```

While more often used to atomically perform writes to multiple tables, transactions can also manually route read queries to the primary as to avoid having to handle replication lag for time-sensitive queries.
While often used to atomically perform multiple changes, transactions can also be used explicitly route read queries to a primary as to avoid having to handle replication lag.

This is useful for time-sensitive workloads, like background jobs that have been triggered by a database change which hasn't propagated to all the replicas yet.

!!! note
This is common with background jobs that get triggered after a row has been inserted by an HTTP controller.
The job queue is often configured to read data from a replica, which is a few milliseconds behind the primary and, unless specifically handled, could run into "record not found" errors.
This behavior often manifests with "record not found"-style errors, e.g.:

```
ActiveRecord::RecordNotFound (Couldn't find User with 'id'=9999):
```


## Configuration

The load balancer is enabled automatically when cluster contains more than
The load balancer is **enabled** automatically when a database cluster contains more than
one database, for example:

```toml
Expand All @@ -134,9 +139,9 @@ role = "replica"
host = "10.0.0.2"
```

### Reads on the primary
### Allowing reads on the primary

By default, the primary is used for serving reads and writes. If you want to isolate your workloads and have your replicas serve all read queries, you can configure it like so:
By default, the primary is used for serving both reads and writes. If you want to isolate these workloads and have your replicas serve all read queries instead, you can configure it, like so:

```toml
[general]
Expand Down
Loading
Loading