diff --git a/docs/configuration/index.md b/docs/configuration/index.md index 7b46fea..1d2ca7f 100644 --- a/docs/configuration/index.md +++ b/docs/configuration/index.md @@ -25,6 +25,8 @@ Since PgDog uses TOML, both `5000` and `5_000` are valid numbers. Configuration |------|-------------| | [General](pgdog.toml/general.md) | General pooler settings like `host`, `port` and various timeouts. | | [Databases](pgdog.toml/databases.md) | PostgreSQL databases proxied by PgDog. | +| [Mirroring](pgdog.toml/mirroring.md) | Configuration to [mirror](../features/mirroring.md) databases for testing | +| [Sharded Tables](pgdog.toml/sharded_tables.md) | Configuration for [sharding](../features/sharding/basics.md) databases | | [Plugins](pgdog.toml/plugins.md) | Plugins configuration. | | [Users](users.toml/users.md) | List of users (with passwords) that are allowed to connect to PgDog. | | [Admin](pgdog.toml/admin.md) | Admin database settings like admin password. | diff --git a/docs/configuration/pgdog.toml/databases.md b/docs/configuration/pgdog.toml/databases.md index ceef26e..d191443 100644 --- a/docs/configuration/pgdog.toml/databases.md +++ b/docs/configuration/pgdog.toml/databases.md @@ -72,14 +72,6 @@ The shard number for this database. Only required if your database contains more Default: **`0`** -### `mirror_of` - -Indicates this database is a mirror of another one. All traffic sent to the `mirror_of` database will also be replayed against this database, subject to [`mirror_exposure`](general.md#mirror_exposure) setting. - -For clusters with multiple databases, `mirror_of` must be identical in all entries. Otherwise, mirroring will be automatically disabled for that database. - -Default: **`none`** - !!! note All settings below take priority over values in [`[general]`](general.md) and [`[[users]]`](../users.toml/users.md) config sections. diff --git a/docs/configuration/pgdog.toml/general.md.bak b/docs/configuration/pgdog.toml/general.md.bak deleted file mode 100644 index 388135f..0000000 --- a/docs/configuration/pgdog.toml/general.md.bak +++ /dev/null @@ -1,259 +0,0 @@ - -# General settings - -General settings are relevant to the operations of the pooler itself, or apply to all database pools. - -### `host` - -The IP address of the local network interface PgDog will bind to listen for connections. - -!!! note - This setting cannot be changed at runtime. - -Default: **`0.0.0.0`** (all interfaces) - -### `port` - -The TCP port PgDog will bind to listen for connections. - -Default: **`6432`** - -!!! note - This setting cannot be changed at runtime. - -### `workers` - -Number of Tokio threads to spawn at pooler startup. In multi-core systems, the recommended setting is two (2) per -virtual CPU. The value `0` means to spawn no threads and use the current thread runtime (single-threaded). The latter option is better on IO-bound systems where multi-threading is not necessary and could even hamper performance. - -Default: **`0`** (current thread runtime) - -!!! note - This setting cannot be changed at runtime. - -### `default_pool_size` - -Default maximum number of server connections per database pool. The pooler will not open more than this many PostgreSQL database connections when serving clients. - -Default: **`10`** - -### `min_pool_size` - -Default minimum number of connections per database pool to keep open at all times. Keeping some connections -open minimizes cold start time when clients connect to the pooler for the first time. - -Default: **`1`** - - -### `pooler_mode` - -Default pooler mode to use for database pools. See [Transaction mode](../../features/transaction-mode.md) and [session mode](../../features/session-mode.md) for more details on each mode. - -Default: **`transaction`** - -## TLS - -### `tls_certificate` - -Path to the TLS certificate PgDog will use to setup TLS connections with clients. If none is provided, TLS will be disabled. - -Default: **none** - -!!! note - This setting cannot be changed at runtime. - -### `tls_private_key` - -Path to the TLS private key PgDog will use to setup TLS connections with clients. If none is provided, TLS will be disabled. - -Default: **none** - -!!! note - This setting cannot be changed at runtime. - -## Healthchecks - -### `healthcheck_interval` - -Frequency of healthchecks performed by PgDog to ensure connections provided to clients from the pool are working. - -Default: **`30_000`** (30s) - -### `idle_healthcheck_interval` - -Frequency of healthchecks performed by PgDog on idle connections. This ensures the database is checked for health periodically when -PgDog receives little to no client requests. - -Default: **`30_000`** (30s) - -#### Note on `min_pool_size` - -[Healthchecks](../../features/healthchecks.md) try to use existing idle connections to validate the database is up and running. If there are no idle connections available, PgDog will create an ephemeral connection to perform the healthcheck. If you want to avoid this, make sure to have `min_pool_size` to be at least `1`. - -### `idle_healthcheck_delay` - -Delay running idle healthchecks at PgDog startup to give databases (and pools) time to spin up. - -Default: **`5_000`** (5s) - -## Timeouts - -These settings control how long PgDog waits for maintenance tasks to complete. These timeouts make sure PgDog can recover -from abnormal conditions like hardware failure. - -### `rollback_timeout` - -How long to allow for `ROLLBACK` queries to run on server connections with unfinished transactions. See [transaction mode](../../features/transaction-mode.md) for more details. - -Default: **`5_000`** (5s) - -### `ban_timeout` - -Connectionn pools blocked from serving traffic due to an error will be placed back into active rotation after this long. This ensures -that servers don't stay blocked forever due to healthcheck false positives. - -Default: **`300_000`** (5 minutes) - -### `shutdown_timeout` - -How long to wait for active clients to finish transactions when shutting down. This ensures that PgDog redeployments disrupt as few -queries as possible. - -Default: **`60_000`** (60s) - - -### `query_timeout` - -Maximum amount of time to wait for Postgres query to finish executing. Use only in unreliable network conditions or when Postgres runs on unreliable hardware. - -Default: **disabled** - -### `connect_timeout` - -Maximum amount of time to allow for PgDog to create a connection to Postgres. - -Default: **`300`** (5s) - -### `connect_attempts` - -Maximum number of retries for Postgres server connection attempts. When exceeded, an error is returned to the pool -and the pool will be banned from serving more queries. - -Default: **`1`** - -### `connect_attempt_delay` - -Amount of time to wait between connection attempt retries. - -Default: **`0`** (0ms) - -### `checkout_timeout` - -Maximum amount of time a client is allowed to wait for a connection from the pool. - -Default: **`300`** (5s) - -## Load balancer - -### `load_balancing_strategy` - -Which strategy to use for load balancing read queries. See [load balancer](../../features/load-balancer.md) for more details. Available options are: - -* `random` -* `least_active_connections` -* `round_robin` - -Default: **`random`** - -### `read_write_split` - -How to handle the separation of read and write queries. - -Available options: - -- `include_primary` -- `exclude_primary` - -Include primary uses the primary database as well as the replicas to serve read queries. Exclude primary will send all read queries to replicas, leaving the primary to serve only writes. - -Default: **`include_primary`** - -## Service discovery - -### `broadcast_address` - -Send multicast packets to this address on the local network. Configuring this setting enables -mutual service discovery. Instances of PgDog running on the same network will be able to see -each other. - -Default: **none** (disabled) - -### `broadcast_port` - -The port used for sending and receiving broadcast messages. - -Default: **`6433`** - -## Monitoring - -### `openmetrics_port` - -The port used for the OpenMetrics HTTP endpoint. - -Default: **unset** (disabled) - -### `openmetrics_namespace` - -Prefix added to all metric names exposed via the OpenMetrics endpoint. - -Default: **none** - -## Authentication - -### `auth_type` - -What kind of authentication mechanism to use for client connections. - -Currently supported: - -- `scram` (SCRAM-SHA-256) -- `md5` (MD5) - -Default: **`scram`** - -`md5` is very quick but not secure, while `scram` authentication is slow but has better security features. If security isn't a concern but latency for connection creation is, consider using `md5`. - -### `passthrough_auth` - -Toggle automatic creation of connection pools given the user name, database and password. See [passthrough authentication](../../features/authentication.md#passthrough-authentication). - -Available options are: - -- `disabled` -- `enabled` -- `enabled_plain` - -Default: **`disabled`** - -## Prepared statements - -### `prepared_statements` - -Enables support for prepared statements. Available options are: - -- `disabled` -- `extended` -- `full` - -Full enables support for rewriting prepared statements sent over the simple protocol. Extended handles prepared statements sent normally -using the extended protocol. `full` attempts to rewrite prepared statements sent over using the simple protocol. - -Default: **`extended`** - -## Misc - -### `mirror_queue` - -Size of requests that can wait while the mirror database processes other requests. Increase this to lose less traffic while replaying, in case the mirror database is slower than production. - -Default: **128** diff --git a/docs/configuration/pgdog.toml/mirroring.md b/docs/configuration/pgdog.toml/mirroring.md new file mode 100644 index 0000000..27eeb68 --- /dev/null +++ b/docs/configuration/pgdog.toml/mirroring.md @@ -0,0 +1,37 @@ +# Mirroring + +Mirroring settings configure a mirroring setup between two databases. This causes traffic to be copied to the destination database for testing purposes: More details on [mirroring](../../features/mirroring.md) + +```toml +[[mirroring]] +source_db = "source" +destination_db = "dest" +queue_depth = 500 # optional, overrides global setting +exposure = 0.1 # optional, overrides global setting +``` + +### `source_db` + +Name of the source database. This should be a `name` set up in +[the `databases` section of the configuration.](./databases.md) + +Default: **none** (required) + +### `destination_db` + +Name of the destination database. This should be a `name` set up +in [the `databases` section of the configuration.](./databases.md) + +Default: **none** (required) + +### `queue_depth` + +The length of the queue to provision. See [mirroring](../../features/mirroring.md) for more details. This overrides the [general](./general.md) setting `mirror_queue` + +Default: **none** (optional) + +### `exposure` + +The percent of transactions to mirror, as a floating point number between 0.0 and 1.0 . See [mirroring](../../features/mirroring.md) for more details. This overrides the [general](./general.md) setting `mirror_exposure` + +Default: **none** (optional) \ No newline at end of file diff --git a/docs/features/mirroring.md b/docs/features/mirroring.md index 2a63a80..7908a64 100644 --- a/docs/features/mirroring.md +++ b/docs/features/mirroring.md @@ -17,7 +17,7 @@ Mirroring in PgDog is asynchronous and should have minimal impact production dat ### Configure mirroring -To use mirroring, first configure both the mirror and the production database in [`pgdog.toml`](../configuration/pgdog.toml/general.md). Once both databases are running, add `mirror_of` to all instances of the mirror database: +To use mirroring, first configure both the mirror and the production database in [`pgdog.toml`](../configuration/pgdog.toml/general.md). Once both databases are running, add a `[[mirroring]]` section: ```toml [[databases]] @@ -27,7 +27,12 @@ host = "10.0.0.1" [[databases]] name = "staging_db" host = "10.0.2.25" -mirror_of = "prod" + +[[mirroring]] +source_db = "prod" +destination_db = "staging_db" +# queue_length = 256 # Optional: overrides general.mirror_queue +# exposure = 0.5 # Optional: overrides general.mirror_exposure ``` !!! note @@ -48,6 +53,17 @@ If the mirror database(s) can't keep up with production traffic, queries will ba mirror_queue = 500 ``` +Or in the individual `[[mirroring]]` section: + +```toml +[[mirroring]] +source_db = 'source' +destination_db = 'dest' +queue_depth = 500 +``` + +Note that local `[[mirroring]]` configuration overrides the `general` settings for that mirror only. + If the queue gets full, all subsequent mirrored transactions will be dropped until there is space in the queue again. !!! note @@ -64,6 +80,17 @@ This is configurable using a percentage, relative to the amount of transactions mirror_exposure = 0.5 # 50% ``` +Or in the specific mirroring section: + +```toml +[[mirroring]] +source_db = 'source' +destination_db = 'dest' +exposure = 0.5 +``` + +Local config in `[[mirroring]]` overwrites the `[general]` value for that mirror. + Acceptable values are between **0.0** (0%) and **1.0** (100%). This is changeable at runtime, without restarting PgDog. When adding a mirror, it's a good idea to start slow, e.g., with only 0.1% exposure (`mirror_exposure = 0.01`), and gradually increase it over time.