Skip to content
This repository was archived by the owner on Apr 30, 2025. It is now read-only.

Commit 8555d7f

Browse files
annanay25pracucci
authored and
jtlisi
committed
Add TLS support to HTTP/GRPC clients (cortexproject#2502)
* Checkpoint Signed-off-by: Annanay <[email protected]> * Add tls options to grpc client Signed-off-by: Annanay <[email protected]> * Add new httpclient util package for use in all client configs Signed-off-by: Annanay <[email protected]> * Change all grpc clients to use grpcclient Signed-off-by: Annanay <[email protected]> * Fix build, add docs Signed-off-by: Annanay <[email protected]> * Fix tests Signed-off-by: Annanay <[email protected]> * Fix lint, add tls to store-gw-client Signed-off-by: Annanay <[email protected]> * Rename config parameters Signed-off-by: Annanay <[email protected]> * Lint Signed-off-by: Annanay <[email protected]> * Nit fix Signed-off-by: Annanay <[email protected]> * Checkpoint Signed-off-by: Annanay <[email protected]> * Checkpoint Signed-off-by: Annanay <[email protected]> * Checkpoint Signed-off-by: Annanay <[email protected]> * Add integration tests for TLS Signed-off-by: Annanay <[email protected]> * Correct package names, fix config file reference Signed-off-by: Annanay <[email protected]> * Fix cert paths Signed-off-by: Annanay <[email protected]> * Fix lint, add sample tls config file Signed-off-by: Annanay <[email protected]> * Crash quickly if certs are bad Signed-off-by: Annanay <[email protected]> * Fixed linter and doc generation Signed-off-by: Marco Pracucci <[email protected]> * Cleaned white noise Signed-off-by: Marco Pracucci <[email protected]> * Address review comments Signed-off-by: Annanay <[email protected]> * Fix docs, flags Signed-off-by: Annanay <[email protected]> * Fix test Signed-off-by: Annanay <[email protected]> * Fix lint, docs Signed-off-by: Annanay <[email protected]> * Do not use TLS options with GCP clients Signed-off-by: Annanay <[email protected]> * Add client auth type, go mod tidy/vendor Signed-off-by: Annanay <[email protected]> * Address comments Signed-off-by: Annanay <[email protected]> * Fix lint, add new integration test Signed-off-by: Annanay <[email protected]> * Revert logging level to warn, add CHANGELOG entry Signed-off-by: Annanay <[email protected]> Co-authored-by: Marco Pracucci <[email protected]> Signed-off-by: Jacob Lisi <[email protected]>
1 parent 9db28c1 commit 8555d7f

26 files changed

+537
-51
lines changed

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ Please make sure to review renamed metrics, and update your dashboards and alert
3737
* [FEATURE] TLS config options added to the Server. #2535
3838
* [FEATURE] Experimental: Added support for `/api/v1/metadata` Prometheus-based endpoint. #2549
3939
* [FEATURE] Add ability to limit concurrent queries to Cassandra with `-cassandra.query-concurrency` flag. #2562
40+
* [FEATURE] TLS config options added for GRPC clients in Querier (Query-frontend client & Ingester client), Ruler, Store Gateway, as well as HTTP client in Config store client. #2502
4041
* [ENHANCEMENT] Experimental TSDB: sample ingestion errors are now reported via existing `cortex_discarded_samples_total` metric. #2370
4142
* [ENHANCEMENT] Failures on samples at distributors and ingesters return the first validation error as opposed to the last. #2383
4243
* [ENHANCEMENT] Experimental TSDB: Added `cortex_querier_blocks_meta_synced`, which reflects current state of synced blocks over all tenants. #2392

‎docs/configuration/config-file-reference.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -646,6 +646,19 @@ The `querier_config` configures the Cortex querier.
646646
# instances form a ring and addresses are picked from the ring).
647647
# CLI flag: -experimental.querier.store-gateway-addresses
648648
[store_gateway_addresses: <string> | default = ""]
649+
650+
store_gateway_client:
651+
# TLS cert path for the client
652+
# CLI flag: -experimental.querier.store-gateway-client.tls-cert-path
653+
[tls_cert_path: <string> | default = ""]
654+
655+
# TLS key path for the client
656+
# CLI flag: -experimental.querier.store-gateway-client.tls-key-path
657+
[tls_key_path: <string> | default = ""]
658+
659+
# TLS CA path for the client
660+
# CLI flag: -experimental.querier.store-gateway-client.tls-ca-path
661+
[tls_ca_path: <string> | default = ""]
649662
```
650663

651664
### `query_frontend_config`
@@ -757,6 +770,19 @@ The `ruler_config` configures the Cortex ruler.
757770
# CLI flag: -ruler.external.url
758771
[external_url: <url> | default = ]
759772
773+
ruler_client:
774+
# TLS cert path for the client
775+
# CLI flag: -ruler.client.tls-cert-path
776+
[tls_cert_path: <string> | default = ""]
777+
778+
# TLS key path for the client
779+
# CLI flag: -ruler.client.tls-key-path
780+
[tls_key_path: <string> | default = ""]
781+
782+
# TLS CA path for the client
783+
# CLI flag: -ruler.client.tls-ca-path
784+
[tls_ca_path: <string> | default = ""]
785+
760786
# How frequently to evaluate rules
761787
# CLI flag: -ruler.evaluation-interval
762788
[evaluation_interval: <duration> | default = 1m]
@@ -1964,6 +1990,18 @@ grpc_client_config:
19641990
# Number of times to backoff and retry before failing.
19651991
# CLI flag: -ingester.client.backoff-retries
19661992
[max_retries: <int> | default = 10]
1993+
1994+
# TLS cert path for the client
1995+
# CLI flag: -ingester.client.tls-cert-path
1996+
[tls_cert_path: <string> | default = ""]
1997+
1998+
# TLS key path for the client
1999+
# CLI flag: -ingester.client.tls-key-path
2000+
[tls_key_path: <string> | default = ""]
2001+
2002+
# TLS CA path for the client
2003+
# CLI flag: -ingester.client.tls-ca-path
2004+
[tls_ca_path: <string> | default = ""]
19672005
```
19682006

19692007
### `frontend_worker_config`
@@ -2025,6 +2063,18 @@ grpc_client_config:
20252063
# Number of times to backoff and retry before failing.
20262064
# CLI flag: -querier.frontend-client.backoff-retries
20272065
[max_retries: <int> | default = 10]
2066+
2067+
# TLS cert path for the client
2068+
# CLI flag: -querier.frontend-client.tls-cert-path
2069+
[tls_cert_path: <string> | default = ""]
2070+
2071+
# TLS key path for the client
2072+
# CLI flag: -querier.frontend-client.tls-key-path
2073+
[tls_key_path: <string> | default = ""]
2074+
2075+
# TLS CA path for the client
2076+
# CLI flag: -querier.frontend-client.tls-ca-path
2077+
[tls_ca_path: <string> | default = ""]
20282078
```
20292079

20302080
### `etcd_config`
@@ -2530,6 +2580,18 @@ The `configstore_config` configures the config database storing rules and alerts
25302580
# Timeout for requests to Weave Cloud configs service.
25312581
# CLI flag: -<prefix>.configs.client-timeout
25322582
[client_timeout: <duration> | default = 5s]
2583+
2584+
# TLS cert path for the client
2585+
# CLI flag: -<prefix>.configs.tls-cert-path
2586+
[tls_cert_path: <string> | default = ""]
2587+
2588+
# TLS key path for the client
2589+
# CLI flag: -<prefix>.configs.tls-key-path
2590+
[tls_key_path: <string> | default = ""]
2591+
2592+
# TLS CA path for the client
2593+
# CLI flag: -<prefix>.configs.tls-ca-path
2594+
[tls_ca_path: <string> | default = ""]
25332595
```
25342596

25352597
### `tsdb_config`

‎docs/configuration/single-process-config-blocks-tls.yaml‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
2+
# Configuration for running Cortex in single-process mode.
3+
# This should not be used in production. It is only for getting started
4+
# and development.
5+
6+
# Disable the requirement that every request to Cortex has a
7+
# X-Scope-OrgID header. `fake` will be substituted in instead.
8+
auth_enabled: false
9+
10+
server:
11+
http_listen_port: 9009
12+
13+
# Configure the server to allow messages up to 100MB.
14+
grpc_server_max_recv_msg_size: 104857600
15+
grpc_server_max_send_msg_size: 104857600
16+
grpc_server_max_concurrent_streams: 1000
17+
grpc_tls_config:
18+
cert_file: "server.crt"
19+
key_file: "server.key"
20+
client_auth_type: "RequireAndVerifyClientCert"
21+
client_ca_file: "root.crt"
22+
23+
24+
distributor:
25+
shard_by_all_labels: true
26+
pool:
27+
health_check_ingesters: true
28+
29+
ingester_client:
30+
grpc_client_config:
31+
# Configure the client to allow messages up to 100MB.
32+
max_recv_msg_size: 104857600
33+
max_send_msg_size: 104857600
34+
use_gzip_compression: true
35+
tls_cert_path: "client.crt"
36+
tls_key_path: "client.key"
37+
tls_ca_path: "root.crt"
38+
39+
ingester:
40+
# Disable blocks transfers on ingesters shutdown or rollout.
41+
max_transfer_retries: 0
42+
43+
lifecycler:
44+
# The address to advertise for this ingester. Will be autodiscovered by
45+
# looking up address on eth0 or en0; can be specified if this fails.
46+
# address: 127.0.0.1
47+
48+
# We want to start immediately and flush on shutdown.
49+
join_after: 0
50+
min_ready_duration: 0s
51+
final_sleep: 0s
52+
num_tokens: 512
53+
54+
# Use an in memory ring store, so we don't need to launch a Consul.
55+
ring:
56+
kvstore:
57+
store: inmemory
58+
replication_factor: 1
59+
60+
storage:
61+
engine: tsdb
62+
63+
tsdb:
64+
dir: /tmp/cortex/tsdb
65+
bucket_store:
66+
sync_dir: /tmp/cortex/tsdb-sync
67+
68+
# You can choose between local storage and Amazon S3, Google GCS and Azure storage. Each option requires additional configuration
69+
# as shown below. All options can be configured via flags as well which might be handy for secret inputs.
70+
backend: s3 # s3, gcs, azure or filesystem are valid options
71+
s3:
72+
bucket_name: cortex
73+
endpoint: s3.dualstack.us-east-1.amazonaws.com
74+
# Configure your S3 credentials below.
75+
# secret_access_key: "TODO"
76+
# access_key_id: "TODO"
77+
# gcs:
78+
# bucket_name: cortex
79+
# service_account: # if empty or omitted Cortex will use your default service account as per Google's fallback logic
80+
# azure:
81+
# account_name:
82+
# account_key:
83+
# container_name:
84+
# endpoint_suffix:
85+
# max_retries: # Number of retries for recoverable errors (defaults to 20)
86+
# filesystem:
87+
# dir: ./data/tsdb
88+
89+
compactor:
90+
data_dir: /tmp/cortex/compactor
91+
sharding_ring:
92+
kvstore:
93+
store: inmemory
94+
95+
frontend_worker:
96+
match_max_concurrent: true
97+
grpc_client_config:
98+
tls_cert_path: "client.crt"
99+
tls_key_path: "client.key"
100+
tls_ca_path: "root.crt"

‎docs/production/tls.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
---
2+
title: "Securing communication between Cortex components with TLS"
3+
linkTitle: "Securing communication between Cortex components with TLS"
4+
weight: 5
5+
slug: tls
6+
---
7+
8+
Cortex is a distributed system with significant traffic between its services.
9+
To allow for secure communication, Cortex supports TLS between all its
10+
components. This guide describes the process of setting up TLS.
11+
12+
### Generation of certs to configure TLS
13+
14+
The first step to securing inter-service communication in Cortex with TLS is
15+
generating certificates. A Certifying Authority (CA) will be used for this
16+
purpose which should be private to the organization, as any certificates signed
17+
by this CA will have permissions to communicate with the cluster.
18+
19+
We will use the following script to generate self signed certs for the cluster:
20+
21+
```
22+
# Refer: github.com/cortexproject/cortex/integration/certs/genCerts.sh
23+
24+
# keys
25+
openssl genrsa -out root.key
26+
openssl genrsa -out client.key
27+
openssl genrsa -out server.key
28+
29+
# root cert / certifying authority
30+
openssl req -x509 -new -nodes -key root.key -subj "/C=US/ST=KY/O=Org/CN=root" -sha256 -days 100000 -out root.crt
31+
32+
# csrs - certificate signing requests
33+
openssl req -new -sha256 -key client.key -subj "/C=US/ST=KY/O=Org/CN=client" -out client.csr
34+
openssl req -new -sha256 -key server.key -subj "/C=US/ST=KY/O=Org/CN=localhost" -out server.csr
35+
36+
# certificates
37+
openssl x509 -req -in client.csr -CA root.crt -CAkey root.key -CAcreateserial -out client.crt -days 100000 -sha256
38+
openssl x509 -req -in server.csr -CA root.crt -CAkey root.key -CAcreateserial -out server.crt -days 100000 -sha256
39+
```
40+
41+
Note that the above script generates certificates that are valid for 100000 days.
42+
This can be changed by adjusting the `-days` option in the above commands.
43+
It is recommended that the certs be replaced atleast once every 2 years.
44+
45+
The above script generates keys `client.key, server.key` and certs
46+
`client.crt, server.crt` for both the client and server. The CA cert is
47+
generated as `root.crt`.
48+
49+
### Load certs into the HTTP/GRPC server/client
50+
51+
Every HTTP/GRPC link between Cortex components supports TLS configuration
52+
through the following config parameters:
53+
54+
#### Server flags
55+
56+
```
57+
# Path to the TLS Cert for the HTTP Server
58+
-server.http-tls-cert-path=/path/to/server.crt
59+
60+
# Path to the TLS Key for the HTTP Server
61+
-server.http-tls-key-path=/path/to/server.key
62+
63+
# Type of Client Auth for the HTTP Server
64+
-server.http-tls-client-auth="RequireAndVerifyClientCert"
65+
66+
# Path to the Client CA Cert for the HTTP Server
67+
-server.http-tls-ca-path="/path/to/root.crt"
68+
69+
# Path to the TLS Cert for the GRPC Server
70+
-server.grpc-tls-cert-path=/path/to/server.crt
71+
72+
# Path to the TLS Key for the GRPC Server
73+
-server.grpc-tls-key-path=/path/to/server.key
74+
75+
# Type of Client Auth for the GRPC Server
76+
-server.grpc-tls-client-auth="RequireAndVerifyClientCert"
77+
78+
# Path to the Client CA Cert for the GRPC Server
79+
-server.grpc-tls-ca-path=/path/to/root.crt
80+
```
81+
82+
#### Client flags
83+
84+
Client flags are component specific.
85+
86+
For an HTTP client in the Alertmanager:
87+
```
88+
# Path to the TLS Cert for the HTTP Client
89+
-alertmanager.configs.tls-cert-path=/path/to/client.crt
90+
91+
# Path to the TLS Key for the HTTP Client
92+
-alertmanager.configs.tls-key-path=/path/to/client.key
93+
94+
# Path to the TLS CA for the HTTP Client
95+
-alertmanager.configs.tls-ca-path=/path/to/root.crt
96+
```
97+
98+
For a GRPC client in the Querier:
99+
```
100+
# Path to the TLS Cert for the GRPC Client
101+
-querier.frontend-client.tls-cert-path=/path/to/client.crt
102+
103+
# Path to the TLS Key for the GRPC Client
104+
-querier.frontend-client.tls-key-path=/path/to/client.key
105+
106+
# Path to the TLS CA for the GRPC Client
107+
-querier.frontend-client.tls-ca-path=/path/to/root.crt
108+
```
109+
110+
TLS can be configured in a similar fashion for other GRPC clients like the
111+
ingester client.

‎integration/certs/genCerts.sh‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
#!/usr/bin/env bash
2+
# Copied from https://github.com/joe-elliott/cert-exporter/blob/5ce49ebf6bfcdcb178d31145ae2a460f3b348cf5/test/files/genCerts.sh
3+
# Copyright [2020] [cert-exporter authors]
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
certFolder=$1
18+
days=$2
19+
20+
pushd $certFolder
21+
22+
# keys
23+
openssl genrsa -out root.key
24+
openssl genrsa -out client.key
25+
openssl genrsa -out server.key
26+
27+
# root cert
28+
openssl req -x509 -new -nodes -key root.key -subj "/C=US/ST=KY/O=Org/CN=root" -sha256 -days $days -out root.crt
29+
30+
# csrs
31+
openssl req -new -sha256 -key client.key -subj "/C=US/ST=KY/O=Org/CN=client" -out client.csr
32+
openssl req -new -sha256 -key server.key -subj "/C=US/ST=KY/O=Org/CN=localhost" -out server.csr
33+
34+
openssl x509 -req -in client.csr -CA root.crt -CAkey root.key -CAcreateserial -out client.crt -days $days -sha256
35+
openssl x509 -req -in server.csr -CA root.crt -CAkey root.key -CAcreateserial -out server.crt -days $days -sha256
36+
37+
popd

‎integration/configs.go‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,16 +22,21 @@ const (
2222
cortexConfigFile = "config.yaml"
2323
cortexSchemaConfigFile = "schema.yaml"
2424
blocksStorageEngine = "tsdb"
25+
clientCertFile = "certs/client.crt"
26+
clientKeyFile = "certs/client.key"
27+
caCertFile = "certs/root.crt"
28+
serverCertFile = "certs/server.crt"
29+
serverKeyFile = "certs/server.key"
2530
storeConfigTemplate = `
2631
- from: {{.From}}
2732
store: {{.IndexStore}}
2833
schema: v9
2934
index:
3035
prefix: cortex_
31-
period: 168h
36+
period: 168h
3237
chunks:
3338
prefix: cortex_chunks_
34-
period: 168h
39+
period: 168h
3540
`
3641

3742
cortexAlertmanagerUserConfigYaml = `route:
@@ -120,7 +125,7 @@ storage:
120125
121126
table_manager:
122127
poll_interval: 1m
123-
retention_period: 168h
128+
retention_period: 168h
124129
125130
schema:
126131
{{.SchemaConfig}}

0 commit comments

Comments
 (0)