Support additional image registries with self-signed CA certificates

[DanielXiao]

Users could have application images in private registries with self-signed CA certificates, this patch configure containerd TLS settings for them.

For image registries with a public CA certificates, no additional configurations are required for containerd.

What this PR does / why we need it

Which issue(s) this PR fixes

Fixes #

Describe testing done for PR

Tested with the following legacy configurations:

ADDITIONAL_IMAGE_REGISTRY_1: "10.92.127.192:8443"
ADDITIONAL_IMAGE_REGISTRY_1_CA_CERTIFICATE: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURMekNDQWhlZ0F3SUJBZ0lVWlRFSTgrZWFhK3dQdHdqUnpBQ0RoNmxSSzR3d0RRWUpLb1pJaHZjTkFRRUwKQlFBd1J6RUxNQWtHQTFVRUJoTUNWVk14RXpBUkJnTlZCQWdNQ2tOaGJHbG1iM0p1YVdFeEVqQVFCZ05WQkFjTQpDVkJoYkc4Z1FXeDBiekVQTUEwR0ExVUVDZ3dHZG0xM1lYSmxNQjRYRFRJek1ETXdOekE0TkRReE9Gb1hEVEkwCk1ETXdPREE0TkRReE9Gb3dSekVMTUFrR0ExVUVCaE1DVlZNeEV6QVJCZ05WQkFnTUNrTmhiR2xtYjNKdWFXRXgKRWpBUUJnTlZCQWNNQ1ZCaGJHOGdRV3gwYnpFUE1BMEdBMVVFQ2d3R2RtMTNZWEpsTUlJQklqQU5CZ2txaGtpRwo5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBempobWh4TXJTRjlPZ3ZmUGFOUEpObGNaWWNJRmU0bk9yMHhDClBSRVQyMGxCc3JWRUF5a2NGRThUaTZGV3ZZNFBPL0VOODJYdXVZU0JhOEMwQ3ZSK0hjaEhmZDJDbTc0KzZNSkEKT3RqdmszV1RzU0hlWjkxbHVEY0dvQ09XeGFENHdyaCtjRnhtS0FscnNpM3RYcUZiMkQyZHk5c0U5dVBTUEltQwpUR2dRQWdZTTFRM0RSRUNqNjUxRUYxVE9ySXRmYzdjeVNUWXpvOHZYbWtNYXBaVmdtdHFoM01UQ3VHSUdpTXhqCnBsb2kvR3RhclZzSHFmSEJCalBPYTVZOHpuNGgySm55SWJlV0JoYWlQeVpRK3BhWmQ3U2U0cDZ0RUpTVzFqOFkKK2pkbW9PUGE3QmQ1K2tHclprb1Z4cTZ3emFXQkl1czE1QnRlb2ZTZVl2d0JlYS9ha1FJREFRQUJveE13RVRBUApCZ05WSFJNQkFmOEVCVEFEQVFIL01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ1YvSmRZbVZWSjIvbVY4ZmF6CkhieVlwTUVkNXBJZHdyY1VRNzFpM2ZRZDZBUndZQjdwKys0R1J0NDdRZ01PUU80VUliKzlORXZCcTA3Q0lBKzcKdHI0MFpOYVdEV3BtRW1Fb1RBS3l6cHV4TWJBMHdJdURLSGo4VmhOUTErS29GWG8xU0RyVnZpNlYrcWk0WkVaYgovNm9mdmd0a1YrajFpa0ZkRWxYY0VlNkE2dUpHMUJ0QTdGK0QrSUMxRCtaclIyc3dYQWxuOVJlUXBhOXUvMHVWCnZ3TkFPUEdpR3BnTXdYaXJSVy8rR3JML0Y2RmNQMUZqcHo3aUxXOFZlWGQ1SWY0cVpkc0FqVGhWcEZtdCtQeDcKanNOUEV5RGw3V2hSY0oxbHlVSXlZTm9XUVBqUDBSMjZzaStWbUM3UDBnSkp5a2NuWCs5eUliQ21RbmFrUGx4WgpqV0h2Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
ADDITIONAL_IMAGE_REGISTRY_2: "10.191.157.222:8443"
ADDITIONAL_IMAGE_REGISTRY_2_SKIP_TLS_VERIFY: true

Tanzu cli generated variable in the cluster manifest

    - name: additionalImageRegistries
      value:
      - caCert: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURMekNDQWhlZ0F3SUJBZ0lVWlRFSTgrZWFhK3dQdHdqUnpBQ0RoNmxSSzR3d0RRWUpLb1pJaHZjTkFRRUwKQlFBd1J6RUxNQWtHQTFVRUJoTUNWVk14RXpBUkJnTlZCQWdNQ2tOaGJHbG1iM0p1YVdFeEVqQVFCZ05WQkFjTQpDVkJoYkc4Z1FXeDBiekVQTUEwR0ExVUVDZ3dHZG0xM1lYSmxNQjRYRFRJek1ETXdOekE0TkRReE9Gb1hEVEkwCk1ETXdPREE0TkRReE9Gb3dSekVMTUFrR0ExVUVCaE1DVlZNeEV6QVJCZ05WQkFnTUNrTmhiR2xtYjNKdWFXRXgKRWpBUUJnTlZCQWNNQ1ZCaGJHOGdRV3gwYnpFUE1BMEdBMVVFQ2d3R2RtMTNZWEpsTUlJQklqQU5CZ2txaGtpRwo5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBempobWh4TXJTRjlPZ3ZmUGFOUEpObGNaWWNJRmU0bk9yMHhDClBSRVQyMGxCc3JWRUF5a2NGRThUaTZGV3ZZNFBPL0VOODJYdXVZU0JhOEMwQ3ZSK0hjaEhmZDJDbTc0KzZNSkEKT3RqdmszV1RzU0hlWjkxbHVEY0dvQ09XeGFENHdyaCtjRnhtS0FscnNpM3RYcUZiMkQyZHk5c0U5dVBTUEltQwpUR2dRQWdZTTFRM0RSRUNqNjUxRUYxVE9ySXRmYzdjeVNUWXpvOHZYbWtNYXBaVmdtdHFoM01UQ3VHSUdpTXhqCnBsb2kvR3RhclZzSHFmSEJCalBPYTVZOHpuNGgySm55SWJlV0JoYWlQeVpRK3BhWmQ3U2U0cDZ0RUpTVzFqOFkKK2pkbW9PUGE3QmQ1K2tHclprb1Z4cTZ3emFXQkl1czE1QnRlb2ZTZVl2d0JlYS9ha1FJREFRQUJveE13RVRBUApCZ05WSFJNQkFmOEVCVEFEQVFIL01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ1YvSmRZbVZWSjIvbVY4ZmF6CkhieVlwTUVkNXBJZHdyY1VRNzFpM2ZRZDZBUndZQjdwKys0R1J0NDdRZ01PUU80VUliKzlORXZCcTA3Q0lBKzcKdHI0MFpOYVdEV3BtRW1Fb1RBS3l6cHV4TWJBMHdJdURLSGo4VmhOUTErS29GWG8xU0RyVnZpNlYrcWk0WkVaYgovNm9mdmd0a1YrajFpa0ZkRWxYY0VlNkE2dUpHMUJ0QTdGK0QrSUMxRCtaclIyc3dYQWxuOVJlUXBhOXUvMHVWCnZ3TkFPUEdpR3BnTXdYaXJSVy8rR3JML0Y2RmNQMUZqcHo3aUxXOFZlWGQ1SWY0cVpkc0FqVGhWcEZtdCtQeDcKanNOUEV5RGw3V2hSY0oxbHlVSXlZTm9XUVBqUDBSMjZzaStWbUM3UDBnSkp5a2NuWCs5eUliQ21RbmFrUGx4WgpqV0h2Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
        host: 10.92.127.192:8443
        skipTlsVerify: false
      - host: 10.191.157.222:8443
        skipTlsVerify: true

After cluster is ready, launch Pods whose images are from above 2 registries:

❯ kubectl run nginx --image=10.92.127.192:8443/library/harbor/nginx-photon:v2.6.3_vmware.1  --restart=Never
pod/nginx created
❯ kubectl run nginx-2 --image=10.191.157.222:8443/library/harbor/nginx-photon:v2.6.3_vmware.1  --restart=Never
pod/nginx-2 created

Log into a node to do validation:

root [ /home/capv ]# crictl images
IMAGE                                                                TAG                 IMAGE ID            SIZE
10.191.157.222:8443/library/harbor/nginx-photon                      v2.6.3_vmware.1     4652f583ba2fe       60.4MB
10.92.127.192:8443/library/harbor/nginx-photon                       v2.6.3_vmware.1     4652f583ba2fe       60.4MB
...

root [ /home/capv ]# cat /tmp/insert_registry_ca_certs.sh
#!/bin/bash
set -e
echo 'LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURMekNDQWhlZ0F3SUJBZ0lVWlRFSTgrZWFhK3dQdHdqUnpBQ0RoNmxSSzR3d0RRWUpLb1pJaHZjTkFRRUwKQlFBd1J6RUxNQWtHQTFVRUJoTUNWVk14RXpBUkJnTlZCQWdNQ2tOaGJHbG1iM0p1YVdFeEVqQVFCZ05WQkFjTQpDVkJoYkc4Z1FXeDBiekVQTUEwR0ExVUVDZ3dHZG0xM1lYSmxNQjRYRFRJek1ETXdOekE0TkRReE9Gb1hEVEkwCk1ETXdPREE0TkRReE9Gb3dSekVMTUFrR0ExVUVCaE1DVlZNeEV6QVJCZ05WQkFnTUNrTmhiR2xtYjNKdWFXRXgKRWpBUUJnTlZCQWNNQ1ZCaGJHOGdRV3gwYnpFUE1BMEdBMVVFQ2d3R2RtMTNZWEpsTUlJQklqQU5CZ2txaGtpRwo5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBempobWh4TXJTRjlPZ3ZmUGFOUEpObGNaWWNJRmU0bk9yMHhDClBSRVQyMGxCc3JWRUF5a2NGRThUaTZGV3ZZNFBPL0VOODJYdXVZU0JhOEMwQ3ZSK0hjaEhmZDJDbTc0KzZNSkEKT3RqdmszV1RzU0hlWjkxbHVEY0dvQ09XeGFENHdyaCtjRnhtS0FscnNpM3RYcUZiMkQyZHk5c0U5dVBTUEltQwpUR2dRQWdZTTFRM0RSRUNqNjUxRUYxVE9ySXRmYzdjeVNUWXpvOHZYbWtNYXBaVmdtdHFoM01UQ3VHSUdpTXhqCnBsb2kvR3RhclZzSHFmSEJCalBPYTVZOHpuNGgySm55SWJlV0JoYWlQeVpRK3BhWmQ3U2U0cDZ0RUpTVzFqOFkKK2pkbW9PUGE3QmQ1K2tHclprb1Z4cTZ3emFXQkl1czE1QnRlb2ZTZVl2d0JlYS9ha1FJREFRQUJveE13RVRBUApCZ05WSFJNQkFmOEVCVEFEQVFIL01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ1YvSmRZbVZWSjIvbVY4ZmF6CkhieVlwTUVkNXBJZHdyY1VRNzFpM2ZRZDZBUndZQjdwKys0R1J0NDdRZ01PUU80VUliKzlORXZCcTA3Q0lBKzcKdHI0MFpOYVdEV3BtRW1Fb1RBS3l6cHV4TWJBMHdJdURLSGo4VmhOUTErS29GWG8xU0RyVnZpNlYrcWk0WkVaYgovNm9mdmd0a1YrajFpa0ZkRWxYY0VlNkE2dUpHMUJ0QTdGK0QrSUMxRCtaclIyc3dYQWxuOVJlUXBhOXUvMHVWCnZ3TkFPUEdpR3BnTXdYaXJSVy8rR3JML0Y2RmNQMUZqcHo3aUxXOFZlWGQ1SWY0cVpkc0FqVGhWcEZtdCtQeDcKanNOUEV5RGw3V2hSY0oxbHlVSXlZTm9XUVBqUDBSMjZzaStWbUM3UDBnSkp5a2NuWCs5eUliQ21RbmFrUGx4WgpqV0h2Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K' | base64 -d > /etc/containerd/10.92.127.192:8443.crt
echo '[plugins."io.containerd.grpc.v1.cri".registry.configs."10.92.127.192:8443".tls]' >> /etc/containerd/config.toml
echo '  ca_file = "/etc/containerd/10.92.127.192:8443.crt"' >> /etc/containerd/config.toml
echo '[plugins."io.containerd.grpc.v1.cri".registry.configs."10.191.157.222:8443".tls]' >> /etc/containerd/config.toml
echo "  insecure_skip_verify = true" >> /etc/containerd/config.toml

root [ /home/capv ]# cat /etc/containerd/config.toml
## template: jinja

# Use config version 2 to enable new configuration fields.
# Config file is parsed as version 1 by default.
version = 2

imports = ["/etc/containerd/conf.d/*.toml"]

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "projects.registry.vmware.com/tkg/pause:3.8"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
    runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".registry.configs."10.92.127.192:8443".tls]
  ca_file = "/etc/containerd/10.92.127.192:8443.crt"
[plugins."io.containerd.grpc.v1.cri".registry.configs."10.191.157.222:8443".tls]
  insecure_skip_verify = true

root [ /home/capv ]# cat /etc/containerd/10.92.127.192:8443.crt
-----BEGIN CERTIFICATE-----
MIIDLzCCAhegAwIBAgIUZTEI8+eaa+wPtwjRzACDh6lRK4wwDQYJKoZIhvcNAQEL
BQAwRzELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExEjAQBgNVBAcM
CVBhbG8gQWx0bzEPMA0GA1UECgwGdm13YXJlMB4XDTIzMDMwNzA4NDQxOFoXDTI0
MDMwODA4NDQxOFowRzELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWEx
EjAQBgNVBAcMCVBhbG8gQWx0bzEPMA0GA1UECgwGdm13YXJlMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzjhmhxMrSF9OgvfPaNPJNlcZYcIFe4nOr0xC
PRET20lBsrVEAykcFE8Ti6FWvY4PO/EN82XuuYSBa8C0CvR+HchHfd2Cm74+6MJA
Otjvk3WTsSHeZ91luDcGoCOWxaD4wrh+cFxmKAlrsi3tXqFb2D2dy9sE9uPSPImC
TGgQAgYM1Q3DRECj651EF1TOrItfc7cySTYzo8vXmkMapZVgmtqh3MTCuGIGiMxj
ploi/GtarVsHqfHBBjPOa5Y8zn4h2JnyIbeWBhaiPyZQ+paZd7Se4p6tEJSW1j8Y
+jdmoOPa7Bd5+kGrZkoVxq6wzaWBIus15BteofSeYvwBea/akQIDAQABoxMwETAP
BgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQCV/JdYmVVJ2/mV8faz
HbyYpMEd5pIdwrcUQ71i3fQd6ARwYB7p++4GRt47QgMOQO4UIb+9NEvBq07CIA+7
tr40ZNaWDWpmEmEoTAKyzpuxMbA0wIuDKHj8VhNQ1+KoFXo1SDrVvi6V+qi4ZEZb
/6ofvgtkV+j1ikFdElXcEe6A6uJG1BtA7F+D+IC1D+ZrR2swXAln9ReQpa9u/0uV
vwNAOPGiGpgMwXirRW/+GrL/F6FcP1Fjpz7iLW8VeXd5If4qZdsAjThVpFmt+Px7
jsNPEyDl7WhRcJ1lyUIyYNoWQPjP0R26si+VmC7P0gJJykcnX+9yIbCmQnakPlxZ
jWHv
-----END CERTIFICATE-----

Release note

Support trusting self-signed image registry for user's application. Users can configure 3 registries with a legacy config file:
ADDITIONAL_IMAGE_REGISTRY_1: ""
ADDITIONAL_IMAGE_REGISTRY_1_SKIP_TLS_VERIFY: false
#! Base64 encoded self-signed CA certificate
ADDITIONAL_IMAGE_REGISTRY_1_CA_CERTIFICATE: ""

ADDITIONAL_IMAGE_REGISTRY_2: ""
ADDITIONAL_IMAGE_REGISTRY_2_SKIP_TLS_VERIFY: false
ADDITIONAL_IMAGE_REGISTRY_2_CA_CERTIFICATE: ""

ADDITIONAL_IMAGE_REGISTRY_3: ""
ADDITIONAL_IMAGE_REGISTRY_3_SKIP_TLS_VERIFY: false
ADDITIONAL_IMAGE_REGISTRY_3_CA_CERTIFICATE: ""

If there are more than 3, user can add them to generated cluster manifest variable additionalImageRegistries before creating the cluster.

Additional information

Special notes for your reviewer

[github-actions]

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4467/20230309102359/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

[codecov]

Codecov Report

Merging #4467 (62f7840) into main (3ae5649) will decrease coverage by 0.86%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #4467      +/-   ##
==========================================
- Coverage   49.67%   48.82%   -0.86%     
==========================================
  Files         452      482      +30     
  Lines       45105    47272    +2167     
==========================================
+ Hits        22406    23079     +673     
- Misses      20577    22018    +1441     
- Partials     2122     2175      +53     

see 37 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[github-actions]

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4467/20230310080114/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

[github-actions]

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4467/20230310084512/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

[alfredthenarwhal]

Created cherry-pick PR #4479