Prevent credential Secret drift in DBaaS Harvester VM provisioning

[Yohansenanayake]

Description:

The DBaaS operator currently has a VM provisioning edge case where the credentials/cloud-init Secret can be created before the Harvester VirtualMachineImage is resolved and validated.

This can lead to two related problems:

1. If image resolution fails after the Secret is created, the Secret may be left behind before status.resources.secretName is recorded.
2. On a later reconcile retry, the operator may generate a fresh CA/password set, ignore AlreadyExists for the existing Secret, and return the newly generated CA. The existing Secret still contains the original CA, so status.caCertPem can drift from the Secret actually consumed by the VM.

This issue was identified during review of the Harvester VM creation path in the DBaaS operator.

Affected area:

• database/internal/harvester/client.go
• CreatePostgresVM
• Credentials/cloud-init Secret creation
• DBInstance status CA reporting

Expected behavior:

• The operator should resolve and validate the Harvester VM image before creating the credentials Secret.
• If the credentials Secret already exists, the operator should treat it as the source of truth and return the stored ca_cert.
• If the existing Secret is malformed and does not contain ca_cert, VM creation should fail clearly instead of publishing unrelated newly generated CA material.

Proposed fix:

• Move resolveVMImage(ctx, p.OSImage) before credential generation and Secret creation.
• Add create-or-reuse behavior for the credentials Secret.
• When the Secret already exists, read and return data.ca_cert.
• Do not overwrite, rotate, or repair existing credentials as part of this fix.
• Add unit tests for:
  • image resolution failure does not create a Secret
  • first successful create returns the CA stored in the Secret
  • retry with existing Secret reuses existing CA
  • malformed existing Secret fails before VM creation
  • existing Kube-OVN VM settings remain preserved

Validation:

• Unit tests should pass for internal/harvester.
• Full make test should pass after envtest setup.
• Manual Harvester validation:
  • creating a DBInstance with an invalid image should not create pg-<name>-credentials
  • creating a DBInstance with a valid image should create a credentials Secret
  • .status.caCertPem should match Secret.data.ca_cert

[Akini98]

Thanks for the analysis @Yohansenanayake

On the proposed fix (Layer 1)

The reorder + create-or-reuse approach is the right fix for both problems. Could you raise a PR for that? @Yohansenanayake

To summarise what it covers for anyone following along:

• Move resolveVMImage before credential generation — image resolution failure no longer leaves an orphaned Secret
• Create-or-reuse the Secret on retry — returns the CA cert already stored in the existing Secret instead of generating a new one, eliminating the status.caCertPem drift

Follow-up: Secret lifecycle improvements (Layer 2+3)

Once that PR lands I will raise a follow-up that addresses two related structural issues:

The current Secret bundles cloud-init bootstrap data (userdata, networkdata) with user-facing DB credentials in one object. These have different lifetimes and different access requirements. The below suggested fix splits them:

• pg--cloudinit — userdata + networkdata, referenced by the VM at boot
• pg--credentials — admin_user, admin_password, ca_cert (public only, no ca_key), endpoint

Both Secrets are currently long-lived with no cleanup. Suggested approach is as mentioned below.

• pg--cloudinit is deleted once the VM reaches Available (cloud-init runs once; keeping it is unnecessary exposure)
• pg--credentials is deleted after the shown-once credentials fetch (makes the shown-once guarantee hold at the Kubernetes layer, not just the API layer)

This can be tracked as a separate issue if that's the preference, or keep it as a follow-up comment here.