terraform: API should support idempotency keys for non-idempotent create endpoints

[h4x3rotab]

Context

The Terraform provider drives app creation and scaling through non-idempotent POST endpoints, notably:

• POST /apps/{id}/cvms/{vm_uuid}/replicas
• POST /cvms (commit a provisioned app/CVM create)

Without an API-level deduplication mechanism, a client cannot safely retry those requests after a transient 409/429/503 or after losing the response.

Problem

The same logical operation can be applied more than once if the request reaches the backend but the client sees a retryable failure or disconnect:

• replica creation can overshoot the desired count
• initial app/CVM creation can create duplicates or leave the client unsure whether creation succeeded

The provider now avoids retrying non-idempotent create POSTs to prevent duplicate resources, but that shifts the failure mode into hard apply failures on transient backend errors. In other words, client-side caution avoids corruption, but we still lack a way to recover cleanly.

Proposed fix

Add support for an optional Idempotency-Key header (or equivalent request field) on non-idempotent create endpoints, starting with:

• POST /apps/{id}/cvms/{vm_uuid}/replicas
• POST /cvms

Suggested semantics:

• identical key + same authenticated caller + same endpoint/payload shape within a bounded window returns the original result instead of creating a second resource
• conflicting payload for the same key returns a clear client error
• response should include enough identity (app_id, vm_uuid, resource id) for the caller to reconcile state

Why this matters

This would let Terraform and other clients safely retry after transient infrastructure failures without risking duplicate creates.

Current workaround

The provider side now:

• retries replay-safe writes such as PATCH, DELETE, and action-style POSTs like /start and /stop
• does not retry non-idempotent create POSTs
• still pre-counts replicas before scaling, which helps normal sequential applies but does not solve lost-response/concurrent-request races

That means transient failures during create/scale remain user-visible until the API provides idempotent create semantics.

References

• PR feat(terraform): introduce Phala Cloud Terraform provider (app-first beta) #195 review
• terraform/internal/provider/resource_app.go reconcileReplicas()
• terraform/internal/provider/resource_shared.go commitAndCreate()

[Leechael]

Investigation Summary

Reviewed the API behavior and the Terraform provider to assess the actual risk surface.

Terraform Provider Current Behavior

The provider already has a retry system (max 30 attempts, exponential backoff) with a whitelist controlling which operations are safe to retry:

• Retried: POST /cvms/provision, POST /start, POST /stop, all PATCH, all DELETE
• Not retried: POST /cvms (commit), POST /replicas — explicitly excluded as non-idempotent

Per-Endpoint Assessment

POST /cvms (commit) — Partially a real problem

• PHALA KMS: The backend already has deduplication logic for app_id. Retrying the same app_id won't create a duplicate CVM, but the error response is unhelpful — it rejects the retry instead of returning the existing resource. Users must manually terraform import to recover.
• Ethereum/Base KMS: No deduplication. Retry will create an orphan CVM. This is a real bug.

POST /replicas — Not a real problem in practice

The provider's reconcileReplicas() always fetches the current replica count before deciding whether to create more. If a replica was created but the response was lost, the next terraform apply sees the correct count and skips. The provider-side reconciliation already handles idempotency.

PATCH /cvms/{id} — Not a real problem

The provider already retries all PATCH operations. The backend rejects concurrent operations on the same CVM with 409, preventing duplicate side effects.

Conclusion

The actual bug surface is narrower than initially described:

1. Real bug: POST /cvms with Ethereum/Base KMS can create orphan CVMs on lost response
2. UX issue: POST /cvms with PHALA KMS rejects retries with an unhelpful error instead of returning the existing CVM
3. Not a problem: Replica creation (provider-side reconciliation) and PATCH operations (already retried with concurrency guard)

Design Options

Option A: Client-side Idempotency-Key / client_token

• Provider generates a UUID per operation, reuses on retry
• API stores key and returns existing result if found
• Pro: Precise — distinguishes "retry" from "new operation with same params"
• Con: Requires Terraform provider changes

Option B: Server-side state-based dedup

• POST /cvms with same app_id + compose_hash detects the existing CVM and returns it
• PATCH returns current operation status instead of 409 when an operation is already in progress
• Pro: Zero Terraform provider changes needed
• Con: Cannot distinguish "retry" from "intentionally deploying same config as a new resource"

[Leechael]

API Behavior Update

POST /cvms now supports idempotent creation. When the same app_id + compose_hash is submitted within the same workspace:

• Same config: Returns the existing CVM (safe to retry after timeout/disconnect)
• Different compose_hash: Returns 409 with ERR-03-007 indicating the app_id is already in use with a different configuration — provision again to get a new app_id
• Previously deleted CVM: Allows new creation as normal

This means the Terraform provider can safely add POST /cvms to its retry whitelist. No client-side changes are required for the fix to take effect, but enabling retries on the provider side would improve resilience.

POST /replicas was also reviewed — reconcileReplicas() already counts existing replicas before creating, so it is not affected by the lost-response problem.

[Leechael]

Update: Verified Behavior

The fix has been manually tested on a local environment. Here's the exact behavior:

Idempotent retry (same app_id + same compose_hash):

POST /cvms {app_id, compose_hash}  →  200, returns existing CVM

No re-provision needed. The check runs before the provision cache lookup, so retries work even after the cache expires or is deleted.

Conflict (same app_id + different compose_hash):

POST /cvms {app_id, different_hash}  →  409, ERR-03-007

With message: "This app_id already has an active CVM with a different configuration."

After deletion (same app_id after CVM is deleted):

DELETE /cvms/{id}
POST /cvms {app_id, compose_hash}  →  200, creates new CVM

The check is scoped to (app_id, workspace) — different workspaces can independently deploy the same app_id (relevant for on-chain KMS where multiple workspaces may use the same contract).