[Bug]: Agent runtime multica CLI fails TLS certificate verification (x509 OSStatus -26276) connecting to API

[forrestchang]

Summary

The agent-side multica CLI intermittently fails TLS certificate verification and cannot reach the API, halting the entire agent loop (cannot read issues, comment, or change status).

Error: resolve issue: Get "https://multica-api.agentic.mioffice.cn/api/issues/<id>":
tls: failed to verify certificate: x509: OSStatus -26276

-26276 = Apple Security.framework errSecInvalidExtendedKeyUsage. The released CLI is built CGO_ENABLED=0, but on darwin Go still routes system-root verification to Security.framework, so the OSStatus surfaces. The literal OSStatus token proves the rejection comes from Apple's SecPolicyCreateSSL server policy, which requires the leaf to assert id-kp-serverAuth EKU.

Environment

• Platform: macOS Darwin 23.3.0
• Domain: multica-api.agentic.mioffice.cn
• Error code: x509: OSStatus -26276 (errSecInvalidExtendedKeyUsage)
• Affected surface: agent runtime / multica CLI HTTP client

Impact

• Agent cannot read issues, post comments, or run any multica CLI operation.
• Intermittent — recovers for a few minutes, then fails again — severely disrupting agent continuity.
• The same command works on the user's own terminal; only the agent runtime/sandbox is affected.

Timeline

• Started intermittently on 2026-06-17.
• Persistent failures 2026-06-18 ~ 2026-06-19; consecutive retries all fail when it triggers.

Root cause (most likely)

"Works for the user, fails only in the sandbox, intermittent" points away from a broken origin cert and toward a TLS-inspection middlebox in the sandbox's egress path (mioffice.cn = corporate network) re-signing the leaf with a serverAuth-deficient cert from some of its fleet nodes. Less likely but possible: missing intermediate/root in the sandbox keychain, or mixed-config LB backends.

Current code state

Both the CLI (server/internal/cli/client.go) and the daemon (server/internal/daemon/client.go) build a plain &http.Client{Timeout: …} with no custom Transport/tls.Config — there is currently no way to supply a CA bundle or relax verification for API traffic. The only InsecureSkipVerify in the tree is the server-side SMTP sender, unrelated.

Proposed remediation

Env-gated knobs on the CLI/daemon HTTP client, following the existing MULTICA_* convention (MULTICA_HTTP_TIMEOUT):

1. MULTICA_CA_BUNDLE=/path/to/roots.pem — add a custom RootCAs pool; fully verifying (chain + hostname + EKU stay on). Clean fix for a missing corporate/intermediate CA.
2. MULTICA_TLS_INSECURE=1 — emergency-only InsecureSkipVerify, off by default, loudly warned. Restores continuity as a last resort.
3. (Optional, under evaluation) relax only the serverAuth-EKU requirement (VerifyOptions.KeyUsages = ExtKeyUsageAny) while keeping hostname + chain trust — strictly safer than full insecure, targets this exact error.

Durable fix: correct the offending cert's EKU at source, or exempt *.agentic.mioffice.cn from corporate TLS inspection, then drop the escape hatch from the sandbox image.

Diagnostics to confirm cause

Run from inside the sandbox and compare with the user's terminal:

openssl s_client -connect multica-api.agentic.mioffice.cn:443 \
  -servername multica-api.agentic.mioffice.cn -showcerts </dev/null 2>/dev/null | tee /tmp/chain.pem
openssl x509 -in /tmp/chain.pem -noout -text | grep -A3 -E "Extended Key Usage|Issuer:|Subject:"
# Compare leaf SHA-256 fingerprint sandbox-vs-user; differing fingerprint => MITM/proxy substitution
# scutil --proxies and `env | grep -i proxy` to detect a forced inspecting proxy

---

Tracked internally as Multica MUL-3553.

[murbern611]

Fix Plan

Thanks for the thorough analysis! I've implemented the three env-gated escape hatches you proposed. The code is ready.

Approach

Created a shared server/internal/util/httptransport.go that provides NewHTTPClient(timeout) and TLSConfig(). The CLI HTTP client, daemon HTTP client, and WebSocket dialer all use these functions instead of bare &http.Client{Timeout: ...}.

Three environment variables

Variable	Maps to your proposal	Behavior
MULTICA_CA_BUNDLE=/path/to/roots.pem	Option 1	Appends a custom CA bundle to the system root pool; full verification stays on (chain + hostname + EKU)
MULTICA_TLS_EKU_ANY=1	Option 3 (targeted relaxation)	Relaxes only the serverAuth EKU requirement via a custom VerifyConnection callback with ExtKeyUsageAny; chain + hostname verification remain active
MULTICA_TLS_INSECURE=1	Option 2	Emergency-only InsecureSkipVerify; loudly warned, off by default

Changes

• New server/internal/util/httptransport.go — shared TLS config builder (~160 lines)
• New server/internal/util/httptransport_test.go — 9 unit tests (~280 lines)
• Modified server/internal/cli/client.go:160 — &http.Client{} → util.NewHTTPClient()
• Modified server/internal/daemon/client.go:106 — same
• Modified server/internal/daemon/wakeup.go:95 — WebSocket dialer gets util.TLSConfig()

Key design decisions

1. Zero behavior change: when no env vars are set, NewHTTPClient returns a client identical to the pre-fix behavior (uses http.DefaultTransport)
2. EKU_ANY uses VerifyConnection callback (not bare InsecureSkipVerify) — keeps chain + hostname verification, only relaxes EKU. Strictly safer than full insecure; targets the exact errSecInvalidExtendedKeyUsage error
3. insecure takes precedence over eku_any — when fully skipping, no VerifyConnection is set
4. CA bundle load failure does not panic — falls back to system roots with an error log

Verification

• 9 unit tests pass (including a self-signed cert integration test)
• All three modified packages compile
• internal/cli/ existing tests — no regression
• No behavior change when env vars are unset

For the exact scenario in this issue (errSecInvalidExtendedKeyUsage), users should try MULTICA_TLS_EKU_ANY=1 first — it precisely relaxes the EKU check that causes the error while keeping chain and hostname verification intact.

PR coming shortly.