feat(controller): support remote worker deploy mode

[shiyiyue1102]

Summary

• add Worker and Team member deployMode / targetCluster / serviceEnabled API fields for remote Kubernetes placement
• route remote members through a remote Kubernetes client cache and remote ServiceAccount provisioning
• support remote-specific pod-template overlays with pod-template-remote.yaml and remote Service reconciliation
• update CLI payloads, CRDs, docs, and focused controller/backend tests

Scope notes

• this PR intentionally does not include Sandbox/OpenKruise backendRuntime changes
• this PR intentionally does not include QwenPaw, TeamHarness, or AppService push URL split changes

Tests

• make generate
• go test ./...

[github-actions]

📊 CI Metrics Report

Summary

Metric	Current	Baseline	Change
LLM Calls	157	81	+76 ↑ +93.8%
Input Tokens	4812587	2803871	+2008716 ↑ +71.6%
Output Tokens	44326	16791	+27535 ↑ +164.0%
Total Tokens	4856913	2820662	+2036251 ↑ +72.2%

By Role

Role	Metric	Current	Baseline	Change
🧠 Manager	LLM Calls	103	68	+35 ↑ +51.5%
	Input Tokens	3507308	2502214	+1005094 ↑ +40.2%
	Output Tokens	24059	13725	+10334 ↑ +75.3%
	Total Tokens	3531367	2515939	+1015428 ↑ +40.4%
🔧 Workers	LLM Calls	54	13	+41 ↑ +315.4%
	Input Tokens	1305279	301657	+1003622 ↑ +332.7%
	Output Tokens	20267	3066	+17201 ↑ +561.0%
	Total Tokens	1325546	304723	+1020823 ↑ +335.0%

Per-Test Breakdown

Test	Mgr Calls	Wkr Calls	Δ Calls	Mgr In	Wkr In	Mgr Out	Wkr Out	Δ Tokens	Trend
02-create-worker	3	0	-9 ↓ -75.0%	81583	0	527	0	-276512 ↓ -77.1%	✅ improved
03-assign-task	12	5	+2 ↑ +13.3%	330645	113356	2269	738	-26648 ↓ -5.6%	⚠️ regressed
04-human-intervene	17	13	+17 ↑ +130.8%	420066	239167	2812	1347	+230394 ↑ +53.2%	⚠️ regressed
05-heartbeat	7	2	+2 ↑ +28.6%	215631	52053	1783	476	-5309 ↓ -1.9%	⚠️ regressed
06-multi-worker	64	34	+64 ↑ +188.2%	2459383	900703	16668	17706	+2114326 ↑ +165.2%	⚠️ regressed

Trends

✅ 1 test(s) improved (fewer LLM calls)
⚠️ 4 test(s) regressed (more LLM calls)

---

Generated by HiClaw CI on 2026-07-03 10:02:19 UTC

---

📦 Download debug logs & test artifacts

[Sunrisea]

Thanks for syncing the remote deploy mode work from the internal branch. The overall direction looks good, but I think this PR should not merge yet. A few lifecycle/security pieces need to come over together with the remote placement support, otherwise the public implementation can create resources that are hard to clean up or can authenticate remote tokens too broadly.

1. Remote TokenReview needs a stricter trust boundary.

Right now the controller takes the caller-provided X-HiClaw-Cluster-ID, resolves that remote cluster, runs TokenReview there, and then parses the returned Kubernetes ServiceAccount username using the normal local identity rules. Since the parser only looks at the SA name pattern, a token from any resolvable remote cluster with an SA named hiclaw-admin, hiclaw-manager, or hiclaw-worker-* can be mapped into a controller admin/manager/worker identity.

That means the remote cluster effectively participates in controller authorization just by matching ServiceAccount names. For remote workers, the identity should be bound back to the expected CR-level placement: clusterID + namespace + serviceAccount should match a Worker/Team member whose deployMode=Remote and targetCluster points to that same cluster/namespace. Also, remote tokens should probably not be allowed to become admin/manager identities at all.

2. The applied deployment target should be pinned after provisioning.

The current reconcile/delete paths continue to derive the target cluster from the current spec.deployMode/spec.targetCluster. This makes target changes unsafe. For example:

• Worker is created with deployMode=Remote, targetCluster=A
• controller creates Pod/SA/Service in cluster A
• user patches the Worker to targetCluster=B
• later reconcile/delete uses cluster B as the target
• resources in cluster A are left behind, and status/cleanup now refer to the wrong cluster

Remote placement needs an applied-target record, e.g. status.deployMode/status.targetCluster, and target changes should be rejected once runtime resources have been provisioned. Delete should clean up using the applied target, not the latest mutable spec.

3. Service cleanup should be part of the member deletion path.

The new ClusterIP Service is created separately and does not have an ownerRef. However, ReconcileMemberDelete currently cleans up rooms, Pod, OSS data, credentials, local/remote ServiceAccounts, etc., but not the Service. This means deleting a Worker/Team member can leave a stale Service behind in the local or remote cluster.

Please either attach a correct owner/GC story where possible, or explicitly delete Services during member finalizer cleanup. Label-based cleanup is safer than name-only cleanup because it can cover both current and legacy naming formats.

4. Clearing spec.expose should remove an existing Service.

When serviceEnabled=true but spec.expose is empty, ensureServiceExists returns early before checking for an existing Service. That is fine for first creation, but not for updates: if a Service already exists from a previous expose config, clearing spec.expose leaves the old Service and old ports active.

Expected behavior should be: no expose ports means no Service, or at least any previously created Service should be deleted.

These are not just polish issues; they affect remote runtime correctness, cleanup guarantees, and controller auth boundaries. I would suggest fixing these in this PR before merging the remote deploy mode support.

[shiyiyue1102]

Thanks for the detailed review. I pushed f609441 to address the scoped fixes here:

• Remote TokenReview still routes to the cluster selected by X-HiClaw-Cluster-ID; additionally, remote-cluster tokens are now rejected if they parse as admin or manager identities. Remote worker tokens remain supported.
• The ClusterIP Service is controller-created in member_reconcile_service.go, so cleanup is now handled in the controller as well: clearing spec.expose deletes a stale Service, and member finalizer cleanup deletes the Service during Worker/Team member deletion.
• Added regression coverage for remote admin/manager token rejection, expose-cleared Service deletion, and finalizer Service deletion.

For targetCluster changes: the current product contract is that targetCluster should not be changed while runtime resources are active. To move placement, the CR should first be brought to Stopped, then targetCluster can be changed and reconciled from that stopped state.

[Sunrisea]

I re-checked the latest revision (f609441). This version does improve two things: remote tokens can no longer authenticate as admin/manager, and service cleanup is now triggered when spec.expose is cleared or when a member is deleted. The direction looks good, but I still see a few blockers before this can be merged.

1. Remote worker auth is still not bound to the deployment target

   After a remote token review succeeds, the worker identity is still mainly derived from the ServiceAccount name, for example hiclaw-worker-xxx -> Worker xxx. This means any resolvable remote cluster with a same-named ServiceAccount could potentially be recognized as the same Worker in the main cluster.

   I think the auth/enrichment path should also verify:

   • the source clusterID
   • the ServiceAccount namespace/name
   • whether the matched Worker/Team is actually using deployMode=Remote
   • whether targetCluster matches the source cluster

   In other words, a remote worker identity should not be determined by SA name alone. It needs to be bound to the Worker’s actual deployment target.

2. deployMode / targetCluster still need status pinning or immutability

   Today, after a Worker is created, if spec.deployMode or spec.targetCluster is patched, later update/delete operations will use the new spec values. This can leave Pods/Services behind in the old remote cluster, or operate on the wrong cluster.

   I’d suggest either persisting the effective deployMode / targetCluster into status and using that for update/delete paths, or making these deployment target fields immutable after creation.

3. CI is currently failing

   The latest controller test job is red. I saw at least:

   • TestWorkerPodDeleted_Recreates: after pod recreation, the phase remains Starting while the test expects Running
   • TestWorkerCreate_Idempotent_NoDoubleProvision: data race between mock reset and async reconcile reads

   This looks related to the new container status / async reconcile behavior. The test waiting logic and mock lifecycle probably need to be adjusted so go test -race is stable.

4. Remote watches are still not wired up

   remoteclient.Cache exposes RegisterWatch, but the app initialization does not appear to register remote Pod/Sandbox watches. Also, the cache is initialized without ControllerName, so the watch selector would not be fully scoped.

   Without this, remote Pod changes will not promptly enqueue reconciliation, and status updates / cleanup will depend on periodic or manual reconciles.

Overall, the feature direction makes sense, but I think the remote auth boundary, deployment target lifecycle, and failing CI should be fixed before merge.

[shiyiyue1102]

Thanks for catching this. Fixed the remote Worker token binding by carrying the TokenReview serviceAccount namespace/name into the identity and requiring the matching local Worker CR to be deployMode Remote with the same targetCluster id, target namespace, and expected ServiceAccount name before authorization. Remote identities without a Worker CR now fail instead of falling back to Team lookup.

[github-actions]

❌ Integration Tests Failed (llm-interaction / mgr=copaw / wk=copaw)

Commit: 569967d
Workflow run: #1405

Test Results

No test output captured.

Debug Log (tail)

No debug logs available.

---

📦 Download full debug logs & test artifacts

[shiyiyue1102]

Thanks for the detailed follow-up. I pushed 3122eda to address the remaining lifecycle/watch blockers in this PR scope: Workers now persist the applied deployment target in status and keep update/delete operations on that applied target; target changes are rejected until the Worker is fully Stopped, so a stopped-first move can switch targets without orphaning the old cluster. I also wired remote Pod watches for Worker and Team member Pods through the remote client cache, including the controller-name selector and local-namespace enqueue mapping.

[github-actions]

❌ Integration Tests Failed (llm-interaction-2 / mgr=copaw / wk=copaw)

Commit: 569967d
Workflow run: #1405

Test Results

No test output captured.

Debug Log (tail)

No debug logs available.

---

📦 Download full debug logs & test artifacts

[Sunrisea]

Re-checked the latest head (3122edac). This revision fixed several earlier concerns: controller tests are green now, image builds pass, remote worker auth now checks cluster/namespace/SA, and standalone Worker deployment target pinning was added. Thanks for the quick iteration.

I still see a few blockers around the Team remote path and the auth target source:

1. Remote Team members cannot authenticate

   CREnricher now rejects any remote identity that does not have a backing Worker CR. However, Team members do not have standalone Worker CRs, while this PR adds deployMode / targetCluster to Team.spec.leader and Team.spec.workers.

   So a remote Team leader/worker token will pass remote TokenReview, then fail identity enrichment before the Team reverse lookup path can run. The remote Team member path looks unusable as-is.

   I think remote identity validation needs to cover both cases:

   • standalone Worker CR
   • Team leader / Team worker resolved from Team CR, with matching clusterID, namespace, and expected ServiceAccount name

2. Team remote members do not have deployment target status pinning

   Standalone WorkerStatus now records deployMode / targetCluster, which is good. But TeamMemberStatus has no equivalent fields.

   This means stale member cleanup and Team deletion cannot know where an already-provisioned remote member actually lives. If a Team member is removed from spec, or the Team is deleted after the spec was changed, cleanup will build a MemberContext without the original remote target and fall back to the local backend. Remote Pod / Service / ServiceAccount resources may be leaked.

   Team members need the same effective deployment target snapshot in status, and cleanup/update paths should use the status-pinned target instead of the mutable spec.

3. Standalone remote worker auth should validate against the applied target, not only current spec

   Worker reconcile now prevents changing spec.deployMode / spec.targetCluster while the Worker is not stopped, and it uses the status-pinned target for runtime operations. However, remote auth validation still reads worker.Spec.TargetCluster.

   If a user patches spec.targetCluster while the Worker is still running, reconcile will reject the move, but the actually running remote worker in the old cluster may immediately fail auth because auth compares against the new spec. Conversely, the new target could be trusted before it is actually applied.

   Auth should prefer status.deployMode / status.targetCluster as the effective applied target, falling back to spec only before the first successful provisioning.

The latest CI state is much better: test, CRD sync, helm lint, and image builds are passing. I’d still request changes for the Team remote lifecycle/auth gaps above before merging.

[shiyiyue1102]

Thanks for the follow-up. The Team CR path has already been split out and will be covered by the next PR. For this PR, I will keep the scope limited to the standalone Worker remote deploy path and will not add the Team remote lifecycle/auth changes here.

[shiyiyue1102]

@Sunrisea @johnlanni @Jing-ze CI is green on the latest head, and I clarified that the Team CR remote path is split to the next PR. Could you take another look when convenient?

[shiyiyue1102]

Thanks for pointing this out. Fixed the standalone remote Worker auth target source in 7f74c12: remote auth now prefers status.deployMode/status.targetCluster as the applied target and falls back to spec only before first provisioning. I also added a regression test for the spec/status divergence case. The Team remote path remains split to the next PR.

[Sunrisea]

After re-checking this with the Team/Worker decoupling direction in mind, I think the remaining concern is mainly about API ownership / source of truth.

The standalone Worker remote deploy path looks much better now: remote auth validates the applied target, Worker status pins deployMode / targetCluster, and CI is green.

The part that still looks risky is that this PR also adds deployMode / targetCluster to Team.spec.leader and Team.spec.workers[].

If the intended model is that Team only owns membership/organization, and every team member has its own Worker CR, then deployment target should be owned by the Worker CR only. Otherwise we end up with two possible sources of truth for the same member’s runtime placement:

• Worker.spec/status.deployMode/targetCluster
• Team.spec.leader/workers[].deployMode/targetCluster

I’d suggest removing or explicitly disabling the Team-level remote deployment fields in this PR, unless Team direct runtime deployment is still meant to be supported. In the decoupled model, auth, cleanup, and applied target pinning should all be driven by the member’s Worker CR.

[shiyiyue1102]

Thanks for the clarification. Fixed in a7bdfc8 by removing Team-level deployMode/targetCluster from the API/CRD schema and Team member mapping. This keeps remote deployment target ownership on the standalone Worker CR path for this PR; Team runtime placement can stay with the follow-up Team/Worker decoupling work.

[github-actions]

❌ Integration Tests Failed (llm-interaction / mgr=copaw / wk=hermes)

Commit: afa96b5
Workflow run: #1416

Test Results

No test output captured.

Debug Log (tail)

No debug logs available.

---

📦 Download full debug logs & test artifacts

[github-actions]

❌ Integration Tests Failed (llm-interaction / mgr=openclaw / wk=openclaw)

Commit: afa96b5
Workflow run: #1416

Test Results

No test output captured.

Debug Log (tail)

No debug logs available.

---

📦 Download full debug logs & test artifacts