feat(controller): support remote worker deploy mode

changelog/current.md

@@ -6,6 +6,8 @@ Record image-affecting changes to `manager/`, `worker/`, `copaw/`, `openclaw-bas
 
 - fix(agent): update file-sharing path guidance for CoPaw and Team Leader agents to use `/root/hiclaw-fs/agents/...` instead of the retired `/root/.hiclaw-worker/...` path.
 - feat(controller): add per-agent `spec.resources` support for Manager, Worker, Team Leader, and Team Worker CRDs.
+- fix(worker): pass `X-HiClaw-Cluster-ID` when remote Workers refresh controller-issued STS credentials for OSS and Nacos AI registry access.
+- feat(hiclaw-controller): support a separate agent-pod-template for `deployMode: Remote` Workers via an optional `pod-template-remote.yaml` key on the controller-scoped pod-template ConfigMap; remote-mode Pod creation prefers this key and transparently falls back to `pod-template.yaml` when it is absent or empty, while non-remote/Sandbox paths keep ignoring it.
 
 - **OpenHuman runtime**: OpenHuman added as the fourth Worker runtime with native Matrix support via `channel-matrix` feature flag; includes controller routing (K8s + Docker backends), Dockerfile, entrypoint script, agent template, Helm chart integration, and Makefile build targets.
 - **Multi model providers**: Worker, Team, and Manager specs can now select a Higress model provider via `spec.modelProvider`; the controller resolves the provider, injects the matching gateway URL into runtime config, and authorizes consumers only on the selected AI route.
@@ -21,3 +23,6 @@ Record image-affecting changes to `manager/`, `worker/`, `copaw/`, `openclaw-bas
 - **CoPaw worker replies**: CoPaw's no-text debounce only recognized copaw `Content` objects, but the custom Matrix channel builds plain-dict content parts, so every message was buffered and the agent never replied. The channel now also recognizes dict-based text/audio parts; workers and team leaders reply as expected.
 - **Worker Matrix token refresh**: `POST /api/v1/credentials/matrix-token` was registered against the `credentials` resource kind, but the worker credentials branch only permitted `ActionSTS`, so the route was dead and workers could not self-refresh on a homeserver 401. The self-scoped credentials branch now also permits `ActionRefreshMatrixToken` (and the misplaced dead entry was removed).
 - **Helm AppService tokens**: The Helm `runtime-env` Secret now generates and preserves the AppService `as_token` / `hs_token` (values override -> existing Secret -> generated) so a default Helm/in-cluster install no longer crash-loops the controller, and a template comment trim-marker that broke `helm lint` YAML parsing was fixed.
+- **Remote Worker authentication boundary**: Remote Worker tokens are now bound to the matching local Worker CR's `deployMode: Remote`, target cluster ID, target namespace, and ServiceAccount name before authorization.
+- **Remote Worker applied target auth**: Remote Worker authentication now prefers the status-pinned deployment target and falls back to spec only before first provisioning, so spec target edits do not immediately break the running remote Worker or trust a target before it is applied.
+- **Remote Worker lifecycle boundary**: Workers now record the applied deployment target in status, reject running target changes until the Worker is Stopped, clean up using the applied target, and register remote Pod watches for Worker/Team status updates.

docs/agent-pod-template.md

@@ -62,6 +62,56 @@ data:
 
 Find a ready-to-apply example at [`docs/examples/agent-pod-template-cm.yaml`](examples/agent-pod-template-cm.yaml).
 
+## Per-deploy-mode override (Remote)
+
+When a Worker is created with `spec.deployMode: Remote`, the controller
+schedules its Pod into a remote cluster fronted by a virtual-kubelet style
+provider, where node taints, image-pull secrets, and tolerations almost
+always differ from the in-cluster defaults. To avoid forcing those
+remote-only fields onto every local Pod, the same ConfigMap may carry an
+additional key:
+
+| Key | Consumed when |
+|---|---|
+| `pod-template.yaml` | Default and `deployMode: Local` only. NOT used as fallback for Remote mode. |
+| `pod-template-remote.yaml` | Only when `deployMode: Remote` and the key is present and non-empty. |
+
+Resolution rules:
+
+- `deployMode: Remote` → if `pod-template-remote.yaml` is present and
+  non-empty, it is used; otherwise the controller applies the default Pod
+  spec with no overlay (it does **not** fall back to `pod-template.yaml`).
+- Any other deploy mode (default / `Local`) → the remote key is ignored
+  even when present, and only `pod-template.yaml` is consulted.
+- Non-K8s backends never consume the remote key.
+
+This lets you keep one ConfigMap per controller while shipping different
+`tolerations`, `nodeSelector`, `imagePullSecrets`, annotations or sidecars
+to remote-cluster Pods, e.g.:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: <controller-name>
+  namespace: <controller-ns>
+data:
+  pod-template.yaml: |
+    spec:
+      nodeSelector:
+        kubernetes.io/os: linux
+  pod-template-remote.yaml: |
+    spec:
+      nodeSelector:
+        type: virtual-kubelet
+      tolerations:
+        - key: virtual-kubelet.io/provider
+          operator: Exists
+          effect: NoSchedule
+      imagePullSecrets:
+        - name: remote-registry-pull-secret
+```
+
 ## Merge semantics
 
 Fields the **template wins**:

helm/hiclaw/crds/teams.hiclaw.io.yaml

@@ -175,6 +175,9 @@ spec:
                             memory:
                               type: string
                               description: Kubernetes memory quantity, e.g. 3Gi
+                    serviceEnabled:
+                      type: boolean
+                      description: "Whether to create a ClusterIP Service alongside the leader pod."
                     labels:
                       type: object
                       additionalProperties:
@@ -390,6 +393,9 @@ spec:
                               memory:
                                 type: string
                                 description: Kubernetes memory quantity, e.g. 2Gi
+                      serviceEnabled:
+                        type: boolean
+                        description: "Whether to create a ClusterIP Service alongside this team worker pod."
                       accessEntries:
                         type: array
                         description: >
@@ -493,6 +499,21 @@ spec:
                       ready:
                         type: boolean
                         description: "Mirrors backend.Status Running/Ready. Aggregates into leaderReady and readyWorkers."
+                      phase:
+                        type: string
+                        enum: [Pending, Starting, Running, Updating, Stopping, Sleeping, Stopped, Failed]
+                      containerState:
+                        type: string
+                      message:
+                        type: string
+                      lastActiveAt:
+                        type: string
+                        format: date-time
+                        description: "Latest runtime-reported business activity time for this member."
+                      lastHeartbeat:
+                        type: string
+                        format: date-time
+                        description: "Latest heartbeat timestamp for this member."
                       exposedPorts:
                         type: array
                         description: "Ports currently exposed via Higress for this member (workers only)."

helm/hiclaw/crds/workers.hiclaw.io.yaml

@@ -184,6 +184,24 @@ spec:
                     ignored with a warning log — the system value always wins.
                   additionalProperties:
                     type: string
+                deployMode:
+                  type: string
+                  enum: ["Local", "Remote"]
+                  description: "Where the worker pod runs. Local (default): controller's own cluster; Remote: cluster identified by targetCluster."
+                targetCluster:
+                  type: object
+                  description: "Remote cluster target for deployment. Required when deployMode is Remote."
+                  properties:
+                    id:
+                      type: string
+                      description: "ACS/ACK cluster ID."
+                    namespace:
+                      type: string
+                      description: "Target namespace in the remote cluster."
+                  required: ["id", "namespace"]
+                serviceEnabled:
+                  type: boolean
+                  description: "Whether to create a ClusterIP Service alongside the worker pod."
                 labels:
                   type: object
                   additionalProperties:
@@ -235,7 +253,7 @@ spec:
                   format: int64
                 phase:
                   type: string
-                  enum: [Pending, Running, Sleeping, Updating, Stopped, Failed]
+                  enum: [Pending, Starting, Running, Updating, Stopping, Sleeping, Stopped, Failed]
                 matrixUserID:
                   type: string
                 roomID:
@@ -257,6 +275,21 @@ spec:
                         type: integer
                       domain:
                         type: string
+                deployMode:
+                  type: string
+                  enum: ["Local", "Remote"]
+                  description: "Applied deployment mode for the current Worker runtime resource."
+                targetCluster:
+                  type: object
+                  description: "Applied remote cluster target for the current Worker runtime resource."
+                  properties:
+                    id:
+                      type: string
+                      description: "ACS/ACK cluster ID."
+                    namespace:
+                      type: string
+                      description: "Target namespace in the remote cluster."
+                  required: ["id", "namespace"]
       subresources:
         status: {}
       additionalPrinterColumns:

hiclaw-controller/api/v1beta1/types.go

@@ -89,6 +89,20 @@ type AgentResourceValues struct {
 	Memory string `json:"memory,omitempty"`
 }
 
+// DeployMode constants define where the worker pod runs.
+const (
+	DeployModeLocal  = "Local"
+	DeployModeRemote = "Remote"
+)
+
+// TargetClusterSpec identifies the remote cluster and namespace for deployment.
+type TargetClusterSpec struct {
+	// ID is the ACS/ACK cluster ID.
+	ID string `json:"id"`
+	// Namespace is the target namespace in the remote cluster.
+	Namespace string `json:"namespace"`
+}
+
 // +genclient
 // +kubebuilder:subresource:status
 // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
@@ -101,7 +115,6 @@ type Worker struct {
 	Status            WorkerStatus `json:"status,omitempty"`
 }
 
-
 type WorkerSpec struct {
 	Model         string                     `json:"model"`
 	ModelProvider string                     `json:"modelProvider,omitempty"` // APIG Model API name for per-worker LLM provider
@@ -136,6 +149,19 @@ type WorkerSpec struct {
 	// scoped to agents/<name>/* and shared/*).
 	AccessEntries []AccessEntry `json:"accessEntries,omitempty"`
 
+	// DeployMode specifies where the worker pod runs.
+	// "Local" (default): created in the controller's own cluster.
+	// "Remote": created in the cluster identified by TargetCluster.
+	DeployMode *string `json:"deployMode,omitempty"`
+
+	// TargetCluster specifies the remote cluster target for deployment.
+	// Required when DeployMode is "Remote".
+	TargetCluster *TargetClusterSpec `json:"targetCluster,omitempty"`
+
+	// ServiceEnabled controls whether a ClusterIP Service is created
+	// alongside the worker pod (same cluster, namespace, name).
+	ServiceEnabled *bool `json:"serviceEnabled,omitempty"`
+
 	// Env holds user-defined environment variables injected into the worker
 	// container. Keys that collide with variables already set by the
 	// controller or backend (HICLAW_*, OPENCLAW_*, HOME, and similar
@@ -152,7 +178,6 @@ type WorkerSpec struct {
 	// embed WorkerSpec-shaped hashes keep a stable spec hash when the
 	// field is absent.
 	Labels map[string]string `json:"labels,omitempty"`
-
 }
 
 // DesiredContainerMan returns the effective desired containerManaged, defaulting to true.
@@ -205,6 +230,12 @@ type WorkerStatus struct {
 	LastHeartbeat      string              `json:"lastHeartbeat,omitempty"`
 	Message            string              `json:"message,omitempty"`
 	ExposedPorts       []ExposedPortStatus `json:"exposedPorts,omitempty"`
+
+	// DeployMode/TargetCluster record where the current backend resource was
+	// actually provisioned. Once set, target changes are only accepted after
+	// the Worker has reached the Stopped phase.
+	DeployMode    string             `json:"deployMode,omitempty"`
+	TargetCluster *TargetClusterSpec `json:"targetCluster,omitempty"`
 }
 
 // ExposedPortStatus records a port that has been exposed via Higress.
@@ -285,6 +316,10 @@ type LeaderSpec struct {
 	// + shared/* + teams/<team>/* on the configured bucket).
 	AccessEntries []AccessEntry `json:"accessEntries,omitempty"`
 
+	// ServiceEnabled controls whether a ClusterIP Service is created
+	// alongside the leader pod (same cluster, namespace, name).
+	ServiceEnabled *bool `json:"serviceEnabled,omitempty"`
+
 	// Env holds user-defined environment variables injected into the
 	// leader container. See WorkerSpec.Env for the collision policy.
 	Env map[string]string `json:"env,omitempty"`
@@ -295,7 +330,6 @@ type LeaderSpec struct {
 	// hashMemberSourceSpec stability for Teams that never set this
 	// field.
 	Labels map[string]string `json:"labels,omitempty"`
-
 }
 
 type TeamLeaderHeartbeatSpec struct {
@@ -328,6 +362,10 @@ type TeamWorkerSpec struct {
 	// + shared/* + teams/<team>/* on the configured bucket).
 	AccessEntries []AccessEntry `json:"accessEntries,omitempty"`
 
+	// ServiceEnabled controls whether a ClusterIP Service is created
+	// alongside the team worker pod (same cluster, namespace, name).
+	ServiceEnabled *bool `json:"serviceEnabled,omitempty"`
+
 	// Env holds user-defined environment variables injected into this
 	// team worker's container. See WorkerSpec.Env for the collision policy.
 	Env map[string]string `json:"env,omitempty"`
@@ -337,7 +375,6 @@ type TeamWorkerSpec struct {
 	// system labels (see WorkerSpec.Labels godoc). omitempty preserves
 	// hashMemberSourceSpec stability for existing Teams.
 	Labels map[string]string `json:"labels,omitempty"`
-
 }
 
 // EffectiveWorkerName returns the runtime identity key for a team leader.
@@ -436,6 +473,17 @@ type TeamMemberStatus struct {
 	// summarizeBackendReadiness on each reconcile pass. Aggregates into
 	// Team.Status.LeaderReady and Team.Status.ReadyWorkers.
 	Ready bool `json:"ready,omitempty"`
+	// Phase is the member lifecycle phase: Pending, Starting, Running,
+	// Updating, Stopping, Sleeping, Stopped, Failed.
+	Phase string `json:"phase,omitempty"`
+	// ContainerState is the raw backend container status.
+	ContainerState string `json:"containerState,omitempty"`
+	// Message holds per-member error detail from reconcile. Cleared on success.
+	Message string `json:"message,omitempty"`
+	// LastActiveAt is the latest runtime-reported business activity time.
+	LastActiveAt string `json:"lastActiveAt,omitempty"`
+	// LastHeartbeat is the latest heartbeat timestamp for this member.
+	LastHeartbeat string `json:"lastHeartbeat,omitempty"`
 	// ExposedPorts records the ports currently exposed via Higress for this
 	// member. Leader members never expose ports (this field stays nil).
 	ExposedPorts []ExposedPortStatus `json:"exposedPorts,omitempty"`
@@ -545,7 +593,6 @@ type ManagerSpec struct {
 	// godoc): pod-template < CR metadata.labels < CR spec.labels <
 	// controller system labels.
 	Labels map[string]string `json:"labels,omitempty"`
-
 }
 
 // DesiredState returns the effective desired state, defaulting to "Running".