Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 2 additions & 21 deletions changelog/current.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,8 @@ Record image-affecting changes to `manager/`, `worker/`, `copaw/`, `openclaw-bas

---

- feat(qwenpaw): add the QwenPaw worker runtime Python package baseline with runtime config sync, storage sync, heartbeat reporting, Matrix channel overlay, and focused unit tests.
- fix(controller): surface Kubernetes Pod container failures in Worker backend status and status API responses.
- feat(controller): expose low-cardinality AgentTeams controller metrics and optional Helm ServiceMonitor.
- feat(controller): add Matrix AppService Human SSO identity provisioning with hash-derived Matrix IDs, AppService login, deletion deactivation, and Team admin/member identity resolution from Human status.
- fix(agent): update file-sharing path guidance for CoPaw and Team Leader agents to use `/root/hiclaw-fs/agents/...` instead of the retired `/root/.hiclaw-worker/...` path.
- feat(helm): add a Helm LLM preflight hook and reusable `hiclaw llm-preflight` command to validate API key/base URL/model before controller startup.

- fix(copaw): harden Matrix channel control-command handling, task-thread routing, NO_REPLY suppression, and cancellation noise handling.
- feat(controller): add OpenKruise Sandbox backend support for Workers via `spec.backendRuntime=sandbox`, including SandboxClaim lifecycle, status watches, CRD schema, and Helm RBAC/env wiring.
- fix(controller): materialize sandbox Worker runtime env/auth material into worker-deps storage before creating SandboxClaim deps and block legacy pod-to-sandbox runtime switches until the Worker is stopped.
- feat(controller): add per-agent `spec.resources` support for Manager, Worker, Team Leader, and Team Worker CRDs.
- fix(worker): pass `X-HiClaw-Cluster-ID` when remote Workers refresh controller-issued STS credentials for OSS and Nacos AI registry access.
- feat(hiclaw-controller): support a separate agent-pod-template for `deployMode: Remote` Workers via an optional `pod-template-remote.yaml` key on the controller-scoped pod-template ConfigMap; remote-mode Pod creation prefers this key and transparently falls back to `pod-template.yaml` when it is absent or empty, while non-remote/Sandbox paths keep ignoring it.
- feat(controller): move Kubernetes resources to the `agentteams.io/v1beta1` API group and decouple Team membership through `spec.workerMembers` references to standalone Worker CRs.
- feat(runtime): let Worker runtimes consume terminal `AGENTTEAMS_*` storage, controller, and auth environment variables while preserving existing `HICLAW_*` deployment compatibility.

- **OpenHuman runtime**: OpenHuman added as the fourth Worker runtime with native Matrix support via `channel-matrix` feature flag; includes controller routing (K8s + Docker backends), Dockerfile, entrypoint script, agent template, Helm chart integration, and Makefile build targets.
- **Multi model providers**: Worker, Team, and Manager specs can now select a Higress model provider via `spec.modelProvider`; the controller resolves the provider, injects the matching gateway URL into runtime config, and authorizes consumers only on the selected AI route.
Expand All @@ -27,18 +14,12 @@ Record image-affecting changes to `manager/`, `worker/`, `copaw/`, `openclaw-bas

**Bug Fixes**

- **Legacy Team channel policy**: Legacy Team reconciliation now writes final Matrix channel allow-lists to member runtime config and re-adds the Team Leader to the Manager allow-list so controller integration tests observe durable policy state.
- **Sandbox worker-deps hardening**: Sandbox-backed Workers now prepare controller-owned worker-deps env/token/data material before claim creation, recycle stale SandboxClaims and bound Sandboxes on runtime-affecting changes, and use bounded ServiceAccount token projection for built-in SandboxClaim mounts.
- **CLI AgentTeams auth env**: The `hiclaw` CLI now discovers `AGENTTEAMS_CONTROLLER_URL`, `AGENTTEAMS_AUTH_TOKEN`, `AGENTTEAMS_AUTH_TOKEN_FILE`, and `AGENTTEAMS_CLUSTER_ID` while preserving legacy `HICLAW_*` fallbacks, so Manager and Worker containers can use the terminal env names for controller calls.
- **CoPaw skill cleanup guardrails**: CoPaw worker skill cleanup now refuses recursive deletion outside managed direct child directories or through symlinks.
- **CoPaw worker runtime environment**: CoPaw workers now prefer AgentTeams storage/runtime environment variables while preserving legacy HiClaw fallbacks, and Qwen-style model health preflights disable thinking for lightweight readiness checks.
- **CoPaw Worker heartbeat**: CoPaw worker templates now seed heartbeat at a 10-minute interval so Team Leader agents created from the worker template can run heartbeat turns without requiring an explicit Team CR heartbeat spec.
- **Helm CRDs**: Removed unsupported `propertyNames` schema fields from Worker and Team CRDs so Kubernetes API servers accept the chart CRDs.
- **CoPaw local runtime paths**: CoPaw direct-run defaults now honor `COPAW_INSTALL_DIR` and `COPAW_WORKING_DIR` before falling back to local home-directory paths, while container entrypoints can continue to pass explicit directories.
- **CoPaw worker replies**: CoPaw's no-text debounce only recognized copaw `Content` objects, but the custom Matrix channel builds plain-dict content parts, so every message was buffered and the agent never replied. The channel now also recognizes dict-based text/audio parts; workers and team leaders reply as expected.
- **Worker Matrix token refresh**: `POST /api/v1/credentials/matrix-token` was registered against the `credentials` resource kind, but the worker credentials branch only permitted `ActionSTS`, so the route was dead and workers could not self-refresh on a homeserver 401. The self-scoped credentials branch now also permits `ActionRefreshMatrixToken` (and the misplaced dead entry was removed).
- **Helm AppService tokens**: The Helm `runtime-env` Secret now generates and preserves the AppService `as_token` / `hs_token` (values override -> existing Secret -> generated) so a default Helm/in-cluster install no longer crash-loops the controller, and a template comment trim-marker that broke `helm lint` YAML parsing was fixed.
- **Remote Worker authentication boundary**: Remote Worker tokens are now bound to the matching local Worker CR's `deployMode: Remote`, target cluster ID, target namespace, and ServiceAccount name before authorization.
- **Remote Worker applied target auth**: Remote Worker authentication now prefers the status-pinned deployment target and falls back to spec only before first provisioning, so spec target edits do not immediately break the running remote Worker or trust a target before it is applied.
- **Remote Worker lifecycle boundary**: Workers now record the applied deployment target in status, reject running target changes until the Worker is Stopped, clean up using the applied target, and register remote Pod watches for Worker/Team status updates.
- **Team Worker CR decoupling**: Worker identity enrichment and Worker REST APIs now resolve `spec.workerMembers` references, and Teams reject sharing the same referenced Worker CR before injecting coordination context.
- **Matrix AppService integration**: SSO Human Team admins now resolve through the Human identity source, Matrix AppService transaction push routes are wired into the controller registration path, and registration keeps the homeserver-facing controller URL as the endpoint base.
- **CoPaw skill cleanup guardrails**: CoPaw worker skill cleanup now refuses recursive deletion outside managed direct child directories or through symlinks.
27 changes: 24 additions & 3 deletions copaw/src/copaw_worker/sync.py
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,27 @@ def _mc(
return result


def _remove_managed_child_dir(parent: Path, child: Path) -> bool:
"""Remove a direct child directory without following links or path escapes."""
try:
parent_resolved = parent.resolve(strict=True)
child_parent = child.parent.resolve(strict=True)
except OSError as exc:
logger.warning("Refusing to remove unresolved path %s: %s", child, exc)
return False

if child_parent != parent_resolved or child.is_symlink() or not child.is_dir():
logger.warning(
"Refusing to remove unmanaged directory: parent=%s child=%s",
parent,
child,
)
return False

shutil.rmtree(child)
return True


def _looks_like_missing_object_error(stderr: str | None) -> bool:
text = stderr or ""
return "Object does not exist" in text or "The specified key does not exist" in text
Expand Down Expand Up @@ -723,9 +744,9 @@ def pull_all(self) -> list[str]:
minio_skill_set = set(minio_skills)
for child in list(local_skills_dir.iterdir()):
if child.is_dir() and child.name not in minio_skill_set:
shutil.rmtree(child)
changed.append(f"skills/{child.name}/ (removed)")
logger.info("Removed local skill no longer in MinIO: %s", child.name)
if _remove_managed_child_dir(local_skills_dir, child):
changed.append(f"skills/{child.name}/ (removed)")
logger.info("Removed local skill no longer in MinIO: %s", child.name)

return changed

Expand Down
36 changes: 28 additions & 8 deletions copaw/src/copaw_worker/worker.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,27 @@
logger = logging.getLogger(__name__)


def _remove_managed_child_dir(parent: Path, child: Path) -> bool:
"""Remove a direct child directory without following links or path escapes."""
try:
parent_resolved = parent.resolve(strict=True)
child_parent = child.parent.resolve(strict=True)
except OSError as exc:
logger.warning("Refusing to remove unresolved path %s: %s", child, exc)
return False

if child_parent != parent_resolved or child.is_symlink() or not child.is_dir():
logger.warning(
"Refusing to remove unmanaged directory: parent=%s child=%s",
parent,
child,
)
return False

shutil.rmtree(child)
return True


class Worker:
def __init__(self, config: WorkerConfig) -> None:
self.config = config
Expand Down Expand Up @@ -519,8 +540,8 @@ def _sync_skills(self) -> None:
keep_names = builtin_names | set(skill_names) | {"file-sync"}
for child in list(active_skills_dir.iterdir()):
if child.is_dir() and child.name not in keep_names:
shutil.rmtree(child)
logger.info("Removed stale active skill: %s", child.name)
if _remove_managed_child_dir(active_skills_dir, child):
logger.info("Removed stale active skill: %s", child.name)

def _dedup_customized_skills(self) -> None:
"""Remove customized skills that shadow CoPaw builtins.
Expand Down Expand Up @@ -554,14 +575,13 @@ def _dedup_customized_skills(self) -> None:
return

# Remove customized copies that duplicate a builtin
import shutil
for child in list(customized_dir.iterdir()):
if child.is_dir() and child.name in builtin_names:
shutil.rmtree(child)
logger.info(
"Removed stale customized skill '%s' (now a builtin)",
child.name,
)
if _remove_managed_child_dir(customized_dir, child):
logger.info(
"Removed stale customized skill '%s' (now a builtin)",
child.name,
)

# ------------------------------------------------------------------
# MatrixChannel installation
Expand Down
34 changes: 33 additions & 1 deletion copaw/tests/test_sync.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,14 @@

import pytest

from copaw_worker.sync import BridgeRuntimeError, FileSync, _mc, push_local, sync_loop
from copaw_worker.sync import (
BridgeRuntimeError,
FileSync,
_mc,
_remove_managed_child_dir,
push_local,
sync_loop,
)


@pytest.fixture
Expand Down Expand Up @@ -91,6 +98,31 @@ def fake_mc(*_args, **_kwargs):
assert "AccessDenied: denied" in caplog.text


def test_remove_managed_child_dir_removes_direct_child_only(tmp_path):
parent = tmp_path / "skills"
child = parent / "stale"
child.mkdir(parents=True)
(child / "SKILL.md").write_text("old")

assert _remove_managed_child_dir(parent, child) is True
assert not child.exists()


def test_remove_managed_child_dir_refuses_symlink(tmp_path, caplog):
parent = tmp_path / "skills"
outside = tmp_path / "outside"
link = parent / "linked"
parent.mkdir()
outside.mkdir()
link.symlink_to(outside, target_is_directory=True)
caplog.set_level(logging.WARNING)

assert _remove_managed_child_dir(parent, link) is False
assert link.exists()
assert outside.exists()
assert "Refusing to remove unmanaged directory" in caplog.text


def test_mirror_all_restores_worker_prefix_and_shared_without_credentials(tmp_path, monkeypatch):
sync = _sync(tmp_path)
commands = []
Expand Down
Loading