Skip to content

test: make multi-worker bob setup deterministic#985

Open
RerankerGuo wants to merge 1 commit into
agentscope-ai:mainfrom
RerankerGuo:fix/test-06-deterministic-bob-setup
Open

test: make multi-worker bob setup deterministic#985
RerankerGuo wants to merge 1 commit into
agentscope-ai:mainfrom
RerankerGuo:fix/test-06-deterministic-bob-setup

Conversation

@RerankerGuo

Copy link
Copy Markdown

Summary

  • make test-06-multi-worker create Bob via hiclaw apply worker instead of a Manager DM request
  • keep the rest of the test focused on multi-worker collaboration after Alice and Bob exist
  • avoid CI flakes where the Manager is still processing heartbeat/task follow-ups or hits tool-guard approval before creating Bob

Why

Recent failures on #972, #983, #984, and older PRs all failed in test-06-multi-worker with the same pattern: the Manager did not produce a matching Bob creation reply, wait_worker_provisioned bob timed out, hiclaw get workers bob returned 404, and hiclaw-worker-bob did not exist. The logs also show the Manager processing unrelated heartbeat/task work or denied shell commands during the Bob creation window.

This makes Bob creation an unreliable LLM interaction inside a test whose stated purpose is multi-worker collaboration. test-02-create-worker already covers Manager-mediated worker creation for Alice; test-06 can use deterministic setup for Bob.

Verification

  • bash -n tests/test-06-multi-worker.sh
  • git diff --check

I could not run the embedded integration test locally because this environment cannot access /var/run/docker.sock (permission denied).

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

📊 CI Metrics Report

Summary

Metric Current Baseline Change
LLM Calls 151 81 +70 ↑ +86.4%
Input Tokens 5183009 2803871 +2379138 ↑ +84.9%
Output Tokens 44558 16791 +27767 ↑ +165.4%
Total Tokens 5227567 2820662 +2406905 ↑ +85.3%

By Role

Role Metric Current Baseline Change
🧠 Manager LLM Calls 82 68 +14 ↑ +20.6%
Input Tokens 3171123 2502214 +668909 ↑ +26.7%
Output Tokens 16922 13725 +3197 ↑ +23.3%
Total Tokens 3188045 2515939 +672106 ↑ +26.7%
🔧 Workers LLM Calls 69 13 +56 ↑ +430.8%
Input Tokens 2011886 301657 +1710229 ↑ +566.9%
Output Tokens 27636 3066 +24570 ↑ +801.4%
Total Tokens 2039522 304723 +1734799 ↑ +569.3%

Per-Test Breakdown

Test Mgr Calls Wkr Calls Δ Calls Mgr In Wkr In Mgr Out Wkr Out Δ Tokens Trend
02-create-worker 9 0 -3 ↓ -25.0% 293327 0 1681 0 -63614 ↓ -17.7% ✅ improved
03-assign-task 11 7 +3 ↑ +20.0% 370798 164610 1613 1083 +64448 ↑ +13.6% ⚠️ regressed
04-human-intervene 15 12 +14 ↑ +107.7% 480042 335865 2508 1623 +387040 ↑ +89.4% ⚠️ regressed
05-heartbeat 8 5 +6 ↑ +85.7% 298259 143655 2125 941 +169728 ↑ +61.7% ⚠️ regressed
06-multi-worker 39 45 +50 ↑ +147.1% 1728697 1367756 8995 23989 +1849303 ↑ +144.5% ⚠️ regressed

Trends

1 test(s) improved (fewer LLM calls)
⚠️ 4 test(s) regressed (more LLM calls)


Generated by HiClaw CI on 2026-07-08 08:53:53 UTC


📦 Download debug logs & test artifacts

@RerankerGuo RerankerGuo force-pushed the fix/test-06-deterministic-bob-setup branch from 3a7a1e4 to 067fd41 Compare July 8, 2026 07:46
@RerankerGuo

Copy link
Copy Markdown
Author

@shiyiyue1102 Hi! This PR makes the multi-worker test-06-multi-worker.sh bob setup deterministic by replacing the LLM-driven DM instruction with a direct hiclaw apply worker --name bob ... call. This bypasses the SHARD_A flake that has been blocking PRs #972, #973, #975, #976, #983, #984 (all hit "Manager never invoked hiclaw create worker --name bob" — confirmed by inspecting #984's downloaded artifacts where the hiclaw-worker-bob session directory did not exist at all).

Just rebased onto current main and CI is 18/18 green (verified just now). If you can land this first, the next CI runs on the six Tier-3 PRs above should turn green automatically — they all hit the same root cause this PR fixes. Branch is MERGEABLE. Could you prioritize this one?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant