Skip to content

Conversation

@echobt
Copy link
Contributor

@echobt echobt commented Feb 5, 2026

Changes

  • Adds a shell script for testing multiple validators locally without Docker containers
  • Enables quick validation of P2P network formation and validator initialization
  • Complements existing Docker-based multi-validator tests

Testing

Verified the following tests pass:

Local Multi-Validator Test:

  • Started 3 validators with --no-bittensor mode
  • All validators initialized successfully
  • Distributed storage created for each validator

Docker Multi-Validator Test:

  • Built and ran 4-validator Docker network
  • All containers healthy within 15 seconds
  • P2P network initialized with proper bootstrap configuration

Unit/Integration Tests:

  • All 249 tests pass
  • Clippy check passes with only warnings (no errors)

Verification

# Run the local test script
bash tests/docker/test-multi-validator.sh

# Run Docker-based tests
docker compose -f tests/docker/docker-compose.multi-validator.yml up -d
docker compose -f tests/docker/docker-compose.multi-validator.yml ps

Summary by CodeRabbit

  • Tests
    • Added integration tests for multi-validator network deployments that orchestrate multiple validator instances in P2P configurations. Tests verify network initialization, peer discovery and connectivity, database state persistence, and validator process health monitoring to ensure distributed validator setups function correctly.

Adds a shell script for testing multiple validators locally without
Docker containers. This enables quick validation of P2P network
formation and validator initialization.
@coderabbitai
Copy link

coderabbitai bot commented Feb 5, 2026

📝 Walkthrough

Walkthrough

A new Bash integration test script that orchestrates a multi-validator P2P network. It builds the validator binary, starts a bootstrap validator, launches additional validators with distinct keys and ports, waits for initialization, verifies all validators are running, checks for distributed.db files, and validates P2P connections and peer discovery in logs.

Changes

Cohort / File(s) Summary
Multi-Validator Test Orchestration
tests/docker/test-multi-validator.sh
New integration test script that builds validator binary, starts bootstrap validator on port 9100, launches additional validators with unique keys/ports, verifies process initialization and distributed.db creation, validates P2P peer connections in logs, and reports overall network health. Includes configurable parameters (NUM_VALIDATORS, VALIDATOR_BINARY) and cleanup on exit.

Sequence Diagram

sequenceDiagram
    participant Script as Test Script
    participant Binary as Validator Binary
    participant Bootstrap as Bootstrap Validator
    participant Others as Additional Validators
    participant FileSystem as File System
    participant Logs as Log Files

    Script->>Script: Check/Build validator binary
    Script->>Bootstrap: Start bootstrap validator (port 9100)
    activate Bootstrap
    Script->>FileSystem: Create test directory
    Loop i = 1 to NUM_VALIDATORS-1
        Script->>Others: Start validator with unique key/port
        activate Others
    end
    Script->>Script: Wait for initialization
    Script->>Bootstrap: Verify process running
    Script->>Others: Verify all processes running
    Script->>FileSystem: Check distributed.db exists
    Bootstrap->>Logs: Write initialization logs
    Others->>Logs: Write initialization logs
    Others->>Logs: Write P2P peer connection logs
    Script->>Logs: Scan for P2P initialization
    Script->>Logs: Scan for peer connections
    Script->>Script: Run network (stabilization period)
    Script->>Script: Count healthy validators
    Script->>Script: Return status (0=all healthy, 1=failures)
    deactivate Others
    deactivate Bootstrap
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Poem

🐰 A dozen validators dance and spin,
Bootstrap leading the P2P din,
Peers connect, logs confirm the sight,
Our test ensures the network's bright!
Hop hop hooray, this script's our friend! 🎉

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: a new test script for running multiple validators locally without Docker containers.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feat/multi-validator-test-script

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@tests/docker/test-multi-validator.sh`:
- Around line 131-136: The test is checking for a file but uses the directory
test operator (-d), causing false negatives; change the check that evaluates
DB_FILE (the variable set as DB_FILE="$TEST_DIR/validator-$i/distributed.db")
from -d to -f (or -e if both file/dir are possible) and update the conditional
that calls log_success/log_warning accordingly; apply the same replacement for
the similar check around lines that reference the same DB_FILE logic later in
the script (the block that also uses log_success/log_warning for validators).
- Around line 86-90: The validator key construction in the loop yields
non‑padded hex for two‑digit indexes; update the VALIDATOR_KEY assignment inside
the for loop (where NUM_VALIDATORS and VALIDATOR_KEY are used) to generate a
64‑hex‑char, zero‑padded value (and prefix with 0x) by formatting the numeric
index with printf to produce a 64‑character hex string (e.g., use printf "%064x"
on the index) instead of concatenating the index to a fixed short prefix.
🧹 Nitpick comments (1)
tests/docker/test-multi-validator.sh (1)

74-110: Replace fixed sleeps with bounded polling to reduce flakiness.

Fixed sleeps can make the test flaky across slower machines. Consider a retry loop with timeout (e.g., wait for log markers or DB file existence) to make readiness checks deterministic.

♻️ Example pattern (optional)
+wait_for_log() {
+  local file="$1" pattern="$2" timeout="${3:-20}"
+  for ((t=0; t<timeout; t++)); do
+    grep -q "$pattern" "$file" 2>/dev/null && return 0
+    sleep 1
+  done
+  return 1
+}

Also applies to: 155-157

Comment on lines +86 to +90
for i in $(seq 2 $NUM_VALIDATORS); do
VALIDATOR_DIR="$TEST_DIR/validator-$i"
mkdir -p "$VALIDATOR_DIR"
VALIDATOR_KEY="0x000000000000000000000000000000000000000000000000000000000000000$i"
VALIDATOR_PORT=$((BASE_PORT + i - 1))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Validator key formatting breaks for NUM_VALIDATORS ≥ 10.

Appending $i to a fixed hex prefix yields non‑padded hex (and non‑64‑byte) keys for two‑digit indexes, which can make keys invalid. Use a zero‑padded 64‑hex format.

🛠️ Suggested fix
-    VALIDATOR_KEY="0x000000000000000000000000000000000000000000000000000000000000000$i"
+    VALIDATOR_KEY=$(printf '0x%064x' "$i")
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for i in $(seq 2 $NUM_VALIDATORS); do
VALIDATOR_DIR="$TEST_DIR/validator-$i"
mkdir -p "$VALIDATOR_DIR"
VALIDATOR_KEY="0x000000000000000000000000000000000000000000000000000000000000000$i"
VALIDATOR_PORT=$((BASE_PORT + i - 1))
for i in $(seq 2 $NUM_VALIDATORS); do
VALIDATOR_DIR="$TEST_DIR/validator-$i"
mkdir -p "$VALIDATOR_DIR"
VALIDATOR_KEY=$(printf '0x%064x' "$i")
VALIDATOR_PORT=$((BASE_PORT + i - 1))
🤖 Prompt for AI Agents
In `@tests/docker/test-multi-validator.sh` around lines 86 - 90, The validator key
construction in the loop yields non‑padded hex for two‑digit indexes; update the
VALIDATOR_KEY assignment inside the for loop (where NUM_VALIDATORS and
VALIDATOR_KEY are used) to generate a 64‑hex‑char, zero‑padded value (and prefix
with 0x) by formatting the numeric index with printf to produce a 64‑character
hex string (e.g., use printf "%064x" on the index) instead of concatenating the
index to a fixed short prefix.

Comment on lines +131 to +136
DB_FILE="$TEST_DIR/validator-$i/distributed.db"
if [ -d "$DB_FILE" ]; then
log_success "Validator $i: distributed.db exists"
else
log_warning "Validator $i: distributed.db not found yet"
fi
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Use -f/-e for distributed.db; -d will mis-detect files.

If distributed.db is a file (as the summary implies), -d will always fail and incorrectly mark validators unhealthy. Switch to -f (or -e if it can be file/dir).

🧩 Suggested fix
-    if [ -d "$DB_FILE" ]; then
+    if [ -f "$DB_FILE" ]; then
         log_success "Validator $i: distributed.db exists"
     else
         log_warning "Validator $i: distributed.db not found yet"
     fi
-        if [ -d "$DB_FILE" ]; then
+        if [ -f "$DB_FILE" ]; then
             log_success "Validator $i: HEALTHY (running + DB initialized)"
             ((HEALTHY++))
         else
             log_warning "Validator $i: RUNNING (no DB yet)"
         fi

Also applies to: 168-174

🤖 Prompt for AI Agents
In `@tests/docker/test-multi-validator.sh` around lines 131 - 136, The test is
checking for a file but uses the directory test operator (-d), causing false
negatives; change the check that evaluates DB_FILE (the variable set as
DB_FILE="$TEST_DIR/validator-$i/distributed.db") from -d to -f (or -e if both
file/dir are possible) and update the conditional that calls
log_success/log_warning accordingly; apply the same replacement for the similar
check around lines that reference the same DB_FILE logic later in the script
(the block that also uses log_success/log_warning for validators).

@echobt echobt merged commit 0adaab1 into main Feb 5, 2026
7 checks passed
@echobt echobt deleted the feat/multi-validator-test-script branch February 5, 2026 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant