Feature Request: Hierarchical Network Topology

[aojea]

Overview

To scale the agent network to millions of nodes, we are moving away from a flat libp2p mesh and avoiding a rigid, hardcoded 3-tier architecture. Instead, we will implement a flexible Routing Domains (or Scopes) model.

Nodes will simply declare which logical routing domains they participate in. The control plane will dynamically compute intersections to assign bootstrap peers and isolate network churn, hiding all underlying libp2p and DHT protocol complexities from the user.

---

Proposed API Schema (Control Plane Source of Truth)

1. Managing Routing Domains (CRUD)

Admins create, update, or delete routing domains dynamically via the API.

// POST /api/v1/mesh/domains
{
  "name": "gke-prod-cluster",
  "description": "Local routing domain for the production GKE cluster agents"
}

2. Provisioning Nodes & Routers

A node can belong to one or more routing domains. The control plane uses the number of assigned domains to infer whether a node behaves as an edge client or a transit router.

• Standard Edge Agent (Single Domain):

// POST /api/v1/nodes/enroll
{
  "node_name": "gke-mcp-agent-01",
  "routing_domains": ["gke-prod-cluster"]
}

• Transit Router (Multiple Domains):

// POST /api/v1/nodes/enroll
{
  "node_name": "hybrid-backbone-router-gcp",
  "role": "router",
  "routing_domains": ["gke-prod-cluster", "global-backbone"]
}

---

Control Plane Backend Translation Pipeline

When nodes heartbeat or configurations change, the control plane processes them through a background pipeline:

1. Protocol Name Hashing: The control plane deterministically maps user domain strings to libp2p protocol strings invisibly (e.g., gke-prod-cluster ➡️ /mesh/domain/gke-prod-cluster/kad/1.0.0).
2. Auto-Intersection Bootstrapping: The control plane queries the database for all nodes sharing a specific domain token. It automatically returns their multiaddresses to each other as bootstrap peers only for that specific domain's DHT pipe.
3. DHT Mode Optimization:

• If a node is in exactly one domain, the control plane configures its enrollment profile to run the DHT in Client Mode (saves CPU/memory).
• If a node is a designated Router in multiple domains, it is configured to run in Server Mode across all of them, enabling it to act as an offline directory and handle cross-domain delegated routing.

---

📖 Concrete Example: Multi-Cloud Agent Mesh

Imagine an enterprise environment with an agent running in Google Kubernetes Engine (GKE) that needs to access an LLM or local MCP server running inside AWS EKS, connected via an On-Premise Global Backbone.

Step 1: Admin Configuration via the API

The administrator creates three domains and registers the nodes.

1. Create Domains: gke-local, eks-local, and global-backbone.
2. Enroll the Edge Agents:

• Agent-GKE is assigned to ["gke-local"].
• Agent-AWS is assigned to ["eks-local"].

3. Enroll the Routers (The Bridges):

• Router-GCP is assigned to ["gke-local", "global-backbone"].
• Router-AWS is assigned to ["eks-local", "global-backbone"].

Step 2: What the Control Plane Generates (Under the Hood)

The control plane reads the database records and generates the following instructions for the data plane when the nodes pull their configuration profiles:

• Agent-GKE: Instantiates one libp2p DHT protocol (/mesh/domain/gke-local/...) in Client Mode. Its bootstrap peer list contains only Router-GCP.
• Router-GCP: Instantiates two separate libp2p DHT networks in Server Mode:
• DHT 1: /mesh/domain/gke-local/... (listens for local cluster pods).
• DHT 2: /mesh/domain/global-backbone/... (bootstraps directly to Router-AWS).

Step 3: Runtime Cross-Domain Search over libp2p

When Agent-GKE wants to talk to Agent-AWS, the query ripples through the routing domains automatically:

[Agent-GKE] 
    |  (1) Looks for Agent-AWS in local DHT '/mesh/domain/gke-local/' -> Fails.
    v
[Router-GCP] 
    |  (2) Receives escalated request. Not found in 'gke-local' table.
    |  (3) Delegates lookup to its second DHT pipe: '/mesh/domain/global-backbone/'.
    v
[Router-AWS]
    |  (4) Hears query on 'global-backbone'. Matches target to its own 'eks-local' table.
    v
[Agent-AWS] (Found!)

A secure, end-to-end libp2p circuit stream is opened directly between Agent-GKE and Agent-AWS. The admin never had to configure network interfaces, firewall rules, or complex trees—they just tagged the nodes with logical routing domains, and the system self-assembled the architecture.

[bharqav]

Hi @aojea, this is a really clean design.

Are you (or anyone else) already working on an implementation for this? If not, I'd like to take a crack at it, happy to start with a specific slice first (e.g. the domain CRUD API in /api/v1/mesh/domains, or the protocol name hashing piece) to make sure I'm aligned with the intended approach before tackling the full pipeline.

Could you assign this to me if it's open? Let me know if there's a preferred order of implementation or any design decisions still up in the air that I should be aware of before starting.

[aojea]

@bharqav as we are working now towards the v0.1.0 we are focusing on stabilize and provide a strong core foundation and apis and interfaces , once we get all those pieces stables we must tackle this problem, but I want to avoid premature optimization so I think we should postpone this for later, although discussions and prototypes are welcome if you don't mind spending the time on rebases later

[bharqav]

Not a problem, also do let me know if I could work towards anything in this repo, would love to contribute

[aojea]

Not a problem, also do let me know if I could work towards anything in this repo, would love to contribute

as we walk towards v0.1.0 the main goal now is to start to encode the user stories and start validating assumptions and defining the contracts, see #294 per example on how to integrate with vscode.

So run an agent , connect to the mesh , use the console, connnect more agents, define policies so you can create kind of sovereign rules ... and provide feedback, make suggestions, ...