Two Azure confidential VMs from two datacenters cannot establish node channel

[Mingyuan Gao (MingyuanGao)]

Hello, CCF Team, thank you for this excellent open-source project.
I have two confidential VMs in Azure, one in East US and the other in Canada Central. Both of them have private and public IPs.
I started "cchost" in the East US VM with the "start" subcomand, and "cchost" in the Canada Central with the "join" subcommand".
The Canada Central node is able to connect to the East US node.
However, the East US node can't establish node channel with the Canada Central node.
I guess it is the IP problem. How should I configure the iptables in these VMs?

Log from the East US node:
| Recovery threshold unset. Defaulting to number of initial consortium members with a public encryption key (3).
| Recovered ledger entries up to 0, committed to 0
| Listening for node-to-node on 10.0.0.4:10000
| Listening for RPCs on 10.0.0.4:443
| Creating new node: new network (with 3 initial member(s) and 3 member(s) required for recovery)
ariable for 'AZDCAP_DEBUG_LOG_LEVEL'
| Created new node
| Setting max open sessions to 1000
| Becoming leader e0e24c746120edc77ef96760b4a807eb8745d2ba57d0a49b653d411d4f718b95: 2
| Created service
| Network TLS connections now accepted
| Created new node e0e24c746120edc77ef96760b4a807eb8745d2ba57d0a49b653d411d4f718b95
| Starting thread: 0
| All threads are ready!
| Node e860b52b81ffcf919a41a731cd529aaac23eb129bf6670992f8146dc92320191 added as TRUSTED
| Initiating node channel with e860b52b81ffcf919a41a731cd529aaac23eb129bf6670992f8146dc92320191.
| Added raft node e860b52b81ffcf919a41a731cd529aaac23eb129bf6670992f8146dc92320191
| Initiating node channel with e860b52b81ffcf919a41a731cd529aaac23eb129bf6670992f8146dc92320191.
| Initiating node channel with e860b52b81ffcf919a41a731cd529aaac23eb129bf6670992f8146dc92320191.
| Initiating node channel with e860b52b81ffcf919a41a731cd529aaac23eb129bf6670992f8146dc92320191.
...

Log from the Canada Central node:
| Recovered ledger entries up to 0, committed to 0
| Listening for node-to-node on 10.2.1.4:10000
| Listening for RPCs on 10.2.1.4:443
| Creating new node - joining existing network at 52.136.126.130:443
| No snapshot found: Node will replay all historical transactions
ariable for 'AZDCAP_DEBUG_LOG_LEVEL'
| Created new node
| Setting max open sessions to 1000
| Node TLS connections now accepted
| Created join node e860b52b81ffcf919a41a731cd529aaac23eb129bf6670992f8146dc92320191
| Starting thread: 0
| All threads are ready!
| Network TLS connections now accepted
| Node has now joined the network as node e860b52b81ffcf919a41a731cd529aaac23eb129bf6670992f8146dc92320191: all domains
| Becoming leader e860b52b81ffcf919a41a731cd529aaac23eb129bf6670992f8146dc92320191: 1
| Becoming candidate e860b52b81ffcf919a41a731cd529aaac23eb129bf6670992f8146dc92320191: 1

[Amaury Chamayou (achamayou)]

Hi Mingyuan Gao (@MingyuanGao),

You are setting up the node-to-node channels to listen on a private IP (10.2.1.4), which will only be visible to machines on the same vnet. Vnets cannot (as far as I know) span regions, they are scoped inside a resource group which is also only ever in a single region. The initial join is an RPC, and therefore happens over the RPC interface where a public (advertised) and private distinction can be made. That is why it's working.

So you either need to set up some routing between the vnets you have set up across regions, or peer them so they effectively function as one: https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview

If you go down this route, you may find the following Azure CLI recipe helpful (adjust regions etc as necessary, don't forget to peer both ways each time):

az group create --name $EU_RG --location westeurope --subscription $SUBSCRIPTION
az group create --name $US_RG --location eastus --subscription $SUBSCRIPTION

az network vnet create \
  --name cc-eu-vnet \
  --resource-group $EU_RG \
  --location westeurope \
  --address-prefix 10.1.0.0/16 \
  --subnet-name cc-eu-subnet \
  --subscription $SUBSCRIPTION

az network vnet create \
  --name cc-us-vnet \
  --resource-group $US_RG \
  --location eastus \
  --address-prefix 10.2.0.0/16 \
  --subnet-name cc-us-subnet \
  --subscription $SUBSCRIPTION

US_VNET_ID=$(az network vnet show \
             --resource-group $US_RG \
             --name cc-us-vnet \
             --query id --out tsv \
             --subscription $SUBSCRIPTION)

az network vnet peering create \
  --name cc-eu-to-us-peering \
  --resource-group $EU_RG \
  --vnet-name cc-eu-vnet \
  --remote-vnet $US_VNET_ID \
  --allow-vnet-access \
  --subscription $SUBSCRIPTION

EU_VNET_ID=$(az network vnet show \
             --resource-group $EU_RG \
             --name cc-eu-vnet \
             --query id --out tsv \
             --subscription $SUBSCRIPTION)

az network vnet peering create \
  --name cc-us-to-eu-peering \
  --resource-group $US_RG \
  --vnet-name cc-us-vnet \
  --remote-vnet $EU_VNET_ID \
  --allow-vnet-access \
  --subscription $SUBSCRIPTION

I hope this helps.

[Mingyuan Gao (MingyuanGao)]

Thanks for your answer. "vnet peering" solved this issue.

[First of ME (IFirstYou)]

I think so...