High Availability

[gsanchietti]

Implement High Availability on a 2 node cluster.
The cluster should be composed by:

• one leader node
• one backup node

Supported scenarios:

• multiple WANs with configured static IPs and DHCP
• multiple LANs
• configuration must be kept in sync from leader do backup node
• automatic switchover from leader to backup
• manual switch from backup to leader

The user should be able to configure the High Availability from the CLI.

References:

• Pull Request #871
• README
• OpenWRT High Availability Documentation
• Keepalived Sync Documentation

[gsanchietti]

Test case 1: clean HA cluster install and initialization

• On two new NethServer nodes with identical hardware, set static LAN IPs (e.g. 192.168.100.238 and 192.168.100.239).
• On the primary node, run ns-ha-config check-primary-node and verify all requirements are met.
• On the primary node, run ns-ha-config check-backup-node 192.168.100.239 and verify all requirements are met.
• On the primary node, run ns-ha-config init-primary-node 192.168.100.238 192.168.100.239 192.168.100.240/24.
• On the backup node, run ns-ha-config init-backup-node and provide the password if prompted.
• On both nodes, run ns-ha-config status and verify:
  • Status: enabled
  • Role: primary/backup
  • Current State: master/backup
  • Last Sync Status: Up to Date (after initial sync)
• Possible issues: SSH connectivity, mismatched hardware, missing static IPs, sync failures.

Test case 2: add and remove WAN interface

• On the primary node, run ns-ha-config add-interface wan 192.168.122.49/24 192.168.122.1.
• On both nodes, verify the WAN interface is present and configured with a fake IP in 169.254.0.0/16.
• On the primary node, run ns-ha-config show-config and verify the WAN interface is listed under "Interfaces".
• On the primary node, run ns-ha-config remove-interface wan.
• On the primary node, run ns-ha-config show-config and verify the WAN interface is removed from HA configuration.
• Possible issues: interface not created, wrong IP assignment, removal not reflected.

Test case 3: add and remove alias

• On the primary node, run ns-ha-config add-alias lan 192.168.100.66/24.
• On the primary node, run ns-ha-config show-config and verify the alias is listed under "Aliases".
• On the primary node, run ns-ha-config remove-alias lan 192.168.100.66/24.
• On the primary node, run ns-ha-config show-config and verify the alias is removed.
• Possible issues: alias not added/removed, alias not reflected in keepalived config.

Test case 4: failover and recovery

• With both nodes running, disconnect the primary node from the network.
• On the backup node, run ns-ha-config status and verify:
  • Role: backup
  • Current State: master
  • Virtual IP is now active on the backup node.
• Reconnect the primary node and verify it regains the master role and virtual IP.
• Possible issues: virtual IP not switching, services not starting on backup, split-brain.

Test case 5: configuration synchronization

• On the primary node, make a configuration change (e.g. add a firewall rule).
• Wait for synchronization to complete.
• On the backup node, verify the configuration change is present (check /usr/share/keepalived/rsync/ or relevant config).
• Check /var/log/messages for sync logs.
• Possible issues: sync delays, missing files, rsync/SSH errors.

Test case 6: DHCP and DNS server HA

• Enable DHCP and DNS server on the primary node.
• Ensure "Force DHCP server start" and "router" options are set to the virtual IP.
• On a client, verify DHCP lease and DNS resolution via the virtual IP.
• Simulate failover and verify DHCP/DNS continue to work via the backup node.
• Possible issues: DHCP not starting, wrong router option, service not failing over.

Test case 7: unsupported configuration detection

• On the primary node, configure a PPPoE WAN interface.
• Run ns-ha-config check-primary-node and verify it reports the unsupported configuration.
• On the backup node, configure a non-static LAN IP and verify detection.
• Possible issues: unsupported configs not detected, misleading error messages.

Test case 8: maintenance and reset

• On the backup node, run /etc/init.d/keepalived stop and perform maintenance (e.g. reboot).
• Start keepalived again and verify the node rejoins the cluster.
• On the primary node, run /etc/init.d/keepalived stop and verify the backup takes over.
• Run ns-ha-config reset on the primary node and verify:
  • keepalived and conntrackd are stopped/disabled
  • configuration files and SSH keys are removed
  • network configuration is unchanged
• Possible issues: services not stopping, reset not cleaning up, backup not taking over.

Test case 9: backup node limitations

• On the backup node, verify:
  • No Internet access (cannot resolve DNS, cannot reach controller/portals)
  • Disabled cronjobs (subscription heartbeat, phonehome, remote backup)
• Possible issues: backup node has Internet access, cronjobs not disabled.

Test case 10: log and troubleshooting checks

• Trigger a configuration sync and check /var/log/messages for ns-rsync.sh entries.
• Add/remove an interface and check for corresponding log entries.
• On both nodes, check the bash prompt for [P] or [B] to distinguish node roles.
• Possible issues: missing logs, prompt not updated, unclear status.

Test case 11: negative test - sync failure

• Block SSH port 65022 between nodes.
• On the primary node, trigger a sync and verify Last Sync Status: SSH Connection Failed.
• Unblock the port and verify sync resumes.
• Possible issues: sync status not updated, no error reported.

Test case 12: OpenVPN RoadWarrior and tunnel HA

• On the primary node, configure an OpenVPN RoadWarrior server and a site-to-site tunnel.
• Connect a client and verify VPN connectivity via the virtual IP.
• Simulate failover (disconnect primary node).
• On the backup node, verify VPN connectivity is maintained and OpenVPN services are running.
• Reconnect the primary node and verify it resumes VPN services.
• Possible issues: VPN not starting on backup, tunnel not restored, client disconnects.

Test case 13: IPsec tunnel (strongSwan) HA

• On the primary node, configure an IPsec tunnel.
• Establish the tunnel and verify connectivity.
• Simulate failover.
• On the backup node, verify the tunnel is re-established and traffic flows.
• Possible issues: tunnel not imported/enabled, routes missing, tunnel not restored after failover.

Test case 14: WireGuard tunnel HA

• On the primary node, configure a WireGuard tunnel.
• Connect a client and verify connectivity.
• Simulate failover.
• On the backup node, verify the WireGuard interface is enabled and connectivity is restored.
• Possible issues: interface not enabled, keys not synced, tunnel not restored.

Test case 15: Threat Shield IP (banip) and DNS (adblock) HA

• On the primary node, enable Threat Shield IP and DNS protection.
• Verify threat lists are downloaded and blocking is active.
• Simulate failover.
• On the backup node, verify threat protection is still enforced.
• Possible issues: lists not synced, blocking not enforced, adblock/banip not running.

Test case 16: Reverse proxy (nginx) HA

• On the primary node, configure a reverse proxy rule.
• Access a backend service via the virtual IP and verify correct proxying.
• Simulate failover.
• On the backup node, verify the reverse proxy is still functional.
• Possible issues: nginx config not synced, proxy not working after failover.

Test case 17: ACME certificates HA

• On the primary node, request and install an ACME certificate.
• Verify the certificate is used by services (e.g., HTTPS).
• Simulate failover.
• On the backup node, verify the certificate is present and used.
• Possible issues: certificate not synced, services using wrong certificate.

Test case 18: Users and objects database HA

• On the primary node, add a user and an object.
• Simulate failover.
• On the backup node, verify the user and object are present.
• Possible issues: database not synced, missing users/objects.

Test case 19: Netmap HA

• On the primary node, configure Netmap.
• Verify mapping is active.
• Simulate failover.
• On the backup node, verify Netmap continues to function.
• Possible issues: config not synced, mapping not enforced.

Test case 20: Flashstart HA

• On the primary node, configure Flashstart.
• Verify filtering is active.
• Simulate failover.
• On the backup node, verify Flashstart continues to function.
• Possible issues: config not synced, filtering not enforced.

Test case 21: SNMP server (snmpd) HA

• On the primary node, enable SNMP server.
• Query SNMP via the virtual IP and verify response.
• Simulate failover.
• On the backup node, verify SNMP server responds via the virtual IP.
• Possible issues: snmpd not running, config not synced.

Test case 22: NAT helpers

• On the primary node, configure NAT helpers.
• Verify NAT helpers are active.
• Simulate failover.
• On the backup node, verify NAT helpers.
• Possible issues: helpers not loaded.

Test case 23: Backup encryption password HA

• On the primary node, set a backup encryption password.
• Simulate failover.
• On the backup node, verify the encryption password is present.
• Possible issues: password not synced.

Test case 24: Controller connection (ns-plug) HA

• On the primary node, connect to the controller.
• Simulate failover.
• On the backup node, verify controller connection is enabled when the node becames active.
• Possible issues: backup node tries to connect to controller.

Test case 25: Active connections tracking (conntrackd) HA

• On the primary node, generate active connections (e.g., ongoing downloads).
• Simulate failover.
• On the backup node, verify connections persist (minimal interruption).
• Possible issues: connections dropped, conntrackd not syncing state.

Test case 26: Connect to backup node

• After the cluster have been configured, on the primary node execute: ns-ha-config ssh-remote
• You must be able to operate on the backup node

Test case 27: Upgrade backup node

• After the cluster have been configured, on the primary node execute: ns-ha-config upgrade-remote
• Check the backup node has been upgraded with latest available image

Test case 28: Sync alert

• Make sure the machine has a valid subscription
• After the cluster have been configured, switch off the backup node
• The ha:sync:failed alert must be present inside the my.nethesis.it
• Connect back the backup node
• The ha:sync:failed alert bust be resolved

Test case 29: Primary node alert

• Make sure the machine has a valid subscription
• After the cluster have been configured, switch off the main node
• The ha:primary:failed alert must be present inside the my.nethesis.it
• Connect back the backup node
• The ha:primary:failed alert bust be resolved

[cotosso]

Test case 1: clean HA cluster install and initialization

ns-ha-config check-primary-node
DHCP options (both force and router) are required even if DHCP is disabled, otherwise the ns-ha-config check-primary-node command ask to configure them.
The documentation currently states that these options only need to be specified if the DHCP server is enabled.
I suggest to force not only the router but also the DNS server in the dhcp options with the VirtualIP.

I'm fine with always setting them regardless, to avoid surprises in case DHCP is enabled later on, as long as this is properly documented.

ns-ha-config status command is ok on both sides

[cotosso]

Test case 2: add and remove WAN interface
Woks as expected

[cotosso]

Test case 3: add and remove alias
All commands work as expected.