feat: rewrite fixed_fallback with background goroutine for independent retry and zero-latency switchback

[itoywh]

Problem

The existing fixed() dialer policy is all-or-nothing: when the configured node dies, traffic either fails immediately (if no fallback) or falls back without any retry window. The original fixed_fallback() (this PR's initial version) improved this but still had three major flaws:

1. Traffic-driven retries: Retry counting depends on natural traffic passing through Select() to compute elapsed/retryCount. If the dead node has no traffic, retries never advance and fallback never happens.
2. First timeout window wasted: After node death, up to one full timeout window's worth of traffic hits the dead node before fallback.
3. Health check dead, but retry unaware: Health check confirms node dead, but fixed_fallback state doesn't update — must wait for natural traffic.

Solution: v2 — Two-Layer Architecture

The rewrite introduces a background goroutine that independently drives retries, completely decoupled from natural traffic.

How It Works

                    .-- MustGetAlive=true  --- Return fixed node normally
                    |
  Select() ---------+                                    Natural Traffic
                    |                                       .
                    .-- MustGetAlive=false --- . goto doFallback . Backup node
                                               |
                                               +--- Start background goroutine (if not running)
                                               |     |
                                               |     +--- Fire immediate probes (NotifyCheckTcp/DnsUdp)
                                               |     |    . immediate probe counts as first retry (retryCount=1)
                                               |     |    . recovers brief transient failures (~300ms) instantly
                                               |     |
                                               |     +--- ticker every timeout ---+--- Probe OK . deadSince=0
                                               |     |                            .   Fail . retryCount++
                                               |     |                                          .   .retries . permanent fallback
                                               |     |
                                               |     .--- Fire NotifyCheckTcp/DnsUdp probes each tick
                                               |
                                               .--- Subsequent Select() . deadSince!=0 . goto doFallback

Two-Layer Architecture

Layer	Responsibility	Sends traffic to dead node?
Natural Traffic (Select())	Fallback immediately on dead; switch back immediately on alive	No (first byte goes to backup)
Background Goroutine	Independent ticker-driven retry cycle; fires immediate probe on start, then periodic probes	Yes, probe-only (health check requests)

Four Key Event Flows

Node Go Dead (two paths)

Path A . Health check detects death:
  connectivity_check: Alive=false
    . aliveTransitionCallback fires
    . CompareAndSwap(false,true)
    . start background goroutine  . no traffic needed!

Path B . Select() discovers death:
  MustGetAlive=false, deadSince==0
    . record deadSince timestamp
    . start goroutine if not already running
    . goto doFallback  . immediately!

Natural Traffic During Death

Select() . MustGetAlive=false . deadSince!=0 . goto doFallback . return backup node

Zero traffic wasted on dead node. Natural traffic does NOT participate in retry counting.

Background Retry Probing

Goroutine starts:
  . Fire immediate NotifyCheckTcp() + NotifyCheckDnsUdp() probes
  . retryCount = 1 (immediate probe counts as first retry)

Ticker fires (e.g. 3s later):
  . MustGetAlive?
       +--- true  . deadSince=0, retryCount=0 . return (goroutine exits)
       .   false
            +--- retryCount >= retries (from immediate reset) . permanent fallback . return
            .   retryCount++ . fire probes . wait next tick

The immediate probe on goroutine start eliminates the first-ticker wait for transient failures. NotifyCheckTcp() and NotifyCheckDnsUdp() are also fired each subsequent tick.

Node Revive

Traffic/probe succeeds . MustGetAlive=true
  +--- Goroutine path . deadSince=0, retryCount=0 . return
  .   Select() path . deadSince=0, retryCount=0 . return fixed node

Zero-latency switch back: recovery within probe RTT (~150ms) for brief glitches, or within ticker cycle for sustained failures.

Retry Counting Semantics

retries	Immediate probe	Ticker probes	Total probes	Recovery window
0	None	None	0	Relies on periodic health check (M1)
1	1 (counts as 1st)	0 (tick exits immediately)	1	~150ms
3	1 (counts as 1st)	2 (ticks 1-2)	3	~6s (3s x 2 ticks)

The immediate probe counts toward the retries budget, so retries=3 means immediate + 2 tickers = 3 total probes.

Key Implementation Details

• aliveTransitionCallback: Health check on dead transition automatically starts the goroutine . no traffic required
• deadSince Timestamp: Records when death was first observed, distinguishing first vs subsequent Select()
• CompareAndSwap: Ensures exactly one goroutine runs
• Probe Cooldown: timeout below 2s clamped to 2s with WARN log, prevents probe storm
• Immediate Probe: NotifyCheckTcp fires before first ticker tick, eliminating 3s first-probe latency
• NotifyCheckTcp 2s Cooldown: Built-in CAS rate limiting prevents probe storms from concurrent callers
• Mutex protection: Retry state (deadSince + retryCount) protected by single sync.Mutex for atomic multi-field reads
• State Cleanup on Revive: MustGetAlive=true clears both deadSince and retryCount

Configuration

Basic: 3s x 3 retry, random fallback

group {
  name 'my-group'
  policy 'fixed_fallback(0, 3s, 3, random)'
}

Behavior: Always use dialer[0]. Dies -> natural traffic -> backup; goroutine probes immediately, then 3s x 2; if all fail, goroutine exits; node recovers -> instant switchback.

Aggressive: Quick fallback

policy 'fixed_fallback(1, 5s, 2, min_moving_avg)'

Behavior: Prefer dialer[1]. Dies -> immediate fallback, goroutine immediate probe + 1 ticker retry.

Conservative: Long retry window

policy 'fixed_fallback(0, 60s, 5)'

Behavior: Prefer dialer[0]. Goroutine immediate probe + 4 ticker retries = ~4 minutes before giving up.

Minimal: Safety net

policy 'fixed_fallback(0)'

Behavior: Same as fixed(0) with safety net . defaults timeout=3s, retries=3, fallback=min_moving_avg.

Parameter Reference

Param	Position	Default	Description
index	1st (required)	-	Dialer index (0-based)
timeout	2nd	3s	Retry interval. Below 2s clamped with WARN. Supports ms/s/m.
retries	3rd	3	Max retries. 0 = no probes, rely on periodic health check.
fallback_policy	4th	min_moving_avg	Policy after exhaust: random, min, min_moving_avg, min_avg10

Benefits vs Original Approach

Aspect	Original (v1)	Improved (v2)
Retry driver	Natural traffic only	Background goroutine (independent)
Dead->fallback delay	Up to 1 timeout window	Zero (immediate)
First probe latency	1 full timeout window (e.g. 3s)	Immediate (~150ms, instant probe before ticker)
Dead->start retry	Wait for traffic	Immediate (health check or traffic)
Retry counting	Traffic-dependent	Immediate probe counts as first retry; clean 1+2=3
Probe storm protection	None	2s minimum + NotifyCheckTcp 2s cooldown CAS
State access	Non-atomic atomic.Int64	Mutex-protected single struct

Changes Summary

• Select(): Natural traffic always falls back immediately on dead node, no timeout window wasted
• Background goroutine: Independent ticker drives retry cycle, fires NotifyCheck probes
• Immediate probe: NotifyCheckTcp fires before first ticker tick, recovers transient failures instantly
• Retry counting: Immediate probe counts as retry 1; retries=N = immediate + (N-1) tickers
• aliveTransitionCallback: Health check dead transition starts goroutine without waiting for traffic
• 2s minimum: timeout clamped to minimum 2s with WARN log
• Mutex state: deadSince + retryCount protected under single sync.Mutex
• Node operational logging: WARN on DEAD, INFO on ALIVE (merged from feat: add operational logs for node alive/dead transitions #1020)
• DNS-UDP false positive fix: Redirect DNS-UDP collection to TCP when no udp_check_dns configured
• gofmt + DRY + tests: Clean code, deduplicated IP-family check, 7 unit tests
• Docs: Flow diagram, two-layer architecture, bilingual (zh/en) documentation

Closes #1020, Closes #1022

[itoywh]

Hi maintainers, I've rebased this PR onto the latest main (as of today). This enhancement addresses a real pain point — when a fixed dialer goes down, the original fixed_fallback has no actual fallback mechanism (latency tracking is broken due to AliveDialerSet being created with FixedWithFallback policy). With this PR, the fixed dialer properly falls back and auto-recovers when healed.

I've been running this in production for a while and it works reliably. Would appreciate a review when you have time. Thanks! 🙏

[itoywh]

Testing update from v2.0-custom deployment

I've been running this PR's logic (ported to v2.0.0rc1) on a production-like environment (192.168.2.4, x86_64) since June 15. During testing I found and fixed 3 additional issues that are also applicable to the v1.x version in this PR:

1. Sub-second timeout precision (critical)

time.Now().Unix() returns seconds — any sub-second timeout (e.g. fixed_fallback(0, 500ms, 3)) gets truncated to 0, making retries fire instantly instead of after the intended delay. Fix: use UnixNano().

// Before (broken for sub-second timeouts)
elapsed := time.Now().Unix() - deadSince.Load()

// After
elapsed := time.Now().UnixNano() - deadSince

2. Non-atomic multi-field read race

fixedFallbackDeadSince and fixedFallbackRetryCount were separate atomic.Int64 fields. A concurrent goroutine could read deadSince from one state and retryCount from another, corrupting the retry decision. Fix: wrap both fields under a single sync.Mutex.

// Before (race-prone)
fixedFallbackDeadSince atomic.Int64
fixedFallbackRetryCount atomic.Int64

// After (atomic across both fields)
fixedFallbackMu sync.Mutex
fixedFallbackDeadSince int64
fixedFallbackRetryCount int

3. Emergency probes should fire on every timeout tick

Original logic only fired emergency probes when retries < maxRetries, skipping them after retries were exhausted. Fix: always fire emergency probes on timeout ticks — this allows the node to recover even after all retries are used up.

These improvements have been battle-tested with real traffic for 2+ days. I can push them to this PR branch if the maintainers are interested.

[ppdragon16]

我觉得其实应该引入dae-next那样的priority的概念，比如：

  mix {
    filter: subtag(ack) && name(keyword:'jui_v6') [priority: '0,2(,100ms),1(100ms,200ms)']
    filter: subtag(evo) && name(keyword:'_v6') [priority: '1']
    filter: subtag(llg) && name(keyword:'Japan 0')
    policy: min_moving_avg(5s, 0.3)
  }

可以自由搭配出各种效果。

[itoywh]

Updated with three review fixes:

1. retries=0 immediate fallback: runFixedFallbackRetry now returns immediately instead of waiting for first ticker tick
2. explicit fixedFallbackDone flag: replaces the deadSince time-reversal trick with a proper boolean flag checked by Select()
3. MovingAverage concurrency: confirmed all read paths are under collectionFineMu.RLock(), no data race

Commit: e30d961

[itoywh]

Review fix: tighten param range to 1-3, drop unused parsePolicyName (points 5)

• 5a: Changed param range from > 4 to > 3 (error message now correctly says expected 1-3 params). The 4th param (fallback_policy) was parsed but never configured in practice.
• 5b: Removed the now-unused parsePolicyName function entirely (the v2.0-custom branch already had "min" as an alias for backward compat, so no breaking change there).

Point 4 (control_plane_core.go isBpfEjected/closeTail): not touched by this PR — these come from upstream baseline, unrelated to fixed_fallback changes.

[itoywh]

Reverted the param range change — the 4th param (fallback_policy) IS used in production configs (e.g. fixed_fallback(3, 3s, 3, random)). My v2.0-custom deployment crashed because of this. Reverted the commit on both v2.0-custom and pr1-fixed-fallback.

The parsePolicyName function is back too with the "min" alias included for backward compatibility (in addition to "min_last_latency").

[itoywh]

Removed dead code (review points A & B)

• Removed fixedFallbackStopCh field + all make(chan struct{}) inits + select case (channel was never closed, so the select path was dead code)
• Removed unused fixedFallbackLastLogTime atomic.Int64 field (never read/written)

Kept fixedFallbackRunning and fixedFallbackLastLogMark which are actively used.

Point C (single-networkType monitoring blind spot): acknowledged as a design trade-off, not a bug — not modified.

[itoywh]

Fixed error message consistency: expected 1-3 params → expected 1-4 params to match the actual validation range > 4. (Residue from the revert.)