feat: characterize and tune machine-ingestion pipeline knobs toward linear scaling

[shayan1995]

Is this a new feature, an enhancement, or a change to existing functionality?

Enhancement

How would you describe the priority of this feature request

Medium

Please provide a clear description of problem this feature solves

Machine ingestion time currently grows superlinearly with fleet size (see #3721): 3,000 machines ingest in ~25 min, but 13,500 take many hours. The pipeline is gated by a handful of configuration knobs whose combined effect has never been measured in isolation, so we don't know which ones buy linearity and which ones destabilize the pipeline (e.g. explorations_per_run set too high stops explore cycles from completing, which starves machine creation entirely).

This epic runs a controlled, one-knob-at-a-time tuning campaign on a machine-a-tron simulated fleet (1,000 hosts × 2 DPUs for iteration; 4,500 × 2 for confirmation) to characterize each knob and produce a recommended scale configuration.

Feature Description

The knobs under test, with code-verified defaults:

Knob	Default	Current scale-profile value	Role
site_explorer.run_interval	120s	120s (untouched)	Period of the explore → identify → create cycle
site_explorer.concurrent_explorations	30	100	Parallel Redfish probes per cycle
site_explorer.explorations_per_run	90	120	Endpoints selected per cycle
site_explorer.machines_created_per_run	4	40	Hard cap on machines created per cycle
firmware_global.concurrency_limit	16	16 (untouched)	Preingestion concurrency width
firmware_global.run_interval	30s	30s (untouched)	Preingestion loop period
state_controller.max_concurrency	10	10 (untouched)	Parallel ingestion state-machine tasks (effective ceiling ~100: COMMAND_BUFFER_SIZE = 100 in crates/api-db/src/work_lock_manager.rs)

Creation-phase throughput model: hosts_per_hour ≈ machines_created_per_run × (3600 / run_interval), provided each cycle completes within the interval. Current scale settings give a ~3.75 h floor for 4,500 hosts, consistent with observed runs.

Each sub-task below changes exactly one knob against the same baseline, with full cleanup between runs and per-phase rate measurements (DHCP / exploration / preingestion / creation / init), so results are attributable.

Describe your ideal solution

Outcome of the epic: a measured knob-by-knob report, an updated scale profile with the winning combination verified at 4,500 hosts, and identification of any code-level bottlenecks that config alone cannot fix (feeding #3721).

Describe any alternatives you have considered

Changing several knobs at once would be faster but makes results unattributable; a knob that silently destabilizes the pipeline (as explorations_per_run ≈ 400 did) would be masked by the others.

Additional context

Method and full run matrix: ingestion tuning plan. Baseline measurements: docs/development/machine-a-tron-scale-testing.md. Related: #3721 (code-side changes for linear ingestion).

Code of Conduct

• I agree to follow NVIDIA Infra Controller's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request