Summary
instrument_system_metrics() can produce very high metric volume on multi-process deployments, because the host-scope system metrics are emitted once per process rather than once per host. On a many-worker server (e.g. Gunicorn/Uvicorn --workers, Celery prefork) running on a high-core machine, this multiplies into a large number of time series.
Where it comes from
The dominant term is the two per-core metrics, system.cpu.time and system.cpu.utilization. Their OTel callbacks iterate psutil.cpu_times(percpu=True), so each emits one time series per (core × CPU-state field):
per-core CPU series = 2 metrics × cores × cpu_state_fields
On a 96-core Linux host (~8 CPU-state fields kept) that is 2 × 96 × 8 ≈ 1,536 series — roughly 95% of everything base='full' emits. base='basic' avoids this entirely (its CPU metric is the single aggregate system.cpu.simple_utilization), but the per-process duplication below still applies to whatever is enabled.
The per-process multiplication
The system metrics are host-scope, but they are emitted by every process that has them enabled:
- Each worker configures independently (
preload_app=False, spawn): each worker process runs instrument_system_metrics() itself, so each gets its own MeterProvider + exporter. The in-process singleton (GLOBAL_CONFIG, and SystemMetricsInstrumentor no-op'ing on re-instrument) only dedupes within a process, not across processes.
- Configure-then-fork (
preload_app=True): the forked child keeps exporting too — OTel's PeriodicExportingMetricReader registers os.register_at_fork(after_in_child=...) and restarts its export thread in the child, and logfire re-stamps process.pid in its own fork hook.
Either way, N workers on a host produce ~N copies of the host-scope series. With the default 60s export interval that is series × N × 1440 points/day.
Suggestion
Host-scope metrics (system.cpu.*, system.memory.*, system.disk.*, system.network.*) only need one emitter per host; process.* metrics are legitimately per-process and should stay per worker. Some options, roughly in order of blast radius:
- Fork-child suppression of host-scope metrics. Extend the
os.register_at_fork(after_in_child=...) hook logfire already registers so a forked child stops emitting host-scope system metrics (e.g. SystemMetricsInstrumentor().uninstrument() + no-op logfire's own system.* gauge callbacks), while keeping process.* per worker. Fixes the configure-then-fork case cleanly.
- Warn when already enabled in an ancestor. Set an inherited marker at instrument time and emit a warning (or no-op) if a descendant calls
instrument_system_metrics() again — makes the duplication self-evident.
- Optional host-level single-writer (file lock / abstract socket) so only the first process per host emits host-scope metrics. This is the only thing that also covers the each-worker-configures-independently case; heavier, so opt-in.
Documentation could also steer users to enable system metrics once per host (a master-only hook or a sidecar) rather than in the per-worker app-import path.
Environment note
Confirmed against the current SDK: default metric export interval is 60s (inherited from OTel; logfire passes no export_interval_millis), and the per-core percpu=True behavior is in opentelemetry-instrumentation-system-metrics.
Summary
instrument_system_metrics()can produce very high metric volume on multi-process deployments, because the host-scope system metrics are emitted once per process rather than once per host. On a many-worker server (e.g. Gunicorn/Uvicorn--workers, Celery prefork) running on a high-core machine, this multiplies into a large number of time series.Where it comes from
The dominant term is the two per-core metrics,
system.cpu.timeandsystem.cpu.utilization. Their OTel callbacks iteratepsutil.cpu_times(percpu=True), so each emits one time series per (core × CPU-state field):On a 96-core Linux host (~8 CPU-state fields kept) that is
2 × 96 × 8 ≈ 1,536series — roughly 95% of everythingbase='full'emits.base='basic'avoids this entirely (its CPU metric is the single aggregatesystem.cpu.simple_utilization), but the per-process duplication below still applies to whatever is enabled.The per-process multiplication
The system metrics are host-scope, but they are emitted by every process that has them enabled:
preload_app=False,spawn): each worker process runsinstrument_system_metrics()itself, so each gets its own MeterProvider + exporter. The in-process singleton (GLOBAL_CONFIG, andSystemMetricsInstrumentorno-op'ing on re-instrument) only dedupes within a process, not across processes.preload_app=True): the forked child keeps exporting too — OTel'sPeriodicExportingMetricReaderregistersos.register_at_fork(after_in_child=...)and restarts its export thread in the child, and logfire re-stampsprocess.pidin its own fork hook.Either way, N workers on a host produce ~N copies of the host-scope series. With the default 60s export interval that is
series × N × 1440points/day.Suggestion
Host-scope metrics (
system.cpu.*,system.memory.*,system.disk.*,system.network.*) only need one emitter per host;process.*metrics are legitimately per-process and should stay per worker. Some options, roughly in order of blast radius:os.register_at_fork(after_in_child=...)hook logfire already registers so a forked child stops emitting host-scope system metrics (e.g.SystemMetricsInstrumentor().uninstrument()+ no-op logfire's ownsystem.*gauge callbacks), while keepingprocess.*per worker. Fixes the configure-then-fork case cleanly.instrument_system_metrics()again — makes the duplication self-evident.Documentation could also steer users to enable system metrics once per host (a master-only hook or a sidecar) rather than in the per-worker app-import path.
Environment note
Confirmed against the current SDK: default metric export interval is 60s (inherited from OTel; logfire passes no
export_interval_millis), and the per-corepercpu=Truebehavior is inopentelemetry-instrumentation-system-metrics.