Skip to content

Add service for Slurm topology collection - #175

Open
giongto35 wants to merge 1 commit into
mainfrom
thanhnh/scontrol-topology-scuba-service
Open

Add service for Slurm topology collection#175
giongto35 wants to merge 1 commit into
mainfrom
thanhnh/scontrol-topology-scuba-service

Conversation

@giongto35

@giongto35 giongto35 commented Aug 14, 2026

Copy link
Copy Markdown

Summary

Package and configure scontrol_topology as a single-node systemd service
that collects and publishes Slurm topology every 60 seconds.

This also adds Debian packaging and operator documentation. The collector
already existed, but it was not packaged or scheduled as a service.

Test Plan

  • git diff --check — passed.
  • black --check gcm/tests/test_scontrol_topology.py — passed.
  • flake8 gcm/tests/test_scontrol_topology.py — passed.
  • Parsed config.toml and verified the scontrol_topology settings — passed.
  • Verified scontrol_topology.service with systemd-analyze verify — passed.
  • Verified the Debian install rule includes scontrol_topology.service — passed.
  • End-to-end test in a development Slurm environment:
    • Deployed the GCM build to the monitoring workload.
    • Collected six topology records from scontrol show topo.
    • Published the records through the configured telemetry pipeline.
    • Confirmed all six records reached the downstream dataset with the expected
      topology fields and node counts.

@giongto35
giongto35 requested a review from calebho as a code owner August 14, 2026 05:21
@meta-cla meta-cla Bot added the cla signed label Aug 14, 2026
@github-actions

Copy link
Copy Markdown

CI Commands

The following CI workflows run automatically on every push and pull request:

Workflow What it runs
GPU Cluster Monitoring Python CI lint, tests, typecheck, format, deb build, pyoxidizer builds
Go packages CI shelper tests, format, lint

The following commands can be used by maintainers to trigger additional tests that require access to secrets:

Command Description Requires approval?
/metaci tests Runs Meta internal integration tests (pytest) Yes — a maintainer must trigger the command and approve the deployment request
/metaci integration tests Same as above (alias) Yes

Note: Only repository maintainers (OWNER association) can trigger /metaci commands. After commenting the command, a maintainer must also navigate to the Actions tab and approve the deployment to the graph-api-access environment before the jobs will run. See the approval guidelines for what to approve or reject.

@giongto35
giongto35 force-pushed the thanhnh/scontrol-topology-scuba-service branch from 30496b1 to e7694dd Compare August 14, 2026 05:44
Summary: Package and configure scontrol_topology as a single-node systemd service that collects and publishes Slurm topology every 60 seconds. Add Debian packaging and operator documentation while leaving exporter-specific routing in deployment configuration.

Test Plan: Static checks: git diff --check; black --check and flake8 on gcm/tests/test_scontrol_topology.py; TOML, Python syntax, systemd-analyze, and Debian packaging verification. End to end: deployed the GCM build to a development Slurm cluster, ran a live one-shot topology collection through the configured telemetry pipeline, and confirmed fresh rows in the target dataset with the expected topology fields and node counts.
@giongto35
giongto35 force-pushed the thanhnh/scontrol-topology-scuba-service branch from e7694dd to 7ab5e62 Compare August 14, 2026 06:20
@giongto35 giongto35 changed the title Deploy Slurm topology collector to Scuba Stream Slurm topology collector Aug 14, 2026
@giongto35 giongto35 changed the title Stream Slurm topology collector Add service for Slurm topology collection Aug 14, 2026
@giongto35
giongto35 requested a review from Yash0270 August 14, 2026 20:06

## Overview

Collects Slurm topology from `scontrol show topo` every 60 seconds. Run one

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

every 60sec is really aggressive? topology changes happens once an hour on avg.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we make it once every 5mins

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants