Feat #99: Add optional Kubernetes Indexed Job template for external_batch simulation execution - #103
Open
anchapin wants to merge 2 commits into
Open
Feat #99: Add optional Kubernetes Indexed Job template for external_batch simulation execution#103anchapin wants to merge 2 commits into
anchapin wants to merge 2 commits into
Conversation
anchapin
force-pushed
the
feat/issue-99
branch
from
August 17, 2026 05:13
c711ddd to
67a4218
Compare
Adds an opt-in batch/v1 Job (templates/job-external-batch.yaml) that
runs external_batch_run analyses as native Kubernetes Indexed Jobs on
the existing cluster, reusing the chart's NFS PVC. Each pod gets a
JOB_COMPLETION_INDEX via the downward API and processes one slice of
the manifest.json chunks array.
Files added:
- templates/job-external-batch.yaml -- rendered only when
external_batch.enabled=true. batch/v1 Job, completionMode: Indexed,
parameterized completions/parallelism/backoffLimit/activeDeadlineSeconds
/ttlSecondsAfterFinished. Pod uses worker node-group affinity
(openstudio-server.nodeGroupAffinity from # PR1), low priority, 5200s
termination grace (matches worker pods). Two volume mounts: the
chart's NFS PVC at storage.mountPath and a runner-script ConfigMap
at /scripts/run_chunk.rb (defaultMode 0o755). Container args invoke
the runner with --package, --results, --chunk JOB_COMPLETION_INDEX,
--openstudio /usr/local/bin/openstudio.
- templates/runner-cm.yaml -- ConfigMap sourced from
configmaps/runner/run_chunk.rb via .Files.Get. Carries the upstream
runner script (plain stdlib Ruby + OpenStudio CLI) so chunk pods
don't depend on the script being baked into the worker image.
- configmaps/runner/run_chunk.rb -- the upstream external_batch runner,
fetched verbatim from commit d63c8eb2e3b37bca2494e66cab3c4457cf5c4e61
of NatLabRockies/OpenStudio-server (the same commit that produced the
digest pinned in external_batch.container.image).
Files modified:
- openstudio-server/values.yaml -- append external_batch block. Image
default is pinned by digest to the runner-supporting upstream tag
(intentionally decoupled from chart appVersion 3.8.0-1, which
predates the runner). All other parameters mirror the issue spec.
- openstudio-server/Chart.yaml -- 0.6.1 -> 0.7.0 (minor). Opt-in
feature; default values render exactly as before.
Notable spec drift from the issue body (commit message for posterity):
The issue body proposed the runner as:
ruby /openstudio-server/external_batch/runner/run_chunk.rb \
--analysis-id <ID> \
--chunk-id <N> \
--storage-dir <DIR>
The actual upstream runner uses (verified by pulling the digest-pinned
image and inspecting /opt/openstudio/server/app/lib/external_batch/* +
raw.githubusercontent.com/NatLabRockies/OpenStudio-server/d63c8eb/
external_batch/runner/run_chunk.rb):
ruby run_chunk.rb \
--package <DIR> --results <DIR> --chunk <N> --openstudio <CMD>
Differences:
* run_chunk.rb is NOT in the nrel/openstudio-server image. It is
fetched from the OpenStudio-server repo at runtime. This PR ships
it via ConfigMap.
* Flags are --package/--results/--chunk/--openstudio (no
--analysis-id, no --storage-dir, --chunk is the chunk index).
* analysis_id is read from manifest.json, not from a CLI flag.
* The runner expects the package at <mountPath>/external_batch/
analysis_<id>/package/ and writes results to <mountPath>/
external_batch/analysis_<id>/results/. The server's
ExternalBatch::Packager writes that layout when an analysis is
submitted with --batch-run-method external_batch_run.
Verified:
* helm lint clean
* helm template at default values: 0 external_batch references
* helm template --set external_batch.enabled=true ...: 1 Job +
1 ConfigMap rendered
* Pod-args template content matches the actual runner CLI
* Storage PVC override (storage.existingClaim) propagates correctly
* Downward API for JOB_COMPLETION_INDEX uses
batch.kubernetes.io/job-completion-index (k8s 1.21+)
Closes #99.
Tests (openstudio-server/tests/job-external-batch_test.yaml, 14 cases): - default-disabled renders nothing - enabled renders Job with correct spec (completions, parallelism, completionMode: Indexed, backoffLimit, activeDeadlineSeconds, ttlSecondsAfterFinished, restartPolicy, priorityClassName, terminationGracePeriodSeconds) - enabled renders ConfigMap carrying run_chunk.rb - container image renders as the pinned digest - JOB_COMPLETION_INDEX is projected via downward API - NFS PVC + runner-script ConfigMap mounts - storage.existingClaim override propagates - container args invoke run_chunk.rb with the correct flag set (snapshot) - resources block renders limits + requests - ConfigMap data.run_chunk.rb starts with #!/usr/bin/env ruby shebang and contains the correct Usage: line (matchRegex) - empty container.image fails the render with a clear error - cluster-autoscaler safe-to-evict annotation is set - pod uses worker node-group affinity (isSubset) helm-unittest gotchas worked through (worth recording): * 'err:' is not a valid field; the right field for failure assertions is 'failedTemplate:' with an errorMessage. * 'service:' under release: is not parsed. Only name/namespace/revision /upgrade are accepted. * documentSelector: only takes path/value/matchMany/skipEmptyTemplates. Use the test-job level 'template:' to scope a single template. * Map keys containing '.' in path expressions use bracket syntax: data["run_chunk.rb"], not data.run_chunk.rb. * 'contains' requires an array path; for string fields use matchRegex. * Go regexp matches ^ at start-of-line by default; use \A for start-of-string anchors. * 'isSubset' matches an object subset; 'contains' matches an array element. The affinity block is an object, so isSubset. Snapshot tests cover the full run_chunk.rb script (truncated on display but verified via diff against the upstream file) and the container args block. Snapshot files live under openstudio-server/tests/__snapshot__/ and are committed. Docs (README.md): - New 'External Batch (Indexed Jobs)' section explaining dual-mode operation, the digest pin rationale, the dispatch workflow (package -> render template -> kubectl apply), the parameter table, and explicit notes on what this is NOT (not a worker replacement, not a substitute for AWS Batch/SLURM/Nomad). Verification: - helm unittest: 14/14 pass - helm lint: clean - README renders correctly
anchapin
force-pushed
the
feat/issue-99
branch
from
August 17, 2026 05:28
67a4218 to
2c85212
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #99
Stacked on #102 (chore/ci-helper-namespace) — PR 1's commit history doesn't appear in this PR's diff.
What
Adds an opt-in
batch/v1Job (templates/job-external-batch.yaml,completionMode: Indexed) that runsexternal_batch_runanalyses as native Kubernetes pods on the existing cluster, reusing the chart's NFS PVC. Standard Resque workers stay intact — both modes coexist.The runner script (
run_chunk.rb) ships via ConfigMap (templates/runner-cm.yaml) sourced fromconfigmaps/runner/run_chunk.rb. Each pod gets a uniqueJOB_COMPLETION_INDEXvia the downward API and processes one slice ofmanifest.json's chunks array.Image
external_batch.container.imagedefaults to a digest-pinnednrel/openstudio-server@sha256:9b0871fc...(tag:3.10.0-179D-test, commitd63c8ebof NatLabRockies/OpenStudio-server). Intentionally decoupled from the chart'sappVersion(3.8.0-1) which predates the runner. Temporary pin per upstream issue #864.Files
openstudio-server/configmaps/runner/run_chunk.rb— upstream runner (15.9 KB, fetched verbatim fromd63c8eb)openstudio-server/templates/runner-cm.yaml— ConfigMap sourced from the script via.Files.Getopenstudio-server/templates/job-external-batch.yaml— the Jobopenstudio-server/tests/job-external-batch_test.yaml— 14 helm-unittest casesopenstudio-server/tests/__snapshot__/job-external-batch_test.yaml.snap— snapshot baselineopenstudio-server/values.yaml—external_batch:blockopenstudio-server/Chart.yaml— 0.6.1 → 0.7.0README.md— new "External Batch (Indexed Jobs)" sectionSpec drift from issue body (recorded for reviewers)
The issue proposed
--analysis-id/--chunk-id/--storage-dirflags at/openstudio-server/external_batch/runner/run_chunk.rb. The actual upstream runner uses--package/--results/--chunk/--openstudio, and isn't in the Docker image at all (fetched from the OpenStudio-server repo). This PR uses the verified actual interface; full diff in the first commit message.Verified
helm lintcleanhelm unittest14/14 passhelm templateat default values: 0 external_batch refs (opt-in verified)helm template --set external_batch.enabled=true ...: 1 Job + 1 ConfigMap render correctlyStacking note
This PR's base is
chore/ci-helper-namespace(PR #102's branch), notdevelop. After #102 merges, retarget this PR's base todevelop(or rebase) before merging. The diff againstchore/ci-helper-namespaceshows only PR 2 changes.CI behavior on stacked PRs
PR #102's CI workflow triggers on
on.pull_request.branches: [develop, main]. This PR's base ischore/ci-helper-namespace, which is not in that list — so the CI workflow may NOT fire on this PR's open. If that's the case, either:developinstead (loses stacked-PR review surface), oron: pull_requestwith nobranches:filter — recommended as a follow-up after Chore: standardize helper namespace, fix PVC hardcode, add CI #102 merges.