Commit 604bca9
committed
Feat #99: add external_batch Indexed Job + runner ConfigMap
Adds an opt-in batch/v1 Job (templates/job-external-batch.yaml) that
runs external_batch_run analyses as native Kubernetes Indexed Jobs on
the existing cluster, reusing the chart's NFS PVC. Each pod gets a
JOB_COMPLETION_INDEX via the downward API and processes one slice of
the manifest.json chunks array.
Files added:
- templates/job-external-batch.yaml -- rendered only when
external_batch.enabled=true. batch/v1 Job, completionMode: Indexed,
parameterized completions/parallelism/backoffLimit/activeDeadlineSeconds
/ttlSecondsAfterFinished. Pod uses worker node-group affinity
(openstudio-server.nodeGroupAffinity from # PR1), low priority, 5200s
termination grace (matches worker pods). Two volume mounts: the
chart's NFS PVC at storage.mountPath and a runner-script ConfigMap
at /scripts/run_chunk.rb (defaultMode 0o755). Container args invoke
the runner with --package, --results, --chunk JOB_COMPLETION_INDEX,
--openstudio /usr/local/bin/openstudio.
- templates/runner-cm.yaml -- ConfigMap sourced from
configmaps/runner/run_chunk.rb via .Files.Get. Carries the upstream
runner script (plain stdlib Ruby + OpenStudio CLI) so chunk pods
don't depend on the script being baked into the worker image.
- configmaps/runner/run_chunk.rb -- the upstream external_batch runner,
fetched verbatim from commit d63c8eb2e3b37bca2494e66cab3c4457cf5c4e61
of NatLabRockies/OpenStudio-server (the same commit that produced the
digest pinned in external_batch.container.image).
Files modified:
- openstudio-server/values.yaml -- append external_batch block. Image
default is pinned by digest to the runner-supporting upstream tag
(intentionally decoupled from chart appVersion 3.8.0-1, which
predates the runner). All other parameters mirror the issue spec.
- openstudio-server/Chart.yaml -- 0.6.1 -> 0.7.0 (minor). Opt-in
feature; default values render exactly as before.
Notable spec drift from the issue body (commit message for posterity):
The issue body proposed the runner as:
ruby /openstudio-server/external_batch/runner/run_chunk.rb \
--analysis-id <ID> \
--chunk-id <N> \
--storage-dir <DIR>
The actual upstream runner uses (verified by pulling the digest-pinned
image and inspecting /opt/openstudio/server/app/lib/external_batch/* +
raw.githubusercontent.com/NatLabRockies/OpenStudio-server/d63c8eb/
external_batch/runner/run_chunk.rb):
ruby run_chunk.rb \
--package <DIR> --results <DIR> --chunk <N> --openstudio <CMD>
Differences:
* run_chunk.rb is NOT in the nrel/openstudio-server image. It is
fetched from the OpenStudio-server repo at runtime. This PR ships
it via ConfigMap.
* Flags are --package/--results/--chunk/--openstudio (no
--analysis-id, no --storage-dir, --chunk is the chunk index).
* analysis_id is read from manifest.json, not from a CLI flag.
* The runner expects the package at <mountPath>/external_batch/
analysis_<id>/package/ and writes results to <mountPath>/
external_batch/analysis_<id>/results/. The server's
ExternalBatch::Packager writes that layout when an analysis is
submitted with --batch-run-method external_batch_run.
Verified:
* helm lint clean
* helm template at default values: 0 external_batch references
* helm template --set external_batch.enabled=true ...: 1 Job +
1 ConfigMap rendered
* Pod-args template content matches the actual runner CLI
* Storage PVC override (storage.existingClaim) propagates correctly
* Downward API for JOB_COMPLETION_INDEX uses
batch.kubernetes.io/job-completion-index (k8s 1.21+)
Closes #99.1 parent ebeab96 commit 604bca9
5 files changed
Lines changed: 606 additions & 1 deletion
File tree
- openstudio-server
- configmaps/runner
- templates
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
| 17 | + | |
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| |||
0 commit comments