Skip to content

Commit a82a006

Browse files
mohitmk01claude
andcommitted
smoke: shrink to mb_mult=4 (matches first working 50M smoke)
Previous cluster smoke OOM'd at startup despite nw=4. Reducing: --mem 48GB → 64GB (more CPU headroom) --train.minibatch-multiplier 25 → 4 (minibatch 20100 → 3216, 6x less GPU memory) --train.max-minibatch-size 20100 → 3216 --train.total-timesteps 5M → 2M (finish faster) --time 30min → 20min This matches the config of the first 50M smoke on the vast box that ran cleanly end-to-end. If THIS still OOMs, the issue is cluster container limits, not our minibatch sizing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent a379013 commit a82a006

1 file changed

Lines changed: 5 additions & 5 deletions

File tree

‎scripts/adaptive/cluster_smoke_test.sh‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@
22
#SBATCH --job-name=ada_k4_smoke
33
#SBATCH --output=/scratch/mmk9418/logs/smoke_%j.out
44
#SBATCH --error=/scratch/mmk9418/logs/smoke_%j.err
5-
#SBATCH --mem=48GB
6-
#SBATCH --time=00:30:00
5+
#SBATCH --mem=64GB
6+
#SBATCH --time=00:20:00
77
#SBATCH --nodes=1
88
#SBATCH --ntasks=1
99
#SBATCH --account=torch_pr_355_tandon_advanced
@@ -35,13 +35,13 @@ singularity exec --nv --overlay "$OVERLAY_FILE:ro" "$SINGULARITY_IMAGE" bash -c
3535
--tag smoke_k4_gb3_\$(date +%s) \
3636
--policy-architecture Transformer --rnn-name Transformer \
3737
--train.horizon 804 \
38-
--train.minibatch-multiplier 25 \
39-
--train.max-minibatch-size 20100 \
38+
--train.minibatch-multiplier 4 \
39+
--train.max-minibatch-size 3216 \
4040
--train.cpu-offload True \
4141
--train.checkpoint-interval 100 \
4242
--train.render-interval 100 \
4343
--train.seed 42 \
44-
--train.total-timesteps 5000000 \
44+
--train.total-timesteps 2000000 \
4545
--vec.num-workers 4 --vec.num-envs 4 \
4646
--env.map-dir resources/drive/binaries/nuplan_201 \
4747
--env.num-maps 4999 \

0 commit comments

Comments
 (0)