Summary
Three related operational rough edges hit while running a private cluster (uw org) with client v5.5.0, offered as UX feedback:
1. Requesting more nodes than an existing stack's capacity can never succeed — and silently skips TTL extension.
Session.scaleout (session.py:382+) discovers available nodes; when available < node_capacity it issues a provisioner deploy. If the stack already exists (e.g. deployed earlier at capacity 8, now requesting 16), deploy returns 500 and scaleout breaks out — without attempting extend. Net effect: the cluster neither grows nor gets its TTL extended, and the session proceeds normally, so a long production run can hit the original auto_shutdown mid-flight with no warning. Suggestions: fall back to extend when deploy fails on an existing stack; and/or support capacity growth on an existing stack; and log the auto_shutdown time at init so users see what they actually got.
2. MAX_PS_CLUSTER_WAIT_SECS = 600 is shorter than a cold CloudFormation deploy (~15–20 min observed).
First init with desired_nodes reports failure ("Maximum time allowed waiting for cluster has been exceeded") while the deploy is in fact succeeding; a later retry finds the cluster up. Suggestion: make the wait configurable and/or default it above the observed cold-start time, and distinguish "deploy accepted, still scaling" from failure in the log message.
3. Minor: freshly created <cluster>.slideruleearth.io Route53 records can be negative-cached by local resolvers (NXDOMAIN for several minutes after IlbRecord CREATE_COMPLETE), which then makes node discovery fail and re-triggers the deploy-500 path in (1). A retry-with-backoff around discovery during the post-deploy window would absorb this.
All three were observed 2026-08-07 while deploying/scaling the uw cluster (8 → attempted 16 nodes). Timeline details available on request.
Summary
Three related operational rough edges hit while running a private cluster (
uworg) with client v5.5.0, offered as UX feedback:1. Requesting more nodes than an existing stack's capacity can never succeed — and silently skips TTL extension.
Session.scaleout(session.py:382+) discovers available nodes; whenavailable < node_capacityit issues a provisionerdeploy. If the stack already exists (e.g. deployed earlier at capacity 8, now requesting 16), deploy returns 500 and scaleout breaks out — without attemptingextend. Net effect: the cluster neither grows nor gets its TTL extended, and the session proceeds normally, so a long production run can hit the originalauto_shutdownmid-flight with no warning. Suggestions: fall back toextendwhen deploy fails on an existing stack; and/or support capacity growth on an existing stack; and log the auto_shutdown time at init so users see what they actually got.2.
MAX_PS_CLUSTER_WAIT_SECS = 600is shorter than a cold CloudFormation deploy (~15–20 min observed).First
initwithdesired_nodesreports failure ("Maximum time allowed waiting for cluster has been exceeded") while the deploy is in fact succeeding; a later retry finds the cluster up. Suggestion: make the wait configurable and/or default it above the observed cold-start time, and distinguish "deploy accepted, still scaling" from failure in the log message.3. Minor: freshly created
<cluster>.slideruleearth.ioRoute53 records can be negative-cached by local resolvers (NXDOMAIN for several minutes afterIlbRecord CREATE_COMPLETE), which then makes node discovery fail and re-triggers the deploy-500 path in (1). A retry-with-backoff around discovery during the post-deploy window would absorb this.All three were observed 2026-08-07 while deploying/scaling the
uwcluster (8 → attempted 16 nodes). Timeline details available on request.