feat(quota): per-region per-machine-type VM quota guard on settings save + grow - #181
Open
JacobZuliani wants to merge 7 commits into
Open
feat(quota): per-region per-machine-type VM quota guard on settings save + grow#181JacobZuliani wants to merge 7 commits into
JacobZuliani wants to merge 7 commits into
Conversation
…ave + grow Blocks the two paths that could ask GCP to boot more VMs of a given (region, machine_type) than quota allows: - `POST /v1/settings` reads `cluster_quota/cluster_quota` and 400s with a structured `quota_exceeded` detail when `machineQuantity > limit`. Frontend catches it: caps quantity to `limit` when `limit >= 1`, or reverts machineType/gcpRegion/machineQuantity to the last-saved snapshot when `limit == 0`, and renders a "GCP Quota Limit Reached" dialog. Ported from PR #141's final quota-working revision. - `POST /v1/jobs/{id}/start` with `grow=True` applies the same per- (region, machine_type) cap to the grow plan, counting existing active nodes against the limit. Over-quota types are soft-capped and their clip recorded in a `warnings` list on the response; the burla client prints those warnings and the job runs at reduced scale. If every machine type in the plan clips to zero, the endpoint 400s with `quota_exceeded` and the client raises the new `QuotaExceeded` exception. Missing (region, machine_type) entries read as unlimited so fresh projects without a populated cluster_quota doc keep working. Local-dev short-circuits both guards. `scripts/backfill_cluster_quotas.py` enumerates every project registered in `burla-prod/backend-service/clusters`, queries live GCP Compute Engine per-region quotas, translates family-level CPU + GPU quotas into per-machine-type VM counts, and writes the result to `cluster_quota/cluster_quota` in each project's `burla` Firestore DB. Idempotent with a `--dry-run` flag. Checked offline against the module-level logic (get_quota_limit, _apply_quota_cap) with NODES_CACHE + DB mocks. `make build-frontend` runs clean. Backfill dry-run against burla-agent-02 produces valid output using real compute.regions.get quotas.
The legacy compute v1 `RegionsClient.get().quotas` only surfaces A2_CPUS and the NVIDIA_A100_* GPU metrics; N4, A3, H100, and H200 quotas are served through the newer `cloudquotas` service under the combined `CPUS-PER-VM-FAMILY-per-project-region` and `GPUS-PER-GPU-FAMILY-per-project-region` entries. Without those, every n4-standard and a3-* row in the written cluster_quota doc was always 0. Script now hits both APIs and merges them: legacy for A2/A100, modern for N4 (CPU-only) and H100/H200 (GPU-gated). Dry-run against burla-agent-02 now correctly shows `n4-standard-2: 150`, etc., in regions where N4 quota is granted.
Replace stored Firestore quota state with live Google quota checks at settings save and boot planning time so quota decisions cannot go stale.
Keep CPU grow quota caps from being dropped when no N4 machines can fit, so users see the intended warning or quota error.
Move N4 quota-aware packing into the shared boot planner so grow only describes desired capacity while boot planning owns quota capping and errors.
Remove unused quota helpers and return plain integer limits from Google quota reads.
# Conflicts: # main_service/src/main_service/endpoints/cluster_lifecycle.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR turns GCP quota failures into clear, early feedback without storing quota state in Firestore.
cluster_quotaFirestore document dependency and deletes the backfill script.BOOTING,READY,RUNNING) as current usage before booting a batch, so quota checks do not depend on Google quota-usage freshness.User-visible quota scenarios
Settings save: selected hardware is impossible
Appears when the selected machine type, region, and quantity exceed the project's live GCP quota even before counting currently running Burla nodes. The settings are not saved. If some machines fit, the form is lowered to that count; if zero fit, the form returns to the last saved hardware config.
Dashboard Start: only part of the boot batch fits
Appears when the saved settings are theoretically possible, but the current active Burla nodes plus the requested Start batch would exceed quota. Burla starts the machines that fit and shows this warning.
Dashboard Start: none of the boot batch fits
Appears when the current active Burla nodes already consume the relevant quota, or the selected region has zero usable quota for the requested hardware. Burla does not schedule an impossible boot.
Python API
grow=True: reduced scaleAppears when
remote_parallel_map(..., grow=True)asks Burla to grow beyond the remaining quota. Burla boots the additional nodes that fit and prints the reduced-scale warning before continuing. The warning includes a self-hosting quota link: https://docs.cloud.google.com/docs/quotas/ .Python API
grow=True: no requested nodes can bootAppears when
grow=Trueneeds new nodes but none can be booted without exceeding quota, and there are no existing compatible nodes that can run the job. The client raisesQuotaExceededimmediately.Implementation notes
regions.get.Verification
./.venv/bin/python tests/test_quota.py./.venv/bin/python -m py_compile src/main_service/quota.py src/main_service/endpoints/client.py src/main_service/endpoints/settings.py src/main_service/endpoints/cluster_lifecycle.pycap_boot_machine_types(['n4-standard-8'], 'us-central1', [])returned['n4-standard-8']with no warnings.npm run buildnpx eslint src/hooks/useClusterControl.ts src/hooks/useSaveSettings.ts src/pages/Index.tsx src/pages/Settings.tsxNote: full
npm run lintstill fails on pre-existing unrelated lint errors outside these files.