Skip to content

fix: TEM-31, TEM-32, TEM-39 pip warm-up, RunPod REST API v2 in smoke tests, proxy-first checks - #30

Open
Chmokachka wants to merge 7 commits into
mainfrom
fix/TEM-31-unavailable-pip-and-uv
Open

fix: TEM-31, TEM-32, TEM-39 pip warm-up, RunPod REST API v2 in smoke tests, proxy-first checks#30
Chmokachka wants to merge 7 commits into
mainfrom
fix/TEM-31-unavailable-pip-and-uv

Conversation

@Chmokachka

@Chmokachka Chmokachka commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Closes TEM-31, TEM-32, TEM-39.

Summary

Image fixes

  • pip warm-up on cold start (TEM-31). ComfyUI-Manager probes python -m pip --version with a 5s timeout; on a cold container the first pip invocation could exceed it and Manager reported "Neither pip nor uv are available". start.sh now warms pip up before ComfyUI starts and logs the wall time. A matching always-on pip check in the smoke tests fails the run if pip is missing or slower than Manager's 5s budget.
  • Trimmed NVIDIA_DRIVER_CAPABILITIES from all to compute,utility,video. Requesting graphics/display makes the container toolkit bind-mount Vulkan/GLX files (e.g. /etc/vulkan/icd.d/nvidia_icd.json), which aborts container start on hosts with headless drivers (the CZ-1 RTX PRO 6000 WK incident). ComfyUI needs no OpenGL/Vulkan; video is kept for NVENC/NVDEC.

Smoke tests: RunPod REST API v2 adoption

  • Fail-fast pod status via GET /v2/pods/{id}. SSH stays the readiness gate, but the v2 status is polled alongside: unlike the legacy desiredStatus (RUNNING from second one, even when the container never starts), v2 distinguishes STARTING/RUNNING and surfaces ERROR — on ERROR/EXITED/TERMINATED we bail immediately instead of sitting out the full CREATE_TIMEOUT.
  • Host-side system logs (GET /v2/pods/{id}/logs?source=system) on stall/timeout/terminal state and in every diagnostic dump. That's where image-pull and container-init failures are reported — container stdout is empty when the container never started. Filtered by the new SYS_LOG_ERROR_PATTERN (error/failed/crashed markers).
  • Container-log error scan now also treats crash(ed/es/ing) as an error marker (LOG_ERROR_PATTERN).

Smoke tests: reliability

  • Proxy-first checks. Jupyter / generic port / ComfyUI reachability probes now hit the public RunPod proxy first (the end-user path); the in-pod SSH probe runs only as a diagnostic when the proxy fails, to distinguish "service never started" from "port not exposed as http".
  • De-flaked the RunpodDirect feature-detect. GET /server_download/folder_paths was a single-shot probe, so a transient proxy 404/5xx was misclassified as "node not installed in this image" (intermittent CI failures). It now retries for up to COMFYUI_ROUTES_TIMEOUT (default 60s) and the FAIL message includes the last HTTP code/error.

Pipeline examples:

@Chmokachka Chmokachka changed the title fix: TEM-31 fix: TEM-31, TEM-32, TEM-39 pip warm-up, RunPod REST API v2 in smoke tests, proxy-first checks Aug 7, 2026
@Chmokachka
Chmokachka marked this pull request as ready for review August 10, 2026 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants