Feat: TEM-14 functional testing

[Chmokachka]

Summary

Adds functional testing for the ComfyUI images on top of the existing smoke
tests: instead of only checking "does the container boot / is the port up?", we
now verify the image can actually generate an image end-to-end — the thing
that actually matters for a ComfyUI release.

The functional check mirrors a real user flow (pick the "1.1 Starter – Text to
Image" template → pull the missing model → run it → get a PNG), driven entirely
over the ComfyUI HTTP API.

What's new

Two ComfyUI test tiers (separate manifest flags, functional implies smoke):

flag	tier	proves	cost
test_comfyui: true	reachability smoke	ComfyUI is up + reachable on :8188 (in-pod + public proxy)	seconds, no GPU work
test_comfyui_functional: true	end-to-end functional	ComfyUI can generate an image	~2 GB model + GPU time

The functional check runs entirely host-side against the public proxy URL
(https://<pod-id>-8188.proxy.runpod.net) — no SSH, no in-pod script. Model
provisioning uses the ComfyUI-RunpodDirect node baked into the image (its
/server_download/* routes live on the same ComfyUI server), exactly like the
"Download to Pod" button in the UI. Flow:

1. wait for /system_stats through the proxy,
2. provision the checkpoint(s) via /server_download/verify_model_integrity →
   /server_download/start → poll /server_download/status (server-side
   size + sha256 verification, 8-connection download on the pod),
3. POST /prompt, poll /history/<id>,
4. GET /view and assert a real, non-empty PNG (magic bytes + IHDR dimensions).

Because it's bounded HTTP polling with per-request timeouts, there's no
hung-SSH risk of leaking a pod.

New files

• tests/runpod_smoke/comfyui.py — the host-side functional check.
• tests/comfyui/workflows/gsl_starter_1_1.api.json — the "1.1 Starter – Text
  to Image" template in API format (fixed seed → reproducible).
• tests/comfyui/models.json — checkpoint manifest (DreamShaper_8_pruned,
  with sha256).
• tests/comfyui/images.example.yaml — example manifest for local runs.

CI wiring

• dev.yml — reachability smoke always on; functional is opt-in via a
  run_functional_tests workflow_dispatch toggle.
• release.yml — functional always on (a release gates on the image
  being able to generate).
• check-incompatibilities.yml — functional on every resolved GPU
  (check_all_gpu) to surface GPU-specific generation failures.

All three ComfyUI workflows use profile: gpu: ComfyUI ships only GPU
images and runpod/comfyui:latest carries no CUDA marker, so base would sort
it into base_cpu and try to generate on a CPU pod.

Generated images are archived as a workflow artifact
(comfyui-generated-images, save-comfyui-images: true) — downloadable from
the run Summary, never committed to the repo. Uploaded on always(), so
even a failed generation surfaces whatever it produced.

New smoke-test action inputs: test-comfyui, test-comfyui-functional,
save-comfyui-images, comfyui-images-artifact-name,
comfyui-images-retention-days.

Review follow-ups (addressed)

• Release functional test is actually on — release.yml sets
  test-comfyui-functional: "true" directly (the old run_functional_tests
  input didn't exist on that workflow, so it was silently off).
• profile: gpu applied across release.yml, dev.yml, and
  check-incompatibilities.yml (was base).
• Release gated on tests — release-comfyui now needs: [build-and-push, test],
  so the GitHub Release isn't published for an image that can't generate.
  (Stable Docker tags are still pushed by build-and-push beforehand; fully
  gating the image push needs the build/test/promote split, to be reconciled
  with chore: automate releases with changesets #17's release.yml rewrite.)
• Docs/comments corrected: the functional check runs host-side via the
  public proxy, not over SSH / in-pod.
• RELEASE_VERSION / IMAGE_REF cross-job fix — these were written to
  $GITHUB_ENV in build-and-push and read from a separate release-comfyui
  job, where they were empty (GitHub Release would get a blank tag/name). Now
  IMAGE_REF is a single workflow-level env, and the dynamic
  release-version is passed as a build-and-push job output.
• Dockerfile: pinned pip==26.1.2 and pip-tools==7.6.0 — the base image
  ships an older pip-tools, and a plain pip install pip-tools won't upgrade
  it; combined with get-pip.py pulling the newest pip, the stale
  (legacy-resolver) pip-tools crashed with
  make_requirement_preparer() missing 'allow_editables' during
  pip-compile. Pinning both makes lock-file generation reproducible.
• ci: refactor — removed dead IS_MANUAL_RELEASE / redundant TAG
  writes and fixed stale copied-from-dev.yml comments in release.yml.

Examples:

• https://github.com/runpod-workers/comfyui-base/actions/runs/30811108255/job/91677806950
• https://github.com/runpod-workers/comfyui-base/actions/runs/30811208969/job/91688077043

[kodxana]

Had a proper look through this, including both linked runs and the generated-image artifact. The functional test itself looks good, the latest run checked out the exact PR head, passed on 15 GPU types with no functional failures, and uploaded 15 real 512×512 PNGs. The remaining skips were capacity/scheduler related.

I don’t think the release wiring is quite ready yet though:

• release.yml passes inputs.run_functional_tests, but that workflow only defines the version input. That means the functional test is currently off for release runs. This should be "true" if it is meant to be always on.
• These are all GPU images, but the workflows use profile: base. On release, runpod/comfyui:latest has no CUDA marker, so it gets placed in base_cpu and would try to generate on a CPU pod once the flag above is fixed. profile: gpu looks like the correct choice here.
• release-comfyui only depends on build-and-push, so it can run while the test is still running or after it fails. The stable Docker tags are also pushed before testing, so the generation check doesn’t currently gate publication as described.

Could you sort those out before merge?

One cross-PR note: #17 also rewrites release.yml and is currently conflicting/outdated. It doesn’t fix this gating issue, and its Changesets flow creates the GitHub Release before release.yml runs, so the two release flows will need to be reconciled.

Small non-blocking cleanup: a few comments/help strings still say the functional check runs over SSH or in-pod, while the implementation now runs host-side through the proxy.

[kodxana]

Thanks, the three points from my previous comment look sorted now: the release functional test is enabled, the ComfyUI workflows use the GPU profile, and the release job waits for the test.

I spotted one more issue while checking the updated release job. RELEASE_VERSION and IMAGE_REF are written to $GITHUB_ENV in build-and-push, but release-comfyui is a separate job, so those values won’t be available there. Could you pass them as job outputs, or calculate them again inside release-comfyui?

Could you also run a fresh build/smoke test against the current head? The linked successful runs are still from 814b363, before the new Docker pins and workflow changes. I don’t think another expensive 15-GPU functional matrix is needed the functional test was already proven there but a current-head build would confirm the final Dockerfile change.

[Madiator2011Work]

LGTM