Skip vmm-tests for lightweight PRs without weakening PR gates
Summary
I want to stop burning vmm-tests capacity on PRs that clearly are not product changes.
The immediate case is Guide/**, but I don’t think we should solve this as a docs-only special case. We should use this to add a small Flowey-level mechanism for classifying PRs into “product” vs “lightweight” buckets, then use that classification to skip the expensive vmm-tests jobs when the PR is lightweight-only.
That keeps the current PR gate model intact, fixes the docs case, and gives us a clean path to exempt other non-product areas later, like repo_support/**/*.py.
What’s wrong today
Right now the main PR workflow is Flowey-generated from .flowey.toml via ci checkin-gates --config=pr, with the logic in flowey/flowey_hvlite/src/pipelines/checkin_gates.rs.
That workflow always emits the full vmm-tests matrix, including the six heavy run vmm-tests [...] jobs. That makes sense for product changes. It does not make sense for changes that only touch docs or other repo-maintenance surfaces.
For example:
Guide/** changes already have their own docs validation pathrepo_support/relabel_backported.py is repo automation, not product behavior
Those changes do not affect guest behavior, device behavior, save/restore behavior, boot behavior, or VM management semantics, but today they still consume the same scarce self-hosted test capacity as a real product change.
That is the real problem here. We are treating every PR like a product PR because we do not have a reusable way to say “this change is lightweight.”
Why I want to solve this more generally
I don’t want to hardcode one exception for Guide/** and then repeat the same discussion the next time we want to exempt another obviously non-product path.
The better model is:
- if a PR touches product code, run full product validation
- if a PR is entirely within approved lightweight buckets, skip
vmm-tests
- if a PR is mixed, take the stricter path and run
vmm-tests
That makes the docs case easy, and it also gives us a clean way to handle future cases like repo_support/**/*.py without inventing another one-off rule.
The default should still be conservative: if a path is not explicitly classified as lightweight, it is treated as product-affecting.
Existing validation we already have
We already have a separate Flowey-generated docs PR workflow:
.github/workflows/openvmm-docs-pr.yaml- generated from
.flowey.toml via ci build-docs --config=pr
- implemented in
flowey/flowey_hvlite/src/pipelines/build_docs.rs
That workflow builds the guide via flowey/flowey_lib_hvlite/src/build_guide.rs, which runs:
So for Guide/**, this is not about removing validation. It is about stopping redundant product validation for a change that already has an appropriate validation path.
Proposed shape
I want to add a small reusable Flowey helper that classifies a PR into named change buckets before we expand the expensive jobs.
Conceptually, the classifier should answer things like:
- does this PR touch product code?
- does this PR touch
Guide/**?
- does this PR touch
repo_support/**/*.py?
- is this PR lightweight-only?
I’m fine with backing that helper using a purpose-built GitHub Action. In fact, that is probably the cleanest implementation, as long as the Flowey interface stays simple and the action contract stays small.
Then checkin_gates.rs can use that classification to gate the six vmm-tests jobs on !lightweight_only.
So the behavior becomes:
Guide/**-only PR → skip vmm-testsrepo_support/**/*.py-only PR → skip vmm-testsGuide/** + product code PR → run vmm-tests- product-only PR → run
vmm-tests
That is the behavior I want.
Why I do not want to skip the whole PR workflow
The tempting shortcut here is to add workflow-level path filtering and just not run the main PR workflow for guide-only changes.
I don’t think that is the right shape.
The current PR pipeline already has an aggregate required-check job, openvmm checkin gates, which depends on the rest of the pipeline and treats only failure and cancelled as blocking. That is a good model. I want to preserve it.
If we skip only the vmm-tests jobs, we keep the existing required check surface and mixed-change behavior stays correct.
If we skip the entire workflow, required-check behavior gets more fragile and the whole thing becomes harder to reason about.
So I want job-level skipping of the expensive matrix, not workflow disappearance.
Goals
- Stop scheduling
vmm-tests for PRs that are entirely within approved lightweight buckets.
- Keep running the existing
vmm-tests matrix for anything that touches product code or unclassified paths.
- Preserve the current required-check model.
- Solve this in Flowey, not by hand-editing generated YAML.
- Make the mechanism reusable so the next lightweight bucket is easy to add.
Non-goals
- I am not trying to automatically treat every markdown file or every script in the repo as lightweight.
- I am not trying to redesign the docs workflow in this change.
- I am not trying to change the
vmm-tests matrix itself.
- I am not trying to pre-solve every future bucket; I just want the mechanism and a conservative initial policy.
Initial bucket policy
I think the initial policy should be conservative and explicit:
Lightweight buckets
Guide/**repo_support/**/*.py
Product / full-validation by default
If we want to expand the lightweight set later, we can do that deliberately.
Tradeoffs
Why not do a one-off Guide/** special case?
Because that only solves today’s symptom.
The real issue is that we do not have a reusable CI concept for lightweight repo changes. If we add a classifier now, Guide/** becomes the first use case rather than a permanent special case.
Why use a purpose-built action?
Because changed-file classification is exactly the kind of thing that is cleaner to encapsulate once and reuse, instead of rebuilding with scattered string conditions.
As long as the action only does the minimal changed-file lookup and bucket evaluation, I think that is acceptable.
What’s the downside?
The initial change is a little bigger than a docs-only hack, because it adds a reusable classification mechanism instead of one condition. I think that extra work is worth it, because it gives us a sane model going forward.
Validation
I’d want to validate at least these cases:
Guide/**-only PR → vmm-tests skippedrepo_support/**/*.py-only PR → vmm-tests skipped- mixed lightweight + product PR →
vmm-tests run
- product-only PR →
vmm-tests run
I’d also want to confirm that the aggregate openvmm checkin gates job still behaves correctly and remains suitable as a required check.
Rough implementation plan
- Add a Flowey-facing helper for lightweight PR classification.
- Back it with a purpose-built GitHub Action that evaluates changed files against named buckets.
- Use that helper in
checkin_gates.rs to skip the vmm-tests fan-out when lightweight_only is true.
- Seed the initial policy with
Guide/** and repo_support/**/*.py.
- Leave everything else on the full-validation path by default.
Open questions
Are Guide/** and repo_support/**/*.py the right initial lightweight buckets, or do we want to start even narrower? (Matt: yes)Do we want the classifier to expose only lightweight_only, or also named per-bucket outputs so future policies can compose more easily? (Matt: let's come up with a different name, that clearly indicates that this is not product, not test, not CI, etc.)Should the purpose-built action live in this repo, or in a shared CI repo? (Matt: in this repo)
Skip
vmm-testsfor lightweight PRs without weakening PR gatesSummary
I want to stop burning
vmm-testscapacity on PRs that clearly are not product changes.The immediate case is
Guide/**, but I don’t think we should solve this as a docs-only special case. We should use this to add a small Flowey-level mechanism for classifying PRs into “product” vs “lightweight” buckets, then use that classification to skip the expensivevmm-testsjobs when the PR is lightweight-only.That keeps the current PR gate model intact, fixes the docs case, and gives us a clean path to exempt other non-product areas later, like
repo_support/**/*.py.What’s wrong today
Right now the main PR workflow is Flowey-generated from
.flowey.tomlviaci checkin-gates --config=pr, with the logic inflowey/flowey_hvlite/src/pipelines/checkin_gates.rs.That workflow always emits the full
vmm-testsmatrix, including the six heavyrun vmm-tests [...]jobs. That makes sense for product changes. It does not make sense for changes that only touch docs or other repo-maintenance surfaces.For example:
Guide/**changes already have their own docs validation pathrepo_support/relabel_backported.pyis repo automation, not product behaviorThose changes do not affect guest behavior, device behavior, save/restore behavior, boot behavior, or VM management semantics, but today they still consume the same scarce self-hosted test capacity as a real product change.
That is the real problem here. We are treating every PR like a product PR because we do not have a reusable way to say “this change is lightweight.”
Why I want to solve this more generally
I don’t want to hardcode one exception for
Guide/**and then repeat the same discussion the next time we want to exempt another obviously non-product path.The better model is:
vmm-testsvmm-testsThat makes the docs case easy, and it also gives us a clean way to handle future cases like
repo_support/**/*.pywithout inventing another one-off rule.The default should still be conservative: if a path is not explicitly classified as lightweight, it is treated as product-affecting.
Existing validation we already have
We already have a separate Flowey-generated docs PR workflow:
.github/workflows/openvmm-docs-pr.yaml.flowey.tomlviaci build-docs --config=prflowey/flowey_hvlite/src/pipelines/build_docs.rsThat workflow builds the guide via
flowey/flowey_lib_hvlite/src/build_guide.rs, which runs:mdbook testmdbook buildSo for
Guide/**, this is not about removing validation. It is about stopping redundant product validation for a change that already has an appropriate validation path.Proposed shape
I want to add a small reusable Flowey helper that classifies a PR into named change buckets before we expand the expensive jobs.
Conceptually, the classifier should answer things like:
Guide/**?repo_support/**/*.py?I’m fine with backing that helper using a purpose-built GitHub Action. In fact, that is probably the cleanest implementation, as long as the Flowey interface stays simple and the action contract stays small.
Then
checkin_gates.rscan use that classification to gate the sixvmm-testsjobs on!lightweight_only.So the behavior becomes:
Guide/**-only PR → skipvmm-testsrepo_support/**/*.py-only PR → skipvmm-testsGuide/**+ product code PR → runvmm-testsvmm-testsThat is the behavior I want.
Why I do not want to skip the whole PR workflow
The tempting shortcut here is to add workflow-level path filtering and just not run the main PR workflow for guide-only changes.
I don’t think that is the right shape.
The current PR pipeline already has an aggregate required-check job,
openvmm checkin gates, which depends on the rest of the pipeline and treats onlyfailureandcancelledas blocking. That is a good model. I want to preserve it.If we skip only the
vmm-testsjobs, we keep the existing required check surface and mixed-change behavior stays correct.If we skip the entire workflow, required-check behavior gets more fragile and the whole thing becomes harder to reason about.
So I want job-level skipping of the expensive matrix, not workflow disappearance.
Goals
vmm-testsfor PRs that are entirely within approved lightweight buckets.vmm-testsmatrix for anything that touches product code or unclassified paths.Non-goals
vmm-testsmatrix itself.Initial bucket policy
I think the initial policy should be conservative and explicit:
Lightweight buckets
Guide/**repo_support/**/*.pyProduct / full-validation by default
If we want to expand the lightweight set later, we can do that deliberately.
Tradeoffs
Why not do a one-off
Guide/**special case?Because that only solves today’s symptom.
The real issue is that we do not have a reusable CI concept for lightweight repo changes. If we add a classifier now,
Guide/**becomes the first use case rather than a permanent special case.Why use a purpose-built action?
Because changed-file classification is exactly the kind of thing that is cleaner to encapsulate once and reuse, instead of rebuilding with scattered string conditions.
As long as the action only does the minimal changed-file lookup and bucket evaluation, I think that is acceptable.
What’s the downside?
The initial change is a little bigger than a docs-only hack, because it adds a reusable classification mechanism instead of one condition. I think that extra work is worth it, because it gives us a sane model going forward.
Validation
I’d want to validate at least these cases:
Guide/**-only PR →vmm-testsskippedrepo_support/**/*.py-only PR →vmm-testsskippedvmm-testsrunvmm-testsrunI’d also want to confirm that the aggregate
openvmm checkin gatesjob still behaves correctly and remains suitable as a required check.Rough implementation plan
checkin_gates.rsto skip thevmm-testsfan-out whenlightweight_onlyis true.Guide/**andrepo_support/**/*.py.Open questions
Are(Matt: yes)Guide/**andrepo_support/**/*.pythe right initial lightweight buckets, or do we want to start even narrower?Do we want the classifier to expose only(Matt: let's come up with a different name, that clearly indicates that this is not product, not test, not CI, etc.)lightweight_only, or also named per-bucket outputs so future policies can compose more easily?Should the purpose-built action live in this repo, or in a shared CI repo?(Matt: in this repo)