Skip to content

chore: update trainer to k8s 1.37 - #4005

Open
kannon92 wants to merge 2 commits into
kubeflow:masterfrom
kannon92:trainer-kube-1-37
Open

chore: update trainer to k8s 1.37#4005
kannon92 wants to merge 2 commits into
kubeflow:masterfrom
kannon92:trainer-kube-1-37

Conversation

@kannon92

@kannon92 kannon92 commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Update kubernetes to 1.37.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #

Help:

#4006 needs to be resolved to fix the failing tests.

Checklist:

  • Docs included if any changes are user facing

Copilot AI balanced review requested due to automatic review settings August 29, 2026 18:35
@github-project-automation github-project-automation Bot moved this to Needs Triage in Kubeflow Trainer Aug 29, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates Trainer’s Kubernetes tooling and generated clients/models to Kubernetes 1.37.

Changes:

  • Targets Kubernetes 1.37 and refreshes the E2E version matrix.
  • Regenerates Go clients and informer interfaces.
  • Regenerates Python models for Kubernetes 1.37 APIs.

Reviewed changes

Copilot reviewed 103 out of 106 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
Makefile Sets Kubernetes 1.37 default.
.github/workflows/test-e2e.yaml Tests Kubernetes 1.34–1.37.
hack/update-codegen.sh Adds scheduling v1alpha3 generation.
pkg/client/informers/externalversions/factory.go Updates generated informer documentation.
pkg/client/informers/externalversions/trainer/v1alpha1/interface.go Returns typed informers.
pkg/client/clientset/versioned/clientset.go Updates discovery interface type.
pkg/client/clientset/versioned/fake/clientset_generated.go Updates fake discovery interface.
pkg/client/applyconfiguration/trainer/v1alpha1/trainjob.go Updates generated JSON tag.
pkg/client/applyconfiguration/trainer/v1alpha1/trainingruntime.go Updates generated JSON tag.
pkg/client/applyconfiguration/trainer/v1alpha1/podgrouppolicy.go Updates generated JSON tag.
pkg/client/applyconfiguration/trainer/v1alpha1/optimizationjob.go Updates generated JSON tag.
pkg/client/applyconfiguration/trainer/v1alpha1/mlpolicy.go Updates generated JSON tag.
pkg/client/applyconfiguration/trainer/v1alpha1/clustertrainingruntime.go Updates generated JSON tag.
api/python_api/kubeflow_trainer_api/models/io_k8s_apimachinery_pkg_runtime_type_meta.py Refreshes TypeMeta documentation.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload.py Adds Workload model.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_reference.py Adds Workload reference model.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_pod_group_scheduling_policy.py Adds workload pod-group policy.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_pod_group_scheduling_constraints.py Adds workload scheduling constraints.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_pod_group_resource_claim.py Adds workload resource claims.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_pod_group_gang_scheduling_policy.py Adds workload gang policy.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_pod_group_disruption_mode.py Adds workload disruption mode.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_list.py Adds Workload list model.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_composite_pod_group_scheduling_policy.py Adds composite workload policy.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_composite_pod_group_scheduling_constraints.py Adds composite workload constraints.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_composite_pod_group_gang_scheduling_policy.py Adds composite workload gang policy.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_composite_pod_group_disruption_mode.py Adds composite workload disruption mode.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_typed_local_object_reference.py Adds typed local reference.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_topology_constraint.py Adds topology constraint.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_pod_group_scheduling_policy.py Adds PodGroup policy.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_pod_group_scheduling_constraints.py Adds PodGroup constraints.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_pod_group_resource_claim.py Adds PodGroup resource claim.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_pod_group_resource_claim_status.py Adds resource claim status.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_pod_group_list.py Adds PodGroup list model.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_gang_scheduling_policy.py Adds gang scheduling policy.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_disruption_mode.py Adds disruption mode.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_composite_pod_group_status.py Adds composite status.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_composite_pod_group_scheduling_policy.py Adds composite scheduling policy.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_composite_pod_group_scheduling_constraints.py Adds composite constraints.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_composite_pod_group_list.py Adds composite list model.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_composite_gang_scheduling_policy.py Adds composite gang policy.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_composite_disruption_mode.py Adds composite disruption mode.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_volume_mount.py Adds bind-mount options.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_volume_health_status.py Adds volume health status.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_volume_health_condition.py Adds volume health condition.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_service_account_token_projection.py Adds projected-file UID.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_secret_volume_source.py Adds default file UID.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_resource_status.py Refreshes DRA documentation.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_projected_volume_source.py Adds default file UID.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_pod_volume_health.py Adds pod volume health.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_pod_status.py Exposes volume health.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_pod_security_context.py Refreshes SELinux documentation.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_pod_certificate_projection.py Adds projected-file UID.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_persistent_volume_claim_status.py Adds volume health status.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_persistent_volume_claim_spec.py Refreshes data-source documentation.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_node_system_info.py Adds user-namespace status.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_node_spec.py Adds preemption policy.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_node_pod_preemption_policy.py Adds node preemption model.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_node_allocatable_resource_claim_status.py Updates allocatable-resource representation.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_node_allocatable_overhead_resources.py Adds overhead resource model.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_node_allocatable_mapped_resources.py Adds mapped resource model.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_key_to_path.py Adds projected-file UID.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_http_get_action.py Adds HTTP protocol selection.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_grpc_action.py Adds gRPC connection mode.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_eviction_responder.py Adds eviction responder model.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_empty_dir_volume_source.py Adds emptyDir mode.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_downward_api_volume_source.py Adds default file UID.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_downward_api_volume_file.py Adds projected-file UID.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_config_map.py Refreshes BinaryData documentation.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_config_map_volume_source.py Adds default file UID.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_config_map_key_selector.py Clarifies key selection.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_config_map_env_source.py Clarifies environment propagation.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_cluster_trust_bundle_projection.py Adds projected-file UID.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_batch_v1_job_status.py Refreshes terminating-pod documentation.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_batch_v1_job_spec.py Adds workload-aware scheduling.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_autoscaling_v2_object_metric_status.py Refreshes metric documentation.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_autoscaling_v2_metric_value_status.py Corrects metric field documentation.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_autoscaling_v2_metric_status.py Refreshes container metric documentation.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_autoscaling_v2_hpa_scaling_rules.py Refreshes tolerance documentation.
api/python_api/kubeflow_trainer_api/models/io_k8s_api_autoscaling_v2_horizontal_pod_autoscaler_condition.py Adds observed generation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/test-e2e.yaml Outdated
Comment thread Makefile
Comment thread .github/workflows/test-e2e.yaml Outdated

@andreyvelich andreyvelich left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this @kannon92!
Could you rebase it after #4007?

@kannon92

Copy link
Copy Markdown
Contributor Author

Thanks for this @kannon92! Could you rebase it after #4007?

done. We need to resolve #4006 before this can merge though.

@mimowo

mimowo commented Sep 1, 2026

Copy link
Copy Markdown

Thank you @kannon92 for moving this forward to enable the update in Kueue 👍

@kannon92
kannon92 force-pushed the trainer-kube-1-37 branch 2 times, most recently from dc4db38 to 4136a89 Compare September 6, 2026 19:15
Signed-off-by: Kevin Hannon <kehannon@redhat.com>
Kubernetes 1.37 added batch/v1.JobSpec.Scheduling, which TrainingRuntime and
ClusterTrainingRuntime inherit through the embedded JobSet spec. Its
ResourceClaims field declares both +k8s:immutable and +k8s:maxItems=4, but
controller-gen only implements the former, so the generated CRDs ended up with a
`self == oldSelf` CEL rule on an unbounded list nested under the unbounded
replicatedJobs list. The API server estimates CEL cost from the declared bounds
and rejected both CRDs outright:

  x-kubernetes-validations estimated rule cost total for entire OpenAPIv3 schema
  exceeds budget by factor of more than 100x

which failed `make test-integration` during envtest CRD installation, and would
equally have failed a real 1.37 install.

Restore the dropped bound after controller-gen runs, and validate the result
with the API server's own CRD validation so a future upstream field that repeats
this pattern fails at generation time rather than at install time. This keeps
the immutability rule intact and changes no Trainer API: the schema now accepts
exactly the claim lists the Job API accepts, where before it advertised an
unbounded list.

The generated diff is a single `maxItems: 4` line per affected CRD, since the
manifest is re-encoded with the same encoder controller-gen uses.

Reported upstream as kubernetes-sigs/controller-tools#1473; the bounds table can
go away once controller-gen emits +k8s:maxItems itself.

Signed-off-by: Kevin Hannon <kehannon@redhat.com>
ndipebot added a commit to NVIDIA/cluster-readiness-engine that referenced this pull request Sep 8, 2026
Kubernetes 1.37 removed the unused corev1.PodStatusResult type
(kubernetes/kubernetes#136271). github.com/kubeflow/trainer/v2 still
references it from its generated openapi, and we import that package
from api/v1alpha1/job_types.go, so the bump in #312 fails to type-check
and takes Lint, Build, Test, CodeQL, govulncheck, Verify and UAT down
with the same error.

Nothing in our graph wants 0.37; only Dependabot proposes it. Trainer
master is still on k8s v0.36.4 and its bump is open in
kubeflow/trainer#4005, blocked on kubeflow/trainer#4006. Without this
hold Dependabot re-raises the same red PR every Monday.

Scoped to minor and major so 0.36.x patch updates keep flowing, and
limited to the modules that move with the Kubernetes minor so that
klog, kube-openapi, utils and the independently versioned sigs.k8s.io
modules are not frozen by accident.

Signed-off-by: Ebot Ndip-Agbor <endipagbor@nvidia.com>
@kannon92

kannon92 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

cc @tenzen-y @mimowo @andreyvelich

Was asked if we can merge this to unblock the Kueue updates to 1.37.

Comment thread hack/crdschema/main.go
@mimowo

mimowo commented Sep 8, 2026

Copy link
Copy Markdown

@andreyvelich just FYI this is currently the main blocker to update K8s in Kueue for v0.20 which is planned last week of September: kubernetes-sigs/kueue#14905

This is not a blocker to release Kueue, but it would be nice to move this forward.

cc @apullo777 @kannon92

@mimowo

mimowo commented Sep 8, 2026

Copy link
Copy Markdown

cc @tenzen-y

@andreyvelich

Copy link
Copy Markdown
Member

I am happy to move this forward. @kannon92 do you know why CI is failing?

@andreyvelich

Copy link
Copy Markdown
Member

/retest

@kannon92

kannon92 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

/retest

The tests are github actions so prow doesn't work. Can you retry them?

@andreyvelich andreyvelich left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/assign @tenzen-y @robert-bell @Sridhar1030 in case you want to have quick look too.

@robert-bell robert-bell left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy to merge this as is. Thanks @kannon92

/lgtm

@mimowo

mimowo commented Sep 9, 2026

Copy link
Copy Markdown

Thank you folks for moving this forward. Once the PR is merged should we use in Kueue commit from the HEAD or you consider releasing this updated version before 20th of Sept?

cc @kaisoz

@andreyvelich

Copy link
Copy Markdown
Member

Once the PR is merged should we use in Kueue commit from the HEAD or you consider releasing this updated version before 20th of Sept?

I think we plan to release Trainer v2.4 in October since we have a few pending PRs we need to merge.
/lgtm
/approve

@google-oss-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mimowo

mimowo commented Sep 9, 2026

Copy link
Copy Markdown

I think we plan to release Trainer v2.4 in October since we have a few pending PRs we need to merge.

I see, would it be feasible to have a release candidate so that we can avoid using the "main" branch?

@tenzen-y

tenzen-y commented Sep 9, 2026

Copy link
Copy Markdown
Member

CI is still failing. Is it just a flake?

@andreyvelich

Copy link
Copy Markdown
Member

I think we plan to release Trainer v2.4 in October since we have a few pending PRs we need to merge.

I see, would it be feasible to have a release candidate so that we can avoid using the "main" branch?

I am not sure if we can do that before end of September.
Could Kueue just rely on commit for now? It should be stable too.

CI is still failing. Is it just a flake?

It looks like, i restarted it.

@mimowo

mimowo commented Sep 9, 2026

Copy link
Copy Markdown

Could Kueue just rely on commit for now? It should be stable too.

I think we are ok to use the "main" commit - we did it in the past, but a named release-candidate would be great.

@andreyvelich

Copy link
Copy Markdown
Member

@kannon92 if you rebase this PR, it should fix the CI.

lalitadithya pushed a commit to NVIDIA/cluster-readiness-engine that referenced this pull request Sep 10, 2026
…325)

* chore(deps): hold the kubernetes group at 0.36.x

Kubernetes 1.37 removed the unused corev1.PodStatusResult type
(kubernetes/kubernetes#136271). github.com/kubeflow/trainer/v2 still
references it from its generated openapi, and we import that package
from api/v1alpha1/job_types.go, so the bump in #312 fails to type-check
and takes Lint, Build, Test, CodeQL, govulncheck, Verify and UAT down
with the same error.

Nothing in our graph wants 0.37; only Dependabot proposes it. Trainer
master is still on k8s v0.36.4 and its bump is open in
kubeflow/trainer#4005, blocked on kubeflow/trainer#4006. Without this
hold Dependabot re-raises the same red PR every Monday.

Scoped to minor and major so 0.36.x patch updates keep flowing, and
limited to the modules that move with the Kubernetes minor so that
klog, kube-openapi, utils and the independently versioned sigs.k8s.io
modules are not frozen by accident.

Signed-off-by: Ebot Ndip-Agbor <endipagbor@nvidia.com>

* docs: make the golden-file test convention reachable without Claude

AGENTS.md described the pattern as what most packages do, then pointed
at the /cre-test slash command for the part that binds: which packages
it applies to. Non-Claude agents cannot invoke that command and were
given no file path to read instead, so they read the rule as optional.
CONTRIBUTING.md said nothing about it at all.

Inline the package list into CLAUDE.md and AGENTS.md, state it as a
requirement, and reference the skill by path. Add a Writing Tests
section to CONTRIBUTING.md so human contributors get the same rule.

Closes #324

Signed-off-by: Ebot Ndip-Agbor <endipagbor@nvidia.com>

* docs: fix golden regeneration command and two unverified claims

make test-integration runs only go test ./cmd/integration/, so it
rewrites the goldens under cmd/integration/testdata/ and never touches
the ones beside a package. Following the old instruction to regenerate a
unit golden left the file unchanged with no indication why. Verified by
corrupting a pkg/report golden: the integration command leaves the hash
untouched, the package command restores it.

Fix it in the unit-test section added here, in the generic instruction
in CLAUDE.md and AGENTS.md, and in the skill this branch points
contributors at.

Also correct two claims in the new CONTRIBUTING.md text that were
written from memory rather than checked. gpu_defaults_test.go is not the
shortest TestCaseParser example (orchestration/bisect_test.go is 34
lines to its 37), and testutil prints full want and got blocks on
mismatch, not a diff. Replace the vague 'most packages' with the counted
figure: 23 of the 24 packages that have tests use no table test.

Signed-off-by: Ebot Ndip-Agbor <endipagbor@nvidia.com>

* docs: correct the table-test survey and widen the required list

The claim that the required packages had zero Go table tests was wrong.
It came from a grep for the inline form, for _, tc := range []struct{,
which never matches the ordinary tests := []struct{...} then range tests.
Re-measured: pkg/controller has three (status_test.go, parser_cache_test.go,
metrics_cleanup_test.go), and four packages repo-wide use tables, not one.

Those three are legitimate under the existing exception for pins, guards
and single-value assertions, so the fix is to state the exception plainly
and name the live examples rather than assert a clean sweep that is not
true.

The required list also omitted packages that already follow the pattern.
Add pkg/cluster/, pkg/podlogs/, pkg/workloadrun/, cmd/manager/ and
test/helm/, all confirmed to carry TestCaseParser tests and a testdata
directory. Note that the trivial-helper packages are themselves already
all TestCaseParser, so the wording no longer reads as if they were
table-test territory.

Add the missing how-to: creating a new case, substituting your own
package into the regeneration command, and the TrimSpace comparison.

Reported by asivanadi0 in review of #325.

Signed-off-by: Ebot Ndip-Agbor <endipagbor@nvidia.com>

---------

Signed-off-by: Ebot Ndip-Agbor <endipagbor@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Needs Triage

Development

Successfully merging this pull request may close these issues.

6 participants