chore: update trainer to k8s 1.37 - #4005
Conversation
There was a problem hiding this comment.
Pull request overview
Updates Trainer’s Kubernetes tooling and generated clients/models to Kubernetes 1.37.
Changes:
- Targets Kubernetes 1.37 and refreshes the E2E version matrix.
- Regenerates Go clients and informer interfaces.
- Regenerates Python models for Kubernetes 1.37 APIs.
Reviewed changes
Copilot reviewed 103 out of 106 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
Makefile |
Sets Kubernetes 1.37 default. |
.github/workflows/test-e2e.yaml |
Tests Kubernetes 1.34–1.37. |
hack/update-codegen.sh |
Adds scheduling v1alpha3 generation. |
pkg/client/informers/externalversions/factory.go |
Updates generated informer documentation. |
pkg/client/informers/externalversions/trainer/v1alpha1/interface.go |
Returns typed informers. |
pkg/client/clientset/versioned/clientset.go |
Updates discovery interface type. |
pkg/client/clientset/versioned/fake/clientset_generated.go |
Updates fake discovery interface. |
pkg/client/applyconfiguration/trainer/v1alpha1/trainjob.go |
Updates generated JSON tag. |
pkg/client/applyconfiguration/trainer/v1alpha1/trainingruntime.go |
Updates generated JSON tag. |
pkg/client/applyconfiguration/trainer/v1alpha1/podgrouppolicy.go |
Updates generated JSON tag. |
pkg/client/applyconfiguration/trainer/v1alpha1/optimizationjob.go |
Updates generated JSON tag. |
pkg/client/applyconfiguration/trainer/v1alpha1/mlpolicy.go |
Updates generated JSON tag. |
pkg/client/applyconfiguration/trainer/v1alpha1/clustertrainingruntime.go |
Updates generated JSON tag. |
api/python_api/kubeflow_trainer_api/models/io_k8s_apimachinery_pkg_runtime_type_meta.py |
Refreshes TypeMeta documentation. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload.py |
Adds Workload model. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_reference.py |
Adds Workload reference model. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_pod_group_scheduling_policy.py |
Adds workload pod-group policy. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_pod_group_scheduling_constraints.py |
Adds workload scheduling constraints. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_pod_group_resource_claim.py |
Adds workload resource claims. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_pod_group_gang_scheduling_policy.py |
Adds workload gang policy. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_pod_group_disruption_mode.py |
Adds workload disruption mode. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_list.py |
Adds Workload list model. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_composite_pod_group_scheduling_policy.py |
Adds composite workload policy. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_composite_pod_group_scheduling_constraints.py |
Adds composite workload constraints. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_composite_pod_group_gang_scheduling_policy.py |
Adds composite workload gang policy. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_workload_composite_pod_group_disruption_mode.py |
Adds composite workload disruption mode. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_typed_local_object_reference.py |
Adds typed local reference. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_topology_constraint.py |
Adds topology constraint. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_pod_group_scheduling_policy.py |
Adds PodGroup policy. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_pod_group_scheduling_constraints.py |
Adds PodGroup constraints. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_pod_group_resource_claim.py |
Adds PodGroup resource claim. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_pod_group_resource_claim_status.py |
Adds resource claim status. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_pod_group_list.py |
Adds PodGroup list model. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_gang_scheduling_policy.py |
Adds gang scheduling policy. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_disruption_mode.py |
Adds disruption mode. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_composite_pod_group_status.py |
Adds composite status. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_composite_pod_group_scheduling_policy.py |
Adds composite scheduling policy. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_composite_pod_group_scheduling_constraints.py |
Adds composite constraints. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_composite_pod_group_list.py |
Adds composite list model. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_composite_gang_scheduling_policy.py |
Adds composite gang policy. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_scheduling_v1alpha3_composite_disruption_mode.py |
Adds composite disruption mode. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_volume_mount.py |
Adds bind-mount options. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_volume_health_status.py |
Adds volume health status. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_volume_health_condition.py |
Adds volume health condition. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_service_account_token_projection.py |
Adds projected-file UID. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_secret_volume_source.py |
Adds default file UID. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_resource_status.py |
Refreshes DRA documentation. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_projected_volume_source.py |
Adds default file UID. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_pod_volume_health.py |
Adds pod volume health. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_pod_status.py |
Exposes volume health. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_pod_security_context.py |
Refreshes SELinux documentation. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_pod_certificate_projection.py |
Adds projected-file UID. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_persistent_volume_claim_status.py |
Adds volume health status. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_persistent_volume_claim_spec.py |
Refreshes data-source documentation. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_node_system_info.py |
Adds user-namespace status. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_node_spec.py |
Adds preemption policy. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_node_pod_preemption_policy.py |
Adds node preemption model. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_node_allocatable_resource_claim_status.py |
Updates allocatable-resource representation. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_node_allocatable_overhead_resources.py |
Adds overhead resource model. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_node_allocatable_mapped_resources.py |
Adds mapped resource model. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_key_to_path.py |
Adds projected-file UID. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_http_get_action.py |
Adds HTTP protocol selection. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_grpc_action.py |
Adds gRPC connection mode. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_eviction_responder.py |
Adds eviction responder model. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_empty_dir_volume_source.py |
Adds emptyDir mode. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_downward_api_volume_source.py |
Adds default file UID. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_downward_api_volume_file.py |
Adds projected-file UID. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_config_map.py |
Refreshes BinaryData documentation. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_config_map_volume_source.py |
Adds default file UID. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_config_map_key_selector.py |
Clarifies key selection. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_config_map_env_source.py |
Clarifies environment propagation. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_cluster_trust_bundle_projection.py |
Adds projected-file UID. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_batch_v1_job_status.py |
Refreshes terminating-pod documentation. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_batch_v1_job_spec.py |
Adds workload-aware scheduling. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_autoscaling_v2_object_metric_status.py |
Refreshes metric documentation. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_autoscaling_v2_metric_value_status.py |
Corrects metric field documentation. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_autoscaling_v2_metric_status.py |
Refreshes container metric documentation. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_autoscaling_v2_hpa_scaling_rules.py |
Refreshes tolerance documentation. |
api/python_api/kubeflow_trainer_api/models/io_k8s_api_autoscaling_v2_horizontal_pod_autoscaler_condition.py |
Adds observed generation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
3387459 to
2322c57
Compare
2322c57 to
d2fd3dc
Compare
|
Thank you @kannon92 for moving this forward to enable the update in Kueue 👍 |
dc4db38 to
4136a89
Compare
Signed-off-by: Kevin Hannon <kehannon@redhat.com>
Kubernetes 1.37 added batch/v1.JobSpec.Scheduling, which TrainingRuntime and ClusterTrainingRuntime inherit through the embedded JobSet spec. Its ResourceClaims field declares both +k8s:immutable and +k8s:maxItems=4, but controller-gen only implements the former, so the generated CRDs ended up with a `self == oldSelf` CEL rule on an unbounded list nested under the unbounded replicatedJobs list. The API server estimates CEL cost from the declared bounds and rejected both CRDs outright: x-kubernetes-validations estimated rule cost total for entire OpenAPIv3 schema exceeds budget by factor of more than 100x which failed `make test-integration` during envtest CRD installation, and would equally have failed a real 1.37 install. Restore the dropped bound after controller-gen runs, and validate the result with the API server's own CRD validation so a future upstream field that repeats this pattern fails at generation time rather than at install time. This keeps the immutability rule intact and changes no Trainer API: the schema now accepts exactly the claim lists the Job API accepts, where before it advertised an unbounded list. The generated diff is a single `maxItems: 4` line per affected CRD, since the manifest is re-encoded with the same encoder controller-gen uses. Reported upstream as kubernetes-sigs/controller-tools#1473; the bounds table can go away once controller-gen emits +k8s:maxItems itself. Signed-off-by: Kevin Hannon <kehannon@redhat.com>
4136a89 to
955bfc9
Compare
Kubernetes 1.37 removed the unused corev1.PodStatusResult type (kubernetes/kubernetes#136271). github.com/kubeflow/trainer/v2 still references it from its generated openapi, and we import that package from api/v1alpha1/job_types.go, so the bump in #312 fails to type-check and takes Lint, Build, Test, CodeQL, govulncheck, Verify and UAT down with the same error. Nothing in our graph wants 0.37; only Dependabot proposes it. Trainer master is still on k8s v0.36.4 and its bump is open in kubeflow/trainer#4005, blocked on kubeflow/trainer#4006. Without this hold Dependabot re-raises the same red PR every Monday. Scoped to minor and major so 0.36.x patch updates keep flowing, and limited to the modules that move with the Kubernetes minor so that klog, kube-openapi, utils and the independently versioned sigs.k8s.io modules are not frozen by accident. Signed-off-by: Ebot Ndip-Agbor <endipagbor@nvidia.com>
|
cc @tenzen-y @mimowo @andreyvelich Was asked if we can merge this to unblock the Kueue updates to 1.37. |
|
@andreyvelich just FYI this is currently the main blocker to update K8s in Kueue for v0.20 which is planned last week of September: kubernetes-sigs/kueue#14905 This is not a blocker to release Kueue, but it would be nice to move this forward. |
|
cc @tenzen-y |
|
I am happy to move this forward. @kannon92 do you know why CI is failing? |
|
/retest |
The tests are github actions so prow doesn't work. Can you retry them? |
andreyvelich
left a comment
There was a problem hiding this comment.
/lgtm
/assign @tenzen-y @robert-bell @Sridhar1030 in case you want to have quick look too.
robert-bell
left a comment
There was a problem hiding this comment.
I'm happy to merge this as is. Thanks @kannon92
/lgtm
|
Thank you folks for moving this forward. Once the PR is merged should we use in Kueue commit from the HEAD or you consider releasing this updated version before 20th of Sept? cc @kaisoz |
I think we plan to release Trainer v2.4 in October since we have a few pending PRs we need to merge. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: andreyvelich The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
I see, would it be feasible to have a release candidate so that we can avoid using the "main" branch? |
|
CI is still failing. Is it just a flake? |
I am not sure if we can do that before end of September.
It looks like, i restarted it. |
I think we are ok to use the "main" commit - we did it in the past, but a named release-candidate would be great. |
|
@kannon92 if you rebase this PR, it should fix the CI. |
…325) * chore(deps): hold the kubernetes group at 0.36.x Kubernetes 1.37 removed the unused corev1.PodStatusResult type (kubernetes/kubernetes#136271). github.com/kubeflow/trainer/v2 still references it from its generated openapi, and we import that package from api/v1alpha1/job_types.go, so the bump in #312 fails to type-check and takes Lint, Build, Test, CodeQL, govulncheck, Verify and UAT down with the same error. Nothing in our graph wants 0.37; only Dependabot proposes it. Trainer master is still on k8s v0.36.4 and its bump is open in kubeflow/trainer#4005, blocked on kubeflow/trainer#4006. Without this hold Dependabot re-raises the same red PR every Monday. Scoped to minor and major so 0.36.x patch updates keep flowing, and limited to the modules that move with the Kubernetes minor so that klog, kube-openapi, utils and the independently versioned sigs.k8s.io modules are not frozen by accident. Signed-off-by: Ebot Ndip-Agbor <endipagbor@nvidia.com> * docs: make the golden-file test convention reachable without Claude AGENTS.md described the pattern as what most packages do, then pointed at the /cre-test slash command for the part that binds: which packages it applies to. Non-Claude agents cannot invoke that command and were given no file path to read instead, so they read the rule as optional. CONTRIBUTING.md said nothing about it at all. Inline the package list into CLAUDE.md and AGENTS.md, state it as a requirement, and reference the skill by path. Add a Writing Tests section to CONTRIBUTING.md so human contributors get the same rule. Closes #324 Signed-off-by: Ebot Ndip-Agbor <endipagbor@nvidia.com> * docs: fix golden regeneration command and two unverified claims make test-integration runs only go test ./cmd/integration/, so it rewrites the goldens under cmd/integration/testdata/ and never touches the ones beside a package. Following the old instruction to regenerate a unit golden left the file unchanged with no indication why. Verified by corrupting a pkg/report golden: the integration command leaves the hash untouched, the package command restores it. Fix it in the unit-test section added here, in the generic instruction in CLAUDE.md and AGENTS.md, and in the skill this branch points contributors at. Also correct two claims in the new CONTRIBUTING.md text that were written from memory rather than checked. gpu_defaults_test.go is not the shortest TestCaseParser example (orchestration/bisect_test.go is 34 lines to its 37), and testutil prints full want and got blocks on mismatch, not a diff. Replace the vague 'most packages' with the counted figure: 23 of the 24 packages that have tests use no table test. Signed-off-by: Ebot Ndip-Agbor <endipagbor@nvidia.com> * docs: correct the table-test survey and widen the required list The claim that the required packages had zero Go table tests was wrong. It came from a grep for the inline form, for _, tc := range []struct{, which never matches the ordinary tests := []struct{...} then range tests. Re-measured: pkg/controller has three (status_test.go, parser_cache_test.go, metrics_cleanup_test.go), and four packages repo-wide use tables, not one. Those three are legitimate under the existing exception for pins, guards and single-value assertions, so the fix is to state the exception plainly and name the live examples rather than assert a clean sweep that is not true. The required list also omitted packages that already follow the pattern. Add pkg/cluster/, pkg/podlogs/, pkg/workloadrun/, cmd/manager/ and test/helm/, all confirmed to carry TestCaseParser tests and a testdata directory. Note that the trivial-helper packages are themselves already all TestCaseParser, so the wording no longer reads as if they were table-test territory. Add the missing how-to: creating a new case, substituting your own package into the regeneration command, and the TrimSpace comparison. Reported by asivanadi0 in review of #325. Signed-off-by: Ebot Ndip-Agbor <endipagbor@nvidia.com> --------- Signed-off-by: Ebot Ndip-Agbor <endipagbor@nvidia.com>
What this PR does / why we need it:
Update kubernetes to 1.37.
Which issue(s) this PR fixes (optional, in
Fixes #<issue number>, #<issue number>, ...format, will close the issue(s) when PR gets merged):Fixes #
Help:
#4006 needs to be resolved to fix the failing tests.
Checklist: