feat(api): add resources field support to RuntimePatch ContainerPatch API

[hrathina]

What you would like to be added?

Add support for patching resource requirements (CPU, memory, GPU) for initContainers (sidecars) via the RuntimePatch API's ContainerPatch model.

Current Limitation

The ContainerPatch model currently only supports 4 fields:

message ContainerPatch {
  string name = 1;                               // Required - container to patch
  repeated EnvVar env = 2;                       // Supported
  SecurityContext security_context = 3;          // Supported
  repeated VolumeMount volume_mounts = 4;        // Supported
  // ResourceRequirements resources = 5;         // NOT supported
}

This limitation affects both containers and initContainers since they share the same ContainerPatch model in PodSpecPatch.

Proposed Change

Add a resources field to ContainerPatch:

message ContainerPatch {
  string name = 1;
  repeated EnvVar env = 2;
  SecurityContext security_context = 3;
  repeated VolumeMount volume_mounts = 4;
  ResourceRequirements resources = 5;           // Add this field
}

This would allow users to override sidecar resources via RuntimePatch:

apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
metadata:
  name: large-dataset-training
spec:
  trainingRuntimeRef:
    name: pytorch-distributed
  runtimePatches:
    - manager: user-controller
      trainingRuntimeSpec:
        template:
          spec:
            replicatedJobs:
              - name: node
                template:
                  spec:
                    template:
                      spec:
                        initContainers:
                          - name: data-loader
                            resources:
                              limits:
                                memory: "64Gi"  # Override from default 8Gi
                              requests:
                                memory: "64Gi"

Implementation Considerations

• Follow Kubernetes strategic merge patch semantics for combining runtime and patch resources
• Support both limits and requests subfields
• Validate resource quantities using standard Kubernetes resource validation
• Document merge behavior when both runtime template and patch specify resources

---

Why is this needed?

ClusterTrainingRuntimes often define initContainers (sidecars) for common tasks like data loading, model downloading, or preprocessing. These sidecars need different resource allocations depending on the specific TrainJob requirements (dataset size, model size, etc.).

Main training containers can set resources via spec.trainer.resourcesPerNode on the TrainJob CRD, but sidecars can only be customized via RuntimePatch. Since RuntimePatch's ContainerPatch doesn't support the resources field, there is no API mechanism to configure sidecar resources per-job.

Use Cases

Primary Use Cases (InitContainers/Sidecars):

1. Data Loading Sidecars

   • Small datasets: 8Gi memory sidecar
   • Large datasets: 64Gi memory sidecar
   • Cannot adjust memory per job today

2. Model Downloading InitContainers

   • Small models (few GB): 4Gi memory
   • Large models (100+ GB): 32Gi memory
   • Different jobs need different download buffer sizes

3. GPU-Accelerated Preprocessing Sidecars

   • Some jobs need GPU preprocessing: 1-2 GPUs
   • Some jobs skip GPU preprocessing: 0 GPUs
   • Cannot conditionally allocate GPU to sidecar

Note on Main Containers:
Main training containers already have spec.trainer.resourcesPerNode field on the TrainJob CRD for setting resources. Adding resources support to RuntimePatch would provide an alternative mechanism, but is not critical since the CRD field works.

Current Workarounds and Limitations

Since the resources field is not supported in RuntimePatch, users must:

Workaround 1: Pre-configure all resource requirements in the ClusterTrainingRuntime template

# Must hardcode resource requirements in the runtime
apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
spec:
  template:
    spec:
      replicatedJobs:
        - name: node
          template:
            spec:
              template:
                spec:
                  containers:
                    - name: pytorch
                      resources:
                        limits:
                          nvidia.com/gpu: "4"  # Fixed - cannot override per TrainJob

• No flexibility: cannot adjust resources per TrainJob
• Forces choice: either create many runtimes OR accept one-size-fits-all resources

Workaround 2: Create duplicate ClusterTrainingRuntimes for each resource configuration

# Must maintain separate runtimes
pytorch-distributed-1gpu
pytorch-distributed-4gpu
pytorch-distributed-8gpu

• Maintenance burden: template changes must be replicated across all variants
• Configuration drift: runtimes can become inconsistent over time

---

References

• Kubernetes Resource Requirements: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
• RuntimePatch API (Training Operator v2.2+)

[hrathina]

/assign