fix(trainer): filter duplicate Pods in get_job() API by HKanoje · Pull Request #160 · kubeflow/sdk

HKanoje · 2025-11-17T03:32:58Z

Description

Problem:

The get_job() API currently returns multiple Pods for the same TrainJob component
(e.g., dataset-initializer, trainer-node-0) when Kubernetes recreates Pods based on
Batch/Job restart policies.

This causes users to see duplicate components with conflicting statuses
—for example, one Pod may show "Failed" while another shows "Running"—leading to
confusion about the actual state of the training job.

Solution:

This PR improves the get_job() API to filter duplicate Pods and display only the most recently created Pod for each TrainJob component.

Key Improvements

1. Groups Pods by role

Uses JOBSET_RJOB_NAME_LABEL for initializer Pods
Uses a combination of JOBSET_RJOB_NAME_LABEL + JOB_INDEX_LABEL for training-node Pods
(ensures correct grouping across multi-node trainer replicas)

2. Selects the most recent Pod

For each group, the API now selects the Pod with the latest creation_timestamp
Eliminates stale or restarted Pods that would otherwise appear as duplicates
(e.g., old Pods in Failed state)

3. Maintains backward compatibility

No changes to the API schema or response format
Behavior only differs when duplicate Pods exist, improving clarity for end users

This ensures users see clean, de-duplicated component statuses that accurately represent the current state of their training job.

Example Impact:

Before this fix:

job = client.get_job("my-job")
# Shows duplicate components with conflicting statuses
job.steps = [
    Step(name='dataset-initializer', status='Failed'),    # Old pod
    Step(name='dataset-initializer', status='Running'),   # New pod
    Step(name='node-0', status='Failed'),                 # Old pod
    Step(name='node-0', status='Running'),                # New pod
]

After this fix:

job = client.get_job("my-job")
# Shows only current components
job.steps = [
    Step(name='dataset-initializer', status='Running'),   # Latest only ✓
    Step(name='node-0', status='Running'),                # Latest only ✓
]

Changes Made

Modified Files

`backend.py`

Updated the __get_trainjob_from_cr() method to implement Pod de-duplication and filtering logic
Added comprehensive inline comments explaining the grouping and selection approach
Groups Pods by component role:
- Initializers grouped by JOBSET_RJOB_NAME_LABEL
- Training nodes grouped by JOBSET_RJOB_NAME_LABEL + JOB_INDEX_LABEL
For each group, selects the most recent Pod based on creation_timestamp

`backend_test.py`

Added a new test: test_get_job_with_pod_restarts()
Simulates Pod restart scenarios where Kubernetes creates duplicate Pods
Verifies that only the most recent Pod per component is returned
Covers mixed scenarios:
- Some components with restarts
- Some components without restarts
Ensures correct behavior and backward compatibility

Testing

All tests passing:

make verify — PASSED (lint + format checks)
test_get_job_with_pod_restarts — PASSED (new test for Pod restart filtering)
test_get_job — PASSED (existing behavior remains compatible)
All 36 Kubernetes backend tests — PASSED
All 163 Python unit tests — PASSED

Test Coverage

Pod restart scenarios with duplicate Pods having different creation_timestamp values
Mixed scenarios where:
- Some components have restarts
- Others have no duplicates
Verified that the API selects only the newest Pod per component
Confirmed that statuses come from the latest Pods, not older failed ones

Checklist

Follows Conventional Commits specification
Code follows project style guidelines (make verify passes)
All tests pass locally (make test-python)
Added comprehensive unit tests for new functionality
Updated documentation and inline comments where needed
No breaking changes to public APIs
Fully backward compatible with existing behavior

Related Issues

Fixes #25

Fiona-Waters · 2025-11-17T09:44:57Z

+                    pod_groups[key] = []
+                pod_groups[key].append(pod)
+
+            # Select the most recently created Pod from each group.


I think to make it more robust we could select the pod based on the status as well as the timestamp something like this Fiona-Waters@b48277f
wdyt?
It will return a pod that actually reflects the true state of each TrainJob component, rather than the newest pod.

Absolutely agree! I've actually implemented exactly that approach from your commit b48277f. The current implementation now:

Prioritizes by status first: Running (4) > Succeeded (3) > Failed (2) > Pending (1) > Unknown (0)
Uses timestamp as tiebreaker: Among pods with the same status, selects the most recent one
This ensures we return a pod that reflects the true state of the TrainJob component (preferring Running/Succeeded pods over Failed ones), rather than blindly picking the newest pod regardless of its state.

For example, if we have:

Pod A: Failed (created at 11:00)
Pod B: Running (created at 10:00)
The old logic would return Pod A (newest), but the new logic correctly returns Pod B (Running status is higher priority).

Fiona-Waters · 2025-11-17T12:20:47Z

+
+        # Sort by creation timestamp (most recent first)
+        candidate_pods.sort(
+            key=lambda p: p.metadata.creation_timestamp or datetime.datetime.min, reverse=True


This could cause an issue in newer python versions - do we want it to be timezone naive or set to utc?

Suggested change

key=lambda p: p.metadata.creation_timestamp or datetime.datetime.min, reverse=True

key=lambda p: (p.metadata.creation_timestamp or datetime.datetime.min.replace(tzinfo=timezone.utc))

Good catch! I've applied your suggestion to use datetime.datetime.min.replace(tzinfo=timezone.utc) instead of the timezone-naive datetime.datetime.min.

Thanks for the review!

Great. Don't forget to import timezone too
from datetime import timezone

Done! Added the timezone import. Thanks for catching that! 👍

Fiona-Waters · 2025-11-17T12:23:24Z

@HKanoje left one more comment but otherwise it looks good to me.
@andreyvelich @astefanutti @kramaranya please review when you can. Thanks.

astefanutti · 2025-11-27T16:19:43Z

/lgtm

Thanks @HKanoje @Fiona-Waters!

/assign @kubeflow/kubeflow-sdk-team

astefanutti · 2025-11-27T16:19:54Z

/ok-to-test

coveralls · 2025-11-27T16:22:24Z

Pull Request Test Coverage Report for Build 19878220233

Details

58 of 61 (95.08%) changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.4%) to 67.024%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
kubeflow/trainer/backends/kubernetes/backend.py	32	35	91.43%

Totals
Change from base Build 19828346095:	0.4%
Covered Lines:	2561
Relevant Lines:	3821

💛 - Coveralls

kramaranya

Thank you @HKanoje!
I've left a few comments

kramaranya · 2025-12-02T22:27:53Z

+        Priority order:
+        1. Running or Succeeded Pods (prefer most recent)
+        2. Failed Pods (prefer most recent)
+        3. Pending Pods (prefer most recent)
+        4. Unknown Pods (prefer most recent)
+        """
+        if not pods:
+            return None
+
+        # Pod status priority (higher number = higher priority)
+        status_priority = {
+            constants.POD_RUNNING: 4,  # Highest priority
+            constants.POD_SUCCEEDED: 3,  # Second highest
+            constants.POD_FAILED: 2,  # Third priority
+            constants.POD_PENDING: 1,  # Low priority
+            constants.POD_UNKNOWN: 0,  # Lowest priority


Do running and succeeded statuses have the same priority? The docstring doesn't match the actual priorities

kramaranya · 2025-12-02T22:34:39Z

+            constants.POD_RUNNING: 4,  # Highest priority
+            constants.POD_SUCCEEDED: 3,  # Second highest


Shall we consider those two to be equal priority? Since both running and succeeded are healthy pods, I think we should care about the most recent one. wdyt @HKanoje @andreyvelich @astefanutti

kramaranya · 2025-12-02T22:43:52Z

                            trainjob.runtime,
                            pod.metadata.labels[constants.JOBSET_RJOB_NAME_LABEL],
-                            int(pod.metadata.labels[constants.JOB_INDEX_LABEL]),
+                            int(pod.metadata.labels.get(constants.JOB_INDEX_LABEL, "0")),


Don't those pods always have this label?

kramaranya · 2025-12-02T22:45:43Z


        self.namespace = cfg.namespace

+    def _select_best_pod_for_role(


Can you move this after public methods?

@kramaranya Thank you for the thorough review! I've addressed all your comments:

Changes Made:

1. Docstring & Priority Design

Updated docstring to explicitly state: "Running or Succeeded Pods (equal priority, prefer most recent)"

Changed POD_SUCCEEDED priority from 3 to 4 (now equal to POD_RUNNING)

Added clarification: "Both Running and Succeeded are considered healthy states with equal priority"

2. JOB_INDEX_LABEL

Removed .get(constants.JOB_INDEX_LABEL, "0") in both locations

Now using direct access: pod.metadata.labels[constants.JOB_INDEX_LABEL]

3. Method Placement

Moved _select_best_pod_for_role after all public methods (after delete_job)

Now positioned before _read_pod_logs, following project convention

Testing:

make verify passes

All 36 Kubernetes backend tests pass

All 163 Python tests pass

@kramaranya

- Give Running and Succeeded pods equal priority (both are healthy states) - Update docstring to clearly explain equal priority and timestamp tiebreaker - Remove JOB_INDEX_LABEL .get() default, use direct access - Move _select_best_pod_for_role method after public methods per convention Addresses review comments from @kramaranya on PR kubeflow#160

google-oss-prow · 2025-12-02T23:48:34Z

New changes are detected. LGTM label has been removed.

@kramaranya

- Give Running and Succeeded pods equal priority (both are healthy states) - Update docstring to clearly explain equal priority and timestamp tiebreaker - Remove JOB_INDEX_LABEL .get() default, use direct access - Move _select_best_pod_for_role method after public methods per convention Addresses review comments from @kramaranya on PR kubeflow#160 Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

HKanoje · 2026-01-01T05:03:33Z

@astefanutti I have made new changes please review and then it can be tested.

HKanoje · 2026-02-01T19:06:31Z

@szaher Please Review whenever you get chance! Thanks!

andreyvelich · 2026-02-23T19:44:48Z

    print("test execution complete")


+def test_get_job_with_pod_restarts(kubernetes_backend):


Please add another test case to test_get_job() API as described here:
https://github.com/kubeflow/sdk/blob/main/AGENTS.md#3-testing-requirements

andreyvelich · 2026-02-23T19:53:27Z

+        candidate_pods.sort(
+            key=lambda p: (
+                p.metadata.creation_timestamp or datetime.datetime.min.replace(tzinfo=timezone.utc)
+            ),
+            reverse=True,
+        )


Can you just simple sort all pods by creation_timestamp after calling self.core_api.list_namespaced_pod() ?
After that, just add items to trainjob.steps.append() list, and if item already exists (e.g. node-1, or model-initializer, just ignore them).

@andreyvelich Thanks for the feedback! I've simplified the implementation as you suggested:

Simplified pod filtering: Removed the _select_best_pod_for_role() helper method with status-priority logic. Now pods are sorted by creation_timestamp after list_namespaced_pod(), and duplicates are skipped using a seen_roles set — first occurrence (newest) wins.

Added test case to test_get_job(): Added a new parametrized test case "filters duplicate pods and returns only newest per role" and removed the standalone test_get_job_with_pod_restarts() function.

krishdef7 · 2026-03-06T06:34:58Z

Thanks @HKanoje for the detailed implementation and the comprehensive test coverage, the test_get_job_with_pod_restarts() scenario makes the duplicate Pod problem very clear.

I noticed the current implementation introduces _select_best_pod_for_role() with status-priority logic to choose between duplicate Pods.

However, in a recent comment, @andreyvelich suggested a simpler approach: sorting Pods by creation_timestamp after list_namespaced_pod() and then adding the first occurrence per role while ignoring duplicates.

That approach might simplify the implementation and avoid maintaining the additional status-priority mapping and helper method.

One question I had while reading the code: if a Pod is recreated quickly and the newest Pod is still in Pending while the previous Pod briefly reached Running, is the expectation that the newest Pod should always represent the authoritative state of that TrainJob component?

Curious whether the timestamp-based approach is preferred mainly for simplicity, or if there are cases where Pod status might still be useful in the selection logic.

@kramaranya

- Give Running and Succeeded pods equal priority (both are healthy states) - Update docstring to clearly explain equal priority and timestamp tiebreaker - Remove JOB_INDEX_LABEL .get() default, use direct access - Move _select_best_pod_for_role method after public methods per convention Addresses review comments from @kramaranya on PR kubeflow#160 Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

Copilot

Pull request overview

This PR updates the Kubernetes Trainer backend’s get_job() path to de-duplicate restarted Kubernetes Pods so users see a single, authoritative component per TrainJob step.

Changes:

Add Pod-phase constants (e.g., POD_RUNNING, POD_FAILED) for consistent phase handling.
Update Kubernetes backend TrainJob parsing to group Pods by component role/index and select a single Pod per component.
Add a unit test that simulates Pod restarts and asserts that only the newest Pod per component is returned.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
`kubeflow/trainer/constants/constants.py`	Adds explicit Pod phase constants used by backend pod-selection logic.
`kubeflow/trainer/backends/kubernetes/backend.py`	Introduces pod grouping/selection logic to filter duplicate Pods returned by `get_job()`.
`kubeflow/trainer/backends/kubernetes/backend_test.py`	Adds coverage for duplicate-Pod/restart scenarios in `get_job()`.

Copilot · 2026-03-07T23:54:47Z


+    def _select_best_pod_for_role(
+        self, pods: list[models.IoK8sApiCoreV1Pod]
+    ) -> Optional[models.IoK8sApiCoreV1Pod]:


Optional is used in the return type but isn’t imported (and this file doesn’t use from __future__ import annotations), which will raise a NameError when importing the module; import Optional from typing or switch the annotation to models.IoK8sApiCoreV1Pod | None to match the rest of the file’s union style.

Suggested change

) -> Optional[models.IoK8sApiCoreV1Pod]:

) -> models.IoK8sApiCoreV1Pod | None:

Copilot · 2026-03-07T23:54:48Z

+    def _select_best_pod_for_role(
+        self, pods: list[models.IoK8sApiCoreV1Pod]
+    ) -> Optional[models.IoK8sApiCoreV1Pod]:
+        """
+        Select the best Pod for a role based on status priority and creation timestamp.
+
+        Priority order (higher priority = preferred):
+        1. Running or Succeeded Pods (equal priority, prefer most recent)
+        2. Failed Pods (prefer most recent)
+        3. Pending Pods (prefer most recent)
+        4. Unknown Pods (prefer most recent)
+
+        Both Running and Succeeded are considered healthy states with equal priority.
+        When multiple pods share the same priority, the most recently created pod is selected.
+        """
+        if not pods:
+            return None
+
+        # Pod status priority (higher number = higher priority)
+        # Running and Succeeded have equal priority as both are healthy states
+        status_priority = {
+            constants.POD_RUNNING: 4,  # Highest priority (healthy)
+            constants.POD_SUCCEEDED: 4,  # Highest priority (healthy)
+            constants.POD_FAILED: 2,  # Lower priority
+            constants.POD_PENDING: 1,  # Low priority
+            constants.POD_UNKNOWN: 0,  # Lowest priority
+        }
+
+        # Group Pods by status priority
+        pods_by_status = {}
+        for pod in pods:
+            status = pod.status.phase if pod.status else constants.POD_UNKNOWN
+            priority = status_priority.get(status, 0)
+
+            if priority not in pods_by_status:
+                pods_by_status[priority] = []
+            pods_by_status[priority].append(pod)
+
+        # Find the highest priority status that has Pods
+        highest_priority = max(pods_by_status.keys()) if pods_by_status else 0
+        candidate_pods = pods_by_status[highest_priority]
+
+        # Among Pods with the same priority, select the most recent one
+        if len(candidate_pods) == 1:
+            return candidate_pods[0]
+
+        # Sort by creation timestamp (most recent first)
+        candidate_pods.sort(
+            key=lambda p: (
+                p.metadata.creation_timestamp or datetime.datetime.min.replace(tzinfo=timezone.utc)
+            ),
+            reverse=True,
+        )
+
+        return candidate_pods[0]
+


_select_best_pod_for_role is defined twice in this class, so the later definition silently overrides the earlier one; please remove one copy and keep a single implementation to avoid dead code and inconsistent behavior/documentation.

Suggested change

def _select_best_pod_for_role(

self, pods: list[models.IoK8sApiCoreV1Pod]

) -> Optional[models.IoK8sApiCoreV1Pod]:

"""

Select the best Pod for a role based on status priority and creation timestamp.

Priority order (higher priority = preferred):

1. Running or Succeeded Pods (equal priority, prefer most recent)

2. Failed Pods (prefer most recent)

3. Pending Pods (prefer most recent)

4. Unknown Pods (prefer most recent)

Both Running and Succeeded are considered healthy states with equal priority.

When multiple pods share the same priority, the most recently created pod is selected.

"""

if not pods:

return None

# Pod status priority (higher number = higher priority)

# Running and Succeeded have equal priority as both are healthy states

status_priority = {

constants.POD_RUNNING: 4, # Highest priority (healthy)

constants.POD_SUCCEEDED: 4, # Highest priority (healthy)

constants.POD_FAILED: 2, # Lower priority

constants.POD_PENDING: 1, # Low priority

constants.POD_UNKNOWN: 0, # Lowest priority

}

# Group Pods by status priority

pods_by_status = {}

for pod in pods:

status = pod.status.phase if pod.status else constants.POD_UNKNOWN

priority = status_priority.get(status, 0)

if priority not in pods_by_status:

pods_by_status[priority] = []

pods_by_status[priority].append(pod)

# Find the highest priority status that has Pods

highest_priority = max(pods_by_status.keys()) if pods_by_status else 0

candidate_pods = pods_by_status[highest_priority]

# Among Pods with the same priority, select the most recent one

if len(candidate_pods) == 1:

return candidate_pods[0]

# Sort by creation timestamp (most recent first)

candidate_pods.sort(

key=lambda p: (

p.metadata.creation_timestamp or datetime.datetime.min.replace(tzinfo=timezone.utc)

),

reverse=True,

)

return candidate_pods[0]

Copilot · 2026-03-07T23:54:48Z

+        Priority order (higher priority = preferred):
+        1. Running or Succeeded Pods (equal priority, prefer most recent)
+        2. Failed Pods (prefer most recent)
+        3. Pending Pods (prefer most recent)
+        4. Unknown Pods (prefer most recent)


The “status priority” selection can return an older Pod instead of the most recently created one (e.g., old Failed pod vs newly recreated Pending pod), which contradicts the PR’s goal of showing the latest pod per component; prefer selecting by latest creation_timestamp first (and only use phase as a tie-breaker if needed).

Copilot · 2026-03-07T23:54:48Z

+# The pending phase of the Pod.
+POD_PENDING = "Pending"
+


POD_PENDING is now defined here but it’s also defined again later in the same module, which creates an unnecessary duplicate and can confuse readers; keep a single POD_PENDING definition and remove the other copy.

Suggested change

# The pending phase of the Pod.

POD_PENDING = "Pending"

…ubeflow#485) Cover all public methods of the optimizer's KubernetesBackend with parametrized unit tests mirroring the trainer backend test structure. Tests added for: optimize, get_job, list_jobs, get_job_logs, get_best_results, wait_for_job_status, delete_job, get_job_events. Each method tested with success paths and error scenarios (TimeoutError, RuntimeError, ValueError) using mock K8s APIs. Address review feedback: - Add payload verification for create_namespaced_custom_object - Add test_get_job_status_conditions for all status-mapping branches - Add get_job_logs branch coverage (trial_name, follow, empty, pending pod) - Restore polling_interval validation cases (zero, equal-to-timeout, negative) - Rework wait_for_job_status with CR-based mock and callback test - Fix docstrings, move TypeVar near imports, add positional-arg comments Signed-off-by: Slowlybomb <hslyusar@redhat.com> Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

Implement support for NPU resource labels in resource limit validation, resolving the existing TODO to support additional accelerator types. Signed-off-by: Sujal Shah <sujalshah28092004@gmail.com> Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

…eflow#495) Replaces the Trivy-based workflows removed in kubeflow#427 with two new tools: - validate-lockfile.yaml: PR check using uv audit to diff-compare vulnerabilities between PR and base branch. Non-blocking (informational comments only). Note: uv audit is experimental; if its text output format changes, the fallback is switching to osv-scanner JSON output (both query the same OSV.dev database). - osv-scanner.yaml: Nightly scan using OSV-Scanner CLI (v2.3.8) with SARIF upload to the GitHub Security tab and auto-fix PRs. Integrates with existing .github/scripts/ utilities (update_overrides.py, compare_versions.py, extract_version.py) and the cleanup-overrides workflow for the full fix lifecycle. - osv-scanner.toml: Minimal config for suppressing false positives. Closes kubeflow#478 Signed-off-by: Fiona-Waters <fiwaters6@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

When Kubernetes recreates Pods due to restart policies, multiple Pods with the same role can exist simultaneously. This causes get_job() to return duplicate TrainJob components with different statuses, creating confusion for users. This change groups Pods by their component role and selects only the most recently created Pod for each component based on creation_timestamp. This ensures users see the current state of their TrainJob after any Pod restarts. Changes: - Group Pods by role identifier (initializer name or node+index) - Select most recent Pod from each group using creation_timestamp - Add comprehensive test for Pod restart scenarios Fixes kubeflow#25 Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

…e safety Apply code quality improvements based on review feedback: - Use status-based priority for pod selection (Running > Succeeded > Failed > Pending > Unknown) - Add datetime.min fallback for safer timestamp sorting (prevents TypeError) - Add precise type hints to internal dicts for better type checking - Use consistent .get() access for JOB_INDEX_LABEL with default fallback - Add pod phase constants (POD_RUNNING, POD_FAILED, POD_PENDING, POD_UNKNOWN) These changes improve robustness, type safety, and maintainability while maintaining the same behavior of selecting the best pod for each role. Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

Use datetime.datetime.min.replace(tzinfo=timezone.utc) instead of datetime.datetime.min to prevent TypeError when comparing timezone-aware and timezone-naive datetimes in Python 3.9+. The Kubernetes API returns creation_timestamp as timezone-aware datetime objects in UTC, so the fallback should also be timezone-aware for safe comparison. Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

Import timezone from datetime module to use timezone.utc directly instead of datetime.timezone.utc for better readability. Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

@kramaranya

- Give Running and Succeeded pods equal priority (both are healthy states) - Update docstring to clearly explain equal priority and timestamp tiebreaker - Remove JOB_INDEX_LABEL .get() default, use direct access - Move _select_best_pod_for_role method after public methods per convention Addresses review comments from @kramaranya on PR kubeflow#160 Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

- Remove _select_best_pod_for_role() helper with status-priority logic - Sort pods by creation_timestamp and use seen set to skip duplicates - Add test case to parametrized test_get_job() per AGENTS.md pattern - Remove standalone test_get_job_with_pod_restarts() function Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

…ubernetes client v35+ - Replace watch.Watch().stream() with response.stream() for pod log streaming - Pass _preload_content=False to read_namespaced_pod_log when following logs - Remove unused watch import that caused ApiTypeError in newer kubernetes versions - Update mock_read_namespaced_pod_log to support streaming responses - Add parameterized test case for get_job_logs with follow=True Fixes the RuntimeError in E2E tests: 'Got an unexpected keyword argument watch to method read_namespaced_pod_log' which occurs with kubernetes client >= 35.0.0 Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

@andreyvelich

Address reviewer feedback from @andreyvelich: - Sort pods by creation_timestamp descending (newest first) - Use set-based deduplication to track seen (role, index) combinations - First occurrence (newest pod) wins, duplicates skipped - Creates unique keys: 'role' for initializers, 'role-index' for trainers - Improve error handling with .get() for label lookups and explicit RuntimeError messages - Remove unused Pod phase constants (POD_RUNNING, POD_FAILED, POD_UNKNOWN) - Remove unused timezone import This simplifies the pod filtering logic while maintaining correctness for duplicate pod detection during Kubernetes restarts. Closes: kubeflow#160 Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

HKanoje · 2026-05-27T02:25:47Z

@Shekharrajak @andreyvelich Just wanted to let you know that after the reset branch to commit 57ce1c8 Spark operator test are running properly now

andreyvelich · 2026-07-18T20:13:39Z

Sorry for the late reply @HKanoje, could you rebase it once again, so we can merge it?
@reckless-sherixx @Himanshujha7 could you also help reviewing this PR please?

reckless-sherixx · 2026-07-19T10:33:26Z

Sorry for the late reply @HKanoje, could you rebase it once again, so we can merge it? @reckless-sherixx @Himanshujha7 could you also help reviewing this PR please?

Currently the pr needs rebase and fixing of the tests failure, otherwise lgtm

reckless-sherixx · 2026-07-21T22:41:37Z

@andreyvelich The pr has yet not been completed so can I complete it and raise a pr?

andreyvelich · 2026-07-22T14:54:51Z

Sure, @reckless-sherixx feel free to finalize it!

andreyvelich · 2026-07-23T00:18:32Z

Fixed by: #632
/close

google-oss-prow · 2026-07-23T00:18:38Z

@andreyvelich: Closed this PR.

Details

In response to this:

Fixed by: #632
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

google-oss-prow Bot requested review from kramaranya and szaher November 17, 2025 03:33

google-oss-prow Bot added the size/L label Nov 17, 2025

HKanoje force-pushed the fix/filter-duplicate-pods-in-get-job branch from 978b209 to faf96a5 Compare November 17, 2025 03:36

Fiona-Waters reviewed Nov 17, 2025

View reviewed changes

google-oss-prow Bot assigned astefanutti Nov 27, 2025

google-oss-prow Bot added the lgtm label Nov 27, 2025

google-oss-prow Bot added the ok-to-test label Nov 27, 2025

kramaranya reviewed Dec 2, 2025

View reviewed changes

google-oss-prow Bot removed the lgtm label Dec 2, 2025

HKanoje force-pushed the fix/filter-duplicate-pods-in-get-job branch from a22ed47 to 3ec8093 Compare December 3, 2025 00:35

kramaranya mentioned this pull request Feb 21, 2026

[SDK] Get the correct TrainJob components using get_job() API #25

Closed

andreyvelich reviewed Feb 23, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings March 7, 2026 23:51

HKanoje force-pushed the fix/filter-duplicate-pods-in-get-job branch from 3ec8093 to ab7d286 Compare March 7, 2026 23:51

Copilot started reviewing on behalf of HKanoje March 7, 2026 23:52 View session

Copilot AI reviewed Mar 7, 2026

View reviewed changes

HKanoje force-pushed the fix/filter-duplicate-pods-in-get-job branch from 00a24f9 to 72f47b5 Compare March 8, 2026 00:29

google-oss-prow Bot added the size/L label May 27, 2026

Slowlybomb and others added 14 commits May 26, 2026 19:12

fix(trainer): import timezone for cleaner code

1981582

Import timezone from datetime module to use timezone.utc directly instead of datetime.timezone.utc for better readability. Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

refactor(trainer): optimize pod filtering to O(n) and clean up constants

d6afcd5

Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

chore: re-trigger Spark Examples E2E test

6d3ef60

Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

fix: resolve lockfile validation workflow conflicts from main

5ab1caa

Signed-off-by: HKanoje <hrithik.kanoje@gmail.com>

HKanoje force-pushed the fix/filter-duplicate-pods-in-get-job branch from 2732c83 to 5ab1caa Compare May 27, 2026 02:13

google-oss-prow Bot added size/XXL and removed size/L labels May 27, 2026

Merge branch 'main' into fix/filter-duplicate-pods-in-get-job

f81b20c

google-oss-prow Bot added size/L and removed size/XXL labels May 27, 2026

reckless-sherixx mentioned this pull request Jul 22, 2026

fix(trainer): deduplicate restarted TrainJob pods #632

Merged

google-oss-prow Bot closed this Jul 23, 2026

	key=lambda p: p.metadata.creation_timestamp or datetime.datetime.min, reverse=True
	key=lambda p: (p.metadata.creation_timestamp or datetime.datetime.min.replace(tzinfo=timezone.utc))

		constants.POD_RUNNING: 4, # Highest priority
		constants.POD_SUCCEEDED: 3, # Second highest


		self.namespace = cfg.namespace

		def _select_best_pod_for_role(

		print("test execution complete")


		def test_get_job_with_pod_restarts(kubernetes_backend):

	) -> Optional[models.IoK8sApiCoreV1Pod]:
	) -> models.IoK8sApiCoreV1Pod \| None:

Uh oh!

Conversation

HKanoje commented Nov 17, 2025

Description

Problem:

Solution:

Key Improvements

1. Groups Pods by role

2. Selects the most recent Pod

3. Maintains backward compatibility

Example Impact:

Before this fix:

After this fix:

Changes Made

Modified Files

backend.py

backend_test.py

Testing

Test Coverage

Checklist

Related Issues

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fiona-Waters commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

astefanutti commented Nov 27, 2025

Uh oh!

astefanutti commented Nov 27, 2025

Uh oh!

coveralls commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 19878220233

Details

💛 - Coveralls

Uh oh!

kramaranya left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Changes Made:

Testing:

Uh oh!

google-oss-prow Bot commented Dec 2, 2025

Uh oh!

HKanoje commented Jan 1, 2026

Uh oh!

HKanoje commented Feb 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krishdef7 commented Mar 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

`backend.py`

`backend_test.py`

Fiona-Waters commented Nov 17, 2025 •

edited

Loading

coveralls commented Nov 27, 2025 •

edited

Loading

reckless-sherixx commented Jul 19, 2026 •

edited

Loading