[GSoC] Add failure-investigation accessors to the core client (stacked on #3348) by Aryan95614 · Pull Request #3349 · Netflix/metaflow

Aryan95614 · 2026-08-23T22:13:07Z

GSoC 2026 project: metadata service request improvements (mentor: @saikonen). Upstream twin of the fork review PR (Aryan95614#2), opened per mentor request for a static PR number.

Rebased onto master after #3348 merged (2026-09-15); the diff is only this PR's own commits.

Summary

Folds bounded failure investigation into the regular Metaflow client as three
accessors on the existing objects, instead of a separate metaflow/agent
module. Built on the core pagination/filtering from #3348.

Context / Motivation

Review on the earlier revision steered the run-listing and pagination pieces
into the core client (that's #3348). Following the same direction, the failure
investigation layer now lives on the client objects rather than a parallel
metaflow.agent package, so there's a single, discoverable API.

Changes Made

Flow.failed_runs(*tags, since=None, max_runs=None) -> Iterator[Run]: failed
runs newest-first, via runs(*tags, _filters={"status:eq": "failed"}); tags
are forwarded exactly as for runs, and since adds an inclusive
ts_epoch:ge bound.
Run.failed_task -> Optional[Task]: the latest task that finished without
success (newest step first), or None. A task counts as failed when it raised
(exception is set) or a decorator such as @catch handled the failure;
tasks that are still running or never started are skipped.
Task.failure_summary -> Optional[FailureSummary]: normalized
{exception_type, message, stacktrace, attempt}, or None when the task has
no exception. Read errors propagate rather than being swallowed.
FailureSummary is exported from metaflow.client and metaflow, next to
Flow, Run and Task.
Removed the metaflow/agent module (failure_finder, pagination,
run_lister, TaskRef/FailureInfo, injectable resolver/fetcher).

Behavior note: failed_runs relies on server-side filtering, so it raises
against the local metadata provider or a service without pagination/filtering
support: the same gate as runs(_filters=...).

Testing

test/unit/test_client_failure_investigation.py: 19 tests covering
failed_runs filter/tags/bounds forwarding (including one through the real
Flow.runs down to the provider call), failed_task iteration order and the
crashed / decorator-handled / still-running cases, failure_summary
construction + error propagation, _normalize_exception across
mapping/object/string inputs, and the package exports.
Combined with the merged [GSoC] Add paginated, filtered run iteration to the core client #3348 run-listing tests (test_client_run_listing.py): 63 passed.
Checked against a local run with one raising step and one @catch step:
failed_task returns the raising task and failure_summary carries its type and message.
black (repo-pinned 25.12.0) clean.

Trade-offs / Design Decisions

failed_task cannot use finished alone: Task.finished reads _task_ok,
which the task runner sets to False when the step raises, so
finished and not successful would match only decorator-handled failures
and never the task that crashed. It checks exception is not None or finished
after ruling out successful tasks.
Dropped the injectable task_resolver / artifact_fetcher and the TaskRef
indirection: unnecessary once this is core; real Task objects are used and
the methods are testable with mocks.
Kept FailureSummary as a small frozen dataclass in core.py rather than
returning a bare dict, so the normalized fields are typed and discoverable.

codecov · 2026-08-24T12:16:20Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (master@8b500da). Learn more about missing BASE report.

Additional details and impacted files

@@            Coverage Diff            @@
##             master    #3349   +/-   ##
=========================================
  Coverage          ?   28.94%           
=========================================
  Files             ?      382           
  Lines             ?    52948           
  Branches          ?     9362           
=========================================
  Hits              ?    15327           
  Misses            ?    36469           
  Partials          ?     1152

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Shriprasad-P

Review

PR: [GSoC] Add failure-investigation accessors to the core client (stacked on #3348)

Touched: metaflow/client/core.py, metaflow/metadata_provider/metadata.py, metaflow/metaflow_config.py, metaflow/plugins/metadata_providers/service.py, test/unit/test_client_failure_investigation.py

Scoped to the stated goal; please confirm CI is green.
Size: +1249/-46.

Drive-by review after reading the diff.

Replaces the separate metaflow/agent module with core-client accessors, per the review that record iteration and normal client operations belong in core: - Flow.failed_runs(*, since=None, max_runs=None): iterate failed runs newest first via runs(filters={"status:eq": "failed"}); raises on local/older services like any other server-filtered listing. - Run.failed_task: the first unsuccessful task in the run (newest step first). - Task.failure_summary -> FailureSummary: normalized exception type/message/ stacktrace/attempt, or None when the task has no exception. Read errors propagate rather than being swallowed into diagnostic strings. Drops the injectable resolver/fetcher and the TaskRef indirection (uses real Task objects). Adds test/unit/test_client_failure_investigation.py.

greptile-apps · 2026-09-15T23:07:43Z

Confidence Score: 3/5

[Medium risk] Adds failure investigation accessors to the client API.

The PR is not yet safe to merge because failed_task excludes decorator-handled failures and still misses terminal tasks killed before completion metadata is persisted.

Findings

Handled Failures Are Excluded ▶
Hard Failures Are Skipped ▶

Summary

The PR adds failure-investigation accessors to the core client and exports the new FailureSummary type.

Adds server-filtered Flow.failed_runs.
Adds Run.failed_task based on per-attempt metadata.
Adds normalized exception details through Task.failure_summary.
Adds focused unit coverage and public package exports.

_{Reviews (3) · Last reviewed commit: "Address review: decide failed_task from ..."}

saikonen · 2026-09-15T23:33:31Z

+        """
+        for step in self:
+            for task in step:
+                if not task.successful:


this expression will be true for tasks that are running, pending, or have not launched because it simply pulls from the task["_success"].data

alternative could be

if task.finished and not task.successful:

Agreed on the problem, but finished on its own does not work here. Task.finished reads _task_ok, and task.py sets that to False when the step raises, so finished and not successful would match only failures a decorator handled (for example @catch) and never the task that actually crashed. I checked on a local run with one raising step and one @catch step: the raising task reads finished=False, successful=False with exception set; the @catch task reads finished=True, successful=False with exception None.

The latest push rules out successful tasks first, then returns the task if task.exception is not None or task.finished. That covers the crashed task and the handled one, and skips tasks that have not written their artifacts yet (running, pending, never launched), which read as unsuccessful, unfinished and without an exception. Tests cover all four cases.

…t FailureSummary - failed_runs passed filters= but runs() only takes _filters=, so every call raised TypeError; the plain Mock used for runs in the tests hid it. Tests now autospec Flow so runs enforces its real signature, and one test goes through the real runs() down to the provider call. - failed_runs takes tags positionally, like runs, and forwards them. - failed_task returned any task whose _success is not True, which is also the case for tasks still running or never started. It now requires the task to have finished without success: an exception recorded, or finished and not successful (a decorator-handled failure). finished on its own does not work because _task_ok is False for a task that raised. - FailureSummary is re-exported from metaflow.client and metaflow, next to Flow, Run and Task.

greptile-apps · 2026-09-17T14:46:34Z

+                # has recorded its exception; a task whose failure a decorator
+                # handled (e.g. @catch) is finished but not successful. A task
+                # that has not written its artifacts yet is neither.
+                if task.exception is not None or task.finished:


Hard Failures Are Skipped

If a task is terminated before its final artifacts are persisted—for example by OOM, SIGKILL, or preemption—it has no _exception or _task_ok artifact. This condition treats that genuinely failed task as unfinished and skips it, so failed_task can return None or an unrelated earlier failure.

saikonen

Apologies that the review turnaround has stretched for this. Hopefully you have a moment still to address the review comments. Otherwise let me know and I'll push some changes on top to get this past the finish line.

One generic note as well is that some of the user-facing docstrings have unnecessary implementation details. These could be streamlined a bit

saikonen · 2026-10-05T15:42:01Z

+            exception_type=normalized["type"],
+            message=normalized["message"],
+            stacktrace=normalized["stacktrace"],
+            attempt=self.current_attempt,


self.current_attempt points to the latest started attempt, so there can be a real mismatch here for example with using @retry

saikonen · 2026-10-05T15:44:57Z

            return False

+    @property
+    def failed_task(self) -> Optional[Task]:


the returned failed task might not be the correct one. This can return a @catch handled task instead of the fatal one depending on timings

saikonen · 2026-10-05T16:51:28Z

+        """
+        for step in self:
+            for task in step:
+                if task.successful:


task.successful along with task.exception and task.finished checks result in a lot of calls to the metadata- and datastore. Can we implement this in a more cost-effective manner?

@Retry

…rt the exception's own attempt - failed_task reads one metadata_dict per task and returns a task only when its latest started attempt is done and recorded attempt_ok=False, so @catch-handled failures and tasks still running a retry are skipped; no per-task artifact loads. - failure_summary takes the attempt from the _exception artifact it reads, not the latest started attempt, so it stays correct under @Retry. - Trim the user-facing docstrings.

Aryan95614 · 2026-10-05T17:36:02Z

Thanks @saikonen, and no worries on the timing. Pushed fixes for all three in 1596f08:

failed_task now decides from each task's status metadata (one metadata_dict read per task): a task is returned only when its latest started attempt is done and recorded attempt_ok=False. That skips @catch-handled failures and tasks still running a retry, and drops the per-task successful/exception/finished artifact loads; the exception artifact is only read for the task that is returned.
failure_summary takes the attempt from the _exception artifact it reads, so under @retry it reports the attempt that actually raised rather than the latest started one.
Trimmed the user-facing docstrings.

Tests cover the retry-in-progress, @catch, fatal and single-artifact-read cases. The failing test job is the minio image pull from quay.io (unauthorized), not this change.

greptile-apps · 2026-10-05T17:36:15Z

+                if (
+                    metadata.get("attempt_ok") == "False"
+                    and metadata.get("attempt-done") is not None
+                    and metadata.get("attempt") == metadata.get("attempt-done")
+                ):


Handled Failures Are Excluded

When a decorator such as @catch handles a task failure, it persists attempt_ok as "True". This condition only selects tasks where attempt_ok == "False", so failed_task skips the handled failure and can return None even though the accessor's stated contract says these tasks count as failed. The revised test also explicitly expects this failure to be skipped.

saikonen requested review from saikonen and talsperre August 23, 2026 23:06

Aryan95614 mentioned this pull request Aug 28, 2026

[GSoC] Add paginated, filtered run iteration to the core client #3348

Merged

Shriprasad-P reviewed Sep 12, 2026

View reviewed changes

Aryan95614 added 2 commits September 15, 2026 10:27

Address review: failed_task returns the latest failed task, say so

8cb149e

Aryan95614 force-pushed the agent-client-mentor-showcase branch from 7ea2f8c to 8cb149e Compare September 15, 2026 14:28

saikonen marked this pull request as ready for review September 15, 2026 23:05

greptile-apps Bot reviewed Sep 15, 2026

View reviewed changes

Comment thread metaflow/client/core.py Outdated

Comment thread metaflow/client/core.py

saikonen reviewed Sep 15, 2026

View reviewed changes

greptile-apps Bot reviewed Sep 17, 2026

View reviewed changes

saikonen reviewed Oct 5, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 5, 2026

View reviewed changes

Conversation

Aryan95614 commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Context / Motivation

Changes Made

Testing

Trade-offs / Design Decisions

Uh oh!

codecov Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Shriprasad-P left a comment

Choose a reason for hiding this comment

Review

Uh oh!

greptile-apps Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 3/5

Findings

Summary

Uh oh!

Uh oh!

Uh oh!

saikonen Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

saikonen Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Aryan95614 Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

saikonen left a comment

Choose a reason for hiding this comment

Uh oh!

saikonen Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

saikonen Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

saikonen Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

Aryan95614 commented Oct 5, 2026

Uh oh!

greptile-apps Bot Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Aryan95614 commented Aug 23, 2026 •

edited

Loading

codecov Bot commented Aug 24, 2026 •

edited

Loading

greptile-apps Bot commented Sep 15, 2026 •

edited

Loading