Skip to content

fix: keep the current plan when LLM refinement drops hunks or does not reduce violations - #121

Open
vitali87 wants to merge 1 commit into
fix/score-plan-cyclefrom
fix/refinement-keeps-valid-plan
Open

fix: keep the current plan when LLM refinement drops hunks or does not reduce violations#121
vitali87 wants to merge 1 commit into
fix/score-plan-cyclefrom
fix/refinement-keeps-valid-plan

Conversation

@vitali87

@vitali87 vitali87 commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Summary

Each LLM refinement iteration replaced the plan with whatever the model returned. There was no coverage check and no comparison with the plan being refined, so a response that dropped a hunk turned a plan that only had LOC warnings into one that fails validate_plan with COVERAGE_GAP (the run aborted), and a response that was no better was accepted anyway.

Now a candidate is kept only if it passes validate_coverage and has strictly fewer LOC-bound violations than the plan it replaces. Otherwise the current plan is kept and refinement stops, with a warning naming the reason:

  • Refinement iteration N produced an invalid plan (Hunk b.py[0] not assigned to any group); keeping the current plan …
  • Refinement iteration N did not reduce violations (2 -> 2); keeping the current plan

Stacked on #63 (stack #75 = #59#63#121); #59 provides the LLMError wrapping of malformed LLM output that this relies on.

Test plan

  • test_refinement_that_drops_hunks_is_rejected and test_refinement_that_does_not_improve_is_rejected — both fail on the base
  • Review probed the overlap path, a multi-iteration improve-then-stall sequence (keeps the improved plan), and the chunked path
  • 451 tests pass, ruff clean
  • Local review: 5/5

@coderabbitai

coderabbitai Bot commented Aug 30, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 76778256-b0e7-45ad-bf36-7045b9e47cc9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@vitali87
vitali87 force-pushed the fix/refinement-keeps-valid-plan branch from 345cb57 to 7813443 Compare August 30, 2026 16:31
@vitali87
vitali87 changed the base branch from fix/llm-parse-errors-retry to fix/guard-saved-plan-dag August 30, 2026 16:32
@vitali87
vitali87 force-pushed the fix/refinement-keeps-valid-plan branch from 7813443 to 7455f28 Compare August 30, 2026 16:40
@vitali87
vitali87 changed the base branch from fix/guard-saved-plan-dag to fix/score-plan-cycle August 30, 2026 16:40
@vitali87
vitali87 force-pushed the fix/refinement-keeps-valid-plan branch from 7455f28 to db00703 Compare August 30, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant