v2: Enhance validation with objective delegation, GitHub checks, and failure recovery

[nam20485]

Adds emphasis on independent qa-test-engineer validation, GitHub API queries for output verification, auto-trigger of recover-from-error on fails, and objective reporting notes. Backward compatible; improves accuracy and automation. Rationale: Reduces bias, ensures real-state checks, and chains recovery seamlessly.

[gemini-code-assist]

Summary of Changes

Hello @nam20485, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the robustness and objectivity of the assignment completion validation process. It introduces a mandatory independent validation step, leverages live GitHub API queries for precise state verification, and implements an automated recovery mechanism for failures. The changes aim to reduce bias, ensure real-state checks, and seamlessly chain recovery workflows, ultimately improving the accuracy and automation of the validation system.

Highlights

• Mandatory Independent Validation: Validation of assignment completion must now be delegated to an independent qa-test-engineer to ensure objective evaluation and prevent self-validation bias, operating autonomously without influence from the implementing agent.
• Enhanced GitHub API Verification: GitHub operations validation now mandates live GitHub API queries by a github-expert to verify the real-state of resources (issues, PRs, projects), including detailed checks for existence, title, body, labels, assignees, milestones, and blocking relationships. Critical failures will auto-trigger recovery.
• Automated Failure Recovery: Upon validation failure, the system will now immediately auto-trigger a recover-from-error workflow with failure details. Progression to the next assignment is blocked until recovery is successful, with specific remediation steps and recovery workflow status provided to the user.
• Objective Reporting Requirements: A new section outlines mandatory objective assessment criteria, requiring all validation decisions to be based solely on verifiable evidence (e.g., file existence, command exit codes, GitHub API responses). It emphasizes documenting evidence, reducing bias, cross-validation, and maintaining a full audit trail.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request significantly improves the validation process for AI workflow assignments. It introduces stricter requirements for independent validation by a qa-test-engineer, adds detailed GitHub API checks for verifying real-world state, and automates the recovery process on failure. The addition of objective reporting requirements further strengthens the process by demanding evidence-based decisions. The changes are well-aligned with the goal of improving accuracy and automation. My feedback focuses on minor formatting inconsistencies to ensure the documentation is clear and uniformly structured.

[gemini-code-assist]

The indentation for this list of steps is inconsistent with other lists in the document. This list uses 4 spaces for indentation, while other lists (e.g., under step 2 on line 98) use 3 spaces. For consistency and correct rendering by markdown parsers, please adjust the indentation to 3 spaces.

Suggested change

	- **MANDATORY:** Delegate a `github-expert` agent to perform live GitHub API queries for real-state verification
	- Query repository state using GitHub APIs: `gh api repos/{owner}/{repo}/issues/{number}`, `gh pr view {number}`, `gh project item-list {project-number}`
	- Verify existence, title, body, labels, assignees, milestone, blocking relationships, and linked issues/PRs
	- Cross-reference with assignment specifications to ensure exact match of requirements
	- **CRITICAL:** If any GitHub resource is missing, mismatched, or incorrectly configured, mark validation as **FAILED** and auto-trigger `recover-from-error` workflow
	- Document all API query results and comparisons in validation report for audit trail
	- **MANDATORY:** Delegate a `github-expert` agent to perform live GitHub API queries for real-state verification
	- Query repository state using GitHub APIs: `gh api repos/{owner}/{repo}/issues/{number}`, `gh pr view {number}`, `gh project item-list {project-number}`
	- Verify existence, title, body, labels, assignees, milestone, blocking relationships, and linked issues/PRs
	- Cross-reference with assignment specifications to ensure exact match of requirements
	- **CRITICAL:** If any GitHub resource is missing, mismatched, or incorrectly configured, mark validation as **FAILED** and auto-trigger `recover-from-error` workflow
	- Document all API query results and comparisons in validation report for audit trail

[gemini-code-assist]

The indentation for this list of steps is inconsistent with other lists in the document. This list uses 4 spaces for indentation, while other lists (e.g., under step 4 on line 347) use 3 spaces. For consistency and correct rendering by markdown parsers, please adjust the indentation to 3 spaces.

Suggested change

	- **AUTO-TRIGGER:** Immediately invoke `recover-from-error` workflow with validation failure details
	- Do NOT proceed to next assignment until recovery is successful
	- Notify user of validation failure with specific remediation steps
	- Provide link to validation report and recovery workflow status
	- Request manual intervention only if automated recovery fails
	- **AUTO-TRIGGER:** Immediately invoke `recover-from-error` workflow with validation failure details
	- Do NOT proceed to next assignment until recovery is successful
	- Notify user of validation failure with specific remediation steps
	- Provide link to validation report and recovery workflow status
	- Request manual intervention only if automated recovery fails

[Copilot]

Pull Request Overview

This PR strengthens the validation process for assignment completion by enforcing independent quality assurance, mandating automated error recovery, and requiring objective evidence-based validation. The changes emphasize separation of concerns between implementation and validation to prevent self-validation bias.

Key Changes:

• Enforces mandatory independent validation by qa-test-engineer agents separate from implementing agents
• Adds detailed GitHub API verification requirements with specific commands and cross-reference checks
• Implements automatic recover-from-error workflow triggering on validation failures instead of manual intervention
• Introduces objective reporting requirements with documented evidence trails and bias reduction measures

[Copilot]

[nitpick] Consider replacing 'qa-test-engineer' with 'QA test engineer' for consistency with typical naming conventions, or ensure it matches the formal agent identifier used elsewhere in the system.

Suggested change

	**MANDATORY INDEPENDENT VALIDATION:** Validation **MUST** be delegated to an independent `qa-test-engineer` agent who was **NOT** responsible for the original implementation work. This ensures objective evaluation, prevents self-validation bias, and provides unbiased quality assessment. The qa-test-engineer must operate autonomously with no influence from the implementing agent.
	**MANDATORY INDEPENDENT VALIDATION:** Validation **MUST** be delegated to an independent QA test engineer agent who was **NOT** responsible for the original implementation work. This ensures objective evaluation, prevents self-validation bias, and provides unbiased quality assessment. The QA test engineer must operate autonomously with no influence from the implementing agent.

[Copilot]

Corrected spelling of 'real-state' to 'real-time state' or 'real state'.

Suggested change

	- **MANDATORY:** Delegate a `github-expert` agent to perform live GitHub API queries for real-state verification
	- **MANDATORY:** Delegate a `github-expert` agent to perform live GitHub API queries for real-time state verification

[Copilot]

The command examples use placeholder syntax without clear indication of how to substitute actual values. Consider adding a note that {owner}, {repo}, {number}, and {project-number} should be replaced with actual values from the repository context.

Suggested change

	- **MANDATORY:** Delegate a `github-expert` agent to perform live GitHub API queries for real-state verification
	- **MANDATORY:** Delegate a `github-expert` agent to perform live GitHub API queries for real-state verification
	- **Note:** In the following command examples, replace `{owner}`, `{repo}`, `{number}`, and `{project-number}` with the actual values from your repository context.

[Copilot]

[nitpick] The phrase 'untrusted' may have negative connotations. Consider using more neutral terminology such as 'unverified' or 'pending verification' to maintain professional tone while conveying the same validation requirement.

Suggested change

	- Bias reduction: Implementing agent outputs are treated as untrusted until independently verified
	- Bias reduction: Implementing agent outputs are treated as unverified until independently verified

[claude]

Pull Request Review: v2 Validation Enhancements

Summary

This PR enhances the validate-assignment-completion.md assignment with stronger emphasis on independent validation, GitHub API verification, and automated error recovery. The changes are well-aligned with the repository's quality objectives and workflow orchestration patterns.

---

Code Quality & Best Practices

Strengths

1. Clear Separation of Concerns: The emphasis on independent qa-test-engineer validation (line 69) prevents self-validation bias and aligns with testing best practices
2. Explicit Requirements: Use of bold **MANDATORY** and **CRITICAL** markers makes requirements unambiguous
3. Backward Compatibility: Changes are additive and enhance existing validation without breaking current workflows
4. Documentation Quality: Excellent use of structured sections with clear examples and rationale

Areas for Improvement

1. Agent Role Dependencies (lines 69, 104, 338):

   • References qa-test-engineer and github-expert agents but these roles are not found in ai_instruction_modules/ai-workflow-roles/
   • Recommendation: Create corresponding role definition files:
     • ai_instruction_modules/ai-workflow-roles/qa-test-engineer/ai-workflow-role.md
     • ai_instruction_modules/ai-workflow-roles/github-expert/ai-workflow-role.md
   • Or document these as agent personas/capabilities rather than formal roles

2. GitHub API Command Examples (line 105):

   • Example uses placeholder syntax: gh api repos/{owner}/{repo}/issues/{number}
   • Recommendation: Add concrete example with actual values for clarity:

     # Example: gh api repos/nam20485/agent-instructions/issues/37

---

Potential Bugs & Issues

Critical Issues

1. Missing Workflow Reference (lines 108, 341):

   • Auto-triggers recover-from-error workflow but doesn't specify HOW to trigger it
   • The recover-from-error.md assignment exists but integration mechanism is unclear
   • Fix Required: Add explicit invocation pattern:

     - Invoke: `assign agent the 'recover-from-error' assignment with context: $validation_failure_details`

2. Circular Dependency Risk (line 341):

   • If recover-from-error fails during validation recovery, what happens?
   • Need max retry limit or escalation path to prevent infinite loops
   • Note: The recover-from-error.md (lines 116-120) does have a 3-attempt limit, but this should be cross-referenced

Medium Issues

1. Undefined Behavior (line 109):

   • "Document all API query results" but doesn't specify WHERE to document
   • Validation report template (lines 224-295) doesn't include GitHub verification section
   • Recommendation: Add GitHub verification section to report template:

     ## GitHub Resource Verification
     
     ### Issues/PRs Checked
     - Issue #X: ✅ Exists, labels match, milestone correct
     - PR #Y: ❌ Missing expected label "enhancement"
     
     ### API Queries Executed
     - `gh api repos/owner/repo/issues/X` - [link to output]

2. Ambiguous Agent Autonomy (line 69):

   • States qa-test-engineer "must operate autonomously with no influence from the implementing agent"
   • In practice, both agents may be Claude instances with shared context
   • Recommendation: Clarify implementation details or add note about context isolation

---

Performance Considerations

Positive

1. Efficient Workflow Gating: Auto-trigger on failure (line 341) prevents wasted work on dependent assignments
2. Audit Trail: Documentation requirements support debugging without re-execution

Concerns

1. GitHub API Rate Limits:

   • No mention of rate limiting considerations for API queries
   • Multiple validation runs could exhaust API quota
   • Recommendation: Add note about caching validation results or rate limit handling

2. Validation Report Storage (line 220):

   • Each validation creates timestamped file in docs/validation/
   • No cleanup policy for old reports
   • Recommendation: Add retention policy or rotation guidance

---

Security Concerns

Low Risk

1. GitHub Token Exposure:

   • gh CLI commands require authentication token
   • No guidance on token security or permissions required
   • Recommendation: Add prerequisite note about gh auth and minimum permissions (read:repo, read:project)

2. Command Injection (line 105):

   • Examples show variable interpolation in shell commands
   • While this is documentation, add sanitization reminder for actual implementations
   • Low priority: This is an instruction file, not executable code

---

Test Coverage

Gap Analysis

1. No Test Cases for New Behavior:

   • PR modifies validation logic but doesn't include test scenarios
   • Recommendation: Add example validation scenarios in PR description or separate test assignment:
     • Scenario 1: GitHub resource exists and matches → PASS
     • Scenario 2: GitHub PR missing expected label → FAIL + auto-trigger recovery
     • Scenario 3: Independent qa-test-engineer detects issue implementing agent missed

2. Integration Testing:

   • Changes reference recover-from-error workflow but no integration test
   • Recommendation: Manual test or add to workflow validation suite

---

Alignment with Repository Guidelines (CLAUDE.md)

Adherence

• ✅ Follows modular instruction system pattern
• ✅ Includes acceptance criteria as Definition of Done
• ✅ Emphasizes objective evidence-based validation
• ✅ Backward compatible (no breaking changes)
• ✅ Supports audit trail requirements

Recommendations

1. Remote Canonical Source (CLAUDE.md lines 137-145):

   • Changes to assignment file take effect immediately when fetched
   • Consider adding version comment to track when changes were introduced
   • Current <!-- v2: ... --> comment (line 3) is good practice ✅

2. Dynamic Workflow Integration:

   • This validation assignment is referenced in dynamic workflows
   • Ensure changes are tested with orchestrate-dynamic-workflow assignment

---

Overall Assessment

Status: ✅ APPROVE WITH RECOMMENDATIONS

Rationale:

• Changes significantly improve validation quality and reduce bias
• Well-structured and aligned with repository architecture
• Issues identified are non-blocking but should be addressed post-merge or in follow-up PR

Priority Fixes (recommended for follow-up):

1. Create or document qa-test-engineer and github-expert agent roles
2. Add GitHub verification section to validation report template
3. Clarify recover-from-error invocation mechanism
4. Add rate limiting guidance for GitHub API queries

Merge Recommendation: ✅ Safe to merge after addressing documentation gaps

---

Next Steps

1. Address critical documentation gaps (agent roles, workflow invocation)
2. Consider adding example validation scenarios
3. Test integration with recover-from-error workflow
4. Update validation report template to include GitHub verification

---

Reviewed by: Claude Code Agent
Review Date: 2025-10-30
PR: #37 - v2: Enhance validation with objective delegation, GitHub checks, and failure recovery