Skip to content

v2: Enhance validation with objective delegation, GitHub checks, and failure recovery #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nam20485 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

v2: Enhance validation with objective delegation, GitHub checks, and failure recovery #37

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 23 additions & 9 deletions ai_instruction_modules/ai-workflow-assignments/validate-assignment-completion.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ You will validate that the just-completed assignment has successfully met all it

It is important to the final quality of our product for everyone to perform their assignment exactly as specified.

Validation **must be delegated to an independent quality agent** (e.g., `qa-test-engineer`) who was not responsible for the original implementation work. This ensures objective evaluation and prevents self-validation bias.
**MANDATORY INDEPENDENT VALIDATION:** Validation **MUST** be delegated to an independent `qa-test-engineer` agent who was **NOT** responsible for the original implementation work. This ensures objective evaluation, prevents self-validation bias, and provides unbiased quality assessment. The qa-test-engineer must operate autonomously with no influence from the implementing agent.

Copilot AI Oct 30, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Consider replacing 'qa-test-engineer' with 'QA test engineer' for consistency with typical naming conventions, or ensure it matches the formal agent identifier used elsewhere in the system.

Suggested change
**MANDATORY INDEPENDENT VALIDATION:** Validation **MUST** be delegated to an independent `qa-test-engineer` agent who was **NOT** responsible for the original implementation work. This ensures objective evaluation, prevents self-validation bias, and provides unbiased quality assessment. The qa-test-engineer must operate autonomously with no influence from the implementing agent.
**MANDATORY INDEPENDENT VALIDATION:** Validation **MUST** be delegated to an independent QA test engineer agent who was **NOT** responsible for the original implementation work. This ensures objective evaluation, prevents self-validation bias, and provides unbiased quality assessment. The QA test engineer must operate autonomously with no influence from the implementing agent.

Copilot uses AI. Check for mistakes.

### Detailed Steps

Expand Down Expand Up @@ -101,9 +101,12 @@ Validation **must be delegated to an independent quality agent** (e.g., `qa-test
- Check documentation is present

3. For GitHub operations (issue/PR/project changes):
- Delegate a `github-expert` (or equivalent) agent to query live repository state (e.g., fetch the created issue or PR)
- Confirm existence, title, labels, blocking relationships, and milestone links
- If the resource is missing or mismatched, mark validation as **FAILED** and trigger `recover-from-error`
- **MANDATORY:** Delegate a `github-expert` agent to perform live GitHub API queries for real-state verification

Copilot AI Oct 30, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrected spelling of 'real-state' to 'real-time state' or 'real state'.

Suggested change
- **MANDATORY:** Delegate a `github-expert` agent to perform live GitHub API queries for real-state verification
- **MANDATORY:** Delegate a `github-expert` agent to perform live GitHub API queries for real-time state verification

Copilot uses AI. Check for mistakes.

Copilot AI Oct 30, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The command examples use placeholder syntax without clear indication of how to substitute actual values. Consider adding a note that {owner}, {repo}, {number}, and {project-number} should be replaced with actual values from the repository context.

Suggested change
- **MANDATORY:** Delegate a `github-expert` agent to perform live GitHub API queries for real-state verification
- **MANDATORY:** Delegate a `github-expert` agent to perform live GitHub API queries for real-state verification
- **Note:** In the following command examples, replace `{owner}`, `{repo}`, `{number}`, and `{project-number}` with the actual values from your repository context.

Copilot uses AI. Check for mistakes.
- Query repository state using GitHub APIs: `gh api repos/{owner}/{repo}/issues/{number}`, `gh pr view {number}`, `gh project item-list {project-number}`
- Verify existence, title, body, labels, assignees, milestone, blocking relationships, and linked issues/PRs
- Cross-reference with assignment specifications to ensure exact match of requirements
- **CRITICAL:** If any GitHub resource is missing, mismatched, or incorrectly configured, mark validation as **FAILED** and auto-trigger `recover-from-error` workflow
- Document all API query results and comparisons in validation report for audit trail
Comment on lines +104 to +109

@gemini-code-assist gemini-code-assist Bot Oct 30, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The indentation for this list of steps is inconsistent with other lists in the document. This list uses 4 spaces for indentation, while other lists (e.g., under step 2 on line 98) use 3 spaces. For consistency and correct rendering by markdown parsers, please adjust the indentation to 3 spaces.

Suggested change
- **MANDATORY:** Delegate a `github-expert` agent to perform live GitHub API queries for real-state verification
- Query repository state using GitHub APIs: `gh api repos/{owner}/{repo}/issues/{number}`, `gh pr view {number}`, `gh project item-list {project-number}`
- Verify existence, title, body, labels, assignees, milestone, blocking relationships, and linked issues/PRs
- Cross-reference with assignment specifications to ensure exact match of requirements
- **CRITICAL:** If any GitHub resource is missing, mismatched, or incorrectly configured, mark validation as **FAILED** and auto-trigger `recover-from-error` workflow
- Document all API query results and comparisons in validation report for audit trail
- **MANDATORY:** Delegate a `github-expert` agent to perform live GitHub API queries for real-state verification
- Query repository state using GitHub APIs: `gh api repos/{owner}/{repo}/issues/{number}`, `gh pr view {number}`, `gh project item-list {project-number}`
- Verify existence, title, body, labels, assignees, milestone, blocking relationships, and linked issues/PRs
- Cross-reference with assignment specifications to ensure exact match of requirements
- **CRITICAL:** If any GitHub resource is missing, mismatched, or incorrectly configured, mark validation as **FAILED** and auto-trigger `recover-from-error` workflow
- Document all API query results and comparisons in validation report for audit trail


4. Record file verification results:
- List of files checked
Expand Down Expand Up @@ -334,11 +337,12 @@ chmod +x deploy.sh

2. Note: This validation must be executed by the independent `qa-test-engineer` (or equivalent) agent. Cross-check executor outputs, but rely on objective evidence such as repository queries, command logs, and validation reports before declaring PASS/FAIL.

3. If failed, block progression:
- Do NOT proceed to next assignment
- Notify user of validation failure
- Provide link to validation report
- Request manual intervention
3. If failed, block progression and auto-trigger recovery:
- **AUTO-TRIGGER:** Immediately invoke `recover-from-error` workflow with validation failure details
- Do NOT proceed to next assignment until recovery is successful
- Notify user of validation failure with specific remediation steps
- Provide link to validation report and recovery workflow status
- Request manual intervention only if automated recovery fails
Comment on lines +341 to +345

@gemini-code-assist gemini-code-assist Bot Oct 30, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The indentation for this list of steps is inconsistent with other lists in the document. This list uses 4 spaces for indentation, while other lists (e.g., under step 4 on line 347) use 3 spaces. For consistency and correct rendering by markdown parsers, please adjust the indentation to 3 spaces.

Suggested change
- **AUTO-TRIGGER:** Immediately invoke `recover-from-error` workflow with validation failure details
- Do NOT proceed to next assignment until recovery is successful
- Notify user of validation failure with specific remediation steps
- Provide link to validation report and recovery workflow status
- Request manual intervention only if automated recovery fails
- **AUTO-TRIGGER:** Immediately invoke `recover-from-error` workflow with validation failure details
- Do NOT proceed to next assignment until recovery is successful
- Notify user of validation failure with specific remediation steps
- Provide link to validation report and recovery workflow status
- Request manual intervention only if automated recovery fails


4. If passed, allow progression:
- Log validation success
Expand Down Expand Up @@ -374,3 +378,13 @@ chmod +x deploy.sh
- Keep validation reports for audit trail
- Validation prevents cascading failures
- Early validation catches issues before they compound

### Objective Reporting Requirements

**MANDATORY OBJECTIVE ASSESSMENT:**
- All validation decisions must be based on verifiable evidence only (file existence, command exit codes, GitHub API responses)
- qa-test-engineer must document all evidence sources and decision rationale
- No subjective interpretation allowed - all criteria must have clear pass/fail evidence
- Bias reduction: Implementing agent outputs are treated as untrusted until independently verified

Copilot AI Oct 30, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] The phrase 'untrusted' may have negative connotations. Consider using more neutral terminology such as 'unverified' or 'pending verification' to maintain professional tone while conveying the same validation requirement.

Suggested change
- Bias reduction: Implementing agent outputs are treated as untrusted until independently verified
- Bias reduction: Implementing agent outputs are treated as unverified until independently verified

Copilot uses AI. Check for mistakes.
- Cross-validation: Multiple independent checks (files + commands + GitHub state) required for high-confidence validation
- Audit trail: All validation steps, evidence, and decisions must be fully documented for review