Skip to content

Conversation

@m7md7sien
Copy link
Contributor

@m7md7sien m7md7sien commented Oct 19, 2025

Description

Update Groundedness and Relevance Evaluators prompts:

  • Groundedness Update without_query guidelines
  • Relevance Evaluator Prompt Multi-turn Conversation Handling

If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

@m7md7sien m7md7sien requested a review from a team as a code owner October 19, 2025 17:07
@Copilot Copilot AI review requested due to automatic review settings October 19, 2025 17:07
@github-actions github-actions bot added the Evaluation Issues related to the client library for Azure AI Evaluation label Oct 19, 2025
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Updates evaluator prompt templates to improve groundedness guidance and add multi-turn conversation handling for relevance scoring. Key changes adjust relevance prompt to use conversation history semantics, revise groundedness rating definitions, and update asset tagging plus a test modification.

  • Relevance prompt now references CONVERSATION_HISTORY with multi-turn example
  • Groundedness (without query) prompt definitions and rubric wording revised
  • Test parameter list altered (RelevanceEvaluator removed) and asset tag updated

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
sdk/evaluation/azure-ai-evaluation/tests/e2etests/test_builtin_evaluators.py Removes RelevanceEvaluator from a parametrized test case.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/relevance.prompty Introduces multi-turn conversation handling and renames semantic concept from QUERY to CONVERSATION_HISTORY while retaining the input variable name.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_groundedness/groundedness_without_query.prompty Refactors groundedness rubric definitions and examples.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_groundedness/_groundedness.py Adjusts query presence check to treat None values like missing keys.
sdk/evaluation/azure-ai-evaluation/assets.json Updates asset tag identifier.
Comments suppressed due to low confidence (1)

sdk/evaluation/azure-ai-evaluation/tests/e2etests/test_builtin_evaluators.py:1324

  • Removing RelevanceEvaluator from this parametrized test reduces coverage for protected material JSON evaluation relative to other evaluators. If still supported, keep it in the list or add a dedicated test validating its updated multi-turn prompt behavior.
            RelevanceEvaluator,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Evaluation Issues related to the client library for Azure AI Evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant