Update Groundedness and Relevance Evaluators Prompts

sdk/evaluation/azure-ai-evaluation/assets.json

@@ -2,5 +2,5 @@
   "AssetsRepo": "Azure/azure-sdk-assets",
   "AssetsRepoPrefixPath": "python",
   "TagPrefix": "python/evaluation/azure-ai-evaluation",
-  "Tag": "python/evaluation/azure-ai-evaluation_b613e35220"
+  "Tag": "python/evaluation/azure-ai-evaluation_86c673042d"
 }

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_groundedness/_groundedness.py

@@ -238,7 +238,7 @@ def _has_context(self, eval_input: dict) -> bool:
 
     @override
     async def _do_eval(self, eval_input: Dict) -> Dict[str, Union[float, str]]:
-        if "query" not in eval_input:
+        if eval_input.get("query", None) is None:
             return await super()._do_eval(eval_input)
 
         contains_context = self._has_context(eval_input)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_groundedness/groundedness_without_query.prompty

@@ -29,11 +29,16 @@ system:
 
 user:
 # Definition
-**Groundedness** refers to how faithfully a response adheres to the information provided in the CONTEXT, ensuring that all content is directly supported by the context without introducing unsupported information or omitting critical details. It evaluates the fidelity and precision of the response in relation to the source material.
+**Groundedness** refers to how well a response is anchored in the provided context, evaluating its relevance, accuracy, and completeness based exclusively on that context. It assesses the extent to which the response directly and fully addresses the information without introducing unrelated or incorrect information.
+
+> Context is the source of truth for evaluating the response. 
+> Evaluate the groundedness of the response message based on the provided context.
 
 # Ratings
-## [Groundedness: 1] (Completely Ungrounded Response)
-**Definition:** The response is entirely unrelated to the CONTEXT, introducing topics or information that have no connection to the provided material.
+## [Groundedness: 1] (Completely Unrelated Response)
+**Definition:** A response that does not relate to the context in any way.
+  - Does not relate to the context at all.
+  - Talks about the general topic but does not respond to the context.
 
 **Examples:**
   **Context:** The company's profits increased by 20% in the last quarter.
@@ -42,8 +47,8 @@ user:
   **Context:** The new smartphone model features a larger display and improved battery life.
   **Response:** The history of ancient Egypt is fascinating and full of mysteries.
 
-## [Groundedness: 2] (Contradictory Response)
-**Definition:** The response directly contradicts or misrepresents the information provided in the CONTEXT.
+## [Groundedness: 2] (Attempts to Respond but Contains Incorrect Information)
+**Definition:** A response that attempts to relate to the context but includes incorrect information not supported by the context. It may misstate facts, misinterpret the context, or provide erroneous details. Even if some points are correct, the presence of inaccuracies makes the response unreliable.
 
 **Examples:**
   **Context:** The company's profits increased by 20% in the last quarter.
@@ -52,18 +57,18 @@ user:
   **Context:** The new smartphone model features a larger display and improved battery life.
   **Response:** The new smartphone model has a smaller display and shorter battery life.
 
-## [Groundedness: 3] (Accurate Response with Unsupported Additions)
-**Definition:** The response accurately includes information from the CONTEXT but adds details, opinions, or explanations that are not supported by the provided material.
+## [Groundedness: 3] (Accurate but Vague Response)
+**Definition:** A response that provides accurate information from the context but is overly generic or vague, not meaningfully engaging with the specific details in the context. The information is correct but lacks specificity and detail.
 
 **Examples:**
-  **Context:** The company's profits increased by 20% in the last quarter.
-  **Response:** The company's profits increased by 20% in the last quarter due to their aggressive marketing strategy.
+  **Context:** The company's profits increased by 20% in the last quarter, marking the highest growth rate in its history.
+  **Response:** The company is doing well financially.
 
-  **Context:** The new smartphone model features a larger display and improved battery life.
-  **Response:** The new smartphone model features a larger display, improved battery life, and comes with a free case.
+  **Context:** The new smartphone model features a larger display, improved battery life, and an upgraded camera system.
+  **Response:** The smartphone has some nice features.
 
-## [Groundedness: 4] (Incomplete Response Missing Critical Details)
-**Definition:** The response contains information from the CONTEXT but omits essential details that are necessary for a comprehensive understanding of the main point.
+## [Groundedness: 4] (Partially Correct Response)
+**Definition:** A response that provides correct information from the context but is incomplete or lacks specific details mentioned in the context. It captures some of the necessary information but omits key elements needed for a full understanding.
 
 **Examples:**
   **Context:** The company's profits increased by 20% in the last quarter, marking the highest growth rate in its history.      
@@ -73,7 +78,7 @@ user:
   **Response:** The new smartphone model features a larger display and improved battery life.
 
 ## [Groundedness: 5] (Fully Grounded and Complete Response)
-**Definition:** The response is entirely based on the CONTEXT, accurately and thoroughly conveying all essential information without introducing unsupported details or omitting critical points.
+**Definition:** A response that thoroughly and accurately conveys information from the context, including all relevant details. It directly addresses the context with precise information, demonstrating complete understanding without adding extraneous information.
 
 **Examples:**
   **Context:** The company's profits increased by 20% in the last quarter, marking the highest growth rate in its history.      

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/relevance.prompty

@@ -20,22 +20,25 @@ inputs:
 ---
 
 system:
-You are a Relevance-Judge, an impartial evaluator that scores how well the RESPONSE addresses the QUERY using the definitions provided.
+You are a Relevance-Judge, an impartial evaluator that scores how well the RESPONSE addresses the user's queries in the CONVERSATION_HISTORY using the definitions provided.
 
 user:
 ROLE
 ====
-You are a Relevance Evaluator. Your task is to judge how relevant a RESPONSE is to a QUERY using the Relevance definitions provided.
+You are a Relevance Evaluator. Your task is to judge how relevant a RESPONSE is to the CONVERSATION_HISTORY using the Relevance definitions provided.
 
 INPUT
 =====
-QUERY: {{query}}
+CONVERSATION_HISTORY: {{query}}
 RESPONSE: {{response}}
 
+CONVERSATION_HISTORY is the full dialogue between the user and the agent up to the user's latest message. For single-turn interactions, this will be just the user's query.
+RESPONSE is the agent's reply to the user's latest message.
+
 TASK
 ====
 Output a JSON object with:
-    1) a concise explanation of 15-60 words justifying your score based on how well the response is relevant to the query.
+    1) a concise explanation of 15-60 words justifying your score based on how well the response is relevant to the user's queries in the CONVERSATION_HISTORY.
     2) an integer score from 1 (very poor) to 5 (excellent) using the rubric below.
 
 The explanation should always precede the score and should clearly justify the score based on the rubric definitions.
@@ -49,13 +52,14 @@ Response format exactly as follows:
 
 EVALUATION STEPS
 ================
-A. Read the QUERY and RESPONSE carefully.
-B. Compare the RESPONSE against the rubric below:
-    - Does the response directly address the query?
+A. Read the CONVERSATION_HISTORY and RESPONSE carefully.
+B. Identify the user's query from the latest message (use conversation history for context if needed).
+C. Compare the RESPONSE against the rubric below:
+    - Does the response directly address the user's query?
     - Is the information complete, partial, or off-topic?
     - Is it vague, generic, or insightful?
-C. Match the response to the best score from the rubric.
-D. Provide a short explanation and the score using the required format.
+D. Match the response to the best score from the rubric.
+E. Provide a short explanation and the score using the required format.
 
 SCORING RUBRIC
 ==============
@@ -64,7 +68,7 @@ SCORING RUBRIC
 Definition: The response is unrelated to the question. It provides off-topic information and does not attempt to address the question posed.
 
 **Example A**
-QUERY: What is the team preparing for?
+CONVERSATION_HISTORY: What is the team preparing for?
 RESPONSE: I went grocery shopping yesterday evening.
 
 Expected Output:
@@ -75,7 +79,7 @@ Expected Output:
 
 
 **Example B**
-QUERY: When will the company's new product line launch?
+CONVERSATION_HISTORY: When will the company's new product line launch?
 RESPONSE: International travel can be very rewarding and educational.
 
 Expected Output:
@@ -89,7 +93,7 @@ Expected Output:
 Definition: The response is loosely or formally related to the query but fails to deliver any meaningful, specific, or useful information. This includes vague phrases, non-answers, or failure/error messages.
 
 **Example A**
-QUERY: What is the event about?
+CONVERSATION_HISTORY: What is the event about?
 RESPONSE: It’s something important.
 
 Expected Output:
@@ -99,7 +103,7 @@ Expected Output:
 }
 
 **Example B**
-QUERY: What’s the weather in Paris?
+CONVERSATION_HISTORY: What’s the weather in Paris?
 RESPONSE: I tried to find the forecast but the query failed.
 
 Expected Output:
@@ -112,7 +116,7 @@ Expected Output:
 Definition: The response addresses the query and includes relevant information, but omits essential components or detail. The answer is on-topic but insufficient to fully satisfy the request.
 
 **Example A**
-QUERY: What amenities does the new apartment complex provide?
+CONVERSATION_HISTORY: What amenities does the new apartment complex provide?
 RESPONSE: The apartment complex has a gym.
 
 Expected Output:
@@ -122,7 +126,7 @@ Expected Output:
 }
 
 **Example B**
-QUERY: What services does the premium membership include?
+CONVERSATION_HISTORY: What services does the premium membership include?
 RESPONSE: It includes priority customer support.
 
 Expected Output:
@@ -137,7 +141,7 @@ Expected Output:
 Definition: The response fully addresses the question with accurate and sufficient information, covering all essential aspects. Very minor omissions are acceptable as long as the core information is intact and the intent is clearly conveyed.
 
 **Example A**
-QUERY: What amenities does the new apartment complex provide?
+CONVERSATION_HISTORY: What amenities does the new apartment complex provide?
 RESPONSE: The apartment complex provides a gym, swimming pool, and 24/7 security.
 
 Expected Output:
@@ -147,7 +151,7 @@ Expected Output:
 }
 
 **Example B**
-QUERY: What services does the premium membership include?
+CONVERSATION_HISTORY: What services does the premium membership include?
 RESPONSE: The premium membership includes priority customer support, exclusive content access, and early product releases.
 
 Expected Output:
@@ -161,7 +165,7 @@ Expected Output:
 Definition: The response not only fully and accurately answers the question, but also adds meaningful elaboration, interpretation, or context that enhances the user's understanding. This goes beyond just listing relevant details — it offers insight into why the information matters, how it's useful, or what impact it has.
 
 **Example A**
-QUERY: What amenities does the new apartment complex provide?
+CONVERSATION_HISTORY: What amenities does the new apartment complex provide?
 RESPONSE: The apartment complex provides a gym, swimming pool, and 24/7 security, designed to offer residents a comfortable and active lifestyle while ensuring their safety.
 
 Expected Output:
@@ -171,11 +175,24 @@ Expected Output:
 }
 
 **Example B**
-QUERY: What services does the premium membership include?
+CONVERSATION_HISTORY: What services does the premium membership include?
 RESPONSE: The premium membership includes priority customer support, exclusive content access, and early product releases — tailored for users who want quicker resolutions and first access to new features.
 
 Expected Output:
 {
     "explanation": "The response covers all essential services and adds valuable insight about the target user and benefits, enriching the response beyond basic listing.",
     "score": 5
 }
+
+### Multi-turn Conversation Example
+When evaluating responses in a multi-turn conversation, consider the conversation context to understand the user's intent:
+
+**Example - Multi-turn Context**
+CONVERSATION_HISTORY: [{"role":"user","content":"I'm planning a vacation to Europe."},{"role":"assistant","content":"That sounds exciting! What time of year are you thinking of traveling?"},{"role":"user","content":"Probably in July. What's the weather like then?"}]
+RESPONSE: [{"role":"assistant","content":"July is summer in Europe with generally warm and pleasant weather. Most countries have temperatures between 20-25°C (68-77°F). It's a popular travel time, so expect crowds at major tourist attractions and higher accommodation prices."}]
+
+Expected Output:
+{
+    "explanation": "The response directly addresses the weather question while providing valuable context about crowds and pricing that's relevant to vacation planning established in the conversation.",
+    "score": 5
+}

sdk/evaluation/azure-ai-evaluation/tests/e2etests/test_builtin_evaluators.py

@@ -1232,7 +1232,6 @@ def test_multimodal_evaluator_protected_material_json(self, request, proj_scope,
     @pytest.mark.parametrize(
         "evaluator_cls",
         [
-            RelevanceEvaluator,
             FluencyEvaluator,
             SimilarityEvaluator,
             CoherenceEvaluator,