Skip to content

Jessli/bugbash1020 #43536

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 99 commits into
base: main
Choose a base branch
from
Draft

Jessli/bugbash1020 #43536

wants to merge 99 commits into from

Conversation

@YoYoJa
Copy link
Contributor

@YoYoJa YoYoJa commented Oct 20, 2025

Description

Please add an informative description that covers that changes made by the pull request and link all relevant issues.

If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

Neehar Duvvuri and others added 30 commits September 30, 2025 12:36
skeleton code
fa2f8aa
add function that logs eval results to app insights
8dfde5b
format
771e19d
log red team data
7174bc4
@YoYoJa
add eval result converter
980e2fa
@YoYoJa
Add result converter
57c73b8
@YoYoJa
update converter params to optional
1730b17
@YoYoJa
add eval meta data
3bf93f7
@YoYoJa
merge converter change to include eval_meta_data
e47639d
@YoYoJa
fix type
5b198b4
@YoYoJa
remove useless file
5fbbabe
@YoYoJa
get eval meta data as input
6ca31a1
@YoYoJa
fix build errors
ea93d1a
@YoYoJa
remove useless import
e6a9caa
@YoYoJa
resolve comments
f24f0e0
@YoYoJa
update
0abddb0
get agent info from app insights config
f25760c
merge with main
08d3e36
@YoYoJa
update comments
518b4af
@YoYoJa
merge otel
74dfddc
@YoYoJa
merge result converter
15d881e
@YoYoJa
fix checker failure
5c44f70
@YoYoJa
add eval result converter (#43233)
73fb9c7 
* add eval result converter

* Add result converter

* update converter params to optional

* add eval meta data

* fix type

* remove useless file

* get eval meta data as input

* fix build errors

* remove useless import

* resolve comments

* update

* update comments
rename function
46e0504
Merge branch 'main' of https://github.com/Azure/azure-sdk-for-python …
9cae8cf 
…into needuv/structured-results-otel-logging
fix a thing
17cdd1e
@YoYoJa
add error msg and error code
2ce023e
@YoYoJa
merge main
654e28f
@YoYoJa
Surface evaluator error msg
32aad08
@YoYoJa
surface out error
d1449f5
YoYoJa and others added 28 commits October 16, 2025 15:21
@YoYoJa
merge and resolve conflict
a9416a8
@YoYoJa
Jessli/convert expose sample data for sdk promty based evaluators (#4…
ad17aa1 
…3474)

* add eval result converter

* Add result converter

* update converter params to optional

* add eval meta data

* fix type

* remove useless file

* get eval meta data as input

* fix build errors

* remove useless import

* resolve comments

* update

* update comments

* fix checker failure

* add error msg and error code

* Surface evaluator error msg

* update UT

* fix usage

* make eval_meta_data optional

* remove useless lines

* update param name to add underscore

* parse updated annotation results

* update trace_id

* expose sample data for sdk evaluators

* update
@YoYoJa
update
3cc294c
@YoYoJa
update
26c8a53
@YoYoJa
Jessli/convert remove token counts from metrics (#43477)
ad66137 
* add eval result converter

* Add result converter

* update converter params to optional

* add eval meta data

* fix type

* remove useless file

* get eval meta data as input

* fix build errors

* remove useless import

* resolve comments

* update

* update comments

* fix checker failure

* add error msg and error code

* Surface evaluator error msg

* update UT

* fix usage

* make eval_meta_data optional

* remove useless lines

* update param name to add underscore

* parse updated annotation results

* update trace_id

* expose sample data for sdk evaluators

* update

* update
@YoYoJa
fix UT
d2e40f9
@YoYoJa
remove print
4a0a86c
@YoYoJa
Jessli/convert remove useless lines and fix UT (#43480)
2cb5c5f 
* add eval result converter

* Add result converter

* update converter params to optional

* add eval meta data

* fix type

* remove useless file

* get eval meta data as input

* fix build errors

* remove useless import

* resolve comments

* update

* update comments

* fix checker failure

* add error msg and error code

* Surface evaluator error msg

* update UT

* fix usage

* make eval_meta_data optional

* remove useless lines

* update param name to add underscore

* parse updated annotation results

* update trace_id

* expose sample data for sdk evaluators

* update

* update

* fix UT
try changing prompty output to dict
aef2668
Merge branch 'needuv/structured-results-otel-logging' of https://gith…
d0323df 
…ub.com/Azure/azure-sdk-for-python into needuv/structured-results-otel-logging
change prompty output to dict
29d44b3
run black
4f05fb5
@YoYoJa
merge remote
3664e9a
fix relevance and prompty test
06e07f4
fix unit tests
de8a43d
@YoYoJa
merge remote
fa3ae5c
@YoYoJa
fix tests
e386fed
fix prompty tests
b82956d
fix similarity test
340b243
move groundedness to actual prompty impl
8b87275
@kdestin
chore: Update assets.json
ffbbacb
@YoYoJa
fix test
023bff7
@YoYoJa
merge remote
244b530
@YoYoJa
Jessli/convert Fix test failure (#43518)
eec6eb2 
* add eval result converter

* Add result converter

* update converter params to optional

* add eval meta data

* fix type

* remove useless file

* get eval meta data as input

* fix build errors

* remove useless import

* resolve comments

* update

* update comments

* fix checker failure

* add error msg and error code

* Surface evaluator error msg

* update UT

* fix usage

* make eval_meta_data optional

* remove useless lines

* update param name to add underscore

* parse updated annotation results

* update trace_id

* expose sample data for sdk evaluators

* update

* update

* fix UT

* fix tests

* fix test
add extra attributes to app insights config, remove agent name/id/ver…
7abf6c3 
…sion/ from app insights config
pin otel<1.39.0 since breaking change coming in that version
36e45f6
merge
201af13
@YoYoJa
merge main
ce3af31
@github-actions github-actions bot added the Evaluation Issues related to the client library for Azure AI Evaluation label Oct 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

Evaluation Issues related to the client library for Azure AI Evaluation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@YoYoJa @kdestin