Question on the difference between ITBench and ITBench-AA

[HowardLi1984]

Dear Artificial Analysis and ITBench Teams,

I hope this email finds you well.

We are currently working on developing AI agents for IT operations and are highly interested in benchmarking our models against the ITBench-AA standard. We have been studying your open-source materials and would like to politely ask a few questions regarding the dataset details and the official submission process:

Dataset Discrepancies: We noticed that the Hugging Face repository (ArtificialAnalysis/ITBench-AA) provides a dataset of 40 SRE evaluation cases. However, upon comparison, we found that these 40 cases are not entirely identical to the 40 cases listed under ITBench/scenarios/sre/library/indexes/scenarios in the official IBM ITBench GitHub repository. Could you kindly clarify the differences between these two sets, and advise on which one we should prioritize for local alignment?

Submission Process for Remote Evaluation: We understand that the official ITBench-AA evaluation is conducted on your remote servers. Could you please guide us on how to submit our agent for this official evaluation? We would appreciate any details on the required formats (e.g., whether we should provide an API endpoint, a Docker image, or model weights).

Evaluation Scope (Diagnosis vs. Resolution): We observed that the ITBench-AA dataset on Hugging Face focuses solely on assessing SRE diagnosis capabilities. Conversely, the official ITBench GitHub repository retains evaluations for both diagnosis and resolution. For standard benchmarking on the ITBench-AA leaderboard, should we treat the diagnosis-only evaluation as the primary metric, or are there plans to incorporate the resolution phase as well?

Thank you very much for your time and for contributing such a valuable benchmark to the community. We look forward to your guidance.

[Red-GV]

Heya @HowardLi1984 and thank you for your interest in ITBench!

1. We'll be adding the scenarios corresponding to the public AA dataset to this repository in the coming days (within two weeks). Soon, you - and others - will be able to see and access them on the live environment in addition to the offline (static) environment. Released in chore: release scenarios used in Artifical Analysis dataset #979 .
2. The AA evaluations are more so targeted towards model providers rather than agents. Currently, we don't have a leaderboard for the offline (static) environments. Though this is something we may consider in the future.
3. Currently, the plan is to only support diagnosis-only evaluations.