Skip to content

Add Vectara Hallucination Leaderboard adapter#157

Open
mohammadrezakarami wants to merge 1 commit into
evaleval:mainfrom
mohammadrezakarami:codex/vectara-hallucination-adapter
Open

Add Vectara Hallucination Leaderboard adapter#157
mohammadrezakarami wants to merge 1 commit into
evaleval:mainfrom
mohammadrezakarami:codex/vectara-hallucination-adapter

Conversation

@mohammadrezakarami

Copy link
Copy Markdown

Add Vectara Hallucination Leaderboard adapter

Summary

Adds a one-off utility adapter for parsing selected Vectara Hallucination
Leaderboard result files from the public vectara/results Hugging Face dataset
into the Every Eval Ever schema.

This is the companion code for the datastore submission:

https://huggingface.co/datasets/evaleval/EEE_datastore/discussions/144

Details

  • Adds utils/vectara_hallucination_leaderboard/adapter.py
  • Pins source result fetching to commit
    7c104699e98ade53dd719f79ae9f7eb281c8107d
  • Keeps the private/proprietary underlying evaluation dataset separate from the
    public structured result source (vectara/results)
  • Records inference platforms for the selected models:
    vertex_ai, azure, and dashscope
  • Includes offline source row snapshots for deterministic smoke tests
  • Adds tests for schema-valid export, scores, platforms, and source-data
    semantics

Validation

PYTHONPATH=/private/tmp/eee_code_pr_deps python3 -m pytest tests/test_vectara_hallucination_adapter.py
# 4 passed

PYTHONPATH=/private/tmp/eee_code_pr_deps python3 -m every_eval_ever validate /private/tmp/eee-vectara-code-pr-smoke
# All 3 file(s) passed validation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant