Description
Add OneIG benchmarks including alignment benchmarks for anime, human, and object generation, plus text rendering evaluation.
Details
OneIG Alignment
- Dataset:
OneIG-Bench/OneIG-Bench + GitHub JSON for questions
- Subcategories: anime_alignment, human_alignment, object_alignment
- Collate:
prompt_with_auxiliaries_collate
OneIG Text Rendering
- Source: GitHub CSV from
OneIG-Bench/OneIG-Benchmark repo
- Collate:
prompt_with_auxiliaries_collate
Implementation
- Add
setup_oneig_alignment_dataset in src/pruna/data/datasets/prompt.py
- Add
setup_oneig_text_rendering_dataset in src/pruna/data/datasets/prompt.py
- Support
category param for filtering (Anime_Stylization, Portrait, General_Object)
- Register in
base_datasets
- Add
BenchmarkInfo entries with appropriate metrics
- Auxiliaries should include
questions, dependencies, category, text_content fields
- Add tests
Acceptance Criteria
OneIG Alignment
OneIG Text Rendering