Skip to content

fix(evaluation): replace bare raises with proper exceptions and add text_generation_quality request#560

Open
zamal-db wants to merge 1 commit intoPrunaAI:mainfrom
zamal-db:fix/evaluation-bare-raise-and-text-gen-quality
Open

fix(evaluation): replace bare raises with proper exceptions and add text_generation_quality request#560
zamal-db wants to merge 1 commit intoPrunaAI:mainfrom
zamal-db:fix/evaluation-bare-raise-and-text-gen-quality

Conversation

@zamal-db
Copy link

@zamal-db zamal-db commented Feb 28, 2026

Description

I was running SharpnessMetric on outputs from a custom diffusion pipeline and got hit with RuntimeError: No active exception to reraise. Took me a while to figure out what was going on since the actual issue was a tensor shape mismatch, but the bare raise on this line was masking the real error. The descriptive message was being logged via pruna_logger.error() right above it, but the raise itself had no exception attached.

Checked the rest of the evaluation metrics and found 3 more of the same pattern across 2 other files. Fixed all 4, also caught a small typo in registry.py while I was in there.

Separately, I noticed Task only supports image_generation_quality as a named request. Added text_generation_quality since perplexity is already available through TorchMetricWrapper and it felt like a natural complement.

Changes

Bug fixes (4 bare raises):

  • metric_sharpness.py: 2x bare raise replaced with raise ValueError(...) for wrong tensor dimensions and unsupported channel count
  • metric_dino_score.py: bare raise replaced with raise ValueError(...) for unsupported device
  • metric_memory.py: bare raise replaced with raise RuntimeError(...) for multi-GPU without device map

Typo fix:

  • registry.py: "dos not inherit" fixed to "does not inherit"

Feature:

  • Add text_generation_quality named evaluation request to Task, returning a perplexity metric to complement the existing image_generation_quality request

Related Issue

N/A

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • Verified all 4 bare-raise paths produce RuntimeError: No active exception to reraise before the fix, and raise the correct typed exception with descriptive message after
  • Added test_task_text_generation_quality_request to verify the named request returns a perplexity TorchMetricWrapper
  • Added test_task_invalid_named_request to verify unknown requests raise ValueError
  • Ran ruff check on all modified source files, all pass

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Additional Notes

None

…ext_generation_quality request

- Fix 4 bare `raise` statements that crash with `RuntimeError: No active exception to re-raise` outside except blocks:
  - metric_dino_score.py: raise -> raise ValueError
  - metric_memory.py: raise -> raise RuntimeError
  - metric_sharpness.py: 2x raise -> raise ValueError
- Fix typo in registry.py: 'dos not' -> 'does not'
- Add 'text_generation_quality' named evaluation request to Task (returns perplexity metric)
- Add tests for new named request and invalid request error handling
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant