fix(evaluation): replace bare raises with proper exceptions and add text_generation_quality request#560
Open
zamal-db wants to merge 1 commit intoPrunaAI:mainfrom
Conversation
…ext_generation_quality request - Fix 4 bare `raise` statements that crash with `RuntimeError: No active exception to re-raise` outside except blocks: - metric_dino_score.py: raise -> raise ValueError - metric_memory.py: raise -> raise RuntimeError - metric_sharpness.py: 2x raise -> raise ValueError - Fix typo in registry.py: 'dos not' -> 'does not' - Add 'text_generation_quality' named evaluation request to Task (returns perplexity metric) - Add tests for new named request and invalid request error handling
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
I was running
SharpnessMetricon outputs from a custom diffusion pipeline and got hit withRuntimeError: No active exception to reraise. Took me a while to figure out what was going on since the actual issue was a tensor shape mismatch, but the bareraiseon this line was masking the real error. The descriptive message was being logged viapruna_logger.error()right above it, but theraiseitself had no exception attached.Checked the rest of the evaluation metrics and found 3 more of the same pattern across 2 other files. Fixed all 4, also caught a small typo in registry.py while I was in there.
Separately, I noticed
Taskonly supportsimage_generation_qualityas a named request. Addedtext_generation_qualitysince perplexity is already available throughTorchMetricWrapperand it felt like a natural complement.Changes
Bug fixes (4 bare raises):
metric_sharpness.py: 2x bareraisereplaced withraise ValueError(...)for wrong tensor dimensions and unsupported channel countmetric_dino_score.py: bareraisereplaced withraise ValueError(...)for unsupported devicemetric_memory.py: bareraisereplaced withraise RuntimeError(...)for multi-GPU without device mapTypo fix:
registry.py:"dos not inherit"fixed to"does not inherit"Feature:
text_generation_qualitynamed evaluation request toTask, returning a perplexity metric to complement the existingimage_generation_qualityrequestRelated Issue
N/A
Type of Change
How Has This Been Tested?
RuntimeError: No active exception to reraisebefore the fix, and raise the correct typed exception with descriptive message aftertest_task_text_generation_quality_requestto verify the named request returns a perplexityTorchMetricWrappertest_task_invalid_named_requestto verify unknown requests raiseValueErrorruff checkon all modified source files, all passChecklist
Additional Notes
None