Skip to content

README.md

Latest commit

 

History

History
27 lines (21 loc) · 978 Bytes

README.md

File metadata and controls

27 lines (21 loc) · 978 Bytes

ThrowBench

This repository contains the data for ThrowBench. Data is given in JSONL format. Records have the following fields:

  • bug_id: The RunBugRun bug id
  • code: Program code
  • exception_type: The name of the exception thrown, or no_exception if none thrown. This is the target label.
  • exception_message: The full message of the exception thrown`
  • language: The langugue, either c_sharp, java, python or ruby
  • input: The program input that triggers the exception
  • inputs: Other triggering inputs (input was randomly selected from this list)
  • locs: Program length (in lines of code)

Model Outputs

Model outputs can be found in the results directory. Files have the following fields:

  • model: Model name
  • predicted_exception: Answer given by model
  • actual_exception: Ground-truth answer
  • bug_id: See above
  • output: Full model output
  • language: See above

Scripts

The evaluation script can be found in run.py