Hi — I maintain EvalPort, an open, versioned JSON spec for LLM evaluation datasets (test suites, test cases, graders, result sets), with JSON Schemas and Python/TypeScript SDKs. I noticed CONTRIBUTING.md doesn't lay out a specific proposal process yet, so an issue seemed like the right first step rather than a surprise PR.
Context, for what it's worth: I've also been working with a maintainer on openai/openai-python#3619, which adds native to_openeval()/from_openeval() helpers to openai.types.evals for converting Chat Completion messages to/from the OpenEval item format — that PR is still open and under active review, not merged, so I'm not claiming any shipped integration there. This cookbook issue is a separate, much lighter-weight ask: just a documentation/example contribution, not tied to that PR's outcome.
What prompted this: examples/evaluation/ already has Getting_Started_with_OpenAI_Evals.ipynb and moving-from-openai-evals-to-promptfoo.md — both about getting eval data in and out of specific tools. That's the gap EvalPort targets: a suite is plain JSON (id, input, expected_output, graders), so the same file can feed the OpenAI Evals API, promptfoo, or a custom harness without a bespoke converter each time.
I'd propose a short new notebook or .md in examples/evaluation/ — something like "Running a portable EvalPort suite against the Evals API" — showing a suite loaded, converted to Evals API data_source format, run, and the results written back out as an EvalPort result set. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
Entirely understand if cookbook content review bandwidth doesn't allow for this right now — no pressure, and thanks for maintaining such a heavily-used resource.
Hi — I maintain EvalPort, an open, versioned JSON spec for LLM evaluation datasets (test suites, test cases, graders, result sets), with JSON Schemas and Python/TypeScript SDKs. I noticed CONTRIBUTING.md doesn't lay out a specific proposal process yet, so an issue seemed like the right first step rather than a surprise PR.
Context, for what it's worth: I've also been working with a maintainer on
openai/openai-python#3619, which adds nativeto_openeval()/from_openeval()helpers toopenai.types.evalsfor converting Chat Completion messages to/from the OpenEval item format — that PR is still open and under active review, not merged, so I'm not claiming any shipped integration there. This cookbook issue is a separate, much lighter-weight ask: just a documentation/example contribution, not tied to that PR's outcome.What prompted this:
examples/evaluation/already hasGetting_Started_with_OpenAI_Evals.ipynbandmoving-from-openai-evals-to-promptfoo.md— both about getting eval data in and out of specific tools. That's the gap EvalPort targets: a suite is plain JSON (id,input,expected_output,graders), so the same file can feed the OpenAI Evals API, promptfoo, or a custom harness without a bespoke converter each time.I'd propose a short new notebook or
.mdinexamples/evaluation/— something like "Running a portable EvalPort suite against the Evals API" — showing a suite loaded, converted to Evals APIdata_sourceformat, run, and the results written back out as an EvalPort result set. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.mdEntirely understand if cookbook content review bandwidth doesn't allow for this right now — no pressure, and thanks for maintaining such a heavily-used resource.