Skip to content

EVAL-USAGE.md

Latest commit

 

History

History
399 lines (293 loc) · 14.6 KB

EVAL-USAGE.md

File metadata and controls

399 lines (293 loc) · 14.6 KB
Raw

Evaluate a Skill with comet eval

This document explains, from the user's point of view, how to evaluate a Skill in the current release. In normal usage, you do not need to understand pytest, task registry details, profiles, treatments, or Docker internals. The user-facing entry point is comet eval.

The shortest path: evaluate your own local Skill

comet eval is a standalone Skill evaluator. You do not need to run /comet-any first or create comet/eval.yaml before trying it:

# POSIX
comet eval ./my-skill --collect
comet eval ./my-skill --html
comet eval ./my-skill --quick --html
# Windows PowerShell
comet eval .\my-skill --collect
comet eval .\my-skill --html
comet eval .\my-skill --quick --html
  • --collect performs static discovery and configuration checks only. It does not start an Agent, Docker, plugins, credentials, or network requests.
  • Without a manifest, a normal run reads the Skill, generates 2–4 deterministic tasks, freezes them, and caches them automatically.
  • --quick uses the fixed generic-skill-smoke task for a low-cost smoke test.

Reports and run state are stored under the Skill directory, or under the project passed with --project:

.comet/eval/
├── generated/   # generated tasks
├── cache/       # uv and plugin caches
├── locks/       # concurrency locks
└── runs/        # summaries, events, raw data, reports, artifacts

A Skill directory, a direct SKILL.md, or a direct comet/eval.yaml can be the target. When you pass a Skill directory, Eval automatically discovers <skill-root>/comet/eval.yaml or .yml. When none exists, it synthesizes a basic manifest in memory without rewriting your Skill:

comet eval ./my-skill/SKILL.md --html
comet eval ./my-skill/comet/eval.yaml --collect

Main Agent and independent LLM-as-Judge

The subject execution and Judge use separate configuration. Set the main Agent, model, and API URL in the manifest or override them from the CLI:

execution:
  agent: codex
  model: subject-model
  baseUrl: https://subject.example/v1
judge:
  agent: claude-code
  model: judge-model
  baseUrl: https://judge.example/v1
comet eval .\my-skill `
  --agent codex --model subject-model --base-url https://subject.example/v1 `
  --judge-agent claude-code --judge-model judge-model `
  --judge-base-url https://judge.example/v1
comet eval ./my-skill \
  --agent codex --model subject-model --base-url https://subject.example/v1 \
  --judge-agent claude-code --judge-model judge-model \
  --judge-base-url https://judge.example/v1

CLI values override the manifest. If only judge.model is configured, the Judge Agent inherits the main Agent, but the Judge model, API URL, and credentials remain independent. Judge credentials use only BENCH_JUDGE_API_KEY / BENCH_JUDGE_AUTH_TOKEN; they never inherit main-Agent credentials. Custom Agents must be explicitly installed in the user adapter registry and are selected through the same --agent / --judge-agent interface.

/comet-any is optional

/comet-any can still generate a compatible comet/eval.yaml; pass that manifest to comet eval when you want to use the generated Skill workflow. It is not a prerequisite for standalone evaluation. comet eval does not publish; if you use the /comet-any Bundle publishing flow, evaluation results can also be recorded as publish-readiness evidence.

The composed Skill Bundle capability set remains skills/scripts/rules/hooks/references, with scripts/rules/hooks as the required control plane. hooks/*.yaml become active after comet publish distribute compiles them for the target platform.

Evaluate a Skill generated by /comet-any (optional)

After /comet-any generates a Skill, first look for:

generated-skill/
  comet/
    eval.yaml

Then run these two commands:

comet eval ./generated-skill/comet/eval.yaml --collect
comet eval ./generated-skill/comet/eval.yaml --html

The first command, --collect, only confirms whether tasks can be discovered. It is a low-cost precheck right after generation.

The second command, --html, performs the real evaluation and produces a browsable report. After it passes, /comet-any can use the result as publish evidence.

How eval results enter publish readiness

After /comet-any or the backend records eval results, they are merged into publish readiness. What the user needs to know is just:

  1. Results from comet eval become evidence for Publish readiness:.
  2. If current-hash eval evidence is missing, User next steps: must point to running comet eval before publish continues.

The usual sequence is:

comet eval ./generated-skill/comet/eval.yaml --collect
comet eval ./generated-skill/comet/eval.yaml --html
comet creator next <name> --json
comet publish review <name> --platform <reference-platform> --json

comet creator next prints only the single recommended user command; comet publish review must directly show Publish readiness:, User next steps:, Readiness:, Blockers:, Warnings:, and Evidence:.

Why run collect first

collect is the cheapest user-facing debugging entry point. It answers:

  • is the comet/eval.yaml path correct
  • can the eval harness read this manifest
  • can the tasks referenced by the manifest be discovered
  • are the current repo's eval dependency paths usable

It should not start a full model evaluation first, and it should not spend long-running cost before manifest and discovery work.

Choose the Local, LangSmith, or Langfuse suite

comet eval uses the local suite by default for day-to-day development and local reports. To sync runs, rubric feedback, costs, and Claude Code trajectories to LangSmith, select the suite explicitly:

comet eval ./my-skill --suite langsmith --html

Both suites reuse the same tasks, treatments, rubric, and manifest. The langsmith suite reads LANGSMITH_API_KEY, LANGSMITH_PROJECT, and LANGSMITH_TRACING, provisions the Claude Code trajectory plugin, and writes reports under the project's .comet/eval/runs/. Without --suite langsmith, enabling tracing in the environment does not make the Local runner create a LangSmith experiment.

To send core task/treatment traces and scores to Langfuse, select langfuse:

LANGFUSE_PUBLIC_KEY=pk-lf-... LANGFUSE_SECRET_KEY=sk-lf-... \
  comet eval ./my-skill --suite langfuse --html

comet eval selects the Langfuse optional extra automatically. The Langfuse suite authenticates before any Agent or Docker workload starts. A failure to write a core trace, score, summary, or final flush fails the suite; detailed trajectory upload remains best-effort. Claude Code and Codex use pinned official Langfuse plugins from an isolated .comet/eval/langfuse/plugins/ cache. Qoder and CodeBuddy use project-local Stop hooks and JSONL transcript adapters. --collect --suite langfuse does not initialize the SDK, access the network, or download plugins.

Choose the evaluation agent

Evaluations use claude-code by default. Select claude-code, codex, qoder, or codebuddy from the CLI:

comet eval ./my-skill --agent codex
comet eval ./my-skill --agent qoder
comet eval ./my-skill --agent codebuddy

You can also set the manifest default:

execution:
  agent: codex

Precedence is CLI --agent > manifest execution.agent > claude-code. Local, LangSmith, and Langfuse share this selection rule; the subject, auto-user simulator, and optional Judge use isolated sessions of the selected agent. --collect validates the agent and manifest without starting an agent, Docker, or credential checks.

What --html prints

During execution, the CLI prints a set of execution facts:

  • Eval root: which eval/ root was actually used
  • Mode: collect or run
  • Suite: local, langsmith, or langfuse
  • Target: whether the target is a manifest or local Skill directory
  • Experiment: the experiment id for this run
  • Profile: which profile the run used
  • Task: which task was run
  • Report path: where the report was written
  • Report config: the temporary report config used when --html is enabled

--html requests both markdown and HTML reports. Reports are stored in the project-local run directory regardless of the selected suite:

.comet/eval/runs/<experiment-id>/summary.html

If CLI output shows the placeholder <experiment-id>, use the Experiment value printed in the same output block to resolve the final path.

How to read the report

Users do not need to read low-level logs line by line. Start with:

  • whether evaluation passed
  • whether failures come from harness, workflow, task, or model
  • whether failures are relevant to the Skill's goal
  • whether expected artifacts are missing
  • whether the problem is path, manifest, or environment related
  • whether token / cost / duration look abnormal

comet eval surfaces failure attribution so the report groups failures under harness, workflow, task, model, and related categories. That attribution helps decide whether the next step is fixing the Skill, the eval config, or the environment.

Reports also distinguish the raw set from the analysis set. The raw set keeps every run for auditability; the analysis set is the default source for headline metrics, pass@k/pass^k, cost, charts, and the verdict. excluded usually means API timeout, rate limiting, auth/network failure, Docker/container failure, or an outer runner timeout; these runs stay visible in the report but do not affect headline metrics. flagged means a harness or task assumption looks suspicious; the run remains in the analysis set but is called out as a risk. Real Skill, workflow, model, or validator failures remain included and are not filtered away just because they lower the score.

If the report says Insufficient clean data or Inconclusive due to data quality, rerun the affected task/treatment pair or inspect the environment before treating the verdict as final.

How /comet-any uses eval results

From the user's point of view, after comet eval finishes you can hand control back to /comet-any or run comet creator next <name> to see the single recommended next step. /comet-any merges eval evidence into readiness:

  • no eval evidence: cannot publish
  • eval failed: cannot publish
  • eval evidence points to an old hash: cannot publish
  • .comet/skill-preferences.yaml changed while in strict mode: cannot publish; confirm or regenerate first
  • eval passed and hash matches: continue into review / publish decisions

Users should not edit Bundle state manually, and they should not write report paths into internal JSON by hand. /comet-any records structured evidence through the Bundle backend.

How to evaluate when you only have a local Skill directory

If you only have a local Skill directory, pass it directly as the target:

comet eval ./my-skill --collect
comet eval ./my-skill --html

For a low-cost smoke test:

comet eval ./my-skill --quick --html

This is appropriate for early validation:

  • whether the Skill directory can be read
  • whether the eval harness can inject it as a dynamic Skill
  • whether the generic smoke task runs

Current quick smoke uses:

generic-skill-smoke

This is an early smoke path, not a complete quality conclusion. Add an optional comet/eval.yaml to author reproducible tasks, or reuse the manifest generated by /comet-any.

If eval.yaml has neither evaluation.tasks nor recommendedTasks, a normal run takes a bounded Skill snapshot, generates 2–4 deterministic tasks, and caches them under .comet/eval/generated/. The cache key includes the snapshot, Agent, profile, and interaction settings, and the generated manifest records its generation metadata. --collect only reads an existing cache and never starts the task-generator Agent; use --quick when you explicitly want to bypass generation.

Projects can also author inline tasks or reference a Skill-local task package with source:

evaluation:
  tasks:
    - name: writes-summary
      prompt: Create summary.md.
      expect:
        files: [summary.md]
        contains:
          summary.md: ['# Summary']

source, workspace, and expected artifact paths must stay within the allowed package/workspace. Inline files, contains, json, and commands checks run inside the isolated Docker workspace.

When to choose manifest vs skill-path

The rule is simple:

  • for a local Skill: pass the Skill directory; Eval discovers an optional manifest automatically
  • if you only have SKILL.md: pass that file directly
  • if comet/eval.yaml exists: pass either the Skill directory or the manifest itself
  • if /comet-any generated the Skill: both its directory and manifest are compatible targets

--quick means the fixed smoke task; it does not replace authored task evaluation.

What to do when it fails

If collect fails

Check first:

  • whether the manifest path is correct
  • whether comet/eval.yaml exists
  • whether the manifest's recommended task exists
  • whether you are in the Comet repo root or passed the right --project

If run fails

Look at failure attribution in the report first:

  • harness: usually eval harness, dependency, Docker, path, or environment issues
  • workflow: usually the Skill execution flow did not behave as intended
  • task: usually task definition, verification rule, or fixture issues
  • model: usually model behavior, tool use, or unstable output

If the HTML report cannot be found

First read the CLI's Experiment and Report path. If the path still contains <experiment-id>, use the real experiment id under:

.comet/eval/runs/

comet eval and comet skill check are different

The commands serve different roles.

comet eval is the shared eval-harness entry point for evaluating a Skill package or comet/eval.yaml.

comet skill check is the local Engine Run completion check used to decide whether a run or change meets the runtime checks in comet/checks.yaml.

If your question is "Can this Skill pass product evaluation?", use:

comet eval ./my-skill --html

If your question is "Is this deterministic Skill Run missing artifacts or status?", then use:

comet skill check --change ./changes/demo --scope completion

What the user needs to remember

In practice, only remember these three points:

  1. Pass your own Skill directly to comet eval
  2. Run --collect first, then --quick or --html as needed
  3. /comet-any is an optional Skill-production flow; Eval is not the publish action itself

Recommended commands:

comet eval ./my-skill --collect
comet eval ./my-skill --html
comet creator next <name> --json