[codex] Tighten explicit scan target and harness resolution

[teilomillet]

What changed

This PR tightens the explicit-target scan path and harness hint resolution in ordeal.

• exact explicit selectors now resolve directly instead of forcing whole-module discovery first
• mined state_factory hints are more conservative and no longer auto-promote pytest fixtures that require injected arguments
• read-only metadata hooks are no longer dry-run as executable harness config during --list-targets
• zero-arg wrapped method targets now remain runnable instead of being skipped for missing inferred strategies
• added regression coverage for exact target selection, state-factory hint filtering, metadata-only config hooks, and zero-arg stateful wrappers

Why

Focused scans of explicit verifiers targets were still paying full-module discovery cost, and weak mined state-factory hints were overriding configured state setup with bad pytest fixtures. The follow-up rerun also exposed two adjacent engine issues in read-only metadata listing and zero-arg wrapper strategy inference.

Impact

• explicit high-risk method scans return much faster and avoid unrelated module discovery noise
• configured object runtime state is preserved for --list-targets and targeted scan probes
• the verifiers RLM explicit-target probes now resolve as runnable instead of blocking on false harness issues

Validation

Local checks run:

• uv lock --check
• uv sync --locked --extra dev
• uv run ruff check --fix .
• uv run ruff format .
• targeted pytest replays for tests/test_auto.py, tests/test_cli.py, tests/test_cli_agent_json.py, and tests/test_state.py
• uv run ordeal benchmark --perf-contract ordeal.perf.toml --tier pr --check --output-json .artifacts/perf-contract-pr.json
• uv sync --locked --extra docs
• uv run mkdocs build --strict
• uv run --with build python -m build

Additional targeted external verification:

• reran ordeal cli scan ... --list-targets --json against the verifiers RLM targets using the local checkout on PYTHONPATH
• RLMEnv._setup_interception_and_register and RLMExecutor.create_rollout_dirs both now report runnable with configured state factories and no dry-run error

Summary by Sourcery

Tighten explicit target resolution and harness handling for ordeal scans and CLI listing.

Bug Fixes:

• Bypass full-module discovery when all requested targets are exact selectors, resolving them directly within the module.
• Preserve configured metadata-only factory and state-factory hooks as read-only during --list-targets so they are not executed during harness verification.
• Ensure zero-argument stateful wrapper callables are treated as runnable instead of being skipped due to missing inferred strategies.
• Make mined state_factory harness hints more conservative, avoiding pytest fixtures that require injected arguments and reducing overconfident signal scoring so configured state setup is not overridden.

Enhancements:

• Improve harness hint mining by tokenizing names/docs for more precise target and state matching and by recognizing additional state-related naming patterns.
• Introduce helper utilities to analyze callable signatures, detect metadata-only hooks, and normalize target selectors for more robust internal resolution logic.

Tests:

• Add regression tests for exact explicit target selection bypassing discovery, conservative stateful fixture hinting, metadata-only config hooks in CLI target listing, and zero-arg stateful wrappers being marked runnable.
• Add coverage to ensure _infer_strategies returns an empty mapping for zero-argument callables rather than None.

[sourcery-ai]

Reviewer's Guide

Tightens ordeal’s explicit-target selection, harness hint mining, and CLI metadata-only behavior so that exact scan targets are resolved directly, weak state_factory fixtures are filtered out, metadata-only hooks remain read-only during --list-targets, and zero-arg stateful wrappers stay runnable; adds focused tests around these behaviors.

Sequence diagram for explicit target resolution with exact selectors

sequenceDiagram
    actor User
    participant CLI as CLI_scan_command
    participant Auto as auto_selected_public_functions
    participant Local as auto_resolve_local_target
    participant Explicit as auto_resolve_explicit_target

    User->>CLI: run ordeal scan --targets exact_selector
    CLI->>Auto: _selected_public_functions(mod, targets)

    Auto->>Auto: normalize targets
    Auto->>Auto: check _is_exact_target_selector for each
    alt all selectors exact
        loop for each target
            alt target contains_colon
                Auto->>Explicit: _resolve_explicit_target(module_target, harness_maps)
                Explicit-->>Auto: name, func
            else local selector
                Auto->>Local: _resolve_local_target(mod, target, harness_maps)
                Local->>Explicit: _resolve_explicit_target(mod_name_colon_selector, harness_maps)
                Explicit-->>Local: name, func
                Local-->>Auto: name, func
            end
            Auto->>Auto: deduplicate by name
        end
        Auto-->>CLI: selected[name, func]
    else non_exact_selector_present
        Auto->>Auto: discovered = _get_public_functions(mod, harness_maps)
        Auto->>Auto: filter discovered by glob targets
        Auto-->>CLI: selected[name, func]
    end

    CLI-->>User: run scan on selected callables

Loading

Sequence diagram for metadata-only hooks during --list-targets

sequenceDiagram
    actor User
    participant CLI as cli_list_targets
    participant Placeholder as cli_placeholder
    participant Auto as auto_verify_auto_object_runtime

    User->>CLI: run ordeal scan --list-targets

    CLI->>Placeholder: _placeholder(kind)
    Placeholder-->>CLI: function with __ordeal_metadata_only__ True

    CLI->>Auto: _verify_auto_object_runtime(factory, setup, scenarios, state_factory, sources)

    Auto->>Auto: check any mined sources
    alt no mined_sources
        Auto-->>CLI: True, None
    else mined_sources_present
        Auto->>Auto: _is_metadata_only_hook on configured factory, setup, scenarios, state_factory
        alt any configured hook is_metadata_only_hook
            Auto-->>CLI: True, None (skip dry-run)
        else real configured hooks
            Auto->>Auto: call factory and setup for dry-run
            Auto-->>CLI: result, error_message_or_None
        end
    end

    CLI-->>User: print targets metadata without mutating runtime state

Loading

File-Level Changes

Change	Details	Files
Resolve exact explicit scan targets directly without full-module discovery.	Introduce helpers to detect exact (non-glob) selectors and resolve local targets relative to a module. Short‑circuit _selected_public_functions to use explicit resolution when all targets are exact selectors, bypassing _get_public_functions. Normalize targets once and enforce that module-qualified targets belong to the current module, deduplicating resolved callables.	ordeal/auto.py tests/test_auto.py
Make mined state_factory hints more conservative and ignore unsuitable pytest fixtures.	Tokenize names/docs/return annotations for coarser but safer harness matching via _searchable_tokens. Require state_factory candidates to support zero/optional instance calls and focus on mapping returns and state-related tokens. Remove strong reliance on returns_target_instance/state_compatible/doc state hints and downgrade scoring/signals so weak hints do not override configured state. Add regression test ensuring pytest fixtures that require injected arguments are not chosen as state_factory while plain helper factories are. Ensure _infer_strategies returns an empty mapping instead of None for zero-arg callables to keep them runnable.	ordeal/auto.py tests/test_auto.py
Treat metadata-only hooks as read-only during CLI listing and harness verification.	Mark _metadata_only_hook placeholders with a ordeal_metadata_only flag in cli._metadata_only_hook. Have _verify_auto_object_runtime treat configured hooks with the metadata-only flag as automatically valid, skipping dry-run execution for factories, setups, scenarios, and state_factories. Add CLI test verifying --list-targets keeps metadata-only hooks configured, verified, and runnable without executing them.	ordeal/cli.py ordeal/auto.py tests/test_cli.py
Keep zero-argument stateful wrappers runnable instead of being skipped over missing inferred strategies.	Adjust _infer_strategies to return the (possibly empty) strategies mapping rather than None, so zero-arg callables are not treated as unsupported. Add CLI test that a zero-arg stateful Env.rollout wrapper reports runnable with configured factory and state_factory.	ordeal/auto.py tests/test_cli.py tests/test_auto.py
Minor refactors and formatting cleanups in catalog/annotation code and package init.	Simplify list comprehensions for examples and call_pattern handling in CLI catalog printing and init catalog annotation. Remove stray blank lines and keep style consistent with existing codebase.	ordeal/cli.py ordeal/__init__.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.