Skip to content

feat: pass through all backend-supported sampling/serving parameters #151

Description

@ZuyiZhou

Problem

Raven forwards only a fixed subset of sampling parameters to the serving backend. Anything outside that subset is silently dropped, even when the backend supports it. Two incidents where this bit us:

  1. repetition_penalty — sglang supports it, but there is no way to get it through Raven's model config. We had to interpose a local proxy shim on the request path just to set one sampling knob.
  2. enable_thinking: false (Qwen3-style chat-template switch, normally passed via extra_body / chat_template_kwargs) — unreachable through Raven config; required patching the package.

Both are legitimate, backend-documented parameters; needing a fork for each new knob doesn't scale.

Proposal

A passthrough dict on the model/backend config, forwarded verbatim into the request payload (merged into the OpenAI-compatible body / extra_body):

model:
  sampling_extra:            # merged into sampling params
    repetition_penalty: 1.05
  request_extra:             # merged into extra_body
    chat_template_kwargs: { enable_thinking: false }

Exact shape is bikesheddable — the ask is: anything the backend accepts should be reachable from config without a fork.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions