Skip to content

Evaluate free open-source models via OpenRouter #308

Description

@ndoschek

Make open-weight models hosted on OpenRouter usable in Theia AI. OpenRouter exposes an
OpenAI-compatible API, so it works today through the existing OpenAI custom endpoint preference
(ai-features.openAiCustom.customOpenAiModels) with no code changes. This sub-issue verifies that
path, tests the candidate models, and documents the setup. Less urgent than Hugging Face, since we
do not have an OpenRouter account yet.

Setup

  1. Create an OpenRouter account (https://openrouter.ai/) and get an API key. Give it an expiration
    date and use it only for testing. Do not use sensitive data in the test cases.
  2. Add an entry to the ai-features.openAiCustom.customOpenAiModels preference:
"ai-features.openAiCustom.customOpenAiModels": [
  {
    "model": "z-ai/glm-5.2",
    "url": "https://openrouter.ai/api/v1",
    "id": "openrouter-glm",
    "apiKey": "sk-or-...",
    "enableStreaming": true
  }
]
  1. Configure an agent to use this model (by its id), then start a chat.

What to test

Look up model ids on the OpenRouter models list (https://openrouter.ai/models). Test the candidate
models and record for each:

  • Does a basic chat response work?
  • Does tool calling work (try with Coder on a prompt that triggers a tool)?
  • Does it need supportsStructuredOutput: false?

Candidate models (confirm exact slugs):

  • GLM (z-ai/glm-5.2)
  • Gemma 4 (google/gemma-4-31b-it)
  • A recent Qwen coding model (e.g. qwen/qwen3-coder, confirm the latest)
  • A Kimi model (moonshotai/kimi-k2)

Deliverables

  • A working OpenRouter configuration in Theia (the customOpenAiModels entry).
  • An assessment of which models work for which agents, covering chat, tool calling, and structured
    output.
  • Setup documentation so users can reproduce it.
  • Findings reported in the parent issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions