Skip to content

MCPServer CRDs applied at runtime are not registered with the orchestrator until a pod restart #889

Description

@teemow

Problem

When an MCPServer CRD is applied after muster serve has started, the new server is never created/registered with the orchestrator, so it stays inactive until the muster pod is restarted. The reconciler logs service <name> not found and requeues indefinitely.

This was found while reproducing the Kubernetes-events work (see PR #888): driving the runtime event paths required a kubectl rollout restart of the muster deployment because a freshly-applied MCPServer CRD did not come up on its own.

Root cause

Orchestrator-managed MCPServer services are only ever created and registered at boot:

  • Orchestrator.processAutoStartMCPServers (runs in Start()) iterates the MCPServer definitions and calls createMCPServerService, which builds the service, registry.Register(...)s it, and starts it.
  • createMCPServerService is only called from processAutoStartMCPServers — there is no runtime path that creates a service for a definition that appeared after boot.

The reconcile create path does not create the service, it only tries to start an already-registered one:

  • internal/reconciler/mcpserver_reconciler.go reconcileCreate calls r.orchestratorAPI.StartService(req.Name).
  • Orchestrator.StartService (internal/orchestrator/orchestrator.go) returns fmt.Errorf("service %s not found", name) when the name isn't in the registry.

So for a CRD applied at runtime: definition is loaded by the MusterClient/manager, the reconciler fires StartService, the orchestrator has no such service registered, returns "not found", the reconciler requeues — and it never resolves because nothing creates the service except the boot-time auto-start loop.

Reproduction

  1. muster serve (Kubernetes mode) with no evidence-* MCPServer defined.
  2. kubectl apply a new MCPServer CRD (autoStart: true) into the muster namespace.
  3. Observe the reconciler logs failed to start service: service <name> not found and the server never becomes active / no runtime events are emitted.
  4. kubectl rollout restart deploy/muster — after the restart the boot-time auto-start loop picks up the definition and the server comes up.

Expected behavior

A runtime-applied MCPServer CRD (with autoStart: true) should be created, registered, and started by the reconciler without requiring a pod restart — the same way it would be if it had existed at boot.

Implementation pointers

  • The orchestrator already has the create+register+start logic in createMCPServerService; the reconcile create path just can't reach it.
  • Options:
    • Expose a CreateService / EnsureService(name) on the OrchestratorAPI (internal/api/orchestrator.go + internal/orchestrator/api_adapter.go) that the reconciler's reconcileCreate calls instead of (or before) StartService; it would build the service from the MCPServerManager definition and register it if not already present.
    • Or make Orchestrator.StartService lazily create+register the service from the manager's definition when the name is not found, instead of erroring.
  • Keep the AutoStart=false short-circuit already present in reconcileCreate.
  • Mind the existing auth-required hook wiring (WithAuthRequiredHook / handleAuthRequiredServer) and the state-change callback so SSO/pending-auth servers still register correctly.

Acceptance criteria

  • Applying an MCPServer CRD with autoStart: true at runtime brings the server up (active/connected, or auth_required for SSO servers) without a pod restart.
  • No service <name> not found requeue loop for runtime-applied definitions.
  • A BDD scenario (or unit test on the reconciler create path) covers runtime creation of a previously-unknown MCPServer.
  • Existing boot-time auto-start behavior and update/delete reconcile paths are unchanged.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions