Problem
When an MCPServer CRD is applied after muster serve has started, the new server is never created/registered with the orchestrator, so it stays inactive until the muster pod is restarted. The reconciler logs service <name> not found and requeues indefinitely.
This was found while reproducing the Kubernetes-events work (see PR #888): driving the runtime event paths required a kubectl rollout restart of the muster deployment because a freshly-applied MCPServer CRD did not come up on its own.
Root cause
Orchestrator-managed MCPServer services are only ever created and registered at boot:
Orchestrator.processAutoStartMCPServers (runs in Start()) iterates the MCPServer definitions and calls createMCPServerService, which builds the service, registry.Register(...)s it, and starts it.
createMCPServerService is only called from processAutoStartMCPServers — there is no runtime path that creates a service for a definition that appeared after boot.
The reconcile create path does not create the service, it only tries to start an already-registered one:
internal/reconciler/mcpserver_reconciler.go reconcileCreate calls r.orchestratorAPI.StartService(req.Name).
Orchestrator.StartService (internal/orchestrator/orchestrator.go) returns fmt.Errorf("service %s not found", name) when the name isn't in the registry.
So for a CRD applied at runtime: definition is loaded by the MusterClient/manager, the reconciler fires StartService, the orchestrator has no such service registered, returns "not found", the reconciler requeues — and it never resolves because nothing creates the service except the boot-time auto-start loop.
Reproduction
muster serve (Kubernetes mode) with no evidence-* MCPServer defined.
kubectl apply a new MCPServer CRD (autoStart: true) into the muster namespace.
- Observe the reconciler logs
failed to start service: service <name> not found and the server never becomes active / no runtime events are emitted.
kubectl rollout restart deploy/muster — after the restart the boot-time auto-start loop picks up the definition and the server comes up.
Expected behavior
A runtime-applied MCPServer CRD (with autoStart: true) should be created, registered, and started by the reconciler without requiring a pod restart — the same way it would be if it had existed at boot.
Implementation pointers
- The orchestrator already has the create+register+start logic in
createMCPServerService; the reconcile create path just can't reach it.
- Options:
- Expose a
CreateService / EnsureService(name) on the OrchestratorAPI (internal/api/orchestrator.go + internal/orchestrator/api_adapter.go) that the reconciler's reconcileCreate calls instead of (or before) StartService; it would build the service from the MCPServerManager definition and register it if not already present.
- Or make
Orchestrator.StartService lazily create+register the service from the manager's definition when the name is not found, instead of erroring.
- Keep the
AutoStart=false short-circuit already present in reconcileCreate.
- Mind the existing auth-required hook wiring (
WithAuthRequiredHook / handleAuthRequiredServer) and the state-change callback so SSO/pending-auth servers still register correctly.
Acceptance criteria
- Applying an
MCPServer CRD with autoStart: true at runtime brings the server up (active/connected, or auth_required for SSO servers) without a pod restart.
- No
service <name> not found requeue loop for runtime-applied definitions.
- A BDD scenario (or unit test on the reconciler create path) covers runtime creation of a previously-unknown MCPServer.
- Existing boot-time auto-start behavior and update/delete reconcile paths are unchanged.
Problem
When an
MCPServerCRD is applied aftermuster servehas started, the new server is never created/registered with the orchestrator, so it stays inactive until the muster pod is restarted. The reconciler logsservice <name> not foundand requeues indefinitely.This was found while reproducing the Kubernetes-events work (see PR #888): driving the runtime event paths required a
kubectl rollout restartof the muster deployment because a freshly-applied MCPServer CRD did not come up on its own.Root cause
Orchestrator-managed MCPServer services are only ever created and registered at boot:
Orchestrator.processAutoStartMCPServers(runs inStart()) iterates the MCPServer definitions and callscreateMCPServerService, which builds the service,registry.Register(...)s it, and starts it.createMCPServerServiceis only called fromprocessAutoStartMCPServers— there is no runtime path that creates a service for a definition that appeared after boot.The reconcile create path does not create the service, it only tries to start an already-registered one:
internal/reconciler/mcpserver_reconciler.goreconcileCreatecallsr.orchestratorAPI.StartService(req.Name).Orchestrator.StartService(internal/orchestrator/orchestrator.go) returnsfmt.Errorf("service %s not found", name)when the name isn't in the registry.So for a CRD applied at runtime: definition is loaded by the MusterClient/manager, the reconciler fires
StartService, the orchestrator has no such service registered, returns "not found", the reconciler requeues — and it never resolves because nothing creates the service except the boot-time auto-start loop.Reproduction
muster serve(Kubernetes mode) with noevidence-*MCPServer defined.kubectl applya newMCPServerCRD (autoStart: true) into the muster namespace.failed to start service: service <name> not foundand the server never becomes active / no runtime events are emitted.kubectl rollout restart deploy/muster— after the restart the boot-time auto-start loop picks up the definition and the server comes up.Expected behavior
A runtime-applied
MCPServerCRD (withautoStart: true) should be created, registered, and started by the reconciler without requiring a pod restart — the same way it would be if it had existed at boot.Implementation pointers
createMCPServerService; the reconcile create path just can't reach it.CreateService/EnsureService(name)on theOrchestratorAPI(internal/api/orchestrator.go+internal/orchestrator/api_adapter.go) that the reconciler'sreconcileCreatecalls instead of (or before)StartService; it would build the service from the MCPServerManager definition and register it if not already present.Orchestrator.StartServicelazily create+register the service from the manager's definition when the name is not found, instead of erroring.AutoStart=falseshort-circuit already present inreconcileCreate.WithAuthRequiredHook/handleAuthRequiredServer) and the state-change callback so SSO/pending-auth servers still register correctly.Acceptance criteria
MCPServerCRD withautoStart: trueat runtime brings the server up (active/connected, or auth_required for SSO servers) without a pod restart.service <name> not foundrequeue loop for runtime-applied definitions.