MCPServer CRs applied at runtime stay Disconnected: reconcileCreate calls StartService without registering the service

[paurosello]

Repro

1. muster serve is running with N MCPServer CRs already in the namespace; processServiceClassRequirements at boot has registered each in the orchestrator's service registry.
2. Apply a brand-new MCPServer CR via kubectl apply -f new.yaml.
3. Observe the reconciler log lines:

   INFO  MCPServerReconciler  Creating MCPServer service: new-mcp
   WARN  ReconcileManager     Reconciliation failed for MCPServer/new-mcp:
                              failed to start service: service new-mcp not found
   …repeated until…
   ERROR ReconcileManager     Max retries exceeded for MCPServer/new-mcp
   

4. CR status.state settles at Disconnected (or Failed if it eventually attempts a dial and fails). The MCPServer is not usable until the muster Deployment is rolled.

Workaround: kubectl rollout restart deploy/muster. After restart, processServiceClassRequirements enumerates all MCPServer CRs and registers each, including the new one.

Root cause

internal/reconciler/mcpserver_reconciler.go:reconcileCreate only calls r.orchestratorAPI.StartService(req.Name):

func (r *MCPServerReconciler) reconcileCreate(ctx context.Context, req ReconcileRequest, info *api.MCPServerInfo) ReconcileResult {
    …
    if err := r.orchestratorAPI.StartService(req.Name); err != nil {
        …
    }
    …
}

Orchestrator.StartService (internal/orchestrator/orchestrator.go) looks the service up in o.registry and returns service %s not found if missing. The registration call that would satisfy this — createMCPServerService — is invoked only from processServiceClassRequirements, which runs once at orchestrator boot.

So new CRs created post-boot never enter the registry. The reconciler retries StartService against a name nothing has ever registered, exhausts retries, and the CR is wedged.

Suggested fix

reconcileCreate (or the path it delegates to) should call something equivalent to createMCPServerService(ctx, info) to register the service before — or instead of — StartService(name). Or, more cleanly: expose a single Orchestrator.RegisterAndStartMCPServer(info) and have both processServiceClassRequirements and reconcileCreate go through it.

Out of scope for the workaround: the reconciler also has no symmetric path for service removal on CR delete (internal/reconciler/mcpserver_reconciler.go notes // If service not found, it's already stopped on the delete path — that branch assumes pre-registration too).

Affected versions

Reproduces on v0.1.174 (currently deployed on graveler) and on main (cabae48). No relevant commits between the tag and main per git log v0.1.174..main -- internal/reconciler/mcpserver_reconciler.go internal/orchestrator/orchestrator.go (empty).