Changelog

All notable changes to this project will be documented in this file.

Unreleased

Added

spec.auth.authorizationServer.expectedIssuer: two clients of one authorization server. A pinned authorization server's issuer was both the identity the grant is filed under and the value the RFC 9207 iss parameter of the authorization response is compared against. GitHub sends iss=https://github.com/login/oauth on every response, so a GitHub App pinned under its own identity (https://github.com/apps/<slug>, to keep its grant apart from another App's) never completed its sign-in: iss mismatch: response carried "https://github.com/login/oauth", expected "https://github.com/apps/<slug>". The new optional field names the issuer identifier the server really publishes; a present iss is compared against it (a trailing slash tolerated) while issuer stays the grant key, so each App keeps its own grant, consent and core_auth_logout. It needs the pinned endpoints; with the field unset nothing changes. Carried through the CRD, the API, core_mcpserver_get and the aggregator's pin; the how-to gained "Two clients of one authorization server" and the CRD reference the pin's fields. (#1277, #1275)
muster auth login renews the stored ID token. A valid session is still reused, but the session's automatic refresh renews the access token only: the OIDC ID token in ~/.config/muster/tokens/<hash>.json kept the exp of the first sign-in, so the CLI carried an ID token expired for weeks -- signed with a key the identity provider no longer publishes -- while muster auth status said Authenticated. Now login signs in again through the browser when the stored ID token has expired (or expires within 30 s), and always with the new --force; the stored session is replaced only when the new sign-in completes. muster auth status and muster auth whoami show the ID token's expiry (ID token: expired 12 days ago (renew with: muster auth login)), and the new muster auth token [--id] prints the access token or the ID token alone on stdout for other clients -- a script calling an API that validates the person's ID token -- and refuses an expired ID token instead of handing it out. (#1268)
muster self-update --check reports the running and the latest release without installing anything and exits with status 125 when a newer one exists (devctl's convention for version check), for scripts. Every command a person runs starts with a one-line hint on stderr while a newer release is out -- a hint, never a gate: the GitHub round trip is capped at two seconds, its answer is cached for an hour under the user cache directory (~/.cache/muster/latest-release.json on Linux, ~/Library/Caches/muster/ on macOS) and a failed attempt is remembered for ten minutes, so a machine without internet is not held up. MUSTER_NO_UPDATE_CHECK=1 silences the hint; serve, standalone, agent, test, version, self-update, help and completion never print it, and dev builds never check. The same internal/update package as agentlab's, so the two CLIs behave alike.
muster test: an installation-shaped scale fixture with budgets enforced in CI. internal/testing/fixtures/scale is a committed, generated fixture -- 87 MCPServers (84 session-authenticated, in five families with an instanceArg on 28 installations, three in-house), 18 distinct capability documents shared by the family members, 282 workflows over their tools, 450 session records; every name invented, nothing from an installation -- rendered by go generate from a seeded generator, with a test that fails when the committed YAML drifts. A scenario boots from it with pre_configuration.fixture: scale and adds or overlays on top. Budgets are hard limits with headroom over the value measured on CI's medium executor, never "faster than last run", and a failing budget names the metric, the measured value and the limit. In Go (internal/aggregator, run by make test): a warm list_tools, filter_tools, describe_tool or call_tool over the fixture costs at most 100 ms (measured 1--2 ms), 2 store commands (HGETALL, HKEYS) and 0 definition reads (API-server requests) -- the catalogue is built once; list_tools' default page stays under 40 KB (measured 14 KB); the capability store holds at most 16 KB per session over 450 sessions (measured 7.2 KB: 84 references and a share of 18 shared documents). On the wire (scenario scale-fixture-budgets, on a Valkey stand-in): the default page under 40 KB, under 32 KB of capability store per session (measured 9.9 KB over 8 sessions), a warm meta-tool request costing the store a constant 16 commands at most (measured 11: the two of the catalogue plus the OAuth middleware's per-request session bookkeeping, nothing per server), and a new session's first request answered within 2 s while its fan-out connects 84 servers two of which take four seconds. New test tools test_measure_meta_tool (duration, response bytes, store commands by name per call) and test_valkey_footprint (the store's content by prefix, capability bytes per session); a new expectation json_path_max bounds numeric response fields, and a step whose expectations are not met now reports which one and how (json_path_max "valkey_commands": measured 177, budget 16) instead of "expectations not met". The scenario fails on the release that had each bug, at the budget's step: list_tools unbounded on v5.15.6 (response_bytes 61,651 against 40,960, #1193); every session storing every server's document inline on v5.19.4 (capability_bytes_per_session 105,332 against 32,768, 672 inline entries and no shared document, #1217); a listing reading every server's entry and document on its own on v5.19.10 (valkey_commands 177 against 16: 84 HGET and 87 GET per call, #1225); a new session's first request held by the fan-out on v5.19.14 (4.02 s against a 2 s max_duration, #1226). Documented in "Installation scale: the fixture and the budgets" of docs/contributing/testing/scenarios.md. (#1240)
Homebrew: the release pipeline tells the tap giantswarm/homebrew-muster about a release once its binaries are on the GitHub Release (a muster-release repository dispatch from .circleci/custom.yml); the tap's workflow verifies every binary against its Sigstore bundle before it regenerates the formula. brew install giantswarm/muster/muster follows each release again; the tap had been stuck at v0.7.5 since June.
muster test: the mock OAuth server takes the profile of a real authorization server. mock_oauth_servers[].profile: github | dex | pro bundles that server's quirks instead of one flag per bug (omit_resource_metadata, omit_token_scope, pin_authorization_server, pin_endpoints_ref, authorize_accepts_any_client, ...; the flags stay and each overrides the profile for itself). github: no RFC 8414 discovery document, no Client ID Metadata Documents and no RFC 7591 registration (a pre-registered client only), scope and expires_in absent from token responses; the MCP servers that reference it answer a bare 401 without RFC 9728 metadata, file their grants under the person and are pinned to it with explicit endpoints. dex: discovery, scope omitted, an id_token with every token, RFC 7591 registration. pro (the MCP TypeScript SDK's authorization server): discovery, registration without registration_client_uri, a client it does not know answered directly at /authorize with invalid_client, the token endpoint refusing unregistered clients, registrations held in memory. Without a profile the mock now hands out the RFC 7592 pair (registration_client_uri, registration_access_token) and answers the client read, so muster's spec-defined registration check runs against the well-behaved default and its authorization-endpoint probe against pro. test_restart_mock_oauth_server replaces a mock authorization server's process behind its port and issuer; under pro the registrations go with the old process. The scenarios that reproduce a named server's bug carry its profile and fail on the release that had the bug: a server's grant surviving the login id_token mirror on a Dex (fails on v5.12.0, #1174), a registration a pro restart forgot detected and made again (fails on v5.7.8, #1128), a pinned bare-401 GitHub-style server connectable after a muster restart with no login in the new process (fails on v5.8.3, #1150). Documented in "Authorization-server profiles" of docs/contributing/testing/oauth-testing.md. (#1239)
muster test: faults are named steps and time is a clock the scenario moves. test_redeploy_mock_server replaces a mock backend's process behind its port -- every MCP session forgotten, tools kept, no refused connection in between, the way a rolled pod takes over behind its Service; test_set_mock_server_auth: {required: true|false} flips a running mock between answering anonymously and 401-with-metadata, the rollover of a backend to an OAuth resource server (a mock with a token validator and oauth.required: false starts anonymous). test_advance_clock: {duration} moves muster serve's clock and every mock authorization server's clock forward together: the reconnect backoff of a remote MCPServer, the orchestrator's retry and health ticks and the age of the core catalogue now read a process clock (internal/clock) that the harness advances through a control socket MUSTER_TEST_CLOCK selects -- a tick that has become due fires at once, nothing waits, and production binaries keep the system time. pre_configuration.intervals: production leaves those timers on their production defaults (30 s initial backoff, 2 min cap, 30 s ticks, 5 min catalogue age) for scenarios that assert the production schedule; the default short keeps the seconds-long environment knobs. The fault steps and the clock are documented together in "Faults and time" of docs/contributing/testing/scenarios.md. Four scenarios: a backend redeployed behind the same port has its session recovered on the next call, no health probe in between (fails on v5.15.0, #999); on the production schedule a remote server whose gateway answers 504 four times waits 30 s, 1 min, 2 min and again 2 min -- never 4 -- and is connected within the cap once the gateway serves (fails on v5.9.10 with the clock, #1163); the core catalogue found older than its five-minute age is served as it is and rebuilt once in the background (fails on v5.19.13 with the clock, #1231); a forwardToken backend rolled over from an anonymous pod to a 401 pod and back is never registered globally and keeps serving the signed-in session (fails on v5.7.17, #1135). (#1238)
Documentation site. docs/ is built with MkDocs (Material) and published at https://giantswarm.github.io/muster/ by the Pages workflow; the site kept serving muster-agent.json, the muster agent's OAuth client metadata document, at its URL. A pull request that touches the documentation builds the site in strict mode, so a broken link, a page missing from the navigation or an unknown anchor fails the check. make docs-serve renders the site locally.
The CLI reference is generated from the command tree. make generate-cli-docs renders docs/reference/cli/ from the Cobra commands (hack/gen-cli-docs); make verify-cli-docs, part of make test, fails when the committed pages are stale. The hand-written pages, which documented flags that never existed (--port, --mcp-port) and a REST API, are gone.
muster test: scenarios can run their instance in Kubernetes mode. pre_configuration.mode: kubernetes starts one envtest control plane per run from the binaries KUBEBUILDER_ASSETS points at, installs the CRDs from helm/muster-crds, applies the scenario's mcp_servers and workflows as CRs into a namespace of its own and runs muster serve with kubernetes: true against it -- informers, reconciler, boot pass and CR status the way an installation has them. A TCP proxy the harness owns sits between muster and the API server: apiserver.reachable_after keeps it closed for that long after the process started (the API server late at start) and test_set_apiserver_reachable: false|true cuts and restores it mid-run. test_patch_cr applies a merge patch to a CR (spec.suspended, labels, spec.auth.authorizationServer) so the reconciler is driven by real CR updates; test_get_cr reads a CR with the status muster wrote; test_set_mcpserver_labels and test_pin_mcpserver_authorization_server update the CR in this mode. muster test --mode filesystem|kubernetes runs one definition source; without KUBEBUILDER_ASSETS the Kubernetes-mode scenarios are reported as skipped with the reason, never as passed, and make test-envtest (the test-envtest CI job) runs them against envtest. Four scenarios reproduce bugs the filesystem-mode suite could not express: a suspended server is not started by the boot pass (fails on v5.19.6, #1216), a suspended remote server is stopped once and not on every resync tick (fails on v5.19.0, #1212), the boot pass and the reconciler's first pass register a dozen CRs without an ERROR (fails on v5.19.8, #1222), and an API server unreachable at start is waited for instead of swapped for the filesystem client (fails on v5.7.18, #1143); two more cover a boot from CRs and an API server gone mid-run. (#1237)
muster test: scenarios can run their instance on Valkey storage and restart it. pre_configuration.storage: {type: valkey} starts an in-process Valkey stand-in (miniredis) per instance and points every backed store -- session auth, capabilities, OAuth tokens, state, client credentials, the OAuth server's own store -- at it, the way an installation's stores outlive a pod; start_delay makes it answer only after muster serve started, for "Valkey is late" scenarios. test_restart_instance stops and starts muster serve on the same configuration while the store and the mock servers keep running and reconnects every user client with the bearer it held, so the steps after it act as the sessions that lived through a rollout; test_stop_valkey / test_start_valkey take the store away and bring it back with its data. Three scenarios reproduce bugs that were first seen on an installation because the suite could not express them: a family tool call after a restart without a listing (fails on v5.18.2, #1204), a Valkey that answers late at start being waited for rather than replaced by memory stores (fails on v5.19.12, #1229), and a pinned bare-401 server staying connectable and its grant revocable after a restart with no login in the new process (fails on v5.8.3, #1150, #1154). (#1236)

Changed

The Go module is github.com/giantswarm/muster/v5; every import path carries the suffix (github.com/giantswarm/muster/v5/pkg/oauth). The module path had no /v5 while the releases were v5.x, and Go only considers v0 and v1 tags for such a path: go install github.com/giantswarm/muster@latest resolved to v1.12.0 from August and installed a month-old v1 binary, and go install github.com/giantswarm/muster@v5.23.2 was refused ("module contains a go.mod file, so module path must match major version"). go install github.com/giantswarm/muster/v5@latest now resolves to the newest release, and Go's own VCS stamping gives a build of a tagged commit its v5 tag instead of a pseudo-version of the last v1 tag. No repository outside this one imported the old path. The goimports -local prefix of the pre-commit hook and the contributor docs follow the module. The OpenTelemetry scope name github.com/giantswarm/muster (observability.TracerName, the otel_scope_name label, the logger scope) is an identifier dashboards filter on, not an import path, and is unchanged.
muster version and muster --version print one line, muster version v5.23.5 (commit 361cdef, built 2026-09-15T19:38:51Z), with the details the binary knows; the commit: and built: lines and their unknown placeholder are gone, and muster version still adds the aggregator's version when one is running. A go build from a checkout reports Go's own stamp -- the tag at a tag, +dirty over local edits, a pseudo-version between tags (v5.23.6-0.20260915195350-977012d023ba: after v5.23.5, before v5.23.6) -- which self-update and the hint compare as what it is; a binary without any version says dev, and self-update refuses it with a pointer to go install github.com/giantswarm/muster/v5@latest. The self-update messages follow agentlab's ("Nothing newer than v5.23.5 on GitHub (latest release v5.23.5)."; a newer release is announced with its URL instead of its full release notes), and its logic moved from cmd to internal/update.

Fixed

A remote MCPServer whose endpoint answers the initialize with a 4xx is retried. A forwardToken server registered while its backend did not serve the path yet -- a rollout in progress, the previous pod answering 404 -- failed its registration probe with server returned 4xx for initialize POST, likely a legacy SSE server, read Failed and was never re-probed: the answer was not classified as a connection failure, so no retry was scheduled, and the sessions connecting later with their own token do not touch the CR's state. The server stayed Failed (and MusterMCPServerFailed fired after 20 minutes) although the backend had been serving the path since seconds after the registration. Such an answer now enters the reconnect schedule like a refused connection or a 5xx: retried with the same backoff, the MCPServerFailed event and status.lastFailureHTTPStatus naming the status mcp-go dropped (endpoint answered HTTP 404, next retry in 30s at ...), and the server settles on its own once the path answers -- in Auth Required for a server connected per session, Connected otherwise. The state rules of a per-session server (Failed only while the endpoint does not answer, Auth Required until the first session connects, Connected from then on, a later session's failure never the server's state) are documented in the CRD reference. (#1295)
muster agent --mcp-server and the REPL wait for a slow tool. The CLI's MCP client bounded every request at 30 s, tool calls included, while a tool behind the aggregator may legitimately take longer -- an MCPServer's spec.timeout allows up to 300 s -- so a tool that answered after a minute succeeded through the aggregator and failed through the bridge with transport error: ... context deadline exceeded after exactly 30 s; muster agent --timeout was bound to nothing. A tool call now runs under a timeout of its own, five minutes by default (the aggregator's largest tool timeout), for the bridge, the REPL's call and every CLI command that calls a tool (muster call among them); the handshake, listings, resources and prompts keep their 30 s. muster agent --timeout sets the call timeout for the REPL and the bridge, and the bridge's call_tool takes an optional timeout argument (seconds) that bounds one call in its place -- for a client that knows a read takes longer; the argument is the bridge's own and never reaches the aggregator. (#1293)
A session connecting to an MCPServer no longer reconciles it. With the reconciler's duplicate cadence gone (#1285), every server was still reconciled -- and its CR status written -- once per session that connected to it: the aggregator reports each successful session connection to api.UpdateMCPServerState, which queued a reconcile unconditionally, so every new session's SSO fan-out cost one pass per server it reached, several times per ten minutes on a busy installation, while nothing changed. The handler now only hands the state to the service. A state that actually changes (Auth Required to Connected on the first session, back on the last session's loss) reaches the CRD status through the service's state-change event and the StateChangeBridge, as every other state change does; an unchanged state is logged at debug and costs nothing. TriggerReconcile is gone from the reconcile manager's API; it had no other caller. (#1290)
Filesystem mode: a status sync no longer rewrites the definition file. The status the reconciler records for an MCPServer or Workflow (state, last error, last connection) was written into the definition's own YAML file: the sync read the file, applied the status and renamed its copy over the original. A spec written by another process between that read and that rename -- an operator's editor, a GitOps sync -- was overwritten with the stale spec, and the reconciler then saw the old definition: the change never happened. In CI this was the flaky oauth-auth-config-change-resets-live-sessions (the pin of an authorization server lost to a status write of the connect it followed, so the session stayed Connected and the reset never came) and oauth-pinned-authorization-server-change-takes-effect (the same loss, surfacing as the sign-in rate limit). Status now lives under status/ at the definition's relative path (status/mcpservers/<name>.yaml), the definition files are only ever written by core_mcpserver_create/core_mcpserver_update and their workflow counterparts, a status: block left in a definition file by an earlier muster is ignored, and a status write no longer fires the change detector that watches the definition directories. (#1288)
An MCPServer in sync is left alone. In Kubernetes mode the reconciler logged Reconciling MCPServer: <name> for every server about every 14 s -- some 250 passes per server per hour, each a definition read and a status write against the API server, while nothing changed. Two periodic sources of the same length ran out of phase: every successful pass requeued itself after 30 s (the status sync from before the periodic resync existed), and the Manager's resync enqueued every server every 30 s on top. A pass that finds the server in sync now asks for nothing further -- the next one comes from a definition change, a runtime state change (StateChangeBridge) or the resync -- and the resync is a safety net for a lost event rather than a heartbeat: its default is ten minutes instead of 30 s. MUSTER_RECONCILER_RESYNC_INTERVAL still overrides it and muster test still runs instances at 2 s. DefaultStatusSyncInterval is gone. (#1285)
An OAuth server's spec.timeout reaches the client a tool call builds on a pool miss. #1284 gave the login-time client of a server with auth.type: oauth the server's timeout; the client the tool-call path builds when a session is already authenticated but holds no pooled connection -- after a restart of the aggregator, after an eviction -- was still built without it (and without spec.meta), so the first call after such a restart was cut at the default 30 s although the server declared more. Both are handed to that client too. (#1283)
An OAuth server's spec.timeout reaches the client a person's grant connects. #1282 wired the budget into the token-exchange and token-forwarding clients only; the client establishConnection builds for a server with auth.type: oauth -- the core_auth_login callback and the grant reuse on first use -- kept the zero timeout, so a tool call on such a server was cut after the default 30 s (no answer within the server's timeout of 30s) although its definition declared more. The entry's spec.timeout is read with its spec.meta and handed to the DynamicAuthClient (new WithTimeout) and to the static-bearer client alike. (#1283)
A remote server's spec.timeout governs each tool call. The value bounded the first connect, the health probe and the session-recovery handshake, while the requests themselves -- a tool call above all -- ran under the caller's context alone: a tool that legitimately blocks for two minutes behind a server declaring timeout: 180 was cut whenever the caller's connection gave up first, and muster reported the cut as a transport failure of the POST (transport error: failed to send request: Post "http://..."). Now every operation on a remote client -- each request, and a recovery handshake on the way -- runs under the server's spec.timeout, the CRD's default of 30 s when unset, so the documented range (1-300) holds end to end; a call that outlasts the budget fails with no answer within the server's timeout of 60s: context deadline exceeded rather than a transport error. The session-scoped clients of servers that need a sign-in get the same budget through their registration. (#1281)
A live session follows its MCPServer's auth configuration. When an MCPServer's spec.auth changed underneath sessions connected to it -- forwardToken replaced by a pinned authorizationServer, another issuer, expectedIssuer, grant scope or client Secret, a changed token exchange -- every live session kept its state for the server from before the change: muster auth status said Connected, a tool call failed with the backend's 401 as Tool execution failed: failed to call tool: transport error: authorization required, muster auth login --server <name> refused (already connected and does not require authentication), and nothing short of signing out of muster ended it, while the server itself read Auth Required all along. Three things changed. The aggregator puts the server back to auth_required for every live session when the changed configuration registers (session_auth_reset in the log; a restart or a retry on an unchanged configuration leaves the sessions be); the session's cached tools stay resolvable, so its next call answers with the sign-in. A tool call the session cannot make -- it is not signed in to the server, or the backend refuses its credential with a 401 during the call -- answers the auth_required challenge with the sign-in link for that server, the answer core_auth_login gives (marked as an error: the tool did not run; structuredContent.authUrl carries the link), never user not authenticated to server or a transport error; the refused connection is retired on that call, without waiting for the transport's three retries, and an SSO server's answer says to sign in to muster again. The streamable HTTP transport's bare 401 (authorization required) is now recognized as one wherever muster tests for a 401. And muster auth logout --server <name> signs the session out of that one server at the aggregator (core_auth_logout) and prints what happened to the person's grant; it used to print guidance and change nothing, and --server next to --endpoint signed out of the aggregator instead. Scenarios oauth-auth-config-change-resets-live-sessions and oauth-backend-401-answers-auth-required; test_pin_mcpserver_authorization_server drops forwardToken/tokenExchange when it pins. (#1276)
muster call -- and every other command that connects to an aggregator -- fails fast with auth_required instead of opening a browser, and a rejected token is no longer removed from the store. Without a usable token a command now exits with status 2 at once; the error starts with auth_required, names muster auth login --context <ctx> (or --endpoint <url>) and the new --login flag, which opens the browser from the command itself for a person at a terminal (--auth auto and MUSTER_AUTH_MODE=auto still do the same; the default --auth mode is now none, and a stored token is used and refreshed in every mode). A script, an agent or a CI job can never complete a browser flow, and inside a tool call the old behaviour hung until killed. The 401 handling of the commands, of auth status and of the agent's re-authentication no longer deletes the token file first: several muster processes share ~/.config/muster/tokens/, the ID token in the file serves other clients, and a sign-in replaces the file only when it completes. The store writes each token file atomically (temporary file and rename), so a process reading an endpoint's file while another refreshes it sees the old or the new token and never a truncated file it would take for a missing one; every write and removal is logged at debug with the pid and the command words of the process. auth logout removes the current context's token only, --all every token. (#1273)
Plain muster auth login renews an expired stored ID token. The renewal decision hung on the handler's status path, which reaches Authenticated only when the stored access token is valid for another minute by the CLI's own reading and otherwise probes the server; the mcp-go transport connects with that same token regardless, so a session the aggregator accepted answered Already authenticated while the decision never saw the ID token (muster auth status then said No authentication required). The decision now reads the ID token's exp from the token file alone, before the "already authenticated" answer and without a probe; a session without an ID token is renewed too. (#1268)
No Warning MCPServerRecoveryFailed for the expected 401 of an OAuth-protected server. When an MCPServer whose callers bring their own credentials (auth.forwardToken, auth.tokenExchange, or an OAuth login through muster) came up together with muster, automatic recovery restarted it as soon as it answered and reported its 401 -- the answer such a server is configured to give a token-less probe -- as a failed recovery, one Warning per OAuth-protected server on every platform install, right after the same reconciler had put the server in Auth Required. Recovery now ends there with a Normal MCPServerRecoveryAwaitingAuth event ("automatic recovery reached the server; it waits for a signed-in caller") and the server connects on the first call that carries a token, as before. A 401 from a machine identity (auth.type: sigv4), a 5xx and a refused connection stay MCPServerRecoveryFailed. (#1265)
A forEach can iterate a step result's field. items: "{{ .results.<id>.<field> }}" failed every workflow with items expression … resolved to string, expected a list: a reference deeper than one key below .input, .results or .vars was rendered as text, so only a list passed in as a workflow argument could be iterated. Template references that are a pure path now keep their Go type at any depth -- the navigator the spec.output template already uses -- so a forEach over the pods a previous step listed works, {{ .vars.pod.name }} inside the loop body is the string it names, and a numeric field referenced as a tool argument arrives as a number. Anything more than a pure path still renders to text. The returned document also showed the loop's last result on every iteration's record (the records share the body step's ID); each record now carries its own iteration's result and an iteration index. Scenario workflow-foreach-step-result-items.
go install github.com/giantswarm/muster/v5@latest builds the release it resolves to. Go refuses go install <package>@<version> for a module whose go.mod carries a replace directive, and go.mod carried one since 2026-08-28: goldmark pinned to a fixed version for the nancy scan, because golang.org/x/tools required a version OSS Index flagged (#1106). Nothing in the build reaches x/tools or goldmark anymore (go mod why -m github.com/yuin/goldmark: the main module does not need it; neither is among the packages go list -deps ./... hands to nancy), so the pin changed nothing but the go install verdict and is gone. The module graph is unchanged.
muster test: two races of the harness under --parallel 50, both seen once in CI on 2026-09-15. test_restart_mock_oauth_server listened again on the port the old server had just released, and any socket on the host could take it in between -- the mocks listen on kernel-assigned ephemeral ports, the range every outgoing connection draws from (failed to listen again on port 45841: bind: address already in use). The mock now keeps its socket bound across the restart and replaces only the server behind it, as a Service keeps its address while the pod is replaced. And a step whose call to muster serve never returns (a wait_for_state poll still pending when its budget ran out; oauth-subject-grant-refresh failed that way with an instance that kept logging and a harness that kept finishing other scenarios) is reported as stalled instead of "expectations not met", and the runner records where everything was: muster serve gets SIGQUIT so its goroutines end the instance stderr of the report, the harness's goroutines are stored as harness_goroutines, and the failure line points at both. The harness also runs the muster binary it is part of before consulting PATH: a stale go install first on PATH ran the suite against the wrong muster serve.
The release binaries report their release tag again. muster version on the v5.22.0 binary printed v1.12.1-0.20260915144925-e6c760a32b48: the CI build stamped the commit and the build time but no version, so the binary fell back to the version Go's own VCS stamping had derived -- and as the module path is github.com/giantswarm/muster with no /v5 suffix, Go only considers v0 and v1 tags and turned every v5 release commit into a pseudo-version of the last v1 tag. Because that version is older than every release, muster self-update re-installed the very release it was running on each time instead of reporting "Current version is the latest." The build now links the tag into the binary (make stamp-version, run by make test between the architect orb writing its link flags and linking with them; a branch build gets git describe, e.g. v5.23.2-1-g4be8379e), a pseudo-version from the build info is no longer shown as the version, and self-update refuses a version that is not a release (dev, a bare commit) instead of panicking on it. (#1256)
muster start workflow <name> no longer passes the CLI's own flags on as workflow input. --endpoint, --auth, --context, --config-path, --debug and the output flags given after the workflow name reached the workflow engine as arguments, were recorded in the WorkflowExecution (auth: none, endpoint: http://...) and would have collided with a workflow argument of the same name. muster start workflow and muster call now tell their own flags apart through cobra's flag set -- one grammar for both, no hand-kept list to drift -- and forward every other --key=value pair. After -- every pair is forwarded as it is, so a workflow or tool argument named like a CLI flag can be passed against a non-default muster: muster start workflow w --endpoint http://muster:8090/mcp -- --endpoint=https://target. muster call did not leak the flags but discarded everything after --. (#1249)
muster list tool --server <name> finds the tools of an aggregated server. The flag matched the name against the start of the exposed tool name, which for a server registered as files is x_files_<tool>, so --server files printed "No tools found" for every aggregated server, and no spelling at all selected a server whose toolPrefix differs from its name. --server now matches the server a tool belongs to as list_tools reports it (files, core, workflow), independent of the tool prefix, and accepts the exposed prefix (x_files) as well; -o wide and -o json show that server instead of the first name segment (x). (#1248)
muster test: the process test_restart_instance starts no longer dies when the restart step returns. It was bound to the step's context, so a restart step with a timeout killed the new muster serve right after it had passed readiness and the next step found the connection reset.
The DPoP replay cache's Valkey client no longer enables valkey-go's client-side cache: nothing reads through it, and its CLIENT TRACKING handshake failed against a server without the feature, which left muster's OAuth server in degraded mode (service_unavailable on every request). The session stores and the mcp-oauth store already ran without it since v5.19.13.
oauth.server.allowPrivateIPRedirectURIs (Helm: muster.oauth.server.allowPrivateIPRedirectURIs): a client's redirect URI may resolve to a private address. On a cluster whose own hostnames resolve to an internal load balancer the authorization-time check rejected every platform-hosted client's callback with redirect_uri: hostname resolves to private IP address (DNS rebinding protection) -- klaus-gateway's Slack sign-in ended in server_error: Failed to start authorization flow right after allowPrivateIPClientMetadata had let the client through. Off by default; emits a startup warning when set; exact redirect-URI matching is unchanged. (#1201)
oauth.server.allowPrivateIPClientMetadata (Helm: muster.oauth.server.allowPrivateIPClientMetadata): a CIMD client_id URL -- the Client ID Metadata Document an OAuth client such as klaus-gateway self-hosts -- may resolve to a private, loopback or link-local address. Needed where the platform's own hostnames resolve to an internal load balancer (a management cluster reachable only over VPN): until now the SSRF guard rejected such a client with invalid_client: ... client_id metadata URL resolves to private/internal IP address and nothing in muster's configuration could lift it, although mcp-oauth has had the switch (AllowPrivateIPClientMetadata). Off by default; emits a startup warning when set; PKCE and redirect-URI validation are unchanged. (#1199)
oauth.mcpClient.tokenExchange.allowPrivateIP (Helm: muster.oauth.mcpClient.tokenExchange.allowPrivateIP): the token endpoint of a remote Dex muster exchanges tokens with (spec.auth.tokenExchange.dexTokenEndpoint) may resolve to a private or loopback address -- a management or workload cluster whose Dex sits behind an internal-only load balancer. Until now only --extra-ca-file lifted the token-exchange client's SSRF guard, so an exchange against such a Dex failed with a DNS-rebinding error although the endpoint was reachable. Off by default; emits a startup warning when set; TLS verification is unchanged.
Core tools declare MCP annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint). Every core_* tool is classified where it is declared: the list/get/validate/probe tools are read-only (core_auth_login included — it issues a sign-in link for the caller and changes nothing on the platform), creates are additive writes, updates/deletes/stops are destructive writes. list_tools, filter_tools and describe_tool report them like a downstream server's, so the built-in read-only toolset preset now includes the 16 read-only core tools and refuses the 13 writes without any tool: selector, and a workflow whose steps call a read-only core tool derives readOnlyHint: true. An agent on preset:read-only no longer needs tool:core_auth_login to connect SSO-protected servers. The table is in docs/reference/mcp-tools.md ("Core tool annotations"). (#1172)
Subject-scoped grants refresh themselves. For an MCPServer whose authorization server is pinned with grantScope: subject (GitHub's hosted MCP server and in-house servers verifying GitHub tokens), muster now redeems the grant's refresh token itself: every lookup that serves a tool call, a core_auth_login or a broker release finds the person's grant, refreshes it when its access token has expired or will within five minutes (capped at half the token's lifetime) using the pinned tokenEndpoint and the pre-registered client, and stores the rotated tokens under the person and under every session copy. Refreshes are single-flighted per person and issuer and re-read the store first, so a rotating refresh token (GitHub) is redeemed exactly once however many sessions hit the expiry together. A refresh the authorization server rejects (invalid_grant, GitHub's bad_refresh_token) removes the grant so the next use asks for a sign-in; a transient failure keeps it. Before, the store hid the token 30 s before expiry and nothing redeemed the refresh token: eight hours after connecting GitHub every session lost the server with authentication required: server returned 401 Unauthorized and had to connect again. The in-memory store keeps expired tokens that carry a refresh token for 30 days instead of sweeping them. Logged as subject_grant_refreshed / subject_grant_refresh_rejected / subject_grant_refresh_failed.
Token broker grant targets (tokenExchangeBroker.targets.<audience>.grantIssuer). A broker target can now release a person's subject-scoped grant instead of exchanging at a downstream Dex: an allow-listed confidential broker client exchanges the person's ID token (audience=<audience>) and receives the access token of the person's own grant from that issuer -- looked up by the subject token's raw sub, the identity the grant was filed under, not by a trustedIssuers subjectClaim mapping -- refreshed first when it is due, with issued_token_type access_token and expires_in the token's remaining lifetime, never the refresh token. A person without a grant gets invalid_target (no_grant in muster's log). clientAudiences gates it and every release is audited (subject_grant_released / subject_grant_release_refused). A target is either a grant target or a Dex exchange target: grantIssuer and dexTokenEndpoint are mutually exclusive and a grant target rejects the exchange settings at startup. Lets a developer portal implement its standard GitHub auth API on the grant muster already holds, with GitHub's own client libraries. See "Releasing a person's grant to a trusted relying party" in docs/how-to/connecting-non-rfc9728-mcp-servers.md.
tokenExchangeBroker.brokerClients.<id>.clientSecretFile: seed a confidential broker client from a file holding its secret, for deployments without a Kubernetes Secret store (muster serve against a local config directory, the test harness). Takes precedence over clientCredentialsSecretRef.
muster test: muster_broker.clients (confidential broker clients with their audiences) and muster_broker.grant_targets (audience → mock authorization server) configure brokered exchange for a scenario; test_broker_token_exchange accepts client_id/client_secret for HTTP Basic client authentication and reports an opaque released token with expires_in, issued_token_type and has_refresh_token; the mock authorization server marks access tokens it issues on a refresh with a refreshed- prefix and OAuth-protected mock backends echo an opaque bearer through echo_token tools. Scenarios: oauth-subject-grant-refresh, oauth-subject-grant-refresh-rotation, oauth-broker-grant-release.
Authorization servers muster cannot discover or register with (GitHub). spec.auth.authorizationServer gains three optional fields: authorizationEndpoint/tokenEndpoint pin the endpoints of an authorization server that publishes no RFC 8414 document (muster then performs no discovery for that issuer and assumes S256 PKCE); clientCredentialsSecretRef names a Secret with a client registered out of band (a GitHub App or OAuth App), which muster uses instead of its CIMD URL or a dynamic registration (clientIdMethod: preregistered); grantScope: subject files the tokens under the user's identity as well as the session, so any later session of the same person reuses the grant without a new consent until core_auth_logout on that server or "sign out everywhere". Together they connect the hosted GitHub MCP server -- or any in-house server that verifies GitHub tokens -- to the person's own GitHub account. See docs/how-to/connecting-non-rfc9728-mcp-servers.md.
Subject-scoped grants serve every session of the person on first use. For an MCPServer whose authorization server has grantScope: subject, a session that never called core_auth_login for it -- another MCP client, a fresh CLI session, a front-end whose bearer rotated, any session after a muster restart with the grant persisted -- is connected with the person's existing grant by the tool call itself, and list_tools shows the server's tools to it instead of listing the server under auth_required. Before, only core_auth_login reused the grant: a plain tool call from such a session failed with user not authenticated to server <name> (or tool not found while no session of the process had connected the server), so every client had to know to call core_auth_login on that error. Concurrent first calls from one session share a single connection; a grant the server rejects with 401 is cleared for the person, as core_auth_login does, so the next login issues a fresh sign-in link. Sessions of a person without a grant fail as before. The same lookup also lets a session that is already authenticated to a server whose authorization server is pinned (spec.auth.authorizationServer, no discovery document) rebuild its connection after a muster restart from the pinned issuer; before, the tool call failed with unable to determine auth method for server <name> until the session ran core_auth_login again.
core_auth_login accepts an optional reset_client_registration: true. It discards muster's stored RFC 7591 dynamic client registration with the server's authorization server and registers again before issuing the sign-in link — the escape hatch for an authorization server whose refusal of the stored client_id the automatic checks below cannot see. Other users' tokens from that authorization server may need a fresh sign-in afterwards, since their refresh tokens are bound to the old client_id.
AWS SigV4 request signing for MCPServers. spec.auth.type: sigv4 with spec.auth.sigv4 (region, optional service, optional roleArn) makes muster sign every request to a streamable-http backend with AWS Signature Version 4, as its own machine identity rather than the caller's, using the default AWS credential chain so IRSA needs no extra configuration; spec.family plus one roleArn per MCPServer puts many AWS accounts behind a single tool set with an account selector. Every signed request carries X-Amz-Content-Sha256 inside its SignedHeaders, so a backend that requires the payload-hash header (S3 and the services sharing its request model, reachable because auth.sigv4.service is user-settable) accepts the signature.
spec.meta for remote MCPServers. The map is merged into the params._meta object of every outbound JSON-RPC request that carries params, for a backend that reads call-scoped configuration from MCP metadata rather than tool arguments — the AWS-hosted server takes the region it operates in from params._meta.AWS_REGION, which is a different value from the SigV4 signing region, and omitting it does not fail but answers about the wrong region. Injection happens in an HTTP round tripper shared by every remote client, so it applies to streamable-http and sse alike, with any auth type and with none, and it survives the switch to the per-session connection that an OAuth server's tool calls run on; for a SigV4 server the round tripper sits in front of the signer, because the body has to be rewritten before it is signed. type: stdio rejects the field (CEL rule plus the same check in filesystem mode): a stdio server speaks over a pipe, so no HTTP transport can inject the entries, and a dropped entry fails nowhere.
Helm: setting muster.observability.metrics.prometheus.serviceMonitor.enabled: true is now sufficient on its own to expose metrics — it implicitly appends prometheus to muster.observability.metrics.exporter, so the chart serves /metrics, adds the metrics container/Service port, and renders the ServiceMonitor from that single toggle (previously the exporter had to be set separately, and forgetting it silently rendered nothing). Explicit exporter lists are preserved: exporter: "otlp" plus the ServiceMonitor toggle yields otlp,prometheus; the "none" no-op sentinel is dropped when the toggle is on, so exporter: "none" yields just prometheus.
OAuth proxy start endpoint (/oauth/proxy/start): auth-challenge login URLs now point the browser at this muster-hosted endpoint, which redirects to the upstream authorization server. The login flow is unchanged for users; the URL in core_auth_login results is now a short muster URL instead of the full upstream authorization URL.
Browser-consent connector logins now push tools/resources/prompts list_changed notifications to the user's live sessions once the OAuth callback connects the backend, matching the SSO connect path. Clients that honor list_changed (e.g. Claude Code) see the newly available tools without re-running core_auth_login.
oauth.mcpClient.postLoginRedirectAllowlist: a list of absolute http(s) URL prefixes. A caller may append a redirect query parameter to the start URL; when the target matches an allowlist entry (exact scheme and host, path extended at a segment boundary; targets with dot segments are rejected), a successful callback redirects the browser there (with the connected server's name appended as a server query parameter) instead of rendering the static success page. Lets a front-end observe connector login completion for the flows it initiated, without affecting other clients of the same muster. Empty (default) rejects all redirect requests; a rejected target is dropped and the login still proceeds. Failed callbacks always render the error page. The Helm chart renders this from muster.oauth.mcpClient.postLoginRedirectAllowlist (default empty).
muster test --readiness-timeout: the per-instance deadline for expected resources (tools, workflows, MCP servers) to become available after startup is now configurable (default 15s, previously hardcoded). (#1101)
muster test no longer starts every scenario's instance at t=0: at high --parallel the cold-start herd (50 Go processes at once) is what starved small CI containers into intermittent 15s readiness timeouts and scenario-timeout kills. Startups are now bounded (--startup-parallel, default 8) without limiting steady-state scenario parallelism, and spawned test instances run with GOMAXPROCS=2 so a container that reports the host's core count (CircleCI docker executors allocate CPU by cgroup shares, which Go cannot right-size against) no longer produces fifty host-sized runtimes on a 4-vCPU slice. The suite banner prints GOMAXPROCS/NumCPU so CI logs show the runtime sizing. (#1101)
MCP structuredContent from downstream tools is now preserved. The call_tool meta-tool propagates structuredContent from the wrapped tool both natively on its own result and as a structuredContent field inside its JSON envelope (previously it was silently dropped), and the agent client restores it when unwrapping. Core tools can opt in via the new StructuredContent field on api.CallToolResult; none set it yet. Text-only consumers are unaffected.

Changed

README and documentation reworked. The README describes muster as it is today: an aggregating MCP server with meta-tool discovery, toolsets, OAuth 2.1 protection and single sign-on through Dex, MCPServer and Workflow custom resources, workflows and observability; installation is from the signed release binaries, go install or the Helm chart in the Giant Swarm catalog. The documentation lost its duplicated getting-started and IDE pages, the invented REST API, service resources, create service, port 8080 and Homebrew instructions, and gained a quick start whose every command is verified, a tutorial for servers and workflows, guides for connecting MCP clients and authenticating the CLI, an installation guide for the chart with Dex and Valkey, an HTTP endpoints reference including the admin listener, and a rewritten contributing guide. The product is spelled muster throughout. muster --help and muster serve --help describe the current product instead of a Giant Swarm port-forwarding helper.
describe_tool states that the tool it describes is only reachable through call_tool. Its response gained an invocation line — Call it through the call_tool meta-tool: call_tool with {"name": "<tool>", "arguments": {...}}, arguments per inputSchema. Tools inside muster are not callable by name directly — only the meta-tools are. — which the meta-tool's own description names, next to what it returns, and call_tool's own description now states the rule itself ("Aggregated tools are callable only this way, never by their own name"), so a model reads it when it decides how to execute rather than only after a lookup. An MCP client sees only the meta-tools, so a model that took a name out of describe_tool and issued it as a tool call of its own got an error with nothing in the detail it had just read to recover from. The field is additive: description, inputSchema, server, kind and annotations are unchanged, and the consumers that parse the response (the muster CLI, the REPL describe) ignore what they do not know. Scenario: list-tools-paged.
list_tools answers one bounded page instead of the whole catalogue. It takes limit (default 50) and offset, reports total and truncated, and projects every entry the way filter_tools does — name, one-line summary, server, kind, annotations — with the full description and the input schema behind describe_tool. Before, it returned every tool of the caller's catalogue with its full description in one response: measured on an internal installation, 423 KB unscoped and 402 KB (about 135k tokens) under X-Muster-Toolset: preset:read-only, growing the calling model's next prompt by 105k–120k tokens and staying in its history for the rest of the session. The tool's description now says what a page costs and points at filter_tools for discovery and describe_tool for detail. servers_requiring_auth is still present (neither paged nor narrowed by a toolset), a header-declared toolset still bounds the listing (preset:none lists nothing) and is echoed in toolset as filter_tools echoes it, the refusal texts are unchanged and list_core_tools is untouched. The muster CLI and REPL listings, tab completion and the test harness page through the catalogue, so they keep listing every tool; the local muster agent MCP server now advertises the aggregator's meta-tool definitions verbatim instead of a hand-maintained copy, so limit/offset and the new description reach the assistant through it. (#1193)
BREAKING (Kubernetes mode): type: stdio MCPServers are rejected. A stdio server is started as a subprocess of the muster process, so in a deployed aggregator "may write MCPServer resources" implied "may execute code in the muster pod as muster's ServiceAccount" (which holds get on Secrets in its namespace and cluster-wide write on the muster CRDs). Kubernetes mode now refuses stdio at every layer: mcpserver_validate / mcpserver_create / mcpserver_update fail the tool call with a message naming streamable-http / sse as the alternative; a stdio MCPServer applied straight through the API server reconciles to status.state: Failed with the same message in status.lastError and is never started (an already-running one from an older muster is torn down); and the orchestrator and service layers refuse to build a stdio client at all, so no path reaches NewStdioMCPClient. Deployments with stdio MCPServers must run those servers as their own workload and re-register them with type: streamable-http or type: sse. Filesystem mode — muster serve against a local config directory, i.e. the CLI, where launching subprocesses is the point — is unchanged. (#1067)
Outbound OAuth flows now send the RFC 8707 resource parameter on both the authorization request and the token request, for backends and for the muster agent login. The value is the resource field of the target's RFC 9728 metadata, sent exactly as declared; when that metadata omits the field, or spec.auth.authorizationServer opts out of discovery, it is derived from the configured URL by dropping the query and the fragment and changing nothing else, which is the value mcp-go derives for the refresh request. A token muster obtains for one backend is no longer accepted at another backend that trusts the same issuer, provided the authorization server honors the parameter. An authorization server that rejects an unknown resource will fail the login.
Authorization server metadata is now rejected when its issuer does not identify the server the document was fetched from (RFC 8414 §3.3, trailing slash ignored), and when it carries no issuer at all. Deployments whose authorization server reports an issuer other than the URL muster fetches it from must correct the configured issuer.
RFC 9728 protected resource metadata advertised by a backend is now bound to that backend. A resource_metadata= pointer in a WWW-Authenticate header is followed only when it is on the backend's own scheme and host, and a resource the document declares is used only when it is on that same origin (the path may differ, so a backend serving at <base>/mcp may declare <base>). A pointer to another origin is ignored and discovery falls through to the well-known path; a foreign declared resource is dropped and the indicator is derived from the backend URL. Without these checks a backend could name a document that binds muster's token to a resource the backend does not own.
Breaking (Go API): pkg/oauth.Client.BuildAuthorizationURL takes an AuthorizationRequest struct instead of six positional arguments, and pkg/oauth.Client.ExchangeCode takes a trailing resource argument. Callers pass the same values through the new shapes; an empty Resource omits the parameter.
OAuth callbacks are now validated per RFC 9207 before anything else on the response is acted on. A present iss must equal the issuer the authorization server publishes in its own metadata (simple string comparison, no normalization), and an absent iss is refused when that server advertises authorization_response_iss_parameter_supported. When the metadata is unreachable the comparison falls back to the issuer recorded with the flow and then ignores a trailing slash on either side, because that value is operator-configured rather than published by the server. A response that fails the check is rejected whole: its error and error_description are neither acted on nor displayed. An error response that carries no valid state now renders "missing required parameters" instead of the OAuth denial page.
The muster and muster-crds chart READMEs no longer render a version badge (chart.badgesSection removed from both README.md.gotmpl files): a release PR bumping Chart.yaml's version or appVersion no longer changes the checked-in README.md, so the helm-docs pre-commit hook no longer fails on it. (giantswarm/devctl#2180)

Removed

BREAKING: JWT mode. muster is not an identity provider: dex is the sole SSO authority. The enableJWTMode and jwtSigningKey/jwtSigningKeyFile config keys, the signing-key loading, and the chart's jwt-signing-key Secret plumbing are removed; muster issues only opaque access tokens and has no signing key, so no muster-signed token can exist. forwardToken backends receive the upstream dex ID token (or, for sessions established by a trusted-issuer bearer, that IdP-issued token) byte-identical — never a muster-signed token — and validate it against the IdP's issuer/JWKS, never against muster. The self-issued RFC 8693 exchange at /oauth/token (which signed muster tokens) is permanently disabled and refuses issuance; agent/OBO flows use dex-issued tokens (exchange at dex, e.g. per-backend tokenExchange / auth.mode: exchange). Deployments that set enableJWTMode must drop the key (the chart schema now rejects it) and reconfigure any backend that trusted muster's JWKS to trust dex instead. This reverses the muster-signed-token direction of the forward-token refactor and supersedes its "backends must trust muster's issuer/JWKS" deployment note; the BDD scenarios obo-token-forward* are replaced by dex-token-forward*, which pin that the forwarded delegation chain is dex-minted and that muster refuses to issue tokens.

Fixed

A new session's first request no longer waits for the SSO fan-out (#1226). A session that arrives with a forwarded or trusted-issuer token -- an agent's session -- had muster connect every session-authenticated server (one RFC 8693 token exchange or ID-token forward per server, in parallel, bounded by the slowest remote authorization server) inside its first request: initialize and the agent's first tools/list answered only after the last server had connected, 2.7-3.2 s on an installation with 81 such servers, although tools/list returns the meta-tools and needs none of the connections. The fan-out now starts on the first request and runs in the background; initialize and tools/list answer at once. A call_tool (or resource read, or prompt) for a server still connecting waits for that server's connect alone, resolved from the tool name's prefix or its family, not for the whole fan-out; list_tools, filter_tools and describe_tool wait for the fan-out so a session's first listing is complete. The person's subject-scoped grants (GitHub-style connectors) are adopted in the same fan-out instead of sequentially on the first call that misses them. Requests arriving while the fan-out runs share it; the sign-in flow still connects before it issues the access token. The fan-out logs its total duration, outcome counts and slowest server (SSO: fan-out finished … duration_s=… slowestServer=…), each connect logs its duration, and a held request logs how long it waited (SSO: tool call waited for its server's connect). muster test: a step's max_duration bounds one invocation and fails the step when it ran longer; an OAuth-protected mock takes connect_delay to stand in for a backend that is slow to connect. Scenario: oauth-sso-session-start-answers-during-fan-out.
A muster whose configured Valkey is unreachable at startup no longer serves sessions on in-memory stores for the life of the pod (#1229). The session auth and capability stores were built once at startup; when the Valkey configured in oauth.server.storage did not answer at that moment -- muster and Valkey starting together, a Valkey OOM or a node roll while muster restarts -- muster logged Failed to create Valkey client for session stores, falling back to in-memory and kept in-memory stores: muster:cap:* and muster:auth:* never appeared in Valkey while sessions worked, the next restart lost every session's view and SSO state, and the OAuth token store, which retries its own connection, ended up in Valkey regardless. Now muster dials the configured Valkey again with backoff (1 s doubling to 8 s) for up to 20 s (under the chart's 30 s liveness window), logging Valkey for the session stores did not answer, retrying in between and serving nothing until it answers; when it stays away the start fails with Configured Valkey did not answer; refusing to serve sessions on in-memory stores and muster serve exits non-zero, so the kubelet restarts the pod. In-memory stores remain the behaviour of a deployment that configured no Valkey. The backend in use is published as the gauge muster_session_store_backend{backend="valkey|memory"} (1 for the backend in use, 0 for the other). The Valkey client's client-side cache, which no store reads through, is off (CLIENT TRACKING is no longer requested per connection).
The core catalogue is refreshed in the background, not on the call that finds it old (#1231). The aggregator's own tools (core_*, workflow_<name>) are kept for five minutes as the backstop for a definition change no invalidation saw; the meta-tool call that found them older paid the rebuild -- one list of every Workflow resource from the API server plus their conversion, 1.9-2.05 s with 282 workflows -- while a warm session had been answering the same calls in 7-20 ms. Now that call is served the catalogue it has and starts one rebuild in its own goroutine; concurrent calls share it, and the rebuilt catalogue is swapped in when it lands. A call without a catalogue -- the first of the process, or after a workflow was created, updated or deleted -- still waits for the rebuild, so a definition change stays visible on the very next call; a change that arrives while a rebuild runs makes it build again before anything is served. A change in the server registry -- a backend server registered, lost, or re-registered by the reconciler's status sync -- no longer invalidates the core catalogue either: it depends on the tool providers alone, not on which servers are registered, yet every registry change ran the capability refresh that threw it away, so on an installation with 87 servers the meta-tool calls after a muster restart paid the rebuild again and again (1.2-3.3 s calls minutes apart, none of them at an age boundary). Every rebuild is logged at debug (Core catalogue rebuilt, with its trigger and duration_s). MUSTER_CORE_CATALOGUE_MAX_AGE overrides the age (the integration test harness runs it at 3 s).
Meta-tool calls no longer rebuild the session's catalogue from the definition source and the capability store on every call (#1225). call_tool, describe_tool, filter_tools, list_tools and the resource/prompt meta-tools each listed the caller's catalogue afresh: every workflow definition was fetched from the API server twice per call (once for the tool's arguments, once to derive its read-only hint -- 8182 GET workflows in the two minutes of one agent turn with 282 workflows), and every session-authenticated server's capabilities were read from Valkey with their own HGET and GET (~160 round trips per call with 81 such servers). Together that was a floor of about 3 s per meta-tool call on a warm session while the backend answered in 22 ms. Now the aggregator's own tools (core_*, workflow_<name>) are built once and kept until a workflow definition changes -- through muster's tools or, for a resource applied outside them, through the workflow reconciler -- with the step tools carried on the tool so the read-only hint is derived in-process; a session's capabilities are read with one HGETALL, and the content-addressed documents behind the references are decoded once per process and shared by every session that references them; the servers a session is authenticated to are read with one HKEYS. A warm meta-tool call costs two Valkey commands regardless of how many servers are registered. Toolset scoping, session-scoped visibility and the capability freshness mechanisms are unchanged.
tools/call response and Error log lines carry duration_s, the time since the request reached the transport, so a slow call is visible in the log without subtracting the request line's timestamp (#1225).
The boot pass no longer reports an MCPServer the reconciler registered a moment earlier as a failure (#1222). The orchestrator's boot pass and the MCPServer reconciler's first pass run concurrently: for a definition the boot loop had not reached yet, the reconciler found no service and StartService registered it lazily, and when the loop then got there the refused duplicate registration was logged as ERROR Failed to create MCPServer service: <name> ... service <name> already registered -- one misleading error per affected server per boot (two of 87 definitions on one installation) for a service that existed and was being started by the caller that registered it. The registry's duplicate error now wraps services.ErrServiceAlreadyRegistered and the boot pass treats it as a debug-level no-op (MCPServer <name> was registered through StartService while the boot pass ran; that caller starts it); every other error is reported as before and the suspended-server skip is unchanged.
mcp-oauth's metrics reach muster's metrics endpoint, the token store's included. The OAuth server's instrumentation now records through muster's own OpenTelemetry meter provider (mcp-oauth 1.4.0, instrumentation.Config.MeterProvider), and the Valkey and in-memory token stores share it. Until now mcp-oauth built a Prometheus collector of its own on the default registerer, which the :9464 endpoint -- served from a registry of its own -- never gathered: not one oauth_* series was ever exported, and the token store was not instrumented at all. Now oauth_http_requests_total, oauth_token_endpoint_failures_total{grant_type,error_code} and the other oauth_* series, storage_operation_total{operation,result} (result is success, error or timeout), the storage_operation_duration_milliseconds histogram and the storage_*_count size gauges are served next to the muster_* series. A Valkey outage shows as storage_operation_total{result="timeout"} and oauth_token_endpoint_failures_total{error_code="temporarily_unavailable"}.
A suspended MCPServer is not started when muster boots (#1216). The orchestrator's boot pass created and started a service for every definition with autoStart: true without looking at spec.suspended (what core_service_stop and a portal's Deactivate write), and the reconciler's first pass then stopped it again: per restart and per suspended server one Creating MCPServer service / Starting MCP server service / Suspending MCPServer service <name> (spec.suspended=true) sequence in the log, one connection attempt against a backend the operator had switched off, a MCPServerStarting and a MCPServerStopped event, status.lastAttempt moved to the boot time, and -- when the connect won the race against the stop -- a window in which the deactivated server's tools were registered in the aggregator and callable. The boot pass now skips a suspended definition the way it skips autoStart: false (Skipping MCPServer <name>: Suspended=true at debug level); the server keeps reading Disconnected (stdio: Stopped) through the reconciler's no-service branch, its status keeps its pre-restart lastAttempt, and spec.suspended: false still starts it. muster test: a pre-configured mock server accepts suspended: true, which also takes it out of the readiness gate. Scenario: mcpserver-suspended-not-started-at-boot.
The Valkey capability store keeps each capability document once, content-addressed, instead of once per session (#1217). A session's hash {prefix}cap:{sessionID} held, per connected server, the server's full tool/resource/prompt list with every input schema — 9–180 KB a server, 0.9 MB a session on one installation — for 30 days, and every forwarded bearer (a Dev Portal, kagent or Backstage ID token) is a session of its own, so identical documents were written again and again: 448 session hashes, 462 MB, and muster-valkey was OOM-killed 19 times in 90 minutes with every token refresh blocked meanwhile. The hash field is now a 71-byte reference sha256:<hex>; the document lives once under {prefix}capblob:<hex> with the store TTL, refreshed by every Set that references it. Reads resolve the references (one pipelined GET per distinct document); a reference whose document has expired is a cache miss, which the aggregator already answers by listing the server again on the next connect. Fields written by an earlier muster still read, and on start the aggregator rewrites them to references in the background (MigrateInlineEntries: one SCAN, idempotent, logged with the session, field, document and byte counts) — an installation shrinks on its first start with this release instead of after 30 days. The store is proportional to the number of distinct documents now (that installation: 84 documents ≈ 1.3 MB, the references ≈ 3.6 MB). Unit tests run the store against an in-process miniredis.
A suspended remote MCPServer is stopped once, not on every resync tick (#1212). The stop of a remote server (streamable-http, sse) settles in Disconnected, never in Stopped, but the reconciler's suspend path only recognised Stopped and Stopping as done -- so a server with spec.suspended: true was stopped again every 30 s for as long as it stayed suspended: Suspending MCPServer service <name> (spec.suspended=true) and Stopped service: <name> in the log, an MCPServerStopped Kubernetes event and a second reconcile from the state change on each tick, 13 days of it for one server on a management cluster. reconcileSuspend now returns early for Disconnected as well; Failed, Error and Unreachable are still stopped, so suspending a server whose endpoint is down still ends its reconnect schedule. Service.Stop has the same guard: called on a remote server that is already Disconnected it returns without a state write and without an event, as it always did for a local server that is Stopped. Resume (spec.suspended: false) is unchanged. Scenario: mcpserver-suspend-remote-stops-once; muster test runs instances with MUSTER_RECONCILER_RESYNC_INTERVAL=2s (a new Go-duration override of the reconciler's 30 s resync) and instance_logs gained occurrences, an exact line count per substring.
A deactivated MCPServer refuses sign-ins and reports itself down (#1211). An OAuth MCPServer with spec.suspended: true (what core_service_stop and the portal's Deactivate write) keeps its pending-auth registry entry so that sign-ins work again once it is activated -- but that entry also let core_auth_login hand out a sign-in link while the server was suspended. The person completed the flow, the session connected, the event handler failed a global registration (Failed to register MCP server … no MCP client available (service state inconsistent)), the reconciler stopped the service again within milliseconds, and auth://status then said connected with a tools count while the session's tool list had nothing for the server; a session that had signed in before the deactivation saw the same contradiction. Now core_auth_login refuses a suspended server with Server '<name>' is deactivated (spec.suspended=true); activate it with core_service_start before signing in. and creates no challenge; the OAuth callback for a server deactivated between challenge and callback stores the token and establishes no connection (the first core_auth_login after core_service_start connects with it, no browser needed); auth://status reports a server whose service is down as disconnected whatever the session's auth mark says, with a new "suspended": true flag naming the reason (muster auth status prints Deactivated); and list_tools no longer names a down server under servers_requiring_auth. The spec is read through the MCPServer manager -- the source the reconciler acts on -- so the aggregator keeps no copy of it. The service state inconsistent ERROR turned out not to be the suspension's doing: the event handler tried a global registration on every per-session OAuth sign-in (the skip covered SSO servers only), failed because such a service holds no client, and emitted a MCPServerToolsUnavailable event the moment a person connected; it now skips a server that authenticates per session the way it skips SSO servers. Scenario: mcpserver-suspended-refuses-auth-login; test_simulate_oauth_callback gains an auth_url argument to complete a challenge an earlier step obtained.
Duration histograms use bucket boundaries for seconds (#995). muster_tool_call_duration_seconds, muster_downstream_tool_call_duration_seconds and muster_workflow_execution_duration_seconds are recorded in seconds but inherited the OTel SDK's default boundaries (0 ... 10000), which are spaced for milliseconds: every observation landed in the first bucket and the other 14 series stayed empty, so histogram_quantile over them could resolve nothing finer than "under 5 seconds". A sdkmetric.View matched on unit s now applies 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300 to every seconds-unit histogram, current and future. The boundaries are documented in docs/explanation/observability.md next to the p95 example query.
A family tool is callable by a session that has not listed tools (#1204). The family routing index is process-global and in memory, and the tools of members that require per-session authentication entered it only when a session authenticated to them ran list_tools. A muster restart empties the index while the sessions' tokens and cached capabilities persist, so every session that held on across a rollout got Tool execution failed: tool not found: x_prometheus_get_rules from call_tool for each family tool -- with core_mcpserver_list reporting the member Connected -- until it listed tools again; a session that signed in and called without ever listing hit the same. call_tool now fills the index from the session's view, the pass list_tools runs, when the name lies in a declared family's name space and the index does not know it, and the call goes through. When the name still cannot be routed, the error says so in terms of the family and the session: tool x_kubernetes_list_podz is not exposed under family "kubernetes" by any member this session is connected to (k8s-a), or, for a session connected to no member, which members exist and that core_auth_login connects one -- instead of tool not found. Scenario: mcpserver-family-tool-call-without-listing.
Helm: MusterMCPServerFailed and MusterMCPServerFlapping survive a muster rollout (#1203). Both expressions now aggregate over cluster_id, installation, pipeline, provider, namespace, mcpserver_namespace, mcpserver_name (max by for the state gauge, sum by for the transition counter) instead of carrying the scrape labels pod, instance, container, endpoint, service, job and otel_scope_name into the alert. Before, every rollout ended the old pod's series -- Alertmanager sent RESOLVED -- and the new pod's series started a fresh for window that fired again 20 minutes later: four rollouts in one day produced seven notifications for a single backend that never came back. The alert's identity is now the MCPServer on a cluster; the for windows, labels and annotations are unchanged, and state is no longer an alert label (both rules only select Failed). The promtool suite covers the series moving from one pod to the next mid-window.
A connected MCPServer whose backend stops answering is detected and reconnected (#493, #999). The orchestrator now probes every connected or running MCPServer with an MCP ping every 30 s (MUSTER_ORCHESTRATOR_HEALTH_CHECK_INTERVAL, a Go duration); each probe gets the server's spec.timeout (default 30 s), and a backend that negotiated protocol 2026-07-28, where ping no longer exists, is probed with tools/list. Three failed probes in a row close the client and move the server to failed with a reconnect due at once: its tools are withdrawn from every session, one MCPServerHealthCheckFailed event names the count and the error, the CR status shows Failed with lastError and nextRetryAfter, and the reconnect loop restarts it on its next tick (MCPServerRecoveryStarted). A reconnect that fails follows the usual MCPServerFailed backoff schedule whatever the endpoint answered -- a 404 from a route being reprogrammed as much as a refused connection -- until the backend serves again. A passing probe, a successful start or a stop reset the count, so a single slow ping costs nothing; a JSON-RPC error reply, a probe cancelled by shutdown, and a server that is being stopped or restarted are not counted. Servers served per session (forwardToken, tokenExchange, OAuth) have no shared client and are not probed. Before, a server that reached connected stayed connected / healthy with its tools listed whatever happened to the connection afterwards -- a redeployed, hung or dead backend meant every call failed until an operator ran core_service_restart; the service's CheckHealth() existed but nothing called it. core_service_status exposes the running count as metadata.consecutiveHealthCheckFailures. The aggregator no longer emits a second MCPServerHealthCheckFailed for the same transition. Scenarios: mcpserver-health-probe-restarts-dead-backend, mcpserver-health-probe-restart-failure-is-retried, mcpserver-health-probe-spares-session-auth-server, mcpserver-health-probe-spares-per-session-oauth-server.
filter_tools echoes a header-declared toolset (#1194). The response's toolset field names the toolset the returned tools were resolved within: the toolset argument when one is given, else the request's X-Muster-Toolset -- as declared, e.g. ["preset:read-only"]. Before, only the argument was echoed: an agent whose toolset the platform sets by header got total/filtered_count scoped and toolset_unmatched filled but no toolset, so it had no way to learn which toolset bounds it short of a refused call_tool, and a transcript did not show which toolset a discovery call ran under. With both header and argument the argument keeps winning (it resolves within the header's toolset). presets stays behind include_presets or the argument, so a header-scoped discovery call does not grow. list_core_tools, which shares the engine, echoes the header the same way. Scenario: toolset-filter-tools-echo.
A remote MCPServer recovers its MCP session after its backend is redeployed (#999). A stateful streamable-http backend hands out an Mcp-Session-Id at initialize; a redeployed pod does not know it and answers 404, mcp-go drops the id and from then on every request went out with no session, so every tool call failed (Bad Request: Missing session ID from a Python backend, Invalid session ID from mcp-go) while core_service_status kept reporting connected / healthy, until an operator ran core_service_restart. On gazelle this took every x_pd_* tool away for hours. The client now recognises a lost session -- the typed ErrSessionTerminated, or a transport whose session id is gone because the notification listener met the 404 first -- re-runs initialize once and retries the call; concurrent callers on the same dead session share one handshake whether it succeeds or fails, a handshake is bounded to the server's spec.timeout (30 s when unset), one that fails is reported to every caller that waited on it together with the lost-session error (a 401 from the backend still reads as authentication required), and is retried by the next operation, and a client that was closed on purpose never reconnects. Tool calls, listings, resource reads, prompts and the health ping all go through the recovery, for streamable-http servers with and without OAuth; stdio and SSE have no session to lose and are unchanged. Scenario: mcpserver-session-recovery-after-redeploy.
A restart request on an MCPServer whose endpoint is down is processed by one attempt (#1166). spec.restartRequestedAt is mirrored into status.lastRestartedAt after the reconciler's first start or restart attempt, whether it succeeded or not; the failure is reported once in the reconcile result, and the following attempts are the service's own, on the reconnect backoff the failed start has already scheduled (status.nextRetryAfter, capped at 2 minutes since #1163). Before, the value was recorded only when the restart succeeded: the failed start moved the service starting -> failed, the state-change bridge queued a reconcile for each transition, the request was still pending, and the reconciler restarted the service again -- 131 Restarting MCPServer service <name> (restartRequestedAt=...) / Connection failure #N pairs in 53 s on a management cluster whose tunnel was scaled to 0, one MCPServerFailed event each, throttled only by the 500 ms debounce, while status.consecutiveFailures climbed to 131 and status.nextRetryAfter said "in 2 minutes" the whole time; the loop stopped only when the field was removed from the CR. core_service_start and core_service_restart write exactly this field in Kubernetes mode, so any restart of an unreachable server from the CLI or the UI triggered it. The same rule applies to a spec.suspended: false resume whose start fails (its marker used to be kept, so every state change resumed again) and to a pending request whose create or update in the same pass already attempted a start: one attempt, then the service's schedule. A request on a server with no registered service (autoStart=false, never started) whose start fails before anything is registered stays pending and is retried on the reconcile queue's own backoff, since no state change can re-trigger it. The reconciler logs Restart of MCPServer <name> requested at <time> failed: <error>; the request is processed, further attempts follow the service's reconnect backoff (next attempt at <time>) once per processed request. Scenario: mcpserver-restart-request-endpoint-down.
The reconnect backoff of a remote MCPServer is capped at 2 minutes (#1163). The wait between connection attempts still doubles from 30 s on every consecutive failure, but it stops at MUSTER_MCPSERVER_MAX_BACKOFF (default 2m; a Go duration) instead of 30 minutes, so a server whose upstream has healed is retried within the cap plus the 30 s retry tick, however many attempts the outage cost. Before, four 504s from a tunnel in front of a healthy server scheduled the fifth attempt 4.5 minutes out and the sixth 8.5, and users saw the server down for that long after the outage had ended. MCPServerFailed events now name the HTTP status the endpoint answered with and the scheduled retry -- server unreachable after 4 consecutive failures (endpoint answered HTTP 504, next retry in 2m0s at 2026-09-05T15:57:34Z): ..., or no HTTP response for a refused connection, a DNS failure or a timeout -- and every failure below the threshold emits connection failure N of 3 before unreachable (...). The aggregator no longer emits a second, bare MCPServerFailed on the same transition, so there is one event per failed attempt and it is the one with the schedule. The MCPServer CR status now carries the schedule too: status.consecutiveFailures, status.lastAttempt and status.nextRetryAfter existed in the CRD but were never written; they are now mirrored from the service on every status sync, together with the new status.lastFailureHTTPStatus, and cleared once the server connects or answers a 401 (which also clears the in-memory schedule, so Auth Required no longer sits next to a stale retry forecast). core_service_status exposes the same fields under metadata, core_mcpserver_get returns lastFailureHTTPStatus, and a Failed server whose error is an HTTP 5xx gets the status message Upstream error (HTTP 5xx) - the server or a gateway in front of it is failing; muster retries with backoff. Scenario: mcpserver-remote-backoff-cap; muster test gained test_set_mock_server_outage (a mock server answers its next N requests with a fixed HTTP status) and runs instances with MUSTER_MCPSERVER_MAX_BACKOFF=3s.
The family tool surface follows what members currently offer (#1162). A spec.family member that kept its registration but re-listed fewer tools -- mcp-capi 0.3.x hides its mutating tools when it runs read-only -- kept advertising the tools it had dropped: tools/list returned 40 x_capi_* names while every member reported 24, describe_tool named members that no longer offered the tool, and calling one failed with tool not found. The family routing index only grew (providers were unioned on every listing and removed only on deregistration) and the per-session listing served the capability store regardless of the member's state. The index is now re-synced from every listing -- a contributing member's entries are replaced by what it contributed, tools it no longer offers lose it as a provider, a tool no member offers disappears, and members that did not take part in a per-session listing keep their entries -- and a member whose service is down (failed, unreachable, stopped, disconnected) contributes nothing to any session's tools, resources or prompts, even though its pending-auth registry entry and cached capabilities survive. A failed SSO reconnection drops the session's cached capabilities for that server, and the person's sessions receive list_changed notifications when a capability list shrinks, as they already did when it grew. Scenario: mcpserver-family-member-relists-subset; muster test gained test_stop_mock_server / test_start_mock_server.
muster test no longer terminates the instances of another muster test running at the same time. The start-up sweep for instances left behind by an earlier run matched every muster serve --config-path .../muster-test-... process on the machine and sent each one SIGTERM, so a second suite started in another terminal -- or the harness's own unit tests under go test ./internal/testing/, which called the sweep for real -- killed the first suite's instances: scenarios then failed with muster instance process exited (code 0) before becoming ready, or an instance shut down cleanly mid-scenario. The sweep now terminates only instances whose parent muster test process is gone (re-parented after a crash or kill), which is the case it exists for; instances of a live harness are left alone, and the unit tests run the sweep against a fabricated process table plus two stand-in processes of their own.
core_auth_logout now finds the server's authorization server when the registry entry does not know it yet. The entry learns the issuer from the connection probe's 401 or from a login's discovery, so after a muster restart it is empty for a backend that publishes no RFC 9728 metadata until someone logs in -- while the person's tokens from before the restart are still in the store. Logout skipped every token step in that state and answered "Successfully logged out" with the tokens (for a grantScope: subject issuer: the person's grant) in place, and did not recognise sibling servers on the same issuer. It now resolves the issuer like a login does -- spec.auth.authorizationServer.issuer, else the server's resource metadata -- and matches siblings by their pin as well. Test framework: mcp_servers[].config.oauth.omit_resource_metadata makes the mock backend answer a bare 401 and serve no well-known document.
core_auth_logout on a server whose authorization server has grantScope: subject now revokes the person's grant even when other MCPServers share the issuer. The shared-issuer guard -- which rightly leaves a session-scoped token alone when several servers depend on it -- also skipped the clear for a subject-scoped issuer, so with GitHub's hosted MCP server and an in-house server both pinned to https://github.com/login/oauth, logging out of either answered "Successfully logged out" while the session token and the person's grant stayed in the store and the next core_auth_login reconnected without a prompt; the grant could only expire or be revoked on GitHub's side. The logout now deletes the tokens for the issuer from every session of the person and from the grant, disconnects every server of the issuer (auth mark, cached capabilities, pooled connection) in the calling session and in the person's other live sessions, tells those sessions their tool lists changed, and names the sibling servers it disconnected in the tool result. A session-scoped shared issuer keeps the previous behaviour. Test framework: mcp_servers[].config.oauth.grant_scope pins the referenced mock OAuth server as the MCPServer's authorization server with that grant scope.
The aggregator's registry entry for an MCPServer now carries the resource's own namespace. No registration path passed metadata.namespace on, so every server was filed under default: a clientCredentialsSecretRef (spec.auth.authorizationServer or spec.auth.tokenExchange) that named the MCPServer's own namespace logged a spurious Cross-namespace secret access ... from MCPServer in namespace default warning on every core_auth_login, and the token-forwarding, token-exchange and authentication-loss events muster files for the server were attached to an MCPServer of that name in default. The warning now fires only when the Secret is read from a namespace other than the MCPServer's; which Secret is read is unchanged. core_mcpserver_get and core_mcpserver_list report the namespace in Kubernetes mode.
muster serve with kubernetes: true no longer falls back to filesystem mode when the Kubernetes client cannot be created at startup. The fallback was silent (one debug-level line) and left muster split in two: the reconciler's change detector still watched the apiserver, so every existing MCPServer CR arrived as a create event, was looked up in the empty config directory as "not found" and had its service deleted (Deleting MCPServer service: … for each CR), while the orchestrator's auto-start listed zero definitions — muster served no server tools, and the CRs kept their stale Connected status because the status writes went to the filesystem as well. It happened after a node restart, when the muster container came back before the apiserver (and the kube-proxy rules in front of it) did and the CRD discovery call failed. Startup now retries the client for about half a minute (client.KubernetesClientBackoff, each attempt logged) and then exits with kubernetes mode is configured but no Kubernetes client could be created … refusing to fall back to filesystem mode, so the kubelet restarts the container; the change detector also takes its mode from the client that was actually created rather than from the configuration flag, so the two can no longer disagree. Automatic detection (no configured mode) keeps the fallback, now logged at warning level. (#1143)
A forwarded-token rejection is now attributed to the token's audiences, not only its issuer. When a forwardToken backend answers the forwarded ID token with 401, the log line and the auth://status detail used to name the token's iss and hint at the issuer's JWKS — which sent an investigation towards JWKS and egress when the backend had in fact trusted the issuer and refused the token's audience (agent-manager on gazelle: the Backstage client's audiences were not in its trusted set). The diagnostic now reads forwarded token iss=…, aud=[…]; the backend must trust this issuer's JWKS and one of these audiences, and when the backend's 401 carries a WWW-Authenticate challenge its error and error_description are quoted as well (agent-manager sent no identity token to act with towards the Kubernetes API; muster used to drop the header at the transport). auth://status returns the same account in the server's error field next to reauth_required / sso_attempt_failed, so a client can show why SSO failed instead of a generic pointer to an administrator; muster auth status --server <name> prints it as Reason:. Audiences are client identifiers, and the token itself is never logged. (#1141)
An MCPServer configured for session-level auth (spec.auth.forwardToken, or tokenExchange.enabled) is no longer registered globally with a token-less client when its backend answers muster's connection probe anonymously — for example the old, anonymous pod during a rollover to an OAuth-protected release. The decision to handle a server per session now comes from the server's own definition rather than from the aggregator registry entry that only exists after a registration (and is gone while the reconciler restarts a changed server): the service treats the accepted anonymous probe like a 401 — the probe client is closed, the server reports Auth Required, and every session connects with the caller's own token — and the aggregator refuses to attach a shared client to such a server on any path. Previously the shared client 401-looped once the protected pod took over and every session's tool call failed with authorization required until muster was restarted. A behavioral scenario (oauth-sso-forwarding-anonymous-backend) covers both a server created with forwardToken and one that gains it through core_mcpserver_update. (#1135)
A pending-auth registration is no longer dropped by the aggregator's handling of a stale service state-change event. The auth-required hook registers the entry synchronously on the server's 401, while the starting event from the same start is still queued for the aggregator's event handler; the handler declines to deregister a server whose entry says auth_required, but that check and the removal were two steps and the registration could land in between (the RegisteredAt guard did not catch it when the entry was created before the deregistration was timestamped). core_auth_login then answered "Server not found" until the periodic pending-auth heal ran, up to 5 s later. State-change-driven deregistration now decides under the registry lock and keeps entries that require per-session authentication; the paths for a deleted service are unchanged.
The workflow executor's debug logging no longer prints workflow arguments, resolved step arguments, step results, template results or the final result. Those are user data -- a token passed as a workflow input, a credential returned by a step -- and were dumped verbatim with %+v at debug level. The lines now carry the workflow name, step id, argument and result key names, value shapes and sizes only. A behavioral scenario (workflow-args-never-logged) runs a secret through a workflow and asserts it never reaches muster serve's output.
Stored RFC 7591 dynamic client registrations now heal themselves when the authorization server forgets them. Registrations without an expiry lived forever in muster's credential store, so an authorization server that lost its client registry (pro keeps registered clients in memory; its pod was evicted during a node roll) answered every sign-in in the user's browser with 400 invalid_client — a failure muster never saw — until an operator deleted the Valkey entry by hand. Before a sign-in link is issued with stored DCR credentials muster now verifies them: via the RFC 7592 client read when the registration response provided registration_client_uri, otherwise by asking the authorization endpoint with a client_id/redirect_uri-only request whose RFC 6749 §4.1.2.1 failure shape reveals whether the client is still known (a redirect to muster's own redirect_uri: alive; a direct invalid_client: gone; anything else: inconclusive, credentials kept). A registration found dead is dropped and re-created within the same core_auth_login, logged at INFO. invalid_client from the token endpoint during the code exchange, or error=invalid_client on the callback, drops the stored registration too, so the user's retry registers again.
Test framework: expectations are now evaluated by a single implementation for both step kinds, so an assertion means the same thing whether it is written on an MCP tool step or a test_* step. The two paths each carried their own copy of the checks and had drifted three times — json_path (#1036), not_contains and wait_for_state (#1078) were each implemented on one path and silently ignored on the other, so a scenario declaring one on the wrong kind of step passed regardless of the response. Each path now only adapts its own response shape and shares the checks. Two behaviours are unified in the process: error_contains is evaluated whenever it is declared (the test-tool path previously skipped it unless success: false), and a success: false payload now fails an MCP step that expected success (previously checked only for test_* steps). All 186 scenarios are unaffected by both. A new test asserts that every field of TestExpectation is enforced on both paths, and fails the build when an expectation kind is added without being wired up — the class of bug behind #1038.
Test framework: status_code in a step's expected block is now rejected at load time, with a message pointing at json_path. It was accepted by the schema and printed by the reporter but never evaluated by either path, so a scenario asserting an HTTP status asserted nothing. No scenario in the repository used it.
Test framework: a step's retry block (count, delay, backoff_multiplier) is now rejected at load time, with a message pointing at expected.wait_for_state. It was declared on TestStep and range checked by the loader, and nothing else ever read it — so a step declaring retry got exactly one attempt while reading as if it polled, and TestStepResult.RetryCount (printed by the reporter) was always zero. The three steps in the dex-token-forward scenario that used it to poll for eventually-consistent state — backend registration, the OBO reconnect, and the forwarded-token assertion — were therefore flake-prone in precisely the place their author had covered; they now use wait_for_state, which is the mechanism that works. RetryConfig and RetryCount are removed. A new test asserts that every field of TestStep names the file that acts on it, and fails the build when a field is declared and validated but never honoured — the same class of bug as #1038, one level up from expectation kinds.
Test framework: wait_for_state on an MCP tool step no longer discards a first response that already satisfies the step's expectations. The polling loop ignored the response its caller had just obtained and judged the step only from the first tick onward, so every such step paid a mandatory poll interval, and a step was failed outright when re-invoking the tool did not return the same answer — reported against the first, passing response, so the output showed a satisfied payload on a failed step. Steps whose invocation is the assertion (a token-forwarding call, anything not idempotent) were the ones at risk. The test_* path already judged the first response; the two now agree, which is the parity TestEveryExpectationKindIsEnforcedOnBothStepKinds exists to defend. dex-token-forward drops from 2.5s to 0.5s as a result.
Helm: values.schema.json no longer rejects valid chart values. podDisruptionBudget.minAvailable accepts a percentage string ("50%") as well as an integer, podDisruptionBudget.maxUnavailable is now a declared value (the PDB template already honoured it, but the schema refused it as an unknown key), and the free-form gatewayAPI.httpRoute.{labels,annotations} / gatewayAPI.backendTrafficPolicy.{labels,annotations} maps accept arbitrary keys. Setting any of these previously failed at helm install/helm template time with values don't meet the specifications of the schema(s).
Security (aggregator): a muster without inbound OAuth no longer shares one client's downstream logins with every other client. Without a bearer token there is no token family to key a session by, so the transport layer injects the default-user placeholder — and that constant was used as the session key, collapsing every concurrent MCP connection onto a single session. Session-scoped state is keyed by that ID: the OAuth token store, the capability store that backs session-scoped tool visibility (ADR 006), and the session connection pool. So once any one client ran core_auth_login against an OAuth-protected backend, that backend's tools appeared in list_tools for — and were callable by — every other client of the same muster, using the first client's token. Requests that carry no authenticated identity are now keyed by their MCP transport session instead (one key per client connection, per ADR 006 §4.1); stdio reports a constant session ID, so the inherently single-user stdio transport keeps one stable session. Consequence for unauthenticated HTTP: a session ends with the connection, so a reconnecting client re-runs core_auth_login for its downstream backends. Enable muster's own OAuth to get durable per-user sessions that survive reconnects. Covered by the session-multi-user-progressive-auth and session-multi-user-tool-isolation scenarios, whose isolation assertions previously passed vacuously (see below).
Test framework: expected.not_contains is now evaluated for test_* steps. The scenario loader accepted the field and the regular tool path implemented it, but the test-tool path only logged a warning and passed, so every absence assertion written on a test_* step was vacuous — including the session-isolation assertions in session-multi-user-progressive-auth and session-multi-user-tool-isolation, which is why the leak above went unnoticed. It now fails the step when the response contains an unexpected string, matching the non-test-tool path.
Dex cross-client audience scopes are resolved per authorization request instead of once when the OAuth server is built. An MCPServer with forwardToken: true that registers after muster starts now reaches the next login, so the forwarded ID token carries the audience its backend expects. Existing sessions are not repaired: affected users must log in again once. A background refresher re-reads the set every 10 seconds, so no login costs an MCPServer list call, and a read that fails serves the last known set instead of an empty one.
A change in the cross-client audiences muster requests from Dex is recorded as a dex_audiences_changed security audit event, alongside a log line. The set comes from MCPServer requiredAudiences and now changes without a muster restart.
Listing MCPServers reports a read failure instead of reporting an empty list. core_mcpserver_list now reports a failure when the MCPServer definitions cannot be read, auto-start reports the failure instead of starting no servers silently, and periodic resync logs it instead of treating it as "no MCPServers exist".
Helm: the NetworkPolicy and CiliumNetworkPolicy now allow ingress to the prometheus metrics port when the prometheus exporter is enabled. Previously both policies allowed ingress only to the aggregator port, so Prometheus / Alloy scrapes of the /metrics endpoint were blocked (up = 0) even with the ServiceMonitor in place.
oauth.mcpClient.cimd settings now reach the OAuth proxy: operator-configured cimd.scopes are advertised in the served CIMD document (previously always the defaults), and a custom cimd.path no longer breaks CIMD self-hosting (the client ID was derived with the configured path while the OAuth manager recomputed it with the dropped, defaulted one, so the document could be skipped or mounted at the wrong path). The CIMD block was lost in the same config conversions as the postLoginRedirectAllowlist fix below; that duplication is now collapsed — the aggregator carries the merged OAuthMCPClientConfig unconverted (the aggregator.OAuthProxyConfig mirror struct is removed), so future oauth.mcpClient fields reach the OAuth manager without per-field plumbing.
oauth.mcpClient.postLoginRedirectAllowlist is now honored in deployments. The parsed value was dropped in the two field-by-field config conversions between the YAML config and the OAuth manager (internal/app → aggregator.OAuthProxyConfig → oauth.NewManager), so the handler's allowlist was always empty and every redirect request on the start URL was rejected with Rejecting post-login redirect target not in allowlist, degrading connector logins to the static success page. A configured allowlist now reaches the handler; the Post-login redirect allowlist enabled with N entries startup log confirms it.
Reading an aggregated resource or prompt from a backend that reported connected but had no live client crashed muster with a nil-pointer panic (AggregatorServer.ReadResource / GetPrompt dereferenced the client). GetClient now returns an error in that state instead of handing back a nil client, so the tool call fails cleanly rather than aborting the connection.
SSO sessions on token-exchange backends no longer deauthenticate under a re-exchange rotation storm. A long-lived SSO session reconnecting after its login-time ID token expired re-inited SSO in initSSOForSession from the live request context but never persisted that token to the OAuth-proxy store, so muster's background re-exchange (running on a detached context.Background() that can only read the store) fell back to the in-process refresher and rotated the client's mcp refresh token on every ~1s continuous-listen retry (~56×/min for ~15 min); two rotations eventually collided and OAuth 2.1 reuse detection revoked the whole token family → deauth. initSSOForSession now persists the request-context ID token, and the background SSO refresher is routed to mcp-oauth's provider-only RefreshSessionProvider, which repopulates the upstream provider token without rotating the client-facing refresh token. (#37164)

Deploy note: the accompanying mcp-oauth bump (v1.0.10 → v1.2.0) changes the Valkey provider-token storage layout to a single shared entry per user with no legacy read-fallback, so every user re-authenticates exactly once on rollout (existing sessions do not carry over). Optionally flush the affected token keys (token:*, refresh:*, and related meta:* / family:* / user-client set keys under the configured prefix) for a clean cutover; skipping the flush is safe — leftover keys are never read and just linger until TTL.
Team ownership label. application.giantswarm.io/team now renders bumblebee instead of an empty string: the labels helper looked up the annotation under the wrong key (application.giantswarm.io/team) instead of the OCI key io.giantswarm.application.team set in Chart.yaml.

Removed

localMint downstream auth. The auth.localMint MCPServer CRD field and its admission rules, the local-mint broker target type and the target type key, and the oauth.server.tokenExchangeBroker.delegateToSelf config key are removed. Backends that used localMint switch to forwardToken: true and validate the forwarded token against muster's JWKS.

Upgrade note: applying the new CRD makes the Kubernetes API server silently prune spec.auth.localMint from existing MCPServer resources — the schema is structural, so there is no validation error and a GitOps apply succeeds. The affected backend is then left with no downstream-auth mode: muster's calls reach it unauthenticated, the backend answers 401, and its tools disappear from sessions without any apply-time failure. Migrate every MCPServer that sets auth.localMint to auth.forwardToken: true (with the backend configured to trust muster's issuer/JWKS) before or together with this upgrade.
The X-Actor-Token request header. The actor token is presented once as the RFC 8693 actor_token parameter at /oauth/token; /mcp requests carry only the issued on-behalf-of token as the bearer.

Changed

muster asks for MCP protocol version 2025-11-25 instead of 2024-11-05 whenever it acts as an MCP client: every downstream transport (stdio, SSE, streamable HTTP, dynamic-auth) and the agent client. A backend that supports only an older revision answers with that one and keeps working. (#1029)
core_service_status and core_mcpserver_get report protocolVersion for a connected MCP server: the revision that backend answered with during the handshake. The value can differ per backend.
The aggregator no longer advertises the resources.subscribe capability. resources/subscribe now returns METHOD_NOT_FOUND. resources.listChanged is unchanged and still fires. (#1030)
muster refuses to start when a tokenExchangeBroker target lacks dexTokenEndpoint, naming the misconfigured audience, instead of surfacing an unattributed error on the first exchange request. The chart's values.schema.json requires the key as well.
A token-forwarding connect failure now logs the forwarded token's issuer (the iss claim only — never the token) with a hint that the backend must trust that issuer's JWKS, so a backend that does not yet trust muster's issuer is attributable from the log instead of a bare initialize error.
The aggregator forwards the validated inbound bearer to forwardToken backends on each request instead of issuing a per-backend token, so the on-behalf-of token (including its nested act delegation chain) reaches the backend byte-identical. When the request carries no forwardable bearer (the background listen stream, opaque-token sessions), the session's stored upstream ID token is forwarded as before.
On-behalf-of token exchange at /oauth/token no longer requires a broker target: a request without an audience takes mcp-oauth's self-issued path and the issued token's aud defaults to muster's resourceIdentifier. Requests with an audience keep the brokered downstream Dex exchange.
The self-issued exchange only mints tokens for muster's own audience: TokenExchangeAllowedResources is pinned to the resourceIdentifier, so a request naming any other RFC 8707 resource is refused with invalid_target. Previously the allowlist was unset (disabled) and any caller holding a trusted-issuer token could obtain a muster-signed token for an arbitrary audience. Tokens for other audiences go through the brokered path, which requires client authentication and a per-client audience allowlist.
M2M (machine-to-machine) token exchange. The oauth.server.tokenExchangeBroker.workloadAudiences, workloadGroupGrants, and actorDelegationPolicy config keys are removed, along with broker-granted identity injection (granted.subject / granted.groups). On-behalf-of (OBO) delegation is unchanged and now accepts any actor validated against the trusted issuers; the impersonated subject's downstream authorization governs access.
The github-app broker target type and its githubApp config block. The only remaining broker target type is oidc-exchange.

Fixed

muster agent --mcp-server now honors MUSTER_OAUTH_CALLBACK_PORT. The agent's OAuth flow hardcoded port 3000 for its AuthManager, while the environment variable was read in exactly one place that this path never consulted — so with 3000 occupied, setting the variable made muster auth login work while the agent still bound 3000 and failed, and the port-in-use message interpolated the env-derived port, naming a port the failing code never used. cli.GetCallbackPort is now the single resolver for every OAuth entry point, and the cmd package's duplicate DefaultOAuthCallbackPort constant is removed. docs/reference/cli/agent.md lists the variable now, instead of leaving it documented as belonging to muster auth login alone. The re-auth failure message the MCP server returns no longer names port 3000 either: the number was written into its text, so it stated the default even when the flow had been configured onto another port -- and when the cause really is a bind failure, the wrapped error printed above it already reports the port that was tried. (#1111)
The CLI auth handler registry no longer serves a closed adapter, and get-or-create no longer orphans one. Get-or-create was a check-then-act -- GetAuthHandler, construct, Register, GetAuthHandler again: handlerMutex makes the individual registry calls atomic but leaves the composite racy, so concurrent callers each built and published their own adapter and every loser was orphaned without Close(). Both sites that spelled it that way -- the auth commands and the ToolExecutor behind muster list/get/call -- now go through the new primitive. (noSilentRefresh still lives on the shared adapter, so the caller that registers it fixes the value for the process; that is unchanged here, and no muster command runs a second registrant.) Separately, AuthAdapter.Close() tore down its managers but never unpublished, so api.GetAuthHandler() kept returning a closed adapter that — carrying no closed flag — silently re-created managers on itself instead of failing. New api.GetOrRegisterAuthHandler runs the check, the construction and the publish under one write lock so the factory runs exactly once, and Close() now clears the registration through api.UnregisterAuthHandler (a no-op when the adapter is not the registered one). (#1111)
Token re-exchange on the persistent oidc-exchange connection now uses the resolved per-connection config. The initial-connection path handed the refresh closure the shared spec-only TokenExchange pointer (which, since #940, deliberately never carries the runtime-resolved credentials or appended requiredAudiences scopes), so every re-exchange after the first token neared expiry ran without client credentials — failing outright against a Dex requiring client auth and evicting the session back to Auth Required — and without the required audiences, minting tokens the downstream server rejects. (#942)
Data race in the token-forwarding connection header function, which mutates a per-connection failure counter that mcp-go invokes concurrently from the listener goroutine and tool-call goroutines, so under load the stale-connection eviction could double-fire (double eviction/revoke). Counter access is now synchronized. (#939)
Token-exchange (oidc-exchange) backends no longer get stuck in Auth Required once their exchanged token expires. The persistent aggregator connection now re-exchanges a fresh token before expiry, and evicts itself only when the subject token can no longer be refreshed.
MCPServers using RFC 8693 token exchange no longer restart on every reconcile pass (~every 10-15s), which previously caused intermittent authentication required / user not authenticated to server errors on tool calls landing in the restart window. (#37060)
filter_tools query ranking now weights tool-name matches over description matches and down-weights the ubiquitous CRUD verbs (list, get), so an intent query is driven by its discriminating noun rather than a generic verb. Previously name and description were concatenated into one BM25 document scored token-equally, so list pods surfaced pagerduty x_pd_list_* and core_*_list above x_kubernetes_list and the pod workflows (the generic list token dominated). The ranker is now a BM25F-style field-weighted lexical scorer: name and description are scored as separate length-normalised fields, name matches weigh higher, and stop-verb contributions are scaled down (down-weighted, not dropped, so a list-only query still ranks list-shaped tools). (#931)
Workflow validation no longer requires a top-level tool on parallel/forEach container steps. The reconciler's validateWorkflow (the path that sets status.valid/status.validationErrors for CRD-applied workflows) unconditionally demanded step.tool, so any workflow whose only tool-less step was a parallel group or forEach loop was wrongly marked invalid (step '<id>': tool is required) even though it executed correctly. Validation now mirrors the structured create/validate path: a step must specify exactly one of tool/forEach/parallel, container sub-steps still require their own tool, and status.referencedTools now includes sub-step tools. (#928)
Workflow listing no longer rebuilds the session tool set once per workflow. getWorkflows evaluated each workflow's availability independently, and every check resolved the caller's full session-scoped tool set (GetAllToolsForSession across all backend MCP servers) from scratch — an O(workflows) blow-up that made core_workflow_list take ~30 s for ~280 workflows. The list path now installs a request-scoped memo (api.SessionToolMemo) so the session tool set is resolved once for the whole request and shared across all per-workflow availability checks. Single-workflow paths are unchanged (no memo, same per-call behavior).
local-mint backends are now treated as session-based for tool registration. isServerSSOBased did not recognize the local-mint auth mode, so the aggregator event handler attempted global registration for a local-mint backend (which has no global persistent client), failed on 401, and the backend's tools never reached the caller. local-mint now joins token forwarding and token exchange as a per-session auth mode.
localMint backends connected during SSO bootstrap now mint on the on-behalf-of delegation path. When initSSOForSession rebuilt its detached background context it carried the subject bearer and ID token but dropped the inbound X-Actor-Token, so a bootstrap-established connection minted with no actor (the human subject alone) rather than the agent acting on the human's behalf. The per-request credential tokens (subject bearer, actor token, ID token) are now carried into the bootstrap context as one unit, so the bootstrap and live-request paths mint identically.
Connected MCP clients now receive notifications/{tools,resources,prompts}/list_changed when a backend connects after the session was opened (localMint/OBO background bootstrap, post-restart re-init), so late-connecting backends no longer stay invisible for the session's lifetime.
OBO sessions now connect localMint backends. After the emission fix that adds email to muster-minted OBO JWTs, fireOnAuthenticated fires for OBO requests, but the onAuthenticated callback returned early at the idToken == "" guard (written to avoid 403-spam for post-restart Dex sessions). The guard is now narrowed: OBO sessions (detected via userInfo.ActorSubject) are allowed through. The inbound OBO bearer is threaded into the detached initSSOForSession background context and used as the RFC 8693 subject token in EstablishConnectionWithLocalMint, falling back from the Dex ID-token lookup when none is present.
On-behalf-of tool calls now reach the backend instead of failing closed. A muster self-issued OBO bearer (sub=human, act=agent) re-presented at the front door is signature-validated with no token-store entry, so it previously carried no session ID; backend tools that require session auth then rejected it and the connection fell back to the agent ServiceAccount. Bumping mcp-oauth to v0.18.0 makes the ValidateToken middleware assign a session to every validated token (FamilyID when present, else the deterministic bearer-derived ID), so the existing session lookup resolves the OBO bearer and the per-backend localMint runs as the human. No muster code change.

Added

Durable workflow execution tracking (#930, supersedes the closed-but-unimplemented #35): workflow execution records now survive process restarts and are visible across replicas when muster runs in Kubernetes.
- WorkflowExecution CRD (muster.giantswarm.io/v1alpha1, kind WorkflowExecution, short name wfe). Each workflow run is persisted as one immutable record (metadata.name is the execution UUID) carrying the workflow name, status, timings, input, result, per-step records, and a truncated flag. It has no status subresource — it is an append-only record, not a reconciled resource. The chart's ClusterRole gains workflowexecutions permissions and the CRD ships in both the muster-crds chart and the app chart's crds/ directory.
- Backend selected by deployment mode. In Kubernetes mode the existing ExecutionStorage seam is backed by the WorkflowExecution CRD (via the controller-runtime client already embedded in MusterClient); standalone muster serve keeps the filesystem backend. core_workflow_execution_list/get behave identically against either backend. List filtering uses muster.giantswarm.io/workflow and muster.giantswarm.io/status label selectors, sorting and paginating in memory.
- Payload guard. Before persisting, an execution record whose marshaled size exceeds 256 KB (well under etcd's object limit) has its oversized result and step results replaced with a truncation marker and is flagged truncated, so a large workflow result can never make a record unpersistable in either backend.
- Retention GC. A background goroutine prunes execution records that are older than 21 days or beyond the newest 10000 (burst safety cap), keeping the store bounded without manual cleanup. The clock is injectable so the prune logic is deterministically unit-tested. Retention bounds are constants for v1.
- Execution metrics. New OTel instruments muster.workflow_executions (→ muster_workflow_executions_total), muster.workflow_execution.duration (→ muster_workflow_execution_duration_seconds, keyed by workflow + status), and muster.workflow_execution.store_errors (→ muster_workflow_execution_store_errors_total) are exported via the existing OTel→Prometheus pipeline. A persistence failure is now logged at error level and counted instead of being silently warned, so empty dashboards become an observable signal.
oauth.server.trustedIssuers[].allowPrivateIPJWKSHosts ([]string): host-scoped alternative to allowPrivateIPJWKS. The issuer's jwksUrl may resolve to a private IP only when its hostname matches one of these values; all other hosts keep the SSRF guard. Maps to mcp-oauth's TrustedIssuer.AllowPrivateIPJWKSHosts. Prefer it over the blanket bool for a known in-cluster JWKS endpoint (e.g. a Dex fronted by an internal LB whose public hostname resolves to a private VIP).
oauth.server.tokenExchangeBroker.delegateToSelf (default false): when enabled, a delegated (on-behalf-of) token exchange that carries an actor_token but omits the RFC 8707 resource is bound to muster's own resourceIdentifier, so an agent STS client that cannot set a resource itself still receives a token muster accepts back and re-mints per backend on the localMint path. Only the delegation path is affected; a resource-less plain exchange still errors. Requires mcp-oauth v0.16.0.
Chart values now document the muster-valkey persistence model as a deliberate cache-only choice (RDB-only is intentional; every critical record is reconstructable at startup), with a per-record-type recovery table, and explain why AOF is not enabled (it does not survive PVC loss, and a config-flip restart is a data-loss footgun). Documentation only; no behaviour change. (#884)
App-owned CRDs: the muster application chart now ships its own MCPServer and Workflow CRDs in helm/muster/crds/ (Helm 3 crds/ directory), with helm.sh/resource-policy: keep baked in. Combined with Flux install.crds: CreateReplace / upgrade.crds: CreateReplace on the muster HelmRelease, the CRDs travel with the app at the same version and upgrade atomically on every release, removing CRD-vs-app drift. The standalone muster-crds chart is retained for non-Flux/standalone consumers; make generate-crds now writes both locations from the same Go-type source. Chart docs (NOTES, README, values) now explain the CRD handling for plain-Helm users: fresh helm install includes the CRDs automatically, while CRD upgrades must be applied out-of-band (helm show crds giantswarm/muster | kubectl apply --server-side -f -) because Helm does not manage crds/ upgrades natively — an intentional, still-current Helm design decision (HIP-0011), unchanged in Helm 4. The previous bundle-split docs that claimed the app chart "no longer ships the CRDs" were corrected.
Workflow result model overhaul (#873, #874, #875): step result referencing, LLM-facing output, and result shaping are now decoupled and use one expression language.
- Referencing decoupled from output (#873). Every step result is now always referenceable by later steps as {{ .results.<id>.<field> }} — regardless of any flag — so chaining a single value from one step into the next no longer forces that step's entire result into the response. A new per-step/sub-step output flag controls whether a result is included in the returned document; store remains as a deprecated, backwards-compatible alias. forEach, parallel, and failure-path results are all referenceable consistently. Workflows that still use store now log a one-line deprecation warning naming the affected steps — both on the structured create/validate path and on the CRD reconciler (so a kubectl apply-ed workflow is nudged too).
- Output template (#874). A workflow may declare a workflow-level output template: a templated object rendered once after all steps complete, against .input / .results / .vars, and returned in place of the default {execution_id, workflow, status, input, steps[], ...} response. It can select nested fields and combine values across steps while preserving JSON structure (numbers stay numbers, arrays stay arrays), letting a workflow return a small, shaped response. When omitted, the default response is returned unchanged. Because the output template replaces the response, per-step output/store flags no longer affect the returned document when an output template is declared; this is flagged with a one-line authoring warning naming the now-inert flags. Type preservation (no lossy coercion): an output template leaf's type comes from the value it evaluates to, never from how its rendered text looks. A bare reference path (e.g. "{{ .results.pods.items }}") keeps its original JSON type; a single-action computed leaf keeps the real type of its result, so a numeric expression like "{{ len .results.events.items }}" is a number while a computed string keeps its exact string form. This means values whose form matters — versions, IDs, zero-padded values like "08" or "1.20" — are preserved as-is, with no coercion and no quote workaround. A leaf that mixes literal text with actions (e.g. "v{{ .v }}") renders to a string. Non-finite values (NaN/Inf) are kept as strings.
- Unified expression language (#875). Condition jsonPath / expectNot.jsonPath paths now use the same path navigator as step args and templates, gaining array indexing (e.g. items[0].name) and an optional full Go-template form where the result is exposed as .result (e.g. "{{ (index .result.items 0).name }}"). Existing dotted paths keep working. The duplicate engine.resolvePath and getValueFromPath navigators were collapsed onto a single implementation.
- Debug response escape hatch and non-discarding output-template errors (#877). A workflow that declares an output template can now be inspected without temporarily removing the template: pass the reserved _debug: true execution argument and the full response (execution_id, status, and steps[] with every recorded step result, not just output-flagged ones) is returned with the rendered output template alongside it under output. The _debug arg is stripped before validation and step execution, so it never collides with a workflow's own arguments and is not passed to step tools. Separately, an output-template render error no longer discards the results of steps that already succeeded: the workflow still fails loud (the error is returned and the result is flagged isError), but the response now carries every recorded step result plus an output_error message, so the underlying data stays recoverable for debugging an output-template typo. Default (non-debug, template-renders-cleanly) behaviour is unchanged: only the rendered output template is returned when output is set, the default response otherwise.
Cheap, ranked, faceted tool discovery tier (#868): filter_tools is now a discovery tier distinct from execution, so finding a tool no longer scales with the full descriptive weight of every candidate. Against a ~280-workflow fleet a broad filter_tools(pattern="*workflow*") call returns a bounded summary page (~3 KB) instead of the full-catalogue dump (~330 KB) — measured ~100x smaller.
- Bounded + summarised pages. filter_tools gains limit (default 25) and offset, and the response carries total and a truncated flag. Discovery now defaults to a one-line summary per tool with no input schema; the authoritative full description and schema remain available via describe_tool, or by passing include_schema=true.
- Ranked query mode. A new query argument relevance-ranks matches with a dependency-free lexical ranker (Okapi BM25 over name + summary) and returns them best-first with a score, dropping non-matching tools. Lexical ranking needs no embedding index; embeddings can be a later upgrade.
- Label facets. Workflow CRD metadata.labels are now propagated onto the workflow's execution tool and can be filtered in discovery via a labels facet (key=value; all must match), letting clients scope a lookup to a labelled subset.
- Agent REPL parity. The filter command (aliases find/search) now exposes the discovery tier: options are given as key=value pairs (pattern, description, query, labels as k=v,k2=v2, case_sensitive, detailed, limit, offset), with a bare token still accepted as the name pattern. It now reports the page size against the total match count and the catalogue size, and prints the next offset when more matches exist (previously it mislabelled the page size as the match count and hid truncation). Unknown options and stray extra arguments now fail loudly with the list of valid keys instead of being silently ignored, the name column auto-sizes to the longest tool name on the page, and tab-completion only offers options that have not been supplied yet.
- Existing list_tools / call_tool / describe_tool behaviour is unchanged, as is list_core_tools (which keeps full descriptions and schemas).
Workflow control flow (#865): four genuinely useful constructs are now implemented end to end (CRD types, internal API types, executor, structured create/validate path, and step JSON schema):
- condition.template — a boolean Go-template gate evaluated in-process (e.g. condition: {template: "{{ eq .input.env \"production\" }}"}).
- forEach — a sequential loop that runs a flat body of sub-steps once per item of a list, binding the current item to {{ .vars.<as> }} (default item) and the index to {{ .vars.<as>_index }}. A stored sub-step is addressable per iteration as {{ .results.<id>_<index> }} (the plain {{ .results.<id> }} keeps the last iteration's result).
- parallel — a group of sub-steps executed concurrently; siblings are independent (each resolves arguments from the pre-group state).
- spec.onFailure — best-effort cleanup/rollback sub-steps run when the workflow fails on a step that does not allow failure.
- The "exactly one of tool, forEach, or parallel" rule is enforced by the CRD itself via a CEL validation rule, so a malformed step is rejected at kubectl apply time, not only through the structured create path.
- Step/sub-step conditions are now validated structurally on every authoring path: a condition must set exactly one of template, tool, or fromStep, and a tool/fromStep condition must declare expect or expectNot. Both rules are enforced at kubectl apply time via CEL on the Workflow CRD and in the structured workflow_create/workflow_validate path. Previously a tool/fromStep condition without an expectation silently fell back to "expect the call to fail", and a kubectl-applied condition was not checked at all.
GET /health now responds 200 on the aggregator port regardless of OAuth configuration, so Kubernetes liveness/readiness probes work without patching the chart.
RegisterServer and DeregisterServer aggregator events and MCPServer reconcile entry are now logged at Info level, making freshly-restarted pod lifecycle visible without --debug.
oauth.server.allowedOrigins (comma-separated) is now wired into the mcp-oauth CORS AllowedOrigins list. Previously declared but never read; empty value keeps CORS disabled (default).
oauth.server.trustedIssuers[].acceptedTypHeaders: accepted JWT typ header values for Bearer tokens from a trusted issuer. Empty keeps the RFC 9068 default (at+jwt). Kubernetes ServiceAccount tokens carry no typ header; use [""] to accept them.
oauth.server.trustedIssuers[].subjectClaim: sources the canonical subject (the sub of any token minted from the identity) from a claim other than sub. Empty keeps the standard sub. Set it to email for Dex, whose sub is opaque, so an OBO token minted from a Dex subject token carries the user's email.
Brokered RFC 8693 token exchange (#831): external confidential clients can POST a token-exchange request with an audience parameter to /oauth/token and receive a token minted by the audience's downstream Dex. New oauth.server.tokenExchangeBroker config block (per-client audience allowlist, audience → downstream Dex target mapping with per-target scopes and credential secret refs). Requires mcp-oauth >= v0.3.0; subject tokens are validated against trustedIssuers.
oauth.server.tokenExchangeBroker.targets[].type: credential provider discriminator for broker targets. Defaults to oidc-exchange (downstream Dex RFC 8693 exchange) when omitted; additional provider types will be added in future releases.
oauth.server.tokenExchangeBroker.targets[].type: github-app mints GitHub App installation tokens. Configure via githubApp.appId, githubApp.installationId (or githubApp.owner + githubApp.repo for auto-discovery), githubApp.privateKeyRef (RSA PEM in a Kubernetes Secret), and optional githubApp.repositories / githubApp.permissions scope restriction.
oauth.server.tokenExchangeBroker.targets[].type: local-mint mints a muster-signed RFC 9068 JWT locally. Requires enableJWTMode: true. The issued token carries sub = the validated human subject, the subject's email and groups claims (plus any broker-granted groups), and act = the validated agent SA nested over any prior delegation chain on the subject token, signed by muster's own access-token key. Downstream services that trust muster as an issuer receive the subject's identity and the full delegation chain without a separate Dex exchange, so group-to-tenant and email-to-org routing resolve correctly.
muster events --follow now honors --output: json streams newline-delimited JSON (one object per line, ready for jq), yaml streams one YAML document per event, and the default table/wide formats stream an aligned one-line-per-event view. Previously the output format was silently ignored in follow mode. The human follow line now orders fields to match the static muster events table (timestamp, type, resource, reason, message) and highlights Warning events when stdout is a terminal.

Fixed

Kubernetes events now carry their structured detail. The api.EventManagerHandler boundary previously dropped everything but the object name/namespace, so every rendered event message lost its contextual fields — failure events showed no error, "created with N steps" showed no count, and step events had empty step IDs/tools. A new CreateEventWithData carries structured EventData (error, step count, duration, step ID/tool, execution ID, condition result, tool names) end-to-end so the message template renders the real detail; all emitters (MCPServer service/aggregator, workflow, token exchange/forwarding) were updated. The event type (Normal/Warning) is now consistently derived from the reason.
Kubernetes events for MCPServer runtime lifecycle (start/stop/fail/health/recovery) are now associated with the MCPServer CRD in the configured muster namespace instead of being hardcoded to default. On a real cluster the events previously landed in default with an empty object UID, orphaned from the CRD and invisible to kubectl describe mcpserver.
muster events --follow now works via real server-side streaming. It previously blocked forever waiting on an MCP notification the server never sent (the streaming path was an unimplemented stub). Follow is now implemented end-to-end: core_events with follow=true returns the events seen so far and registers a per-session watch on the aggregator; subsequent events are pushed to the client as notifications/muster/event MCP notifications, sourced from a native Kubernetes watch (Kubernetes mode) or an fsnotify watch on the on-disk event log (filesystem mode) — no client-side polling. The stream is torn down when the client disconnects, starts a new follow, or the server shuts down.
localMint now refuses to mint when the subject token carries a non-empty email whose email_verified is not true, on both the delegation (X-Actor-Token) path and the no-actor path. Previously the check ran only on the delegation path, so a pre-exchanged subject token already carrying an act chain (a single bearer, no actor header) could mint with an unverified email. A ServiceAccount token, which carries no email, is unaffected.
Workflow CRD CEL cost budget (#865 follow-up): the Workflow CRD from 0.8.0 was rejected by the Kubernetes API server (k8s >= 1.34) with "x-kubernetes-validations estimated rule cost total for entire OpenAPIv3 schema exceeds budget" — the WorkflowStep and WorkflowCondition "exactly one of" guards used a [...].filter(x, x).size() == 1 list-comprehension form whose estimated cost, multiplied across the four nested condition sites (steps[], steps[].forEach.steps[], steps[].parallel[], onFailure[]), blew the schema-wide budget (3.2x over). Rewritten to the cheap additive form (has(self.a)?1:0) + (has(self.b)?1:0) + (has(self.c)?1:0) == 1, which estimates far lower and applies cleanly on k8s 1.34 while enforcing identical semantics. Without this, muster-crds 0.8.0 cannot install and the muster 0.8.0 rollout stalls.
Workflow documentation (#865): docs/how-to/workflow-creation.md and docs/how-to/ai-workflow-optimization.md were rewritten to describe only features the engine implements; all example templates now use the correct {{ .input.<arg> }} context (the engine renders with missingkey=error, so the previously documented {{ .<arg> }} form errored at runtime). docs/reference/crds.md template syntax was corrected, the stray spec.name removed from examples, and the dead outputs field replaced with store: true guidance.
Documentation: removed all references to a hallucinated muster configure ... CLI from docs/how-to/ai-troubleshooting.md and docs/how-to/ai-agent-integration.md. muster has no configuration CLI — configuration is file-based (~/.config/muster/config.yaml), entities are created with muster create, and debugging uses muster serve --debug. The invented context/tool-suggestion and alert/cache/limit configuration blocks built around that command were dropped.
Documentation: completed a repo-wide accuracy pass removing the remaining hallucinated CLI surface and config from the docs. docs/how-to/ai-troubleshooting.md and docs/how-to/troubleshooting.md were rewritten around the real command set; the invented muster status, muster logs, muster describe, muster validate, muster restart, muster metrics, muster backup/restore, muster support-bundle, muster config show, the fictional muster serve --port/--host flags, and the kind: Config logging CRD were replaced with the real equivalents (muster get/list/check/call, muster serve --debug/--silent, the /health endpoint, OpenTelemetry/OTLP for metrics and traces, and file-based config). The same fixes were applied in docs/how-to/mcp-server-management.md (muster metrics/muster logs mcpserver, and a Cursor mcpServers entry that used curl as the MCP command) and docs/reference/events.md (muster restart mcpserver).
Documentation: corrected workflow template/feature hallucinations beyond #865 in docs/how-to/advanced-scenarios.md, docs/how-to/ai-agent-integration.md, docs/reference/mcp-tools.md, docs/reference/configuration.md, docs/reference/api.md, docs/explanation/orchestration.md, and docs/explanation/design-principles.md: example templates now use the real {{ .input.<arg> }} / {{ .results.<step-id> }} context (engine renders with missingkey=error), the unsupported enum/examples/pattern arg keywords and stray spec.name/outputs fields were removed, and the fabricated spec.triggers event-driven workflows, custom TemplateFuncs map (templates use the Sprig library), invented muster_* metric names, and non-existent per-step retry/on_failure/error_handling fields were replaced with the real condition.template, workflow-level onFailure, and OpenTelemetry behaviour.
Cross-cluster RFC 8693 token exchange now requests an id_token (was access_token). The exchanged token is forwarded as the downstream bearer and must serve as the user's OIDC identity; Dex's default access token is opaque, so mcp-kubernetes (strict --downstream-oauth) could not use it for Kubernetes OIDC and denied tool calls with authentication required: please log in to access this resource, even though the connection reported Connected [SSO: Exchanged]. Requesting an id_token yields a JWT whose aud carries the configured requiredAudiences, so mcp-oauth accepts it via the forwarded-ID-token (SSO) path — mirroring the token-forwarding behaviour. mcp-prometheus and other identity-only downstreams were unaffected and keep working.
localMint now carries a human on-behalf-of identity to the backend. When an agent reaches muster with a muster-issued on-behalf-of token (sub=human, act=agent), localMint presents it as the exchange subject and the broker re-binds it to the backend audience, preserving the delegation chain, so the backend sees sub=human, act=agent instead of falling back to the agent ServiceAccount.
localMint connections no longer 401-loop on their background listen stream. The streaming listener runs on a context with no inbound headers, so the mint previously failed closed and the backend rejected the unauthenticated stream once per second. The connection's subject and actor are now bound at creation and used as the fallback when the request context carries none.

Changed

Token-exchange spec-vs-runtime handling consolidated onto api.TokenExchangeConfig (#942): the new WithResolvedRuntime method is the single place that stamps the per-connection runtime state (resolved client credentials, appended requiredAudiences scopes) onto a value copy, and SpecOnly is the single place that strips it for CR comparison. Replaces the duplicated build logic in the aggregator's connection and tool-call paths and the hand-rolled field clearing in the reconciler comparison. WithResolvedRuntime returns a distinct ResolvedTokenExchangeConfig type which the token-refresh closures now require, so handing them the shared spec-only registry definition (the #944 bug) no longer compiles; the tool-call path's refresh wiring is now pinned by its own regression test as well. No behavior change.
Kubernetes event emission is now always on. The previous opt-in gate (--enable-events flag and events: config field) and its "disabled by default" framing were removed; events are a core observability feature that works in both Kubernetes (native Events) and filesystem (on-disk log) modes. The muster serve --enable-events flag is retained as a hidden, deprecated no-op so existing scripts and unit files keep working after upgrade (it prints a deprecation notice and has no effect); the removed events: config key is silently ignored.
Kubernetes event spam reduction: the high-volume per-session MCPServerTokenForwarded/MCPServerTokenExchanged Normal events now log at debug level instead of writing a Kubernetes Event on every session's connection to every SSO server. Token failures are still surfaced as Warning events. The Kubernetes backend now emits events through a client-go EventRecorder/EventBroadcaster so duplicate events aggregate into a single object with a Count and get per-key rate limiting, and the per-poll MCPServerHealthCheckFailed emission is gated on the healthy→unhealthy transition rather than firing every 30s for every unhealthy server. These were the dominant event-volume sources that previously made the feature noisy.
Update mcp-oauth to v0.13.2: the trusted-issuer JWKS client used for allowPrivateIPJWKS now honors the process CA bundle, so a backend validating muster's in-cluster JWKS over TLS with an internal CA no longer fails with x509: unknown authority.
Workflow field-name casing is now consistent across authoring surfaces (#865): the structured workflow_create/workflow_update/workflow_validate tool path now accepts the canonical camelCase names used by the CRD and the documentation (allowFailure, fromStep, expectNot, jsonPath) in addition to the previously released snake_case aliases (allow_failure, from_step, expect_not, json_path), and the advertised tool JSON schema now lists the camelCase names. Previously a workflow authored from the (camelCase) documentation was silently mis-parsed through the MCP tool path — e.g. allowFailure was dropped, so a step meant to tolerate failure halted the workflow.
Broker credential minting extracted behind a CredentialProvider interface and an oidc-exchange provider dispatched through a registry (internal/oauth). No behaviour change; the oidc-exchange provider preserves per-(endpoint, connector, user) token caching.
Update mcp-oauth to v0.9.0: server.TrustedIssuer.SubjectClaim sources the canonical subject from a configurable claim, wired through oauth.server.trustedIssuers[].subjectClaim.
Update mcp-oauth to v0.8.0: server.AcceptTrustedIssuerToken for accepting a TrustedIssuers-validated bearer as a forwarded credential with the same ext-<hex> session-ID derivation as AcceptForwardedIDToken.
Update mcp-oauth to v0.7.1: server.LocalMintExchanger for local RFC 9068 JWT minting; RFC 8693 actor_token validation and act claim support; providers.UserInfo.ActorIssuer/ActorSubject; oidc.IDTokenClaims.Act auto-decoded from act.
Update mcp-oauth to v0.4.0.
Update mcp-oauth to v0.3.1: forwarded ID tokens (trustedAudiences) are no longer hard-rejected by the trusted-issuer Bearer branch when the same issuer is also configured in trustedIssuers — fixes Backstage AI-chat SSO token forwarding returning 401 (typ header is "", expected "at+jwt") on deployments with the token-exchange broker enabled.
Update mcp-oauth to v0.3.0 (server-side RFC 8693 token-exchange grant with pluggable Exchanger).
muster version now derives its version from the Go build info (runtime/debug) stamped from the VCS tag, instead of a hand-maintained literal in pkg/project. Release builds run on the tagged commit, so the binaries report the clean tag version; off-tag builds report a pseudo-version and tag-less builds report dev. gitSHA/buildTimestamp are still injected by architect's go-build ldflags. Removes the need to bump the version literal on every release. The scratch files architect's go-build writes into the worktree (the per-arch muster-<os>-<arch> binaries, .ldflags, and .platforms) are now gitignored so an untracked artifact doesn't mark the build +dirty in the embedded version. Validated end-to-end in CI: all six architectures embed the clean tag version with vcs.modified=false.

Removed

Removed the --enable-events flag on muster serve and the events: config/Helm value. Event emission is no longer gated; the flag and field are gone (existing configs that still set events: are harmlessly ignored).
Pruned six event reasons that were defined, templated, and documented but never emitted anywhere: MCPServerReconnected, WorkflowAvailable, WorkflowToolsDiscovered, WorkflowToolsMissing, WorkflowToolUnregistered, and the legacy WorkflowExecuted. Their constants, message templates, and reference-doc entries were removed so the documented event set matches what muster actually emits.
Removed the dead CreateEvent/CreateEventForCRD methods from the api.EventManagerHandler interface (replaced by CreateEventWithData); the documented doc.go examples that called them with the invalid reason "Created" were corrected.
WorkflowStep.outputs (#865): the field was present on the CRD and copied by the adapter but never read by the executor (dead code). Removed from the CRD, internal types, conversion, and step JSON schema. Use store: true and reference the result as {{ .results.<step_id> }} instead.
MCPServer.status.consecutiveFailures, .lastAttempt, and .nextRetryAfter are no longer updated by the reconciler; the retry state machine that drove them was removed in a prior release. The fields remain on the CRD for forward compatibility.
oauth.server.enableHSTS, oauth.server.tlsCertFile, and oauth.server.tlsKeyFile config fields removed; they were declared and YAML-parsed but never read anywhere in the codebase.

Fixed

forwardToken: true (and tokenExchange) MCPServers now work for callers presenting a forwarded ServiceAccount token with no browser auth-code flow. The first request for such a new session connects the session's SSO backends synchronously before the request's MCP handler runs, so the caller's initial tools/list and call_tool succeed instead of returning no tools (auth_required / "authentication context missing, no active session"). Concurrent first requests for the same session are deduplicated.
Raw Kubernetes ServiceAccount projected tokens validated via oauth.server.trustedIssuers now drive per-target RFC 8693 token exchange directly. Previously, injectExternalIDToken only tried the TrustedAudiences path (AcceptForwardedIDToken), which returns ErrTrustedAudienceMismatch for SA tokens (their aud is muster's own resource identifier, not a TrustedAudiences entry); the bearer was dropped and the downstream exchange failed with an empty subject token. The fix adds a fallback to the new AcceptTrustedIssuerToken API (mcp-oauth >= v0.8.0) on mismatch. The same ext-<hex> session-ID derivation is used, preserving cross-hop audit-log correlation. Closes #805 Issue 3.
Bump mcp-oauth to v0.4.2, which makes the trusted-issuer JWKS cache rotation-safe: a subject token presenting a kid absent from the cached JWKS now triggers a single bounded refetch (rate-limited per JWKS URI) and retries verification before rejecting. Previously, a Dex signing-key rotation made the token-exchange broker reject every current user token with subject_token_validation_failed until the muster pod was restarted (the shared broker took down all downstream audiences at once). Closes #847.
Bump mcp-oauth to v0.4.1, which RFC 6749 §2.3.1-encodes client credentials in token-exchange Basic auth. Cross-cluster SSO token exchange previously failed with invalid_client for downstream clusters whose muster-token-exchange-* Dex client secret contained + (decoded to a space on the wire); base64-std secrets with only / and = were unaffected, which is why some clusters worked and others did not.
ssoPoolMissNeedingInit now detects pool misses for token-forwarding servers in addition to token-exchange servers, so warm sessions (authAlive=true after pod restart) trigger initSSOForSession for forwarding servers with empty connection pools. Previously, forwarding servers registered during a restart were inaccessible until the user manually re-authenticated to muster.
establishSSOConnection now treats a pool miss as stale state for token-forwarding servers (clearing the Valkey auth entry and re-establishing the connection), matching the existing behaviour for token-exchange servers.
After an idle period, getIDTokenForForwarding now attempts an in-process upstream provider refresh (Server.RefreshSession) when the proxy store has no valid ID token. On success the store is repopulated by TokenRefreshHandler and the fresh token is forwarded, avoiding 401 Unauthorized errors without requiring re-authentication. Closes #549.

0.3.12 - 2026-06-10

Changed

Release binaries now include darwin/amd64, darwin/arm64, windows/amd64, and windows/arm64 alongside the existing linux targets. Windows binaries are named muster-windows-<arch>.exe.

Fixed

Update mcp-oauth to v0.2.199: JWT access tokens issued for grants without an RFC 8707 resource parameter now carry an aud claim defaulting to the server's resource identifier (RFC 9068 §2.2), instead of an empty audience that JWT-validating gateways (e.g. agentgateway) reject with 401 InvalidAudience. Existing grants self-heal on their next token refresh.

Added

AllowedClaims in TrustedIssuer, drop KubernetesSATrusts, fix JWT signing key wiring (#772) (04b5bd2)

muster.oauth.server.dex.allowPrivateIPOIDC: allows Dex OIDC discovery to reach issuer URLs that resolve to private/loopback IPs (e.g. Azure internal load balancers). Requires mcp-oauth#427. Emits a CWE-918 startup warning.

Fixed

deps: update module github.com/giantswarm/mcp-oauth to v0.2.186 (ca46984)
deps: update module github.com/giantswarm/mcp-toolkit to v0.2.5 (#780) (bcce33a)

CiliumNetworkPolicy egress now reaches an OIDC issuer (Dex) / HTTP MCP server fronted by a Cilium-managed ingress gateway VIP (LB-IPAM / L2, typical on-prem). New networkPolicy.cilium.ingressGateway rule allows egress to the gateway backend endpoints on their target ports (default: 10080/10443, selector: app.kubernetes.io/name=envoy in envoy-gateway-system).

Changed

attach release binaries to GitHub releases (#785) (77dbb0f)
deps: update go toolchain directive to v1.26.4 (#783) (ba9c3fd)

0.1.231 (2026-06-03)

Fixed

deps: update module github.com/giantswarm/mcp-oauth to v0.2.185 (#769) (ca46984)
deps: update module github.com/giantswarm/mcp-toolkit to v0.2.5 (#780) (bcce33a)

Changed

attach release binaries to GitHub releases (#785) (77dbb0f)
deps: update dependency architect to v9 (#768) (a4c790a)
deps: update go toolchain directive to v1.26.4 (#783) (ba9c3fd)
main: release 0.1.227 (#779) (84fca35)
main: release 0.1.228 (#781) (05f5aeb)
main: release 0.1.229 (#782) (3cf5800)
main: release 0.1.230 (#784) (7975401)

0.1.230 (2026-06-03)

Fixed

deps: update module github.com/giantswarm/mcp-oauth to v0.2.185 (#769) (ca46984)
deps: update module github.com/giantswarm/mcp-toolkit to v0.2.5 (#780) (bcce33a)

Changed

deps: update dependency architect to v9 (#768) (a4c790a)
deps: update go toolchain directive to v1.26.4 (#783) (ba9c3fd)
main: release 0.1.226 (#778) (a0ea312)
main: release 0.1.227 (#779) (84fca35)
main: release 0.1.228 (#781) (05f5aeb)
main: release 0.1.229 (#782) (3cf5800)

0.1.229 (2026-06-02)

Fixed

deps: update module github.com/giantswarm/mcp-oauth to v0.2.185 (#769) (ca46984)
deps: update module github.com/giantswarm/mcp-toolkit to v0.2.4 (#777) (9f915d6)
deps: update module github.com/giantswarm/mcp-toolkit to v0.2.5 (#780) (bcce33a)

Changed

deps: update dependency architect to v9 (#768) (a4c790a)
main: release 0.1.226 (#778) (a0ea312)
main: release 0.1.227 (#779) (84fca35)
main: release 0.1.228 (#781) (05f5aeb)

0.1.228 (2026-06-02)

Fixed

deps: update module github.com/giantswarm/mcp-oauth to v0.2.185 (#769) (ca46984)
deps: update module github.com/giantswarm/mcp-toolkit to v0.2.4 (#777) (9f915d6)

Changed

deps: update dependency architect to v9 (#768) (a4c790a)
main: release 0.1.225 (#776) (d00cc90)
main: release 0.1.226 (#778) (a0ea312)
main: release 0.1.227 (#779) (84fca35)

0.1.227 (2026-06-02)

Fixed

deps: update module github.com/giantswarm/mcp-toolkit to v0.2.3 (#770) (7649ee8)
deps: update module github.com/giantswarm/mcp-toolkit to v0.2.4 (#777) (9f915d6)

Changed

deps: update dependency architect to v9 (#768) (a4c790a)
main: release 0.1.225 (#776) (d00cc90)
main: release 0.1.226 (#778) (a0ea312)

0.1.226 (2026-06-02)

Fixed

deps: update module github.com/giantswarm/mcp-toolkit to v0.2.3 (#770) (7649ee8)
deps: update module github.com/giantswarm/mcp-toolkit to v0.2.4 (#777) (9f915d6)

Changed

main: release 0.1.224 (#775) (311f7b5)
main: release 0.1.225 (#776) (d00cc90)

0.1.225 (2026-06-02)

Fixed

deps: update module github.com/giantswarm/mcp-toolkit to v0.2.3 (#770) (7649ee8)

Changed

deps: update actions/checkout action to v6.0.3 (#774) (59a91dc)
main: release 0.1.224 (#775) (311f7b5)

0.1.224 (2026-06-02)

Changed

deps: update actions/checkout action to v6.0.3 (#774) (59a91dc)
main: release 0.1.223 (#771) (4b20779)

0.1.223 (2026-06-02)

Changed

align files according to platform standards (#767) (d7b7c9a)

Shipped between v0.1.223 and v0.3.11 (previously misfiled as Unreleased)

Fixed

enableJWTMode: true now issues RFC 9068 signed JWT access tokens. Set muster.oauth.server.jwtSigningKey (PEM-encoded EC P-256 or RSA key) or existingSecret with key jwt-signing-key; helm template fails if neither is provided when enableJWTMode: true.
CiliumNetworkPolicy egress now reaches an OIDC issuer (Dex) / HTTP MCP server that is fronted by a Cilium-managed ingress gateway VIP (LB-IPAM / L2, typical on-prem). With kube-proxy-replacement, Cilium DNATs the LoadBalancer VIP to the gateway backend pod on its target port (e.g. 443→10443) before egress policy is evaluated, so neither toEntities: world nor toEntities: cluster on 443 matched and OIDC discovery failed with context deadline exceeded. A new networkPolicy.cilium.ingressGateway rule allows egress to the gateway backend endpoints on their target ports (default: Giant Swarm envoy-gateway proxies on 10080/10443). Clusters whose gateway VIP is an external cloud LB (e.g. AWS ELB) were already covered by the world rule and are unaffected (the new rule is a no-op there); set ingressGateway: null to disable. Fixes the OAuth/OIDC discovery failed startup warning on affected clusters.
Workflow availability (core_workflow_available / core_workflow_list / workflow_available) is now session-aware. Previously, availability for SSO / auth-protected family tools (e.g. multi-instance kubernetes / prometheus servers) was computed from the process-global family routing index, which is unioned across sessions and only populated as a side effect of a prior list_tools call. This produced two symmetric defects: a false negative — muster list workflows / muster get workflow reported workflows Unavailable until some session listed tools, while muster agent (which lists tools on connect) reported them available, so the answer depended on call ordering; and a false positive — once any session listed tools, the family entry leaked process-wide, so a session that never authenticated to the family still saw the workflow as available. When the request carries a session, availability now resolves each step tool against that session's own accessible tools (hydrated from the CapabilityStore); core / meta tools resolve by name, and only session-less calls fall back to the process-global view. Closes #764.
Workflow availability is now transitive across nested workflows. A workflow step that calls another workflow (workflow_<name>) was always treated as available because the availability check matched the workflow_ prefix without consulting the registry, so a workflow referencing a non-existent or transitively broken nested workflow was wrongly reported Available and only failed at execution time. Nested workflow steps now require the referenced workflow to exist and to be itself available; the check descends through the whole chain (with cycle detection) and reports the actual unavailable tool. The workflow_ management meta-tools (workflow_list, workflow_available, ...) are unaffected.

Changed

Bump giantswarm/mcp-oauth to v0.2.184. New Helm values muster.oauth.server.{trustedIssuers,trustedProxyCIDRs,enableJWTMode,resourceIdentifier} wire trusted external OIDC issuers for RFC 8693 token exchange (id_token / access_token / jwt), DPoP trusted-proxy CIDRs, RFC 9068 JWT access tokens, and RFC 8707 resource-server audience binding. trustedIssuers entries now support allowedClaims (claim name to glob-pattern map) for Kubernetes ServiceAccount and GitHub Actions trust. Also enables the OIDC userinfo endpoint, PII-redacted audit logging, and CIMD metadata-fetch rate limiting. Encryption-at-rest is now wired on the store constructor (valkey.WithEncryptor / memory.WithEncryptor) rather than as a server option.

Added

New standalone muster-crds Helm chart (helm/muster-crds) shipping the MCPServer and Workflow CustomResourceDefinitions. The CRDs are loaded from files/crds/*.yaml by templates/crds.yaml (regular chart templates, not the Helm 3 crds/ directory), so they remain upgradable on helm upgrade and keep the helm.sh/resource-policy: keep annotation. This decouples the CRD lifecycle from the application chart so a downstream agentic-platform-crds umbrella can own it independently. Install or upgrade muster-crds before muster.
Degraded-mode startup when the Dex/OIDC issuer is unreachable at boot time. muster now starts immediately and serves MCP aggregation, reconcilers, and all non-OAuth paths regardless of Dex availability. A background goroutine retries OIDC discovery with exponential backoff (1 s → 30 s cap); once discovery succeeds the OAuth server activates transparently. Until then, OAuth and MCP-over-OAuth endpoints return 503 Service Unavailable with a Retry-After: 30 header. The /health endpoint always returns 200 with {"status":"degraded","reason":"oidc-discovery-pending"} during the window. Closes #730.
networkPolicy.flavor selects between cilium (CiliumNetworkPolicy) and kubernetes (networking.k8s.io/v1 NetworkPolicy). The kubernetes flavor is best-effort: no entity selectors, no FQDN egress. CIDR replacements live under networkPolicy.kubernetes.{apiServerCIDR,clusterCIDR,worldExcludedCIDRs}. clusterCIDR: "" disables the in-cluster ingress egress rule (kubernetes-flavor equivalent of cilium allowClusterIngress).
crds.annotations (object) is merged into each CRD's metadata.annotations by the loader. Default {helm.sh/resource-policy: keep} keeps CRDs (and the MCPServer / Workflow CRs that depend on them) around on helm uninstall.
revisionHistoryLimit (default 3) on the muster Deployment.
resources.{requests,limits}.ephemeral-storage (50Mi / 100Mi) — Kyverno's resource-limits policy on Giant Swarm workload clusters audits / rejects pods without explicit ephemeral-storage when /tmp is an emptyDir.
Egress to app.kubernetes.io/name=agentgateway:8080 in the release namespace so muster can dial the agentgateway data-plane on the upstream-proxy path (both NetworkPolicy flavors). No-op when agentgateway isn't deployed.

Removed

muster.oauth.server.kubernetesSATrusts Helm value and K8sSATrustConfig Go type are removed. Kubernetes ServiceAccount trust is now expressed via trustedIssuers with an allowedClaims entry (sub: "system:serviceaccount:<namespace>:*") and allowPrivateIPJWKS: true when the JWKS endpoint is in-cluster. The jwt subject_token_type covers projected SA tokens without a separate trust list.
ciliumNetworkPolicy.* is replaced by networkPolicy.*. ciliumNetworkPolicy.enabled → networkPolicy.enabled + networkPolicy.flavor: cilium (default). ciliumNetworkPolicy.allowClusterIngress → networkPolicy.cilium.allowClusterIngress. ciliumNetworkPolicy.{labels,annotations} → networkPolicy.{labels,annotations}.

Changed

The muster application chart no longer renders the CRDs. helm/muster/templates/crds.yaml was removed and the CRDs moved to the new muster-crds chart. crds.install now defaults to false and the whole crds block is deprecated (inert compatibility shim, removed next release) — it is kept only so a downstream that explicitly sets muster.crds.install: false still validates. Operators must install/upgrade muster-crds before muster.
CRD source files moved from helm/muster/files/crds/ to helm/muster-crds/files/crds/. files/ has no Helm 3 special-case, so the CRDs stay upgradable on helm upgrade. controller-gen output path updated in Makefile.crd.mk; CI drift check in .github/workflows/ci.yaml follows the new path.
Container image build no longer compiles the Go binary inside docker buildx. go-build now produces both muster-linux-amd64 and muster-linux-arm64 in one job (architect-orb architectures parameter) and the Dockerfile copies the matching binary from the workspace. Removes the duplicate compile and the QEMU-emulated arm64 cross-build on tag releases; push-to-registries auto-derives --platform from the workspace .platforms file.
Build identifiers (version, gitSHA, buildTimestamp) now live in pkg/project instead of main. Both injection paths populate the same vars: goreleaser writes the semver tag + short commit + date for release archives, architect-orb's go-build writes the commit SHA + UTC timestamp for container images. muster version prefers the tag, falls back to the SHA, falls back to dev, and additionally prints the commit SHA and build timestamp on dedicated lines.
Bump giantswarm/architect orb to 8.2.2 and re-enable cosign keyless chart signing (sign: false removed from every push-to-app-catalog* invocation). v8.2.2 ships architect-orb#772 which upgrades the app-build-suite executor image from 1.8.0-circleci to 1.8.1-circleci -- the new image includes the cosign binary that v8.2.0's chart signing defaults require. Closes architect-orb#769.
Bump giantswarm/architect orb to 8.2.1 to pick up architect-orb#767: image-login-to-registries is now POSIX-portable, unblocking architect/sync-china-registry (the gsoci -> Aliyun mirror via the in-China giantswarm/galaxy-runner). The v8.1.0 refactor accidentally introduced bash-only ${!var} indirect expansion in the shared login command, which BusyBox /bin/sh (used by the regctl executor) rejected with bad substitution -- so no Aliyun mirror has been happening since the migration to split-china-push: true. v8.2.x also enables cosign keyless signing, SLSA provenance, and SBOM attestations by default for public images and charts.
Disable cosign keyless chart signing on the push-to-app-catalog* jobs (sign: false). The architect orb's push-to-app-catalog defaults sign to true since v8.2.0 and shells out to cosign, but this repo uses executor: app-build-suite (so the app_build_suite Python CLI is available to package the chart with metadata) and the app-build-suite image doesn't ship cosign. Without this opt-out, every chart push fails on the Mint Sigstore OIDC token step with cosign: command not found. To be removed once architect-orb makes cosign-prepare resilient to a missing binary (or ships cosign in the app-build-suite executor) -- tracked in architect-orb#769.
Replace the push-to-gsoci-release + push-to-all-registries-release workaround pair with a single push-to-registries-release job using split-china-push: true and a companion sync-china-registry job. The cross-Pacific docker buildx push to the Aliyun mirror is replaced with regctl image copy (gsoci -> Aliyun) executed on the in-China giantswarm/galaxy-runner self-hosted CircleCI runner via the Singapore geo-replica. The chart catalog publish still does not gate on Aliyun.
Migrate image pushes from the deprecated architect/push-to-registries-multiarch job to push-to-registries with multiarch: true. Picks up the orb v8.1.0 QEMU/binfmt auto-registration, hardened buildx bootstrap, and standard OCI image labels.

Added

muster serve --extra-ca-file <path> flag: appends a PEM file to the system trust pool at startup, so outbound HTTP (MCP backends, token exchange, OAuth proxy) trusts an internal CA without per-MCPServer plumbing. Exposed in the chart as muster.extraCaFile.{path,secret.name,secret.key}; the chart mounts the named Secret and passes the flag when secret.name is set. Use case: tunnelport's SPIFFE-issued tunnel certificates on a Giant Swarm consumer MC.
MCPServer.spec.family — optional object {name, instanceArg} grouping equivalent MCPServers under a shared exposed surface. When set, the aggregator exposes tools as x_<family.name>_<tool> with a required parameter (named by family.instanceArg) selecting the providing instance. Both fields are required when family is set. The parameter is always required even for single-instance families so skills written against the family name remain stable as instances are added or removed. When unset, today's per-server prefixing applies (no behavior change for existing CRs).
MCPServer.spec.family is configurable via the core_mcpserver_create / core_mcpserver_update / core_mcpserver_validate tools.
muster.oauth.server.trustedPublicRegistrationRedirectURIs — HTTPS redirect-URI allowlist for unauthenticated dynamic client registration, passed through to mcp-oauth (Config.TrustedPublicRegistrationRedirectURIs). Strict exact-match after RFC 3986 normalization. Default: [] (opt-in per URI).
oauth-secret fail guard accepts a non-empty trustedPublicRegistrationRedirectURIs as a third valid escape valve.

Changed

The shared OpenTelemetry identifiers (TracerName, AttrToolName) move to pkg/observability, a leaf package with no internal/* dependencies that any package can import without going through the service locator. The internal/aggregator/instrument subpackage is flattened into internal/aggregator: Logging, Metrics, StartToolSpan, and the formerly-exported MCPServerOptions (now unexported mcpServerOptions) all live alongside server.go so the aggregator's MCP-server middleware sits in one place. External imports of github.com/giantswarm/muster/internal/aggregator/instrument move to github.com/giantswarm/muster/pkg/observability (constants only).
aggregator.Register / aggregator.RegisterPendingAuth and their manager-level / api.AggregatorHandler counterparts now take a ServerRegistration / PendingAuthRegistration struct rather than five-to-six positional (name, url, toolPrefix, family, authInfo, authConfig) arguments. The previous RegisterServerPendingAuthWithConfig is collapsed into the single RegisterServerPendingAuth(registration) form — AuthConfig is now a nullable field inside the struct. Internal API change; no behavior change for existing CRs.
Aggregator OpenTelemetry tracing adopts mcp-go's native server-tracing hooks via the github.com/mark3labs/mcp-go/otel adapter, replacing muster's per-tool-handler middleware. The aggregator now emits mcp.<method> spans (server kind) around every dispatched JSON-RPC method and tool.<name> spans (internal kind) around tool handlers, with W3C trace-context propagation extracted from inbound headers. Custom instrument.Tracing() middleware is removed; instrument.StartToolSpan is retained for the internal CallToolInternal dispatch path used by workflows and direct API entries.
Outbound MCP clients (stdio, SSE, streamable-http, dynamic-auth) install mcp-go's OTEL tracer via the mcp-go/otel.WithClientTracing adapter so the muster → backend leg inherits the inbound trace context and a W3C traceparent is emitted on every outgoing JSON-RPC frame. Combined with server-side tracing, a single trace now covers the whole caller → muster → backend chain.
mcp-oauth bumped to v0.2.140. The OAuth HTTP handler (Handler, New, OAuthRoutesOptions, UserInfoFromContext, SessionIDFromContext) moved to a handler subpackage; muster's internal/server and internal/aggregator import the new path. No user-facing config change.
mcp-oauth bumped to v0.2.125. Internal API migrated to functional options; server.NewOAuthHTTPServer now takes ...oauth.ServerOption. Security-event log emission is rate-limited (1/s, burst 5). No user-facing config change.

Fixed

MCPServer.spec.family tool emission now deep-copies nested JSON schema sub-trees (object properties, array items, nested required) so caller mutations of an exposed tool's schema no longer leak into the per-server cache and corrupt later tools/list results.
tools/list order is now deterministic across calls for family-grouped tools. Previously the assembly iterated Go maps directly, producing shifting orders between calls and spurious tools/list_changed diffs downstream.
When family.instanceArg collides with a property name already declared in the tool's own InputSchema.Properties, the aggregator now falls back to per-server prefixing for that specific tool. Previously the family-grouped emission silently overwrote the operator-declared property with the instance-selector enum, losing the original property's description, type, and constraints.
Workflow execution tools were advertised twice — both as the documented workflow_<workflow-name> and as core_action_<workflow-name> — through list_tools / list_core_tools / filter_tools. The core_action_* variant is not part of the public surface and the aggregator's call routing does not recognize it (calls fail with "no handler found"), so clients that picked it up from discovery hit non-functional tools. The aggregator now rewrites the workflow provider's internal action_<name> tools to workflow_<name> (no core_ prefix) when listing, matching the architecture spec; management tools (workflow_list, workflow_get, …) continue to be advertised as core_workflow_*. Pure listing fix — execution routing was already correct.
Workflow and ServiceClass CRD validation rejected scalar values in step args (spec.steps[*].args.<key>: must be of type object), making the documented YAML form (namespace: kube-system, limit: 30, allNamespaces: false) unusable through kubectl apply. Step args, condition args, JSONPath maps, and ArgDefinition.Default now use apiextensionsv1.JSON instead of runtime.RawExtension, which controller-tools emits as additionalProperties: {x-kubernetes-preserve-unknown-fields: true} (no type: object constraint), so scalars, objects, and arrays all validate. Wire format and stored values are unchanged; existing workflows with object-only args keep working.
OAuth server initialisation built its own text-format slog.Logger writing to stdout when --debug was set, so in-pod log lines from the mcp-oauth library (Valkey storage, redirect-URI security, OIDC discovery, rate limiters, audit, instrumentation) appeared as text on stdout instead of flowing through the project's JSON handler. createOAuthServer now uses slog.Default() and inherits the level set by logging.Init, so all in-pod log lines share one format and one writer.

Removed

Breaking (MCPServer CRD): Teleport authentication support removed from muster — moved to a separate operator. MCPServerAuth.type no longer accepts teleport; the teleport field (TeleportAuthConfig with identityDir / identitySecretName / identitySecretNamespace / appName) is removed from the CRD. Existing CRs with auth.type: teleport or an auth.teleport block will be rejected by validation and must be migrated to the new operator. The internal/teleport package, the api.TeleportClientHandler / api.RegisterTeleportClient / api.GetTeleportClient / api.TeleportClientConfig / api.TeleportAuth / api.AuthTypeTeleport API surface, the OAuthHandler.ExchangeTokenForRemoteClusterWithClient method, the TokenExchanger.ExchangeWithClient method, the mcpserver.MCPClientConfig.HTTPClient field, and the NewStreamableHTTPClientWithHTTPClient / NewStreamableHTTPClientWithHeaderFuncAndHTTPClient constructors are removed.
Breaking (external consumers of pkg/oauth): pkg/oauth.IDTokenClaims struct and ParseIDTokenClaims function removed. Replaced by typed accessors in pkg/oauth/jwt.go — Subject, Email, Expiry, Issuer, IsExpired — each returning (value, error) so callers can distinguish "missing claim" from "decode failed".
Per-config CA-file knobs removed: OAuthMCPClientConfig.CAFile, DexConfig.CAFile (Go config), and the muster.oauth.server.dex.caFile Helm value. These were redundant after --extra-ca-file (which augments the system trust pool) and the OAuth one was a footgun: it built a fresh cert pool from a single file, silently dropping system roots and narrowing trust to that one CA. Operator caveat — trust scope changes: the removed paths narrowed trust to a single CA; the replacement --extra-ca-file / muster.extraCaFile is additive on top of the system pool. Anyone who deliberately relied on the narrower scope no longer has that option here. None of giantswarm's deployed configs set these values, so no migration is required.

Changed

Logging bootstrap now lives in cmd/serve.go. The serve command calls logging.Init once at startup, defers the Shutdown, then constructs the application. NewApplication no longer touches the logger; non-serve muster subcommands rely on the nil-guard in pkg/logging (the previous in-bootstrap init was vestigial there too).
app.NewConfig signature drops the silent parameter and the corresponding Config.Silent field. Both were set but never read — --silent is enforced by swapping the writer to io.Discard directly in cmd/serve.go. Module-internal change (internal/app is not importable from outside the module).
In-pod muster logs are now JSON by default (auto-detected via KUBERNETES_SERVICE_HOST) instead of text. The text path remains for local muster CLI invocations and tests.
The aggregator's Hooks (AddAfterInitialize, AddAfterListTools, AddBeforeCallTool, AddAfterCallTool, AddOnError) emit log lines via the new *WithAttrsCtx variants so per-request trace correlation lands on the MCP-Protocol subsystem.
The Valkey storage URL in startup logs is now redacted via mcp-toolkit/logging.RedactHost, which strips IPv4/IPv6 addresses and URL userinfo. The local redactURL helper, which only stripped userinfo, is removed.
Consolidated scattered JWT-claim decoders into typed accessors in pkg/oauth/jwt.go: Subject, Email, Expiry, Issuer, IsExpired, plus ErrTokenExpMissing for callers that need to distinguish "missing exp" from "decode failed". The accessors share a single golang-jwt/jwt/v5 parser; consumers in internal/aggregator, internal/cli, and internal/oauth no longer touch encoding/base64, encoding/json, or the JWT library directly. The admin diagnostic UI keeps its own segment decoder (different concern: lenient display of operator-pasted tokens, including 2-part inputs missing the signature segment). The previous defensive RawStdEncoding fallback for non-spec base64 is intentionally dropped — every IdP muster integrates with emits RFC 7515-compliant RawURLEncoding.

Added

pkg/logging.Init(ctx, level, output, serviceName, serviceVersion) (Shutdown, error) initialises logging via mcp-toolkit/logging and returns a Shutdown for the OpenTelemetry LoggerProvider. When any of OTEL_EXPORTER_OTLP_LOGS_ENDPOINT, OTEL_EXPORTER_OTLP_ENDPOINT, or OTEL_LOGS_EXPORTER is set, log records flow through OTLP and carry the active span's TraceID/SpanID for log ↔ trace correlation in Grafana. Otherwise the handler auto-selects JSON (in a Kubernetes pod) or text (local dev) and the Shutdown is a no-op. InitForCLI stays as a non-OTLP convenience.
pkg/logging.{Debug,Info,Warn,Error}Ctx and {Debug,Info,Warn}WithAttrsCtx thread a context.Context through the slog handler so the OTLP path can pull the active span's TraceID/SpanID off the call's context. Existing ctx-less variants remain.
OpenTelemetry tracing for every MCP tool call. The aggregator's mcp-go middleware opens a tool.<name> span at the meta-tool layer, and CallToolInternal opens an inner span carrying the real workload tool name (x_kubernetes_*, workflow_*, service_*, …) so Tempo trace trees pivot on the actual tool. W3C TraceContext + Baggage propagators are installed unconditionally so inbound traceparent headers are honoured even when no exporter is configured. Tracer- and meter-provider lifecycles are handled by mcp-toolkit/tracing and mcp-toolkit/metrics at the composition root.
OpenTelemetry metrics for every MCP tool call: muster.tool_calls (counter, exports as muster_tool_calls_total) and muster.tool_call.duration (histogram, exports as muster_tool_call_duration_seconds), each with tool and outcome attributes (ok / error / error_result).
Helm: muster.observability.metrics.exporter switches the metric backend (otlp, prometheus, console, none, comma-combinations). Selecting prometheus exposes /metrics on port 9464 and (when muster.observability.metrics.prometheus.serviceMonitor.enabled) renders a ServiceMonitor.
Structured per-tool-call log line on subsystem MCP-Tool with tool, outcome, duration_s, and error fields, for log/metric/trace correlation in dashboards.
muster.observability.otel.{endpoint,protocol,headers,resourceAttributes} Helm values configuring the OTLP exporter. Empty endpoint (default) leaves muster in propagator-only mode; setting it enables both traces and metrics over the same OTLP endpoint. K8S_NODE_NAME is now exposed via the downward API alongside the existing K8S_NAMESPACE / K8S_POD_NAME so resource attributes carry k8s.node.name. OTEL_RESOURCE_ATTRIBUTES is set whenever either OTLP or a metrics exporter is configured (previously OTLP-only), so Prometheus-only mode also gets k8s.namespace.name / k8s.pod.name / k8s.node.name resource attribution.
docs/explanation/observability.md documenting the trace contributions, signal configuration, instrument and log-field shape, and a Tempo/Mimir/Loki query catalog.
Add muster call command for direct MCP tool invocation from the CLI. Supports --key=value arguments and --json for complex payloads, with tab completion for tool names.
Add ciliumNetworkPolicy.allowClusterIngress Helm value to allow egress to in-cluster services on HTTP/HTTPS ports (e.g. Dex OIDC via ingress LoadBalancer IP).
OAuth encryption keys can now be supplied as either base64 (openssl rand -base64 32) or hex (openssl rand -hex 32); the format is auto-detected.
Agent OAuth client now validates the RFC 9207 iss parameter on the authorization callback (defense-in-depth against AS mix-up attacks). Servers that omit iss are still accepted.
Authorization-server discovery now also serves /.well-known/openid-configuration and per-path Protected Resource Metadata at /.well-known/oauth-protected-resource/mcp (additive — RFC 9728 / OpenID Connect Discovery).
BDD scenarios workflow-conditional-static and service-state-static to preserve coverage of workflow conditional features (inline tool conditions, from_step, allow_failure, expect_not, condition_evaluation, step skipping) and static-service state-machine semantics (core_service_restart happy-path, core_service_start on already-running, core_service_stop on already-stopped). The deleted ServiceClass-based scenarios bundled this coverage with SC-instance lifecycle; the replacements drive the same workflow-engine and orchestrator code paths against a static MCPServer service.
MCPServer.spec.auth.authorizationServer lets operators pin the OAuth issuer when the backend doesn't publish RFC 9728 metadata (Atlassian's hosted MCP being the prompting case). The override applies to core_auth_login only and is verified against the AS metadata's issuer field per RFC 8414 §3.3 to fail closed on a wrong pin. Fixes #599.

Changed

Extracted validation and template-resolution helpers from the 930-line internal/workflow/executor.go into dedicated validation.go and template.go files. executor.go is now ~600 lines; ExecuteWorkflow itself remains 500 lines and a follow-up tracks breaking it into per-concern helpers (executeStep, evaluateCondition, processStepResult) — that's a behavioral refactor with test impact, not a file split. (#140)
Move the 547-line internal/client/kubernetes_client.go into a new internal/client/kubernetes/ subpackage, split per domain (client.go, mcpserver.go, serviceclass.go, workflow.go, events.go). The core file keeps the type, constructor, scheme, lifecycle methods, and discovery-based CRD validation. Pure refactor: the MusterClient interface stays in the parent client package, the dispatcher calls kubernetes.New(restConfig) directly, and external consumers are unaffected. (#140)
Move the 1233-line internal/client/filesystem_client.go into a new internal/client/filesystem/ subpackage, split per domain (client.go, mcpserver.go, serviceclass.go, workflow.go, events.go). Each file is under 400 lines. Pure refactor: the MusterClient interface stays in the parent client package, the dispatcher calls filesystem.New(basePath) directly, and external consumers are unaffected. (#140)
Collapse per-CRD duplication in both client adapters into shared store.go helpers built on client.Object / client.ObjectList. Each per-CRD file shrinks from ~165 LOC (filesystem) / ~70 LOC (kubernetes) to ~40 LOC of thin wrappers; error wrapping and namespace handling are now uniform across both surviving CRDs. The kubernetes CreateEventForCRD double switch (kind→GVK + kind→Get-method) collapses to a single crdFactories map. Pure refactor: no public method signatures change, no behaviour change. (#140)
Restore groups scope in DefaultOAuthCIMDScopes -- required for group-based RBAC in downstream services. Provider-level scope filtering in mcp-oauth (e.g., filterGoogleScopes, filterDexScopes) handles provider differences.
Bump mcp-oauth to v0.2.117. Adopts oauth.NewServerWithCombined and Handler.RegisterOAuthRoutes to simplify server wiring; the authorization callback now includes the RFC 9207 iss parameter automatically. Operational note: mcp-oauth now rejects low-entropy AES-256 token-encryption keys (fewer than 16 distinct byte values). Real keys generated with openssl rand -base64 32 or openssl rand -hex 32 are unaffected; placeholder keys (all zeros, repeated bytes) will fail at startup with a clear error — rotate before upgrading.

Removed

ServiceClass CRD, API types, and Helm RBAC narrowing (third PR of the ServiceClass removal — see #632).
- Deleted: pkg/apis/muster/v1alpha1/serviceclass_types.go, helm/muster/crds/muster.giantswarm.io_serviceclasses.yaml, internal/api/{serviceclass,serviceinstance}.go, ServiceClassManagerHandler interface and Register/GetServiceClassManager.
- Helm RBAC drops serviceclasses and serviceclasses/status from the ClusterRole's resources lists.
- Operational note (REQUIRED before upgrading past this PR): delete any ServiceClass custom resources in your cluster — they will be orphaned when the CRD is removed:
```
kubectl delete serviceclasses.muster.giantswarm.io --all -A
```
  MCPServer and Workflow CRs are unaffected.
ServiceClass-related MCP tools and CLI surface (first PR of the ServiceClass removal — see #632 for the rest).
- MCP tools: core_serviceclass_*, core_service_create, core_service_delete, core_service_get, core_service_validate. Service inspection still works via core_service_status.
- CLI subcommands: muster create service, muster create serviceclass, muster check serviceclass, muster get serviceclass, muster list serviceclass. The service and serviceclass values for muster events --resource-type and muster test --concept are also gone.
- BDD scenarios: 22 serviceclass-* / serviceclass_* scenarios, 20 service-* scenarios that exercised user-creatable service instances (service-create-*, service-delete*, service-get*, service-validate, service-lifecycle, service-persistence, service-restart, service-state-transitions, service-{start,stop}*), and 6 cross-cutting end-to-end scenarios that depended on ServiceClass (behavior-developer-onboarding-journey, example_with_mock, reconciler-status-sync, user-journey-platform-setup, workflow-conditional-service-check, workflow-run-with-serviceclass) — 48 scenarios total. service-get-non-existent is renamed to service-status-non-existent and now exercises core_service_status.
Note: the ServiceClass runtime, CRD, and Helm RBAC are still in place after this PR; they are removed in subsequent PRs tracked in #632.
api.RegisterConfig and api.GetConfig deprecated wrappers (use RegisterConfigHandler / GetConfigHandler directly). All call sites already suppressed with //nolint:staticcheck; both are gone now along with the suppressions. (#140)

Fixed

Aggregator-side PRM discovery (used by core_auth_login) now follows the MCP 2025-11-25 spec: it parses WWW-Authenticate: ... resource_metadata= from a 401, probes the path-based well-known URL (<host>/.well-known/oauth-protected-resource<path> — using the raw MCP URL path so /v1/mcp is preserved) before the root form, and exposes the RFC 9728 resource field on the parsed result. The previous implementation was root-only and silently dropped both signals.
pkg/oauth.Client.DiscoverMetadata now handles path-bearing issuer URLs (e.g. https://login.microsoftonline.com/<tenant>/v2.0, Auth0 / Okta orgs with paths) per MCP 2025-11-25 §"Authorization Server Metadata Discovery": tries RFC 8414 path-insert, OIDC path-insert, then OIDC append. Previously these issuers fell through to a single no-path probe and failed; now they succeed.
SSO token forwarding no longer hands downstream MCP servers a JWT whose exp is past the current time. Both ID-token storage paths — storeIDTokenForSSO (muster-issued tokens) and the forwarded-bearer mirroring in injectExternalIDToken (SSO-passthrough) — read the JWT's exp claim and persist it as the entry's ExpiresAt, so IsExpiredWithMargin evicts stale tokens after idle periods instead of treating zero ExpiresAt as never-expiring. Tokens without a parseable exp are refused (logged at warn level) — they were always malformed for muster's flow but the previous shape would have stored them with zero ExpiresAt, recreating the same leak. (#549)
Bump mcp-oauth to v0.2.86 with Dex scope filtering: non-standard client scopes like claudeai (sent by Claude) are now stripped before forwarding to Dex, preventing invalid_scope errors. Also includes Google scope filtering and openid force-merge from v0.2.84.
CRD validation now uses the discovery API instead of listing MCPServer resources in the default namespace. With namespace-scoped RBAC (a Role limited to muster's own namespace), the previous probe failed with Forbidden, silently fell back to filesystem mode, and left configured MCPServer CRs unstarted (visible in logs as Found 0 MCPServer definitions for auto-start processing followed by Deleting MCPServer service: <name>).
call_tool meta-tool now forwards the underlying tool's isError flag on the outer response. Previously the top-level isError was always false even when the wrapped tool returned an error, which was misleading for MCP clients that only inspect the top-level flag.
Per-server OAuth flow and agent OAuth flow both now refuse to proceed when the authorization server's metadata does not advertise S256 in code_challenge_methods_supported, per MCP 2025-11-25 §"Authorization Code Protection". Previously an absent list was treated as "S256 OK" (the OAuth 2.1 default), which let muster start a flow that the AS could silently downgrade or reject at the token endpoint with a confusing error. Metadata.SupportsPKCE is renamed to SupportsS256PKCE to match the new semantics — only pkg/oauth-internal callers existed.

0.1.0 - 2026-02-23

Changed

Session duration reduced from 90 days to 30 days. The refresh token TTL now matches Dex's absoluteLifetime (720h). Previously, muster's 90-day refresh token outlived Dex's 30-day session, causing confusing failures when auto-refresh silently stopped working after day 30. Users who were logging in once every ~2 months will now need to re-authenticate every 30 days.
muster auth status now shows session expiry. Instead of Refresh: Available, the output now shows Session: ~29 days remaining (auto-refresh), giving users a concrete estimate of when re-authentication will be required.
Access token TTL is now explicitly set to 30 minutes (matching Dex's idTokens expiry) instead of relying on the library default of 1 hour.
Session duration is now configurable via oauth.server.sessionDuration in config.yaml (default: 720h / 30 days).
Kubernetes event emission is now disabled by default (alpha feature). Use --enable-events flag on muster serve or set events: true in config.yaml to opt in.
Switch CI to push-to-registries-multiarch (architect-orb@6.14.0) with amd64-only on branches for faster PR feedback and full multi-arch on release tags. Chart tests now run before publishing to the app catalog.
Update Dockerfile to multi-stage build with native cross-compilation support for multi-architecture images.

Note: The Server-Side Meta-Tools Migration below is a breaking change that will be released as part of the next major version. External integrations should prepare for this change.

Breaking Changes

Server-Side Meta-Tools Migration

Meta-tools (list_tools, call_tool, describe_tool, etc.) have moved from the agent to the aggregator server. This is a fundamental architectural change.

What Changed:

Component	Before	After
Agent	Exposed 11 meta-tools + bridged to aggregator	Transport bridge only (OAuth shim + stdio↔HTTP)
Aggregator	Exposed 36+ core tools directly	Exposes ONLY meta-tools - no direct tool access
Tool Access	Direct tool calls to aggregator	All tool calls go through `call_tool` meta-tool

What Continues Working (Transparent Migration):

CLI commands (muster list, muster get, etc.) - client wraps calls automatically
Agent REPL (muster agent --repl) - uses same client with transparent wrapping
BDD test scenarios - test client wraps calls automatically
MCP native protocol methods (tools/list, resources/list) - not affected

What Breaks (Requires Update):

External integrations calling tools directly via HTTP
Custom clients connecting directly to aggregator

Migration for External Clients:

// Before: Direct tool call
{"method": "tools/call", "params": {"name": "core_service_list", "arguments": {}}}

// After: Wrap through call_tool
{"method": "tools/call", "params": {
  "name": "call_tool",
  "arguments": {"name": "core_service_list", "arguments": {}}
}}

Benefits:

OAuth-capable clients can connect directly to server without agent
Simpler agent architecture (~200 lines vs ~700 lines)
Consistent tool visibility across all clients
Centralized meta-tool logic

See ADR-010 for design details.

Known External Integrations Affected:

Any HTTP clients calling the aggregator directly
Custom MCP clients not using muster agent
CI/CD pipelines with direct tool calls

Recommended Migration Timeline:

Review your integration code for direct tool calls
Update to wrap calls through call_tool meta-tool
Test with the new Muster version before deploying

Changed

MCPServer CRD State Exposes Auth Required - The MCPServer CRD now shows Auth Required state when a remote server returns 401 Unauthorized (#337)
- Before: 401 response mapped to Connected (hiding auth requirement)
- After: 401 response shows as Auth Required in CRD state
- This gives operators clear visibility into which servers need authentication
- CLI output updated: muster list mcpserver now shows Auth Required state
- SESSION column values updated: OK → Authenticated, Required → Pending Auth
- Column header renamed: AUTH → SESSION to match muster auth status output

Added

Reconciliation Framework - Automatic synchronization between resource definitions (CRDs/YAML) and running services
- Supports both Kubernetes mode (using controller-runtime informers) and filesystem mode (using fsnotify)
- Auto-detects operating mode based on environment
- Configurable per-resource-type enable/disable
- Work queue with deduplication and exponential backoff
- Status tracking and API for observability
- See ADR 007 for design details
StateChangeBridge - Real-time sync of runtime state changes to CRD status subresources
- Subscribes to orchestrator service state changes
- Triggers reconciliation to update CRD status when services start/stop/crash

Changed

BREAKING: Consolidated OAuth Configuration Naming - OAuth configuration structure has been reorganized for clarity (#324)
- Before: aggregator.oauth (client/proxy) + aggregator.oauthServer (server protection)
- After: aggregator.oauth.mcpClient (MCP client/proxy) + aggregator.oauth.server (server protection)
- Both OAuth roles now live under a single oauth section with explicit mcpClient/server sub-sections
- The mcpClient name makes it clear this is for authenticating TO remote MCP servers
- CLI flags renamed: --oauth → --oauth-mcp-client, --oauth-public-url → --oauth-mcp-client-public-url
- Helm values updated: muster.oauth.* → muster.oauth.mcpClient.*, muster.oauthServer.* → muster.oauth.server.*
- CIMD configuration moved to nested structure: cimdPath/cimdScopes → cimd.path/cimd.scopes
- Migration: Update configuration files and Helm values to use the new structure
BREAKING: CRD Status Field Changes - Status fields have been redesigned for session-aware tool availability
- MCPServerStatus: Removed availableTools (session-dependent), added lastConnected and restartCount
- ServiceClassStatus: Replaced available/requiredTools/missingTools/toolAvailability with valid/validationErrors/referencedTools
- WorkflowStatus: Replaced available/requiredTools/missingTools/stepValidation with valid/validationErrors/referencedTools/stepCount
- Tool availability is now computed per-session at runtime, not stored in CRs
- Existing CRs will have stale status fields that will be updated on first reconciliation
Added Chart annotations to support OCI repositories.

Fixed

Helm CiliumNetworkPolicy: Fixed incorrect values path for OAuth storage check (now uses .Values.muster.oauth.server.storage)

Added

Remote MCP Server Support for Kubernetes Environments
- Added comprehensive support for stdio, streamable-http and sse transport protocols
- Enhanced CRD Schema: Updated MCPServerSpec to support all MCP server types
  - Added new config for streamable-http and sse: url, headers and timeout fields
  - Added mutual exclusion validation and required field validation using kubebuilder annotations
- New CLI Commands: Added subcommands to use new type system
  - muster create mcpserver <name> --type stdio for local MCP servers
  - muster create mcpserver <name> --type streamable-http for HTTP remote servers
  - muster create mcpserver <name> --type sse for SSE remote servers
- Updated Examples: Enhanced example files to demonstrate both local and remote configurations
- Kubernetes Deployment Ready: Enables deployment patterns where Muster aggregator runs in cluster and connects to MCP servers deployed as separate Kubernetes services
Systemd Socket Activation Support
- Added muster.socket unit file for socket-activated systemd deployment
- Modified muster.service to use socket activation on localhost:8090
- Updated scripts/setup-systemd.sh and scripts/dev-restart.sh to handle socket activation
- Make use of new dependency github.com/coreos/go-systemd to handle socket activation
Service Health Monitoring
- Added health checks for MCP servers using the tools/list JSON-RPC method
- Added health checks for port forwards by testing TCP connectivity
- Health checks run every 30 seconds for all running services
- Health status is reported through the StateStore and displayed in the TUI
- Created ServiceHealthChecker interface for extensible health checking
Improved State Reconciliation
- Implemented proper ReconcileState() method that syncs TUI state with StateStore
- Updates service statuses, ports, PIDs, and error states from centralized store
- Synchronizes cluster health information from K8sStateManager
- Ensures UI consistency after startup and state changes
K8s Connections as Services
- Kubernetes connections are now modeled as services in the dependency graph
- K8s connection health monitoring is now handled by dedicated K8s connection services
- Unified service management architecture - all services (K8s, port forwards, MCPs) follow the same lifecycle
- K8s connections can be stopped/restarted like other services with proper cascade handling
Cascading stop functionality: stopping a service automatically stops all dependent services
K8s connection health monitoring with automatic service lifecycle management
Port forwards now depend on their kubernetes context being authenticated and healthy
The kubernetes MCP server depends on the management cluster connection
When k8s connections become unhealthy, dependent services are automatically stopped
Manual stop (x key) now uses cascading stop to cleanly shut down dependent services
New StartServicesDependingOn method in ServiceManager to restart services when dependencies recover
New orchestrator package that manages application state and service lifecycle for both TUI and non-TUI modes
New HealthStatusUpdate and ReportHealth for proper health status reporting
Health-aware startup: Services now wait for their K8s dependencies to be healthy before starting
Add comprehensive dependency management system for services
- Services now track why they were stopped (manual vs dependency cascade)
- Automatically restart services when their dependencies recover
- Ensure correct startup order based on dependency graph
- Prevent manually stopped services from auto-restarting
Phase 1 of Issue #45: Message Handling Architecture Improvements
- Added correlation ID support to ManagedServiceUpdate for tracing related messages and cascading effects
- Implemented configurable buffer strategies for TUI message channels:
  - BufferActionDrop: Drop messages when buffer is full
  - BufferActionBlock: Block until space is available
  - BufferActionEvictOldest: Remove oldest message to make room for new ones
- Added priority-based buffer strategies to handle different message types differently
- Introduced BufferedChannel with metrics tracking (messages sent, dropped, blocked, evicted)
- Enhanced orchestrator with correlation tracking for health checks and cascading operations
- Updated service manager to use new correlation ID system for better debugging
- Added comprehensive test coverage for buffer strategies and correlation tracking
Phase 2 of Issue #45: State Consolidation
- Implemented centralized StateStore as single source of truth for all service states
- Added ServiceStateSnapshot for complete state information with correlation tracking
- Introduced state change subscriptions with StateSubscription for reactive updates
- Enhanced ServiceReporter interface with GetStateStore() method for direct state access
- Updated TUIReporter and ConsoleReporter to use centralized state management
- Migrated ServiceManager from local state tracking to centralized StateStore
- Added comprehensive metrics tracking for state changes and subscription performance
- Implemented state change event system with old/new state tracking
- Added support for filtering services by type and state
- Maintained full backwards compatibility while eliminating state duplication
Phase 3 of Issue #45: Structured Event System
- Implemented comprehensive event hierarchy with semantic event types:
  - ServiceStateEvent for service lifecycle changes with old/new state tracking
  - HealthEvent for cluster health status updates
  - DependencyEvent for cascade start/stop operations
  - UserActionEvent for user-initiated actions
  - SystemEvent for system-level operations
- Added EventBus interface with publish/subscribe functionality
- Implemented flexible event filtering system with composable filters:
  - Filter by event type, source, severity, correlation ID
  - Combine filters with AND/OR logic for complex subscriptions
- Created EventBusAdapter for backwards compatibility with existing ServiceReporter interface
- Added comprehensive event metrics tracking (published, delivered, dropped events)
- Implemented both handler-based and channel-based event subscriptions
- Added event severity levels (trace, debug, info, warn, error, fatal) for better categorization
- Enhanced correlation tracking with event metadata support
- Provided thread-safe concurrent event publishing and subscription management
- Added extensive test coverage for all event types and bus functionality
Phase 4 of Issue #45: Testing and Polish
- Added comprehensive integration tests covering end-to-end event flows
- Implemented performance monitoring utilities with PerformanceMonitor and metrics tracking
- Created event batching system with EventBatchProcessor for high-volume scenarios
- Built OptimizedEventBus with configurable performance optimizations
- Added object pooling system with EventPoolManager to reduce GC pressure
- Implemented extensive error recovery testing including panic handling
- Added memory usage monitoring and subscription cleanup verification
- Created comprehensive documentation covering architecture, usage, and best practices
- Fixed race conditions in event bus concurrent access patterns
- Enhanced thread safety across all components with proper synchronization
- Provided migration guides and troubleshooting documentation
- Achieved high test coverage with robust integration and unit tests
Improved Dependency Management for Service Restarts
- When restarting a service, its dependencies are now automatically restarted if they're not active
- This ensures services always have their requirements satisfied (e.g., restarting Grafana MCP will also restart its port forward if needed)
- Dependencies are restarted regardless of their stop reason to guarantee service requirements
- Clear manual stop reason when restarting a service to allow proper dependency management
Implemented Issue #46: Improved State Management Between TUI and Orchestrator
- Phase 1: Unified State Management
  - Added helper methods to TUI Model to use StateStore as single source of truth
  - Implemented state reconciliation on TUI startup to ensure consistency
  - Updated TUI controller to use StateStore instead of directly updating model maps
  - Eliminated state duplication between TUI Model and StateStore
- Phase 2: Message Sequencing
  - Added sequence numbers to ManagedServiceUpdate for proper message ordering
  - Implemented MessageBuffer for handling out-of-order messages
  - Added global sequence counter with atomic operations for thread safety
- Phase 3: Enhanced Correlation Tracking
  - Added CascadeInfo type for tracking cascade relationships between services
  - Added StateTransition type for tracking state changes with full context
  - Enhanced StateStore to record state transitions and cascade operations automatically
  - Updated orchestrator to record cascade operations for better observability
- Phase 4: Improved Error Handling
  - Added retry logic for critical updates that are dropped due to buffer overflow
  - Implemented BackpressureNotificationMsg for user notifications about dropped messages
  - Added configurable retry attempts with exponential backoff
  - Enhanced TUIReporter with retry queue processing and user feedback
Comprehensive Documentation Suite
- Added Architecture Overview documenting system design, components, and principles
- Created Quick Start Guide for new users to get up and running quickly
- Added Troubleshooting Guide with common issues and solutions
- Enhanced development documentation with recent architectural improvements
- Documented dependency management, state management, and message flow in detail
Configurable Namespace for CR Discovery
- Added namespace configuration option to config.yaml for Kubernetes CR discovery
- Allows specifying which namespace to use for MCPServer, ServiceClass, and Workflow resources
- Defaults to "default" when not specified
- Enables muster to work properly in multi-namespace Kubernetes environments

Changed

Aggregator Config
- Drop the "Enabled" field (always enabled in modes where it's used)
Service Manager Refactoring
- ServiceManager now accepts an optional KubeManager parameter for K8s connection services
- Added support for K8s connection services in the service lifecycle management
- Improved service stop handling to report "Stopping" state before closing channels
Orchestrator Improvements
- Removed old health monitoring methods in favor of K8s connection services
- Updated dependency graph to use service labels for K8s connections (e.g., "k8s-mc-mymc" instead of "k8s:context-name")
- Improved service restart logic to properly handle dependencies
Dependency graph now includes K8sConnection nodes as fundamental dependencies
Service manager's StopServiceWithDependents method handles cascading stops
Health check failures trigger automatic cleanup of dependent services
Non-TUI mode now uses the orchestrator for health monitoring and dependency management
TUI mode no longer performs its own health checks - the orchestrator handles all health monitoring and the TUI only displays results
Proper separation of concerns: orchestrator manages health checks and service lifecycle, TUI only displays status
Orchestrator now performs initial health check before starting services
Refactored TUI message handling system
- Introduced specialized controller/dispatcher for better separation of concerns
- Controllers now focus on single responsibilities
- Better error handling and logging throughout the message flow
Improved startup behavior - the UI now shows loading state until all clusters are fully loaded
Port forwards no longer start before K8s health checks pass - orchestrator now checks K8s health before starting dependent services
ManagedServiceUpdate now includes CorrelationID, CausedBy, and ParentID fields for tracing
TUIReporter now uses configurable buffered channels instead of simple channels
Service state updates now include correlation information in logs
Orchestrator operations (stop/restart) now generate and track correlation IDs
Removed unused DependsOnServices field from MCPServerDefinition - MCP servers never depend on other MCP servers
Enhanced RestartService to use the new startServiceWithDependencies method for dependency-aware restarts
Updated handleServiceStateUpdate to properly restart services with their dependencies
Improved Service Monitoring
- Fixed monitorAndStartServices to respect StopReasonDependency - services stopped due to dependency failure won't be restarted until their dependencies are restored
- Added automatic restart of dependent services when a dependency becomes healthy again
- Added 1-second delay before restarting services to ensure ports are properly released

Fixed

Exit CLI on standalone server failure
- When the mcp-aggregator service (server) fails, the CLI now terminates gracefully
Port Forwarding State Issue
- Fixed issue where port forwarding services would get stuck in "Stopping" state
- ServiceManager now properly reports the "Stopping" state before closing the stop channel
- Port forwarding processes correctly transition to "Stopped" state
Code Cleanup
- Removed commented-out mcpServerProcess struct that was marked for deletion
- Removed duplicate updatePortForwardFromSnapshot and updateMcpServerFromSnapshot methods
- Cleaned up unused code and improved code organization
Dependency-Related Fixes
- Fixed issue where MCP servers would restart even when their port forward dependencies were stopped
- Services with StopReasonDependency now properly wait for their dependencies to be restored
- When a service becomes healthy, its dependent services that were stopped due to dependency failure are automatically restarted
- Fixed "address already in use" errors by adding proper restart delay
Fixed spurious error logs when stopping MCP servers
- Suppressed expected "file already closed" errors that occurred when stopping MCP server processes
- Added proper error handling for both stdout and stderr pipe closures during shutdown
- These were harmless errors but created unnecessary noise in the logs
Fixed cascade stops not triggering when K8s connections fail
- When a K8s connection transitions to Failed state (e.g., due to network issues), all dependent services (port forwards and MCP servers) are now properly stopped
- This prevents orphaned services from continuing to run when their underlying K8s connection is no longer healthy
- Services will automatically restart when the K8s connection recovers
Set config directory early to avoid bugs handling the empty string (those should be fixed with this change as well)

Documentation

Added comprehensive documentation about dependency graph implementation
Enhanced dependency management documentation with detailed examples
Added explanation of dependency rules and startup/restart behavior
Documented the relationship between stop reasons and automatic recovery
Created comprehensive architecture documentation covering all major components
Added troubleshooting guide with detailed debugging techniques
Created quick start guide for new users
Updated development guide with recent architectural improvements
Documented the entire dependency management system with visual diagrams
Updated outdated documentation sections
- Removed obsolete "Package Design for Shared Core Logic" section from development.md
- Updated development.md to reference the unified service architecture
- Fixed test examples in development.md to match current implementation
- Updated README.md prerequisites to remove mcp-proxy requirement
- Clarified non-TUI mode behavior in README.md
- Rewritten MCP Integration Notes in README.md to reflect YAML configuration system

Technical Details

New helper functions: NewManagedServiceUpdate(), WithCause(), WithError(), WithServiceData()
New types: BufferStrategy, BufferedChannel, ChannelMetrics, ChannelStats
Backwards compatibility maintained for existing interfaces
All existing tests updated and new comprehensive test suite added

[Previous]

Added

Enhanced MCP server configuration and management capabilities

Changed

MCP server configuration now only supports localCommand type for simplicity and reliability

Technical Details

Streamlined MCP server architecture by removing container support
Simplified MCP server lifecycle management

FilesExpand file tree

CHANGELOG.md

Latest commit

History

CHANGELOG.md

File metadata and controls

Changelog

Unreleased

Added

Changed

Fixed

Changed

Removed

Fixed

Removed

Changed

Fixed

Added

Fixed

Changed

Removed

Fixed

0.3.12 - 2026-06-10

Changed

Fixed

Added

Fixed

Changed

0.1.231 (2026-06-03)

Fixed

Changed

0.1.230 (2026-06-03)

Fixed

Changed

0.1.229 (2026-06-02)

Fixed

Changed

0.1.228 (2026-06-02)

Fixed

Changed

0.1.227 (2026-06-02)

Fixed

Changed

0.1.226 (2026-06-02)

Fixed

Changed

0.1.225 (2026-06-02)

Fixed

Changed

0.1.224 (2026-06-02)

Changed

0.1.223 (2026-06-02)

Changed

Shipped between v0.1.223 and v0.3.11 (previously misfiled as Unreleased)

Fixed

Changed

Added

Removed

Changed

Added

Changed

Fixed

Removed

Changed

Added

Changed

Removed

Fixed

0.1.0 - 2026-02-23

Changed

Breaking Changes

Server-Side Meta-Tools Migration

Changed

Added

Changed

Fixed

Added

Changed

Fixed

Documentation