Skip to content

troubleshooting.md

Latest commit

 

History

History
231 lines (191 loc) · 11.1 KB

troubleshooting.md

File metadata and controls

231 lines (191 loc) · 11.1 KB
Raw

Troubleshooting

Common startup errors

  • "control plane API key is required"

    • Set CONTROL_PLANE_API_KEY, set OPENAI_API_KEY as a fallback, or use --control-plane.api-key=env:.../file:....

  • "tunnel ID is required"

    • Set CONTROL_PLANE_TUNNEL_ID or --control-plane.tunnel-id=....

  • "invalid tunnel ID ... must match tunnel_<32 lowercase hexadecimal characters>"

    • Use a tunnel ID shaped like tunnel_0123456789abcdef0123456789abcdef.

  • "MCP server URL is required"

    • Set MCP_SERVER_URL or --mcp.server-url=....

macOS Gatekeeper blocks a downloaded archive

Directly downloaded macOS release ZIPs are not currently notarized and can be blocked by Gatekeeper before tunnel-client --version or tunnel-client help quickstart runs. Do not use Open Anyway, xattr, or spctl to bypass the check.

Use the supported Homebrew installation path instead:

brew install openai/tools/tunnel-client
tunnel-client --version
tunnel-client help quickstart

Unexpected URLs / 404s

  • Ensure CONTROL_PLANE_BASE_URL is the host root (for example https://api.openai.com) and not a pre-prefixed path.

Authorization failures: identify the failing surface first

A Platform page error and a tunnel-client runtime 401/403 are different failures. tunnel-client doctor is a local preflight: it validates config, key presence, MCP/OAuth reachability, and the local health listener. It does not query the Platform UI or prove that a key can poll the target tunnel.

  • Platform Tunnels page says "Tunnels access required", "not authorized", or that Tunnels Read access is required
    • This is Platform UI authorization, not a tunnel-client runtime or MCP failure. Tunnel permissions are organization-level, not project-level.
    • Select the intended organization. Ask an organization owner or RBAC administrator to add you to a role or group with Tunnels Read to view tunnels, or Read + Manage to create, edit, or delete them. If no matching role exists, they can create one, assign it to a group, and add you to that group. Allow up to 30 minutes for a new role assignment to propagate, then reload the page.
  • tunnel-client doctor --explain fails
    • Fix the named local preflight check before starting the daemon.
  • doctor passes, but tunnel-client admin tunnels get <tunnel_id> returns 401 or 403
    • First identify the credential this probe used. admin tunnels get uses --admin-key or OPENAI_ADMIN_KEY when either is configured; only without an admin key does it fall back to CONTROL_PLANE_API_KEY or OPENAI_API_KEY. To test the runtime key, leave --admin-key and OPENAI_ADMIN_KEY unset and set CONTROL_PLANE_API_KEY to the same key used by tunnel-client run.
    • If that same runtime key gets 401/403, confirm the tunnel ID and organization/workspace association.
  • The same runtime key can read tunnel metadata, but tunnel-client run or /ui#logs shows a polling 401/403
    • Metadata read works, but the runtime path likely lacks Tunnels Use. Grant the runtime-key principal Tunnels Read + Use, then restart.
  • Platform lists the tunnel, but ChatGPT cannot select it
    • Check the tunnel workspace ID, connector operator's Tunnels Read + Use, and /readyz; this is not proof of a daemon failure.

See permissions.md for role and group setup.

Debug why /readyz is failing

For component evidence, read the local /health?details=true or /health/mcp snapshot:

curl -fsS 'http://127.0.0.1:8080/health?details=true'
curl -fsS 'http://127.0.0.1:8080/health/mcp'
  • Live and ready, MCP not observed: stdio startup skips discovery. Only ordinary forwarded initialize and tools/list traffic establishes same-child evidence. Repeated health reads cannot make discovery happen.
  • Poll failures: inspect control-plane for completed failures and backoff. A current long poll or paused polling under queue pressure is not an outage.
  • Results missing despite successful polling: inspect response-delivery. Polls and response uploads can fail independently; a benign 404 completion does not mean a new upload was accepted.
  • Backlog: queue depth counts local waiting commands. A full queue means backpressure and does not change readiness by itself.
  • Empty queue but work in progress: inspect dispatcher active operations and oldest active age. This counts real work, not idle worker goroutines; age by itself does not establish that an operation is stuck.
  • Old or partial evidence: check observation timestamps, child generation, and complete/partial/limited fields. Missing names from a partial catalog are not proof that the server lacks those tools.

Each component endpoint returns its value at /health/{component}. These snapshots return 200 even for degraded components; use /readyz for an HTTP readiness gate. Older runtimes may return 404 for the new diagnostics.

If you are debugging why /readyz is failing or why the client never becomes "healthy/ready", start here:

  • tunnel-client health --url-file "$health_url_file" is the fastest structured probe when you already have a health URL file from tunnel-client run; create that path with mktemp instead of a fixed shared /tmp filename.
  • /healthz is liveness only. A 200 live response means the process is up.
  • /readyz includes startup gating:
    • 503 oauth discovery pending while OAuth discovery is still in flight, including timeout-only startup cycles that are waiting to retry.
    • 503 oauth discovery failed: ... when required OAuth discovery reaches a non-timeout failure.
    • 503 mcp probe failed: ... when the MCP startup probe fails.
    • 200 ready (mcp initialize requires auth: ...) when the MCP endpoint is reachable but requires auth during initialize.
    • 200 ready (mcp startup probe timed out: ...) when the probe times out but startup should continue.
  • For a Kubernetes sidecar or other local listener that can bind after tunnel-client starts, set MCP_STARTUP_WAIT_TIMEOUT to a positive duration. During that window, pre-connect connection refused and missing Unix-socket failures keep readiness pending and delay the first control-plane poll; if the wait expires, polling resumes with the legacy behavior while /readyz remains non-ready for the startup failure.
  • If the process is live but /readyz stays non-200, check logs for:
    • OAuth discovery failures
    • control-plane connectivity errors
    • MCP server connectivity errors
  • tunnel-client health --port 8080 is the quickest loopback check when the daemon is bound to the default port.

Export recent logs

  • The admin UI logs panel can download a redacted support archive from /api/logs/export?minutes=30.
  • To save the same archive into the current working directory without using a browser:
curl -fsSJO "http://127.0.0.1:8080/api/logs/export?minutes=30"
  • To capture one archive every five minutes until stopped:
while :; do
  curl -fsSJO "http://127.0.0.1:8080/api/logs/export?minutes=30"
  sleep 300
done
  • The archive contains manifest.json, README.txt, tunnel-client.logs.ndjson, tunnel-client.metrics.prom, admin/status.json, admin/system.json, and admin/oauth.json.
  • tunnel-client.metrics.prom is a point-in-time Prometheus text snapshot captured from /metrics at export time.
  • The admin/*.json files are point-in-time copies of /api/status, /api/system, and /api/oauth at export time, so support can review the configured tunnel_id, process-scoped client_instance_id, route state, probe status, and OAuth discovery state alongside the log stream.
  • The archive is redacted before it is returned.

Connector setup and runtime pitfalls

  • ChatGPT connector setup cannot discover tools

    • Keep tunnel-client run ... running while creating or testing the connector. The remote tunnel object can exist even when no local runtime is polling it.
    • Confirm the connector selected the same CONTROL_PLANE_TUNNEL_ID that the daemon is using.
    • Check /readyz, not only /healthz; liveness does not prove MCP probing or OAuth discovery finished.

  • Connector URL returns 404 or does not stream on GET

    • Connector MCP traffic is POST-based JSON-RPC. GET requests to /v1/mcp/... are not a diagnostic SSE stream.
    • If client logs show doubled paths, set CONTROL_PLANE_BASE_URL to the host root, for example https://api.openai.com, not a /v1/tunnels/... URL.

  • unsupported_channel from the connector path

    • The incoming command named a channel that is not configured. Add a channel-qualified --mcp.server-url / --mcp.command entry, or update the product configuration to send main.
    • For harpoon, register at least one --harpoon.target / HARPOON_TARGETS entry. Harpoon intentionally stays unroutable with an empty target registry.

  • OAuth-protected connector succeeds locally but fails in product

    • The MCP server can stay private, but the authorization server itself is not automatically tunneled. It must be reachable wherever the OAuth browser flow and metadata fetches require it.
    • Issuer mismatch diagnostics are allowed for external enterprise IdPs; focus first on wrong URLs, unreachable metadata endpoints, or missing Authorization forwarding.

See connectors.md for the full connector request lifecycle, channel routing model, and environment-variable checklist.

MCP connectivity issues

  • Verify MCP_SERVER_URL is reachable from where tunnel-client runs.
  • If your MCP server uses a private CA, ensure the OS/container trust store includes it.
  • Consider temporarily enabling --log.http-raw-unsafe and --log.level=debug in a controlled environment to debug handshake issues.

Harpoon channel disabled (unsupported_channel)

  • harpoon commands return unsupported_channel when there are no registered Harpoon targets.
  • Confirm you have configured at least one --harpoon.target or HARPOON_TARGETS entry and that it passes validation.

Performance / backlog

  • Increase --mcp.max-concurrent-requests / MCP_MAX_CONCURRENT_REQUESTS to raise active MCP execution concurrency, but only if the MCP server can safely handle the additional parallelism. When every worker is busy, the dispatcher removes one command from the local queue and waits for a worker slot. It does not drain another command until a slot is free.
  • Increase --control-plane.max-inflight / CONTROL_PLANE_MAX_INFLIGHT_REQUESTS only to increase the local prefetch backlog. It does not increase MCP execution concurrency. A full buffer pauses polling until a queue slot is free; each poll requests at most 25 commands.
  • Account for both independent limits when sizing the process. With the defaults, tunnel-client can hold up to 10 active MCP requests, 20 commands in the local queue, and one dispatcher-held command waiting for a worker slot.