Skip to content

[codex] Add LLM observability guards#435

Open
Dreamstore2046 wants to merge 1 commit into
harystyleseze:mainfrom
Dreamstore2046:codex/careguard-llm-observability-243
Open

[codex] Add LLM observability guards#435
Dreamstore2046 wants to merge 1 commit into
harystyleseze:mainfrom
Dreamstore2046:codex/careguard-llm-observability-243

Conversation

@Dreamstore2046

Copy link
Copy Markdown

Summary

  • add LLM call counters and latency histograms with success/error labels
  • log LLM failures as structured Pino events with event, model, latency_ms, requestId, and serialized error
  • remove the remaining console.* use from agent/ and add an agent-specific ESLint no-console guard
  • add a Prometheus alert for agent_llm_error_total / agent_llm_calls_total > 5% over 10 minutes
  • document the new LLM observability metrics

Validation

  • npm test -- shared/__tests__/metrics.test.ts agent/__tests__/llm-error-fallback.test.ts agent/__tests__/no-console.test.ts
  • npm run lint:agent
  • npx tsc --noEmit --target ES2022 --module ESNext --moduleResolution bundler --allowImportingTsExtensions --esModuleInterop --allowSyntheticDefaultImports --strict --skipLibCheck --types node,vitest shared/metrics.ts shared/__tests__/metrics.test.ts
  • node -e "const fs=require('fs'); const yaml=require('js-yaml'); for (const f of ['docker/prometheus/prometheus.yml','docker/prometheus/alerts.yml']) { yaml.load(fs.readFileSync(f,'utf8')); } console.log('prometheus yaml ok')"
  • git diff --cached --check

Notes:

  • A broader focused tsc including agent/server.ts is still blocked by existing unrelated project errors in agent/tools.ts x402 template-literal arguments and shared/rate-limit.ts typings.
  • docker compose config --quiet could not run in this local environment because Docker is not installed.

Closes #243.

@vercel

vercel Bot commented Jun 26, 2026

Copy link
Copy Markdown

@Dreamstore2046 is attempting to deploy a commit to the Harrison Eze's projects Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LLM errors logged as console strings — un-queryable in production

1 participant