Observability: Local development stack

[i5okie]

Description

Add the grafana/otel-lgtm observability stack to the local development environment so developers can validate instrumentation locally before deploying to OpenShift.

Context

acapy-agent uses a VS Code devcontainer (.devcontainer/) with docker-in-docker. Developers run aca-py directly in the devcontainer shell — not as a compose service. There is no root-level docker-compose.yml for general local dev.

Requirements

• Create docker-compose.observability.yml at the repo root with only the grafana/otel-lgtm service (ports 3000:3000, 4317:4317, 4318:4318)
• Set the following env vars before running aca-py in the devcontainer shell (e.g. exported in the terminal or sourced from a .env.observability file):
  • OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 (LGTM port is mapped to the devcontainer localhost via docker-in-docker)
  • OTEL_SERVICE_NAME=acapy-agent
  • OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
  • OTEL_TRACES_EXPORTER=otlp
  • OTEL_METRICS_EXPORTER=otlp
  • OTEL_PYTHON_LOG_CORRELATION=true
• Update README or CONTRIBUTING with local observability instructions

Acceptance Criteria

• docker compose -f docker-compose.observability.yml up -d starts the LGTM stack
• Grafana accessible at http://localhost:3000
• Traces visible in Tempo after instrumentation is added and test requests are made locally
• Developer docs updated

[swcurran]

Is this work BC Gov-specific or of general value to the ACA-Py community. Given the reference to OpenShift, I wonder if this should be targetted to the BC Gov deployment repo for ACA-Py vs. here.