Observability Stack

Production-ready monitoring and telemetry stack using open-source observability tools. Supports both production deployment with Traefik and local development environment.

Overview

Complete observability platform covering the three pillars:

• Metrics: Prometheus + Node Exporter + cAdvisor
• Logs: Loki + Promtail
• Tracing: Jaeger + OpenTelemetry Collector
• Visualization: Grafana

Architecture

graph TB
    %% Styles
    classDef proxy fill:#00bfff,stroke:#0080ff,stroke-width:3px,color:#fff
    classDef visualization fill:#ff6b35,stroke:#ff4500,stroke-width:2px,color:#fff
    classDef metrics fill:#4caf50,stroke:#2e7d32,stroke-width:2px,color:#fff
    classDef logs fill:#ffc107,stroke:#ff9800,stroke-width:2px,color:#000
    classDef tracing fill:#9c27b0,stroke:#7b1fa2,stroke-width:2px,color:#fff
    classDef collector fill:#00acc1,stroke:#00838f,stroke-width:2px,color:#fff
    classDef exporter fill:#78909c,stroke:#546e7a,stroke-width:2px,color:#fff
    
    %% Entry Layer
    Traefik[Traefik<br/>Reverse Proxy + SSL<br/>:80, :443]:::proxy
    
    %% Visualization
    Grafana[Grafana<br/>Dashboards & Alerts<br/>:3000]:::visualization
    
    %% Storage Layer
    Prometheus[Prometheus<br/>Metrics & Alerts<br/>:9090]:::metrics
    Loki[Loki<br/>Log Aggregation<br/>:3100]:::logs
    Jaeger[Jaeger<br/>Distributed Tracing<br/>:16686]:::tracing
    
    %% Collector
    OTEL[OTEL Collector<br/>Unified Telemetry<br/>:4317, :4318]:::collector
    
    %% Exporters
    NodeExp[Node Exporter<br/>System Metrics<br/>:9100]:::exporter
    cAdvisor[cAdvisor<br/>Container Metrics<br/>:8080]:::exporter
    Promtail[Promtail<br/>Log Collection<br/>Agent]:::exporter
    
    %% Data Sources
    Apps[Applications<br/>Microservices]
    System[System<br/>Host + Docker]
    
    %% Entry Flows
    Traefik -->|HTTPS| Grafana
    Traefik -->|HTTPS| Prometheus
    Traefik -->|HTTPS| Jaeger
    Traefik -->|HTTPS| Loki
    
    %% Grafana Connections
    Grafana -->|Query| Prometheus
    Grafana -->|Query| Loki
    Grafana -->|Query| Jaeger
    
    %% Metrics Flow
    NodeExp -->|Scrape| Prometheus
    cAdvisor -->|Scrape| Prometheus
    OTEL -->|Export| Prometheus
    
    %% Logs Flow
    Promtail -->|Push| Loki
    OTEL -->|Export| Loki
    System -->|Read| Promtail
    
    %% Traces Flow
    OTEL -->|OTLP| Jaeger
    
    %% Apps to OTEL
    Apps -->|OTLP| OTEL
    
    %% System Data
    System -->|Metrics| NodeExp
    System -->|Metrics| cAdvisor

Loading

Components

Grafana

Port: 3000
Unified visualization dashboard with pre-configured datasources (Prometheus, Loki, Jaeger), alerting support, and auto-provisioning.

Prometheus

Port: 9090
Metrics collection and alerting system. Scrapes metrics from Node Exporter, cAdvisor, and OTEL Collector. Default retention: 2 days.

Loki

Port: 3100
Log aggregation system with filesystem storage, BoltDB schema v11, and 7-day retention.

Jaeger

Port: 16686
Distributed tracing system with native OTLP support and gRPC/HTTP collectors.

OpenTelemetry Collector

Ports: 4317 (gRPC), 55681 (HTTP), 8889 (metrics)
Unified telemetry collector with pipelines for metrics, traces, and logs.

Promtail

Log collection agent that reads system logs (/var/log) and Docker container logs, pushing them to Loki.

Node Exporter

Port: 9100
System metrics exporter for CPU, memory, disk, network, and processes.

cAdvisor

Port: 8080
Container metrics exporter for Docker containers.

Traefik

Ports: 80 (HTTP), 443 (HTTPS)
Reverse proxy with automatic SSL/TLS via Let's Encrypt and HTTP to HTTPS redirection.

Quick Start

Production (Docker Swarm + Traefik)

# Create networks and volumes
docker network create -d overlay network_public
docker volume create certificates

# Deploy stack
docker stack deploy -c docker-compose.yml metrics

Access via:

• Grafana: https://metrics.your-domain.com
• Prometheus: https://metrics.your-domain.com/prometheus
• Jaeger: https://metrics.your-domain.com/jaeger

Local Development

docker-compose -f docker-compose.local.yml up -d

Access via:

• Grafana: http://localhost:3000 (admin/admin)
• Prometheus: http://localhost:9090
• Jaeger: http://localhost:16686
• cAdvisor: http://localhost:8080

Grafana

• Datasources: grafana/provisioning/datasources/datasource.yml
• Dashboards: Add JSON files to grafana/provisioning/dashboards/
• Environment variables: See environment section in docker-compose

Loki

Edit loki/loki-config.yml to:

• Adjust log retention
• Configure ingestion limits
• Change storage backend

OTEL Collector

Edit otel-collector/otel-config.yml to:

• Add new receivers
• Configure custom processors
• Adjust exporters

Configuration

Directory Structure

├── docker-compose.yml           # Production stack
├── docker-compose.local.yml     # Local development
├── grafana/provisioning/
│   ├── dashboards/
│   └── datasources/
├── prometheus/
│   ├── prometheus.yml
│   ├── rules.yml
│   └── web.yml
├── loki/loki-config.yml
├── otel-collector/otel-config.yml
└── promtail/promtail-config.yml

Customization

Prometheus - Edit prometheus/prometheus.yml to add scrape targets or alert rules.

Monitoring and Alerts

Check Service Health

# Via Docker Swarm
docker service ls
docker service ps metrics_prometheus

# Via Docker Compose
docker-compose -f docker-compose.local.yml ps
docker-compose -f docker-compose.local.yml logs -f

Prometheus Targets

Access Prometheus → Status → Targets to verify all exporters are UP.

Recommended Dashboards

Import these dashboards in Grafana (Dashboard → Import):

ID	Name	Description
1860	Node Exporter Full	Detailed system metrics
893	Docker and System Monitoring	Container overview
13639	Loki Dashboard	Log visualization
13407	Jaeger Dashboard	Trace analysis

Alert Rules Example

groups:
  - name: service-alerts
    interval: 30s
    rules:
      - alert: ServiceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical

Integration Examples

Send Metrics

from prometheus_client import Counter, push_to_gateway
counter = Counter('my_metric', 'Description')
counter.inc()
push_to_gateway('prometheus:9091', job='my_job', registry=registry)

Send Traces

const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');

const exporter = new OTLPTraceExporter({
  url: 'http://otel-collector:4317',
});

Useful Commands

# Production
docker stack deploy -c docker-compose.yml metrics
docker stack rm metrics
docker service logs -f metrics_grafana

# Development
docker-compose -f docker-compose.local.yml up -d
docker-compose -f docker-compose.local.yml down
docker-compose -f docker-compose.local.yml logs -f

# Maintenance
docker exec -it prometheus promtool tsdb analyze /prometheus

Documentation

• Grafana
• Prometheus
• Loki
• Jaeger
• OpenTelemetry