Skip to content

Commit 1ca8116

Browse files
EnricoBausenhartEnricoBausenhart
authored
Feature/monitoring with compose (#93)
2 parents 8f70c5e + 1c3b95c commit 1ca8116

21 files changed

Lines changed: 3280 additions & 1 deletion

File tree

‎deployment/README.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,3 +43,30 @@ kubectl get all -n devoops # Shows all Kubernetes resources dep
4343
helm uninstall meetatmensa -n devoops # Deletes all resources deployed with the Helm chart
4444
```
4545
If you want to access cluster-internal services, you can utilize port-forwarding: `kubectl port-forward svc/meetatmensa-<service> <port>:80 -n devoops`. Do not use ports 8080, 8081 or 8082 - they are reserved as entry points to the application (client service).
46+
47+
## AWS Deployment
48+
49+
For AWS deployment, use the `compose.aws.yml` file which is configured for production deployment with SSL certificates and monitoring.
50+
51+
### Access Points
52+
- **Client Application**: https://client.54.204.29.206.sslip.io
53+
- **API Gateway**: https://api.54.204.29.206.sslip.io
54+
55+
56+
57+
## 📊 Monitoring
58+
59+
Meet@Mensa includes a comprehensive monitoring stack with Prometheus for metrics collection, Grafana for visualization, and Loki for log aggregation.
60+
61+
### Local Deployment (Docker Compose)
62+
63+
When running locally with Docker Compose, monitoring services are available at:
64+
65+
- **Grafana Dashboard**: http://localhost:3000
66+
- Default credentials: `admin` / `admin`
67+
- Access dashboards for application metrics, logs, and system health
68+
69+
- **Prometheus**: http://localhost:9090
70+
- View metrics, alerts, and targets
71+
- Access alert rules and their current state
72+

‎deployment/docker/compose.yml‎

Lines changed: 66 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,10 +79,75 @@ services:
7979
networks:
8080
- backend
8181

82+
prometheus:
83+
image: prom/prometheus:v2.52.0
84+
container_name: prometheus
85+
ports:
86+
- "9090:9090"
87+
volumes:
88+
- ../../prometheus:/etc/prometheus
89+
- prometheus_data:/prometheus
90+
command:
91+
- '--config.file=/etc/prometheus/prometheus.yml'
92+
- '--storage.tsdb.path=/prometheus'
93+
- '--web.console.libraries=/etc/prometheus/console_libraries'
94+
- '--web.console.templates=/etc/prometheus/consoles'
95+
- '--web.enable-lifecycle'
96+
networks:
97+
- backend
98+
restart: unless-stopped
99+
100+
grafana:
101+
image: grafana/grafana-oss:latest
102+
container_name: grafana
103+
ports:
104+
- "3001:3000"
105+
volumes:
106+
- grafana-storage:/var/lib/grafana
107+
- ../../grafana/provisioning:/etc/grafana/provisioning:ro
108+
environment:
109+
- GF_SECURITY_ADMIN_USER=admin
110+
- GF_SECURITY_ADMIN_PASSWORD=admin
111+
- GF_USERS_ALLOW_SIGN_UP=false
112+
- GF_FEATURE_TOGGLES_ENABLE=logsInExplore
113+
- GF_LOG_CONSOLECOLORS=true
114+
depends_on:
115+
- prometheus
116+
- loki
117+
networks:
118+
- backend
119+
restart: unless-stopped
120+
121+
promtail:
122+
image: grafana/promtail:latest
123+
volumes:
124+
- /var/lib/docker/containers:/var/lib/docker/containers:ro
125+
- ../../promtail/promtail.yml:/etc/promtail/promtail.yml
126+
command: -config.file=/etc/promtail/promtail.yml
127+
depends_on:
128+
- loki
129+
networks:
130+
- backend
131+
restart: unless-stopped
132+
133+
loki:
134+
image: grafana/loki:2.9.0
135+
ports:
136+
- "3100:3100"
137+
volumes:
138+
- ../../loki/loki-config.yaml:/etc/loki/loki-config.yaml:ro
139+
command: -config.file=/etc/loki/loki-config.yaml
140+
networks:
141+
- backend
142+
restart: unless-stopped
143+
82144
networks:
83145
backend:
84146
driver: bridge
85147

86148
volumes:
87149
matchdb_data:
88-
userdb_data:
150+
userdb_data:
151+
prometheus_data:
152+
grafana-storage:
153+
loki-data:

‎grafana/provisioning/dashboards/README.md‎

Lines changed: 243 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,243 @@
1+
# Meet@Mensa Grafana Dashboards Documentation
2+
3+
This directory contains Grafana dashboards for monitoring the Meet@Mensa microservices application. Each dashboard is designed to provide specific insights into different aspects of the system.
4+
5+
**Business Request**: Any HTTP/API request that directly relates to core business functionality (e.g., creating match requests, user registration, GenAI interactions) - distinct from technical requests like health checks.
6+
7+
## Dashboard Overview
8+
9+
### 1. **Meet@Mensa Overview** (`meetatmensa-overview.json`)
10+
**Purpose**: High-level system overview for executive and operational monitoring
11+
12+
**Key Metrics Monitored**:
13+
- **Request Rate (req/sec)**: Total requests per second across all services
14+
- *Why monitor*: Indicates system load and user activity
15+
- *Alert threshold*: >80 req/sec (red), >100 req/sec (yellow)
16+
- **Active Requests**: Number of concurrent requests being processed
17+
- *Why monitor*: System load and capacity utilization
18+
- *Alert threshold*: >100 active requests (warning)
19+
- **Memory Usage**: Memory consumption across services
20+
- *Why monitor*: Resource utilization and potential memory leaks
21+
- *Alert threshold*: >80% (warning), >90% (critical)
22+
- **Active Threads**: Number of active threads per service
23+
- *Why monitor*: Thread pool utilization and performance
24+
25+
**Alerts Configured**:
26+
- `ServiceDown`: Triggers when any service is down for >1 minute
27+
- `HighErrorRate`: Triggers when error rate >5% for 5 minutes
28+
- `CriticalErrorRate`: Triggers when error rate >10% for 3 minutes
29+
30+
---
31+
32+
### 2. **Meet@Mensa Application** (`meetatmensa-application.json`)
33+
**Purpose**: Business-focused metrics and application performance
34+
35+
**Key Metrics Monitored**:
36+
- **Business Request Rate**: Requests per second for business operations
37+
- *Why monitor*: Track user engagement and feature usage
38+
- **Active Business Requests**: Number of concurrent business operations
39+
- *Why monitor*: Business activity and system load
40+
- **Database Connections by Service**: Active database connections per service
41+
- *Why monitor*: Database connection pool utilization
42+
- **Database Idle Connections by Service**: Idle database connections per service
43+
- *Why monitor*: Connection pool efficiency and resource management
44+
45+
**Alerts Configured**:
46+
- `LowRequestRate`: Triggers when request rate <0.1 req/sec for 10 minutes
47+
- `HighActiveRequests`: Triggers when active requests >100 for 5 minutes
48+
49+
---
50+
51+
### 3. **Meet@Mensa Microservices** (`meetatmensa-microservices.json`)
52+
**Purpose**: Detailed service-level monitoring and troubleshooting
53+
54+
**Key Metrics Monitored**:
55+
- **Request Rate by Service**: Individual service request rates
56+
- *Why monitor*: Identify bottlenecks and service-specific issues
57+
- **JVM Memory Usage (%)**: Java Virtual Machine memory utilization per service
58+
- *Why monitor*: Memory leaks and resource management
59+
- **Active Threads**: Number of active threads per service
60+
- *Why monitor*: Thread pool utilization and performance
61+
- **Database Active Connections**: Active database connections per service
62+
- *Why monitor*: Database performance and connection management
63+
64+
**Alerts Configured**:
65+
- `HighMemoryUsage`: Triggers when memory usage >80% for 10 minutes
66+
- `CriticalMemoryUsage`: Triggers when memory usage >90% for 5 minutes
67+
- `DatabaseConnectionPoolExhausted`: Triggers when pool usage >80% for 5 minutes
68+
- `DatabaseConnectionPoolFull`: Triggers when pool usage >95% for 2 minutes
69+
70+
---
71+
72+
### 4. **Meet@Mensa Logs** (`meetatmensa-logs.json`)
73+
**Purpose**: Centralized log analysis and error tracking
74+
75+
**Key Metrics Monitored**:
76+
- **Error and Warning Rate**: Log-based error and warning frequency
77+
- *Why monitor*: Proactive error detection and debugging
78+
- **Error and Warning Logs**: Detailed error and warning log entries
79+
- *Why monitor*: Identify problematic services and error patterns
80+
- **Info Logs**: Information-level log entries
81+
- *Why monitor*: General system activity and debugging
82+
83+
**Data Source**: Loki (log aggregation)
84+
**Query Examples**:
85+
- `sum(rate({job="meetatmensa-app"} | json | level="ERROR" [5m])) by (container_name)`
86+
- `sum(rate({job="meetatmensa-app"} | json | level="WARN" [5m])) by (container_name)`
87+
88+
---
89+
90+
### 5. **Meet@Mensa Custom Metrics** (`meetatmensa-custom-metrics.json`)
91+
**Purpose**: Business-specific metrics and KPIs
92+
93+
**Key Metrics Monitored**:
94+
- **Total Users Created**: Cumulative user registrations
95+
- *Why monitor*: Business growth and user acquisition
96+
- *Metric name*: `users_total`
97+
- **Total Match Requests Created**: Cumulative match requests
98+
- *Why monitor*: Core business activity and user engagement
99+
- *Metric name*: `match_requests_total`
100+
- **Total GenAI Requests**: Cumulative AI-powered conversation starter requests
101+
- *Why monitor*: Feature adoption and AI service performance
102+
- *Metric name*: `genai_requests_total`
103+
- **Registration and Request Rates**: User registration and match request rates over time
104+
- *Why monitor*: Business activity trends and user engagement
105+
- **GenAI Request Rate**: AI service requests per time period
106+
- *Why monitor*: AI feature usage patterns and performance
107+
108+
**Note**: These metrics require custom implementation in the application code.
109+
110+
---
111+
112+
### 6. **Sample Dashboard** (`sample-dashboard.json`)
113+
**Purpose**: Template and example dashboard for reference
114+
115+
**Key Metrics Monitored**:
116+
- **New panel**: Example panel for testing and learning
117+
- *Why monitor*: Template for creating new dashboard panels
118+
119+
**Usage**:
120+
- Reference for creating new dashboards
121+
- Testing dashboard configurations
122+
- Learning Grafana dashboard structure
123+
124+
---
125+
126+
## Alert Strategy
127+
128+
### Alert Severity Levels
129+
130+
1. **Critical (Red)**: Immediate action required
131+
- Service down
132+
- High error rates (>10%)
133+
- Critical response times (>2s)
134+
- Memory usage >90%
135+
136+
2. **Warning (Yellow)**: Attention needed
137+
- High error rates (>5%)
138+
- High response times (>1s)
139+
- Memory usage >80%
140+
- Database connection pool >80%
141+
142+
3. **Info (Blue)**: Monitoring
143+
- Low request rates
144+
- High active requests
145+
146+
### Alert Response Actions
147+
148+
**Service Down**:
149+
1. Check container status: `docker ps`
150+
2. Check service logs: `docker logs <container>`
151+
3. Restart service if needed: `docker-compose restart <service>`
152+
153+
**High Error Rate**:
154+
1. Check application logs in Grafana
155+
2. Review recent deployments
156+
3. Check database connectivity
157+
4. Scale service if needed
158+
159+
**High Response Time**:
160+
1. Check CPU and memory usage
161+
2. Review database performance
162+
3. Check network connectivity
163+
4. Consider service scaling
164+
165+
**Memory Issues**:
166+
1. Check for memory leaks
167+
2. Review garbage collection logs
168+
3. Increase memory limits if needed
169+
4. Restart service if critical
170+
171+
---
172+
173+
## Data Sources
174+
175+
### Prometheus
176+
- **Purpose**: Metrics collection and storage
177+
- **URL**: `http://prometheus:9090`
178+
- **Metrics**: System metrics, application metrics, custom business metrics
179+
180+
### Loki
181+
- **Purpose**: Log aggregation and querying
182+
- **URL**: `http://loki:3100`
183+
- **Data**: Application logs, error logs, access logs
184+
185+
---
186+
187+
## Dashboard Refresh and Updates
188+
189+
### Automatic Refresh
190+
- Dashboards refresh every 30 seconds by default
191+
- Time range: Last 1 hour (configurable)
192+
193+
### Manual Updates
194+
To apply dashboard changes:
195+
```bash
196+
# Restart Grafana container
197+
docker-compose -f deployment/docker/compose.yml restart grafana
198+
199+
# Or restart the entire stack
200+
docker-compose -f deployment/docker/compose.yml down
201+
docker-compose -f deployment/docker/compose.yml up -d
202+
```
203+
204+
### Dashboard Provisioning
205+
Dashboards are automatically provisioned from JSON files in this directory. Changes to JSON files require a Grafana restart to take effect.
206+
207+
---
208+
209+
## Troubleshooting
210+
211+
### No Data in Dashboards
212+
1. **Check Prometheus targets**: Visit `http://localhost:9090/targets`
213+
2. **Verify service endpoints**: Check `/actuator/prometheus` endpoints
214+
3. **Check time range**: Ensure dashboard time range includes data
215+
4. **Verify metric names**: Use Prometheus UI to search for metrics
216+
217+
### Missing Custom Metrics
218+
1. **Check application code**: Ensure metrics are properly implemented
219+
2. **Verify metric names**: Match exact names in dashboard queries
220+
3. **Check Prometheus configuration**: Ensure services are scraped
221+
4. **Restart services**: Reload metrics after code changes
222+
223+
### Log Issues
224+
1. **Check Promtail configuration**: Verify log collection setup
225+
2. **Check Loki connectivity**: Ensure Loki is running and accessible
226+
3. **Verify log format**: Ensure logs are in expected JSON format
227+
4. **Check container logs**: Verify logs are being generated
228+
229+
---
230+
231+
## Best Practices
232+
233+
1. **Monitor Key Business Metrics**: Focus on metrics that impact user experience
234+
2. **Set Appropriate Thresholds**: Base thresholds on historical data and SLAs
235+
3. **Use Meaningful Alert Messages**: Include actionable information in alerts
236+
4. **Regular Dashboard Reviews**: Update dashboards based on changing needs
237+
5. **Document Changes**: Update this README when adding new metrics or alerts
238+
239+
---
240+
241+
## Contact
242+
243+
For questions about monitoring setup or dashboard configuration, refer to the team documentation or contact the DevOps team.

‎grafana/provisioning/dashboards/dashboards.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
apiVersion: 1
2+
3+
providers:
4+
- name: 'default'
5+
orgId: 1
6+
folder: ''
7+
type: file
8+
disableDeletion: false
9+
updateIntervalSeconds: 10
10+
allowUiUpdates: true
11+
options:
12+
path: /etc/grafana/provisioning/dashboards
13+
foldersFromFilesStructure: true
14+
15+
- name: 'meetatmensa-dashboards'
16+
orgId: 1
17+
folder: 'Meet@Mensa'
18+
type: file
19+
disableDeletion: false
20+
updateIntervalSeconds: 10
21+
allowUiUpdates: true
22+
options:
23+
path: /etc/grafana/provisioning/dashboards
24+
foldersFromFilesStructure: false

0 commit comments

Comments
 (0)