Skip to content

Commit 049787d

Browse files
Cdeth567Cdeth567
authored
Merge pull request #18 from Cdeth567/lab16
lab16
2 parents 070ba9d + 22f3d28 commit 049787d

11 files changed

Lines changed: 312 additions & 0 deletions

‎k8s/MONITORING.md‎

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
# Kubernetes Monitoring & Init Containers
2+
3+
## 1. Stack Components
4+
5+
### Prometheus Operator
6+
Prometheus Operator manages the monitoring stack through Kubernetes custom resources. It reconciles Prometheus, Alertmanager, ServiceMonitor, PodMonitor, and related objects so the configuration stays declarative.
7+
8+
### Prometheus
9+
Prometheus scrapes metrics endpoints, stores time-series data, and provides the query engine used by dashboards and alerting rules.
10+
11+
### Alertmanager
12+
Alertmanager receives alerts from Prometheus, groups and deduplicates them, and exposes the active alert state in its web UI.
13+
14+
### Grafana
15+
Grafana is the visualization layer of the stack. It uses Prometheus as a data source and provides dashboards for pods, namespaces, nodes, kubelet internals, and alerts.
16+
17+
### kube-state-metrics
18+
kube-state-metrics exposes Kubernetes object state such as pods, StatefulSets, PVCs, deployments, and nodes.
19+
20+
### node-exporter
21+
node-exporter exposes host-level metrics from the Kubernetes node, including CPU, memory, filesystem, and network statistics.
22+
23+
---
24+
25+
## 2. Installation Evidence
26+
27+
### Helm installation
28+
29+
```bash
30+
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
31+
helm repo update
32+
helm upgrade --install monitoring prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace -f ./k8s/monitoring/values.yaml
33+
```
34+
35+
### Verification
36+
37+
```bash
38+
kubectl get po,svc -n monitoring
39+
NAME READY STATUS RESTARTS AGE
40+
pod/alertmanager-monitoring-kube-prometheus-alertmanager-0 2/2 Running 2 (86s ago) 2d14h
41+
pod/monitoring-grafana-777595984c-8zfk4 3/3 Running 3 (94s ago) 2d14h
42+
pod/monitoring-kube-prometheus-operator-54f68d65b4-9j4px 1/1 Running 1 (80s ago) 2d14h
43+
pod/monitoring-kube-state-metrics-5957bd45bc-47zrk 1/1 Running 1 (80s ago) 2d14h
44+
pod/monitoring-prometheus-node-exporter-h8n62 1/1 Running 1 (80s ago) 2d14h
45+
pod/prometheus-monitoring-kube-prometheus-prometheus-0 2/2 Running 2 (95s ago) 2d14h
46+
47+
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
48+
service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 2d14h
49+
service/monitoring-grafana ClusterIP 10.109.86.73 <none> 80/TCP 2d14h
50+
service/monitoring-kube-prometheus-alertmanager ClusterIP 10.96.207.158 <none> 9093/TCP,8080/TCP 2d14h
51+
service/monitoring-kube-prometheus-operator ClusterIP 10.108.249.248 <none> 443/TCP 2d14h
52+
service/monitoring-kube-prometheus-prometheus ClusterIP 10.106.216.232 <none> 9090/TCP,8080/TCP 2d14h
53+
service/monitoring-kube-state-metrics ClusterIP 10.106.107.219 <none> 8080/TCP 2d14h
54+
service/monitoring-prometheus-node-exporter ClusterIP 10.110.85.180 <none> 9100/TCP 2d14h
55+
service/prometheus-operated ClusterIP None <none> 9090/TCP 2d14h
56+
```
57+
58+
---
59+
60+
## 3. Dashboard Answers
61+
62+
### 3.1 Pod Resources: StatefulSet
63+
Dashboard: **Kubernetes / Compute Resources / Pod**
64+
65+
Observed pod:
66+
- namespace: `dev`
67+
- pod: `lab15-devops-info-service-0`
68+
69+
Observed values:
70+
- CPU request: **0.0500 cores**
71+
- CPU limit: **0.100 cores**
72+
- Memory request: **64 MiB**
73+
- Memory limit: **128 MiB**
74+
75+
![StatefulSet pod resources](./screenshots/monitoring/02-statefulset-pod-resources.png)
76+
77+
### 3.2 Namespace Analysis: default namespace
78+
Dashboard: **Kubernetes / Compute Resources / Namespace (Pods)**
79+
80+
At the time of observation, the active demo workload visible in the `default` namespace dashboard was `traffic-demo-85bbcb8477-c9hxk`. In that recorded state it was both the highest and the lowest CPU consumer among the active demo pods shown for the namespace, because it was the only active workload listed in the quota section.
81+
82+
![Default namespace CPU analysis](./screenshots/monitoring/03-default-namespace-cpu.png)
83+
84+
### 3.3 Node Metrics
85+
Dashboard: **Node Exporter / Nodes**
86+
87+
Observed values:
88+
- Memory usage: **30.3%**
89+
- Approximate memory usage: **about 5 GiB used**
90+
- Logical CPU cores: **20**
91+
92+
![Node metrics](./screenshots/monitoring/04-node-metrics.png)
93+
94+
### 3.4 Kubelet
95+
Dashboard: **Kubernetes / Kubelet**
96+
97+
Observed values:
98+
- Running kubelets: **1**
99+
- Running pods: **45**
100+
- Running containers: **108**
101+
102+
![Kubelet summary](./screenshots/monitoring/05-kubelet-summary.png)
103+
104+
### 3.5 Network: default namespace
105+
Dashboard: **Kubernetes / Compute Resources / Pod**
106+
107+
A demo pod in the `default` namespace was opened in Grafana to inspect live activity:
108+
- namespace: `default`
109+
- pod: `traffic-demo-85bbcb8477-c9hxk`
110+
111+
During the recorded interval, the dashboard showed CPU and memory request/limit series for the pod. No separate non-zero network traffic series were visible in the captured window.
112+
113+
![Default namespace pod observation](./screenshots/monitoring/06-default-pod-resources.png)
114+
115+
### 3.6 Alerts
116+
UI: **Alertmanager**
117+
118+
Observed values:
119+
- Active alerts: **1**
120+
- Visible alert: **Watchdog**
121+
122+
![Alertmanager active alerts](./screenshots/monitoring/07-alertmanager-active-alerts.png)
123+
124+
---
125+
126+
## 4. Init Containers
127+
128+
### 4.1 Download file pattern
129+
Manifest: `k8s/init-containers/download-demo.yaml`
130+
131+
The init container downloads `https://example.com` with `wget` into a shared `emptyDir` volume. The main container mounts the same volume at `/data` and reads the downloaded file.
132+
133+
Verification:
134+
135+
```bash
136+
kubectl apply -f ./k8s/init-containers/download-demo.yaml
137+
kubectl get pods -w
138+
kubectl logs init-download-demo -c init-download
139+
kubectl exec init-download-demo -- cat /data/index.html
140+
```
141+
142+
Observed output:
143+
144+
```bash
145+
kubectl logs init-download-demo -c init-download
146+
Connecting to example.com (104.20.23.154:443)
147+
wget: note: TLS certificate validation not implemented
148+
saving to '/work-dir/index.html'
149+
index.html 100% |********************************| 528 0:00:00 ETA
150+
'/work-dir/index.html' saved
151+
Download complete
152+
```
153+
154+
### 4.2 Wait-for-service pattern
155+
Manifest: `k8s/init-containers/wait-for-service-demo.yaml`
156+
157+
The init container waits until `monitoring-grafana.monitoring.svc.cluster.local` becomes resolvable. Only after that does the main container start.
158+
159+
Verification:
160+
161+
```bash
162+
kubectl apply -f ./k8s/init-containers/wait-for-service-demo.yaml
163+
kubectl get pods -w
164+
kubectl logs wait-for-service-demo -c wait-for-grafana
165+
kubectl logs wait-for-service-demo -c main-app
166+
```
167+
168+
Observed output:
169+
170+
```bash
171+
kubectl logs wait-for-service-demo -c wait-for-grafana
172+
Server: 10.96.0.10
173+
Address: 10.96.0.10:53
174+
175+
Name: monitoring-grafana.monitoring.svc.cluster.local
176+
Address: 10.109.86.73
177+
178+
Dependency is ready
179+
```
180+
181+
```bash
182+
kubectl logs wait-for-service-demo -c main-app
183+
Main container started after dependency check
184+
```
185+
186+
---
187+
188+
## 5. Result
189+
190+
This lab demonstrates:
191+
- successful installation of the Kube-Prometheus monitoring stack in the `monitoring` namespace;
192+
- Grafana and Alertmanager access through `kubectl port-forward`;
193+
- dashboard-based inspection of pod, namespace, node, kubelet, and alert data;
194+
- one active alert visible in Alertmanager;
195+
- two working init container patterns:
196+
- downloading data before the main container starts;
197+
- waiting for a service dependency before startup.
198+
199+
The bonus task with custom metrics and `ServiceMonitor` was not implemented.

‎k8s/init-containers/download-demo.yaml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
apiVersion: v1
2+
kind: Pod
3+
metadata:
4+
name: init-download-demo
5+
labels:
6+
app: init-download-demo
7+
spec:
8+
restartPolicy: Never
9+
initContainers:
10+
- name: init-download
11+
image: busybox:1.36
12+
command:
13+
- sh
14+
- -c
15+
- |
16+
wget -O /work-dir/index.html https://example.com
17+
echo "Download complete"
18+
volumeMounts:
19+
- name: workdir
20+
mountPath: /work-dir
21+
containers:
22+
- name: main-app
23+
image: busybox:1.36
24+
command:
25+
- sh
26+
- -c
27+
- |
28+
echo "Main container started"
29+
sleep 3600
30+
volumeMounts:
31+
- name: workdir
32+
mountPath: /data
33+
volumes:
34+
- name: workdir
35+
emptyDir: {}

‎k8s/init-containers/wait-for-service-demo.yaml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
apiVersion: v1
2+
kind: Pod
3+
metadata:
4+
name: wait-for-service-demo
5+
labels:
6+
app: wait-for-service-demo
7+
spec:
8+
restartPolicy: Never
9+
initContainers:
10+
- name: wait-for-grafana
11+
image: busybox:1.36
12+
command:
13+
- sh
14+
- -c
15+
- |
16+
until nslookup monitoring-grafana.monitoring.svc.cluster.local; do
17+
echo "Waiting for monitoring-grafana.monitoring.svc.cluster.local..."
18+
sleep 2
19+
done
20+
echo "Dependency is ready"
21+
containers:
22+
- name: main-app
23+
image: busybox:1.36
24+
command:
25+
- sh
26+
- -c
27+
- |
28+
echo "Main container started after dependency check"
29+
sleep 3600

‎k8s/monitoring/values.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
grafana:
2+
adminPassword: prom-operator
3+
service:
4+
type: ClusterIP
5+
6+
alertmanager:
7+
alertmanagerSpec:
8+
replicas: 1
9+
10+
prometheus:
11+
prometheusSpec:
12+
retention: 3d
13+
serviceMonitorSelectorNilUsesHelmValues: false
14+
podMonitorSelectorNilUsesHelmValues: false
15+
16+
kube-state-metrics:
17+
enabled: true
18+
19+
nodeExporter:
20+
enabled: true

‎k8s/screenshots/monitoring/02-statefulset-pod-resources.png‎

251 KB
Loading

‎k8s/screenshots/monitoring/03-default-namespace-cpu.png‎

327 KB
Loading

‎k8s/screenshots/monitoring/04-node-metrics.png‎

469 KB
Loading

‎k8s/screenshots/monitoring/05-kubelet-summary.png‎

508 KB
Loading

‎k8s/screenshots/monitoring/06-default-pod-resources.png‎

243 KB
Loading

‎k8s/screenshots/monitoring/07-alertmanager-active-alerts.png‎

166 KB
Loading

0 commit comments

Comments
 (0)