# Day 31 - KUBERNETES MONITORING USING PROMETHEUS & GRAFANA

In this session, we learn how to monitor a Kubernetes cluster using **Prometheus** and **Grafana**.  
This is not just theory — there is a GitHub repository containing all installation commands and demo steps.  
The repo will also be enhanced later with *advanced Kubernetes monitoring* and *custom metric server* topics, so you can star it and follow future updates.

---

## ✅ **WHY MONITORING?**

If you have only **one Kubernetes cluster**, monitoring is easy.  
But in a real company:

* Multiple teams use the same cluster.
    
* Teams complain: *“My deployment is not receiving requests”,* or *“Service was down for some time.”*
    
* You may have multiple clusters: **dev**, **staging**, **prod**.
    

As clusters increase, **you need a monitoring solution** to understand:

* What is happening inside your clusters?
    
* Which deployment is down?
    
* Is API server healthy?
    
* Are replica counts matching?
    
* Are nodes running or not?
    

---

## ✅ **WHY PROMETHEUS?**

Prometheus was created by SoundCloud and is now completely open-source.

Prometheus:

* Scrapes metrics from Kubernetes.
    
* Stores metrics in a **Time Series Database**.
    
* Can trigger alerts using **Alertmanager**.
    
* Provides a UI for running **PromQL** queries.
    

---

## ✅ **WHY GRAFANA?**

Prometheus gives output but not visually appealing.

Grafana:

* Connects to Prometheus as a **data source**.
    
* Visualizes data using dashboards.
    
* Makes metrics easy to understand.
    

---

# ✅ **PROMETHEUS ARCHITECTURE (Simple Explanation)**

Prometheus includes:

### **1️⃣ Prometheus Server**

* Scrapes metrics from Kubernetes API Server.
    
* Stores metrics in time-series format on disk.
    

### **2️⃣ Kubernetes API Server**

* Exposes built-in metrics at:  
    `/metrics`
    
* Shows default cluster metrics.
    

### **3️⃣ Alertmanager**

* Prometheus pushes alerts to it.
    
* Alertmanager sends notifications (Slack, Email, etc.).
    

### **4️⃣ PromQL Interface**

* Used in Prometheus UI or Grafana to run queries.
    

### **5️⃣ External Access**

* Grafana pulls data from Prometheus.
    
* Tools like curl or Postman can also query Prometheus APIs.
    

---

# ✅ **GRAFANA USE**

Grafana helps visualize Prometheus data through graphs and dashboards.

---

# ✅ **DEMO: INSTALL PROMETHEUS + GRAFANA ON MINIKUBE**

We create a Kubernetes cluster using Minikube:

```plaintext
minikube start --memory=4096 --driver=hyperkit
```

(Use hyperkit on Mac, VirtualBox or Docker on other systems.)

---

# 🔥 **INSTALL PROMETHEUS USING HELM**

1. Add the Prometheus Helm repo:
    

```plaintext
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
```

2. Install Prometheus:
    

```plaintext
helm install prometheus prometheus-community/prometheus
```

3. Check pods:
    

```plaintext
kubectl get pods -A
```

You will see:

* Prometheus server
    
* Prometheus alertmanager
    
* kube-state-metrics
    
* node-exporter
    

### ❗ kube-state-metrics

This component provides **extra Kubernetes metrics** NOT available in the Kubernetes API server.  
It exposes metrics for:

* Deployments
    
* Daemonsets
    
* Pods
    
* ReplicaSets
    
* Services
    
* Replica count
    
* Desired vs Actual state
    

Without kube-state-metrics you only get basic Kubernetes metrics.

---

# 🔥 **EXPOSE PROMETHEUS SERVER USING NODEPORT**

Default service is ClusterIP, so expose it:

```plaintext
kubectl expose service prometheus-server --type=NodePort --name=prometheus-server-ext
```

Get service:

```plaintext
kubectl get svc
```

Use Minikube IP:

```plaintext
minikube ip
```

Open Prometheus:

```plaintext
http://<minikube-ip>:<node-port>
```

You can now run PromQL queries.

---

# 🔥 **INSTALL GRAFANA USING HELM**

1. Add Grafana repo:
    

```plaintext
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
```

2. Install Grafana:
    

```plaintext
helm install grafana grafana/grafana
```

3. Get admin password:
    

```plaintext
kubectl get secret -n default grafana -o jsonpath="{.data.admin-password}" | base64 --decode
```

4. Expose Grafana:
    

```plaintext
kubectl expose service grafana --type=NodePort --name=grafana-ext
```

5. Access Grafana:
    

```plaintext
http://<minikube-ip>:<node-port>
```

Login using:

* **User**: admin
    
* **Password**: (from command above)
    

---

# 🔥 **ADD PROMETHEUS AS A DATASOURCE IN GRAFANA**

Grafana → *Data Sources* → Add → Select **Prometheus**

Enter URL:

```plaintext
http://<minikube-ip>:<prometheus-nodeport>
```

Save & Test → should show **Data source is working**.

---

# 🔥 **IMPORT KUBERNETES DASHBOARD (ID: 3662)**

Grafana → Dashboards → Import → Enter ID:

```plaintext
3662
```

Select data source = Prometheus → Import.

A beautiful Kubernetes dashboard appears showing:

* API server health
    
* Nodes
    
* CPU & Memory usage
    
* Cluster info
    
* Pod counts
    
* Node uptime
    

---

# 🔥 **ENABLE kube-state-metrics FOR DEPLOYMENT-LEVEL METRICS**

Expose kube-state-metrics:

```plaintext
kubectl expose service prometheus-kube-state-metrics --type=NodePort --name=kube-state-metrics-ext --target-port=8080
```

Get NodePort:

```plaintext
kubectl get svc
```

Open:

```plaintext
http://<minikube-ip>:<nodeport>/metrics
```

Now you will see **deployment-level, pod-level, and service-level metrics**.

These metrics appear inside your Grafana dashboards.

---

# 🎉 **END RESULT**

You successfully installed:

✔️ Kubernetes Cluster (Minikube)  
✔️ Prometheus  
✔️ Grafana  
✔️ kube-state-metrics  
✔️ Kubernetes monitoring dashboard (ID: 3662)

Now your Grafana shows:

* Nodes
    
* API Server
    
* Pods
    
* Deployments
    
* Resource usage
    
* Replica counts
    
* Cluster uptime
    
* Realtime metrics