Day 22 - How to Manage Hundreds of Kubernetes Clusters — Using KOPS

🎯 1. What Is the Problem?

In real-world production environments, DevOps engineers must:

• Create

• Upgrade

• Configure

• Delete
  Kubernetes clusters — across multiple environments (dev, staging, prod).

Managing these life cycles manually (especially at scale) is complex.
Hence, automation tools like KOPS are used.

---

🧩 2. Why Not Minikube, Kind, K3s, or MicroK8s?

• These are lightweight, single-node setups meant for learning and development only.

• They lack:

  • High availability

  • Multi-node support

  • Production-grade fault tolerance

  • Scalability and security controls

📌 In short:

Minikube / K3s / Kind = Local dev use only
KOPS / EKS / OpenShift / Rancher = Production-ready systems

---

🏗️ 3. Kubernetes in Production (Distributions)

Just like Linux has distributions (Ubuntu, Red Hat, Amazon Linux),
Kubernetes also has multiple distributions.

💡 Why use distributions?

• Provide enterprise support

• Manage security patches

• Simplify setup and upgrades

• Offer ready-to-use integrations

---

🧠 4. Common Production Scenarios

• Organizations may have:

  • Hundreds of Kubernetes clusters

  • Or one large cluster with thousands of nodes

• Managed solutions (EKS, GKE, AKS) cost a lot when scaled.

• Hence, many companies use open-source Kubernetes with tools like KOPS to manage lifecycle operations.

---

⚖️ 5. Kubernetes vs EKS

🟩 Key Point:
EKS = Kubernetes + AWS management + Paid support
KOPS = Kubernetes + Full control + Open source management

---

⚙️ 6. What Is KOPS?

KOPS = Kubernetes Operations Tool

A CLI tool that automates the creation, management, and lifecycle of Kubernetes clusters on AWS and other clouds.

✳️ KOPS manages:

• Cluster creation

• Configuration changes

• Upgrades

• Node scaling

• Cluster deletion

KOPS stores all cluster configuration in an S3 bucket, which acts as the cluster state store.

---

🧰 7. Why KOPS Is Popular

---

🪜 8. Pre-requisites Before Using KOPS

Before creating a cluster with KOPS, ensure you have:

🔧 Software Requirements:

1. Python 3

2. AWS CLI

3. kubectl

4. KOPS

☁️ AWS Requirements:

• AWS account access

• IAM user (Admin or with following policies):

  • AmazonEC2FullAccess

  • AmazonS3FullAccess

  • IAMFullAccess

  • AmazonVPCFullAccess

• AWS CLI configured via:

    aws configure
  

---

📦 9. Step-by-Step Setup with KOPS

Step 1️⃣: Create an S3 Bucket for Cluster State

KOPS stores cluster metadata in S3.

aws s3 mb s3://kops-state-store-1

Step 2️⃣: Export the S3 Bucket Path

export KOPS_STATE_STORE=s3://kops-state-store-1

Step 3️⃣: Create the Kubernetes Cluster Definition

kops create cluster \
--name=k8s.local \
--zones=us-east-1a \
--node-count=2 \
--node-size=t2.micro \
--master-size=t2.micro \
--state=s3://kops-state-store-1

Step 4️⃣: Build (Launch) the Cluster

kops update cluster k8s.local --yes

⏱️ This process takes several minutes — KOPS provisions EC2 instances, networking, security groups, IAM roles, etc.

---

🧩 10. Domain Considerations

KOPS requires a domain name (for the cluster API endpoint).
You can use:

If using a real domain:

• Purchase it (e.g., GoDaddy)

• Configure DNS in AWS Route 53

• Create a hosted zone:

    aws route53 create-hosted-zone --name dev.example.com
  

---

💰 11. Cost Caution

⚠️ KOPS uses AWS resources:

• EC2 instances

• EBS volumes

• S3 buckets

• Route 53 entries

🧾 These all incur AWS billing, even in free-tier accounts.

Tip:
If you only want to learn, stop after the “create cluster” step — do not run the final “update cluster” command.

---

🧱 12. KOPS in the Real World

• Used by DevOps teams for multi-environment orchestration.

• Commonly manages:

  • Dev, QA, Staging, and Production clusters

  • Clusters across multiple AWS accounts or regions

• Supports upgrades via:

    kops upgrade cluster
    kops rolling-update cluster
  

• Supports deletion via:

    kops delete cluster --name=<cluster-name> --yes
  

---

🧭 13. Comparison: Other Installation Tools

---

🧩 14. Interview Tip

When asked about Kubernetes setup in production, say:

“In our organization, we manage multiple Kubernetes clusters using KOPS on AWS.
KOPS handles the full lifecycle — creation, configuration, upgrades, and deletion.
For staging and testing, we use .k8s.local domains, and for production we use Route 53 hosted domains like prod.company.com.”

---

🧠 15. In Summary

---

✅ Final One-Liner Summary:

KOPS is a powerful open-source tool used by DevOps engineers to create, manage, and scale hundreds of production-grade Kubernetes clusters — providing automation, versioning, and reliability without managed-service costs.