Architecture Stack: Kubernetes & Container Orchestration

This is a conversation between Alex (Engineering Manager) and Jordan (Senior Architect) exploring Kubernetes and container orchestration.

Part 1: The Container Problem at Scale

Alex: "Jordan, we've got AWS infrastructure. Now we're deploying microservices. Each service is a container. How do we manage hundreds of containers across multiple machines?"

Jordan: "That's exactly what Kubernetes solves. Let me paint the problem first."

Alex: "Okay."

Jordan: "You have 20 microservices. Each runs in a Docker container. You have 10 servers. You need to:

Decide which server runs which container
If a server crashes, restart its containers on another server
Scale services independently (Order service handles 2x traffic, needs 10 copies)
Update containers without downtime
Manage networking between containers across servers
Monitor health and restart dead containers"

Alex: "That's a lot of manual work."

Jordan: "Exactly. Without automation, you'd need a full-time ops team managing this. Kubernetes automates all of it."

Alex: "How?"

Jordan: "By being a container orchestrator. You tell Kubernetes 'run 10 copies of Order service'. Kubernetes figures out which servers, handles failures, scales up/down automatically."

Part 2: What Is Kubernetes?

Alex: "Define it precisely."

Jordan: "Kubernetes is an open-source platform for automating deployment, scaling, and management of containerized applications. You define desired state (I want 10 copies of Service A). Kubernetes continuously works to maintain that state."

Alex: "Like a thermostat?"

Jordan: "Perfect analogy. You set desired temperature. Thermostat monitors actual temperature. If it's too cold, heat turns on. If too hot, AC turns on. Always pushing toward desired state. Kubernetes does the same with containers."

Alex: "How does it know what's running?"

Jordan: "It talks to the container runtime (Docker) on each server. Container runtime reports: 'Server A has 3 running containers'. Kubernetes keeps inventory and makes decisions."

Alex: "What kind of decisions?"

Jordan: "Placement (which server gets which container), scaling (add more copies or remove), health (restart dead containers), updates (replace old container with new without downtime)."

Part 3: Problems Kubernetes Solves

Alex: "What concrete problems does this solve?"

Jordan: "Several:

Placement Complexity: With 20 services and 10 servers, manually deciding where each container runs is tedious and error-prone. Kubernetes does it intelligently based on resource requirements.

High Availability: Container crashes. Server fails. Without Kubernetes, you manually restart and reschedule. Kubernetes detects and fixes automatically.

Scaling: Traffic spikes. You need 2x more Order service capacity. Manually spinning up new containers is slow. Kubernetes scales in seconds.

Rolling Updates: New version of Payment service. Without Kubernetes, you manually stop old, start new, pray nothing breaks. Kubernetes does gradual replacement with health checks. If new version is bad, it rolls back.

Resource Efficiency: Containers have different resource needs. Some are CPU-heavy, others memory-heavy. Kubernetes packs them efficiently, reducing wasted capacity."

Alex: "These are all automation problems?"

Jordan: "Exactly. At scale, manual management is impossible. Kubernetes is the automation engine."

Part 4: Why Kubernetes Matters

Alex: "In business terms?"

Jordan: "Directly impacts costs and reliability. Manual management requires large ops teams. Kubernetes reduces that to a few platform engineers. You're cutting overhead by 80%."

Alex: "What about reliability?"

Jordan: "If a container crashes, Kubernetes restarts it instantly. Users don't notice. Uptime improves significantly. Less human error, more automation."

Alex: "And agility?"

Jordan: "New service? Just describe it to Kubernetes. It handles all the operational details. Teams can focus on features, not infrastructure."

Part 5: Containers vs Traditional VMs

Alex: "I've heard containers are lighter than VMs. Why?"

Jordan: "Containers and VMs are different approaches to isolation.

VMs: Full operating system (Windows, Linux). Hypervisor virtualizes hardware. Takes 1GB+ disk space, seconds to start. But complete isolation: app can do almost anything.

Containers: Share OS kernel. Only package application and dependencies. Takes 100MB, starts in milliseconds. But less isolated: containers can see the same OS kernel."

Alex: "So containers are faster but less isolated?"

Jordan: "Exactly. For most applications, container isolation is fine. You get speed and efficiency. If you need strict isolation, use VMs."

Alex: "What's the relationship to Kubernetes?"

Jordan: "Kubernetes orchestrates containers. It assumes container workloads. You package your service as a container, tell Kubernetes about it, and Kubernetes manages deployment across many servers."

Part 6: Kubernetes Architecture (Conceptual)

Alex: "How does Kubernetes work internally?"

Jordan: "Kubernetes has a control plane and worker nodes.

Control Plane: The brain. API server accepts requests. Scheduler decides where to place containers. Controller manager watches services and maintains desired state.

Worker Nodes: The muscles. Each node runs a kubelet (agent) that talks to control plane and manages containers on that node.

Flow: You submit request to API server: 'I want 10 copies of Service A'. Scheduler looks at worker nodes and their current load. Decides node 1 gets 4 copies, node 2 gets 3, node 3 gets 3. Tells kubelets to start containers. They do. Done."

Alex: "That's the basic flow?"

Jordan: "Yes. But there's complexity around networking, storage, persistence, monitoring. Each is a separate concern Kubernetes handles."

Part 7: Key Kubernetes Concepts

Alex: "What vocabulary do I need?"

Jordan: "Core concepts:

Pod: Smallest deployable unit. Usually one container per pod. Pods on same node can share storage and networking.

Deployment: Describes desired state: run 10 copies of this container, expose port 8000, etc. Kubernetes maintains that state. If a pod dies, Deployment spawns a new one.

Service: Networking abstraction. Exposes pods to each other and outside world. Internal service for Order service to call Payment service. External service to expose API to users.

ConfigMap: Key-value configuration. Instead of hardcoding settings in containers, store in ConfigMap. Containers read at startup.

Secret: Like ConfigMap but for sensitive data (passwords, API keys). Encrypted storage.

PersistentVolume: Storage that persists across container restarts. Database data lives here.

Namespace: Logical isolation. Two separate deployments can run in different namespaces with same names. Useful for multi-tenancy or separating environments."

Alex: "That's substantial vocabulary."

Jordan: "It is. But most of it is abstraction. You rarely interact with it directly. Tools handle complexity."

Part 8: EKS (Elastic Kubernetes Service)

Alex: "So Kubernetes is open-source. How does AWS fit?"

Jordan: "EKS is AWS's managed Kubernetes service. You don't manage the control plane. AWS does. You just provide worker nodes and submit manifests."

Alex: "What does AWS manage?"

Jordan: "Control plane availability. High availability (runs across multiple availability zones). Updates and patches. Monitoring. Backups. You just focus on applications."

Alex: "Why not manage Kubernetes ourselves?"

Jordan: "Control plane needs to be highly available. You'd need 3+ masters, load balancers, storage for state. Complex. AWS abstracts that. You get Kubernetes without the ops burden."

Alex: "Do we use EKS?"

Jordan: "Yes. It's standard for us. We have 3 EKS clusters: one for development, one for staging, one for production."

Part 9: Deployment Strategy

Alex: "How do we actually deploy to Kubernetes?"

Jordan: "You write YAML manifests describing your desired state.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  replicas: 10
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
        - name: order
          image: myrepo/order:v1.2.3
          ports:
            - containerPort: 8000

This says: Run 10 copies of the order service image. Expose port 8000. Kubernetes reads this and makes it happen."

Alex: "You write this manually?"

Jordan: "You can. But usually tools generate it. Helm (which we'll discuss later) templates this. ArgoCD deploys it. Most teams don't write raw Kubernetes YAML."

Alex: "What does deployment actually look like?"

Jordan: "You push new container image to registry. Update YAML to reference new image. Submit to Kubernetes. Kubernetes slowly replaces old pods with new ones. Health checks ensure new ones work. If new version is bad, you roll back."

Alex: "How slow is slow?"

Jordan: "Configurable. Typically one pod at a time. So 10 pods take 10 update cycles. Might be 2-3 minutes total. Zero downtime."

Part 10: Scaling & Load Balancing

Alex: "How does Kubernetes scale?"

Jordan: "Several mechanisms:

Horizontal Pod Autoscaler (HPA): Watches CPU/memory metrics. If Order service crosses 80% CPU, add more pods. If drops below 30%, remove pods. Automatic.

Manual Scaling: You say 'run 20 copies instead of 10'. Kubernetes spins up 10 new pods instantly.

Service Load Balancing: Service object distributes traffic across pods. User hits Service IP. Service picks a pod randomly. All pods see equal traffic (load balanced).

External Load Balancer: ALB (AWS) sits in front of EKS. Distributes traffic across services. Users hit ALB, ALB distributes to Services, Services distribute to Pods."

Alex: "Multiple layers of load balancing?"

Jordan: "Yes. Traffic path: User -> ALB (AWS) -> Service (Kubernetes) -> Pod (container). Each layer balances."

Alex: "Is that efficient?"

Jordan: "Efficient enough. Latency added is negligible (microseconds). For high-volume systems, latency matters but is managed."

Part 11: Storage & Persistence

Alex: "What about databases? Do they run in Kubernetes?"

Jordan: "Databases can run in Kubernetes via StatefulSets (like Deployments but ordered, unique identities). But most teams don't. Databases are managed by AWS (RDS, DynamoDB). They're too critical and complex to run yourself."

Alex: "Why not self-managed databases?"

Jordan: "Backup strategy is complex. Replication is complex. Failover is complex. AWS handles all that. You get reliability guarantees (99.95% uptime SLA). Worth paying for."

Alex: "What about application data that needs to persist?"

Jordan: "Use PersistentVolumes in Kubernetes. Maps to EBS volumes on AWS. Container can read/write data. If container dies, data persists. If you need to move container to different server, volume goes with it."

Alex: "Seamlessly?"

Jordan: "As seamlessly as possible. Kubernetes handles the mechanics."

Part 12: Networking in Kubernetes

Alex: "How do services communicate?"

Jordan: "Services expose pods internally. Pod A in deployment wants to call Pod B. It uses the Service name as hostname. Kubernetes DNS resolves Service name to internal IP. Traffic goes to any pod behind that Service."

Alex: "No hardcoded IPs?"

Jordan: "Never. Services are abstractions. Pod IPs are ephemeral (change when pods restart). Service IPs are stable."

Alex: "What about external traffic?"

Jordan: "Ingress object. Defines URL routing: /orders -> order-service, /payments -> payment-service. Kubernetes ingress controller (nginx, AWS ALB) implements this. Users hit domain, get routed appropriately."

Alex: "So networking is abstracted?"

Jordan: "Completely. You define relationships. Kubernetes handles routing."

Part 13: Monitoring & Observability in Kubernetes

Alex: "How do you know what's running?"

Jordan: "Several tools:

kubectl: CLI tool to query Kubernetes. 'kubectl get pods' shows all running pods. 'kubectl logs' shows logs. 'kubectl describe' shows detailed info.

Dashboard: Web UI showing cluster state, resource usage, logs.

Prometheus: Collects metrics from all pods. CPU, memory, requests per second, errors.

Grafana: Visualizes Prometheus metrics in dashboards.

Jaeger: Distributed tracing. See request flow through microservices."

Alex: "Do you need all of these?"

Jordan: "kubectl is essential. Dashboard is convenient. Prometheus + Grafana is standard. Jaeger for debugging complex flows."

Alex: "How detailed can you get?"

Jordan: "Very. You can see metrics for each pod, each container, each request. Drill down from cluster-level down to individual container memory usage over time."

Part 14: Common Challenges & Pitfalls

Alex: "What goes wrong?"

Jordan: "Several common mistakes:

Resource Requests Not Set: Containers don't specify memory/CPU needs. Scheduler can't make good decisions. Nodes get overloaded. Pods crash.

No Liveness/Readiness Probes: Kubernetes doesn't know if container is healthy. Dead containers keep running. Service sends traffic to dead pods.

Stateful Applications: Apps that depend on being on same server. Kubernetes moves them. App breaks. Solution: design stateless or use StatefulSets.

Configuration Management: Hardcoding settings in images. Can't change config without new image. Use ConfigMaps/Secrets.

Storage Strategy: Not planning for persistence. Pods die, data lost. Solution: PersistentVolumes for anything important.

Namespace Pollution: Everything in default namespace. Hard to manage. Solution: use namespaces for separation (prod/dev/staging)."

Alex: "These are all design issues?"

Jordan: "Exactly. Kubernetes will let you do wrong things. It's powerful but needs discipline."

Part 15: When to Use Kubernetes

Alex: "Is Kubernetes always the answer?"

Jordan: "No. Kubernetes adds complexity. Worth it when:

Multiple services need to coordinate
Scale requirements are variable
Need high availability
Team has infrastructure expertise

Not worth it when:

Single monolithic application
Consistent traffic (auto-scaling unnecessary)
Small team (ops burden too high)
Simple workloads (Fargate or Lambda sufficient)"

Alex: "So it's for serious applications?"

Jordan: "Yes. Kubernetes is for companies investing in platform infrastructure. Startups or simple apps should use simpler approaches."

Part 16: Kubernetes on AWS (EKS)

Alex: "Specifically for AWS, what's the story?"

Jordan: "EKS is AWS's managed Kubernetes. You create an EKS cluster. AWS manages control plane. You bring worker nodes (EC2 instances). You deploy containers. Kubernetes orchestrates."

Alex: "Why not just EC2?"

Jordan: "EKS abstracts away server thinking. You think about services and containers, not servers and SSH keys. Plus networking, storage, and monitoring are integrated. It's orchestration, not just IaaS."

Alex: "Is there a simpler AWS option?"

Jordan: "Fargate. You submit containers. Fargate runs them without thinking about servers. But less control and less suitable for stateful workloads."

Alex: "When do we use Fargate vs EKS?"

Jordan: "Fargate for simple workloads (background jobs, webhooks). EKS for main applications where we need control and coordination."

Part 17: Bridging to ArgoCD

Alex: "We've covered infrastructure (AWS) and orchestration (Kubernetes). What's next?"

Jordan: "Deployment. How do you continuously update applications in Kubernetes? That's ArgoCD. GitOps approach: Git as source of truth, ArgoCD keeps cluster in sync with Git."

Alex: "So the flow is: update Git, ArgoCD deploys?"

Jordan: "Exactly. We'll dig into that next."

Part 18: The Bottom Line

Alex: "If I pitch this?"

Jordan: "Kubernetes is the control plane for containerized applications at scale. Instead of manually managing which service runs where, Kubernetes handles placement, scaling, health, and updates automatically. This means services scale independently based on demand, failures are handled automatically, and deployments are instant with zero downtime. Your teams focus on applications, not infrastructure."

Alex: "And the commitment?"

Jordan: "We're committing to containerization and orchestration. No more traditional VMs. Everything runs in containers managed by Kubernetes. The investment in learning Kubernetes pays off in agility and reliability."

Key Takeaways

| Aspect | Details | | ------------------- | ------------------------------------------------------------------------------- | | Core Concept | Automate deployment, scaling, and management of containerized applications | | Primary Benefit | Automatic scaling, self-healing, zero-downtime updates, resource efficiency | | Best For | Microservices, multi-service systems, variable load applications | | Main Challenge | Complexity, learning curve, requires discipline in design | | Key Concepts | Pods, Deployments, Services, ConfigMaps, Secrets, PersistentVolumes, Namespaces | | Scaling | Horizontal Pod Autoscaler (HPA) auto-scales based on metrics | | AWS Option | EKS (managed Kubernetes) provides control plane, you manage worker nodes | | Deployment | YAML manifests define desired state, Kubernetes maintains it |

L2 Series Progress

AWS Fundamentals ✓ (infrastructure foundation)
Kubernetes & EKS ✓ (container orchestration)
ArgoCD (GitOps deployments)
Helm (application packaging)
DevSecOps (security automation)

Ready for ArgoCD and GitOps deployments? Or explore Kubernetes deeper?