Architecture Stack: ArgoCD & GitOps Deployments

This is a conversation between Alex (Engineering Manager) and Jordan (Senior Architect) exploring GitOps and ArgoCD deployments.

Part 1: The Deployment Problem

Alex: Jordan, we have Kubernetes clusters running. Now when we want to deploy a new version of a service, what's the process?

Jordan: Great question. This is where deployment strategy becomes critical. Let me describe the old way first.

Alex: Okay.

Jordan: Traditional deployment: Developer commits code. CI pipeline runs tests, builds container image, pushes to registry. Then someone (ops engineer) manually runs kubectl commands to update the cluster. Or they log into CI/CD system (Jenkins, GitLab CI) and click 'deploy'. Human decides when and how to deploy.

Alex: What's wrong with that?

Jordan: Several problems:

1. Deployment is manual. Prone to human error. Wrong version deployed? Network misconfigured? Only the person who deployed knows what happened.
2. Cluster state diverges from code. Ops engineer manually runs kubectl commands. Now cluster state is different from what's in Git.
3. Disaster recovery is manual. Cluster fails? Rebuild manually by remembering what commands were run. If you don't remember, good luck.
4. No audit trail. Who deployed what when? Why did deployment fail? Hard to trace.

Alex: So it's disorganized?

Jordan: Chaotic is more accurate. At scale, manual deployments cause constant incidents.

---

Part 2: Enter GitOps

Alex: What's GitOps?

Jordan: A philosophy: Git is the single source of truth. Everything about your infrastructure and applications is defined in Git. The actual cluster continuously syncs with Git. If cluster diverges, automatic reconciliation brings it back.

Alex: Everything in Git?

Jordan: Everything. Service configurations, replicas, resource limits, networking, storage. If it's infrastructure, it's in Git. You never manually kubectl apply anything. Never. You push to Git, and GitOps tools automatically deploy.

Alex: How is that better?

Jordan: Huge benefits:

1. Reproducible: Cluster state is 100% defined in Git. Recreate identical cluster from Git anytime.
2. Auditable: Every change is a Git commit. Author, timestamp, review process. Complete audit trail.
3. Reversible: Bad deployment? Git revert. Cluster reverts automatically.
4. Collaborative: Pull requests for infrastructure changes. Review before deploying.
5. Safer: No manual access to production cluster. Only Git commits trigger changes.

Alex: This is a cultural shift?

Jordan: Absolutely. It says: trust the system, not manual operations.

---

Part 3: What Is ArgoCD?

Alex: So GitOps is the philosophy. What's ArgoCD?

Jordan: ArgoCD is a Kubernetes-native GitOps tool. It runs inside your cluster. It watches a Git repository. When Git changes, ArgoCD detects it and syncs the cluster to match Git.

Alex: How does it detect changes?

Jordan: It polls Git periodically (every few seconds) or uses webhooks (Git pushes a notification instantly). Either way, ArgoCD sees the change, compares cluster state to Git, and applies differences.

Alex: Applying differences means kubectl apply?

Jordan: Exactly. But automated. ArgoCD does it, not a human. No human SSH access to production.

Alex: That's secure?

Jordan: Very. Human error is eliminated. Git history is complete. Every deployment is reproducible.

---

Part 4: Problems ArgoCD Solves

Alex: What concrete problems does this solve?

Jordan: Many:

Deployment Reliability: Manual deployments fail mysteriously. Deployments are automated, repeatable, reliable.

Disaster Recovery: Cluster dies. Redeploy from Git instantly. Identical state reconstruction in minutes.

Compliance & Auditing: Every change tracked in Git. Audit trail for compliance (SOC2, PCI-DSS). Deployment history searchable.

Rollback Speed: Bad deployment? Git revert. Cluster reverts in seconds. No messing with kubectl commands, no confusion about rollback.

Team Collaboration: Pull requests for infrastructure changes. Multiple reviewers. No surprise deployments.

Multi-Environment Management: Dev, staging, production are separate Git directories. Same deployment process, different configs. Consistency.

Alex: These are all about safety and auditability?

Jordan: Exactly. GitOps prioritizes safe, trackable deployments.

---

Part 5: Why This Matters

Alex: What's the business impact?

Jordan: Deployments are faster and safer. Teams can push code multiple times per day without fear. A bad deployment is reverted in seconds. Compliance is easier (Git is the audit log). Infrastructure knowledge doesn't live in people's heads; it's in Git, accessible to everyone.

Alex: Faster and safer? That's rare.

Jordan: It is. Usually you trade speed for safety. GitOps gets both because automation removes human error.

Alex: What about employee retention?

Jordan: People prefer working in systems where mistakes are reversible and tracked, not blamed. GitOps culture is better for teams.

---

Part 6: Traditional Deployment Approach

Alex: What was the old way?

Jordan: Varied. Some teams used CI/CD pipelines (Jenkins, GitLab CI) and deployed manually via API calls. Others SSH'd into servers and ran commands. Some kept deployment scripts but ran them occasionally.

Alex: All manual?

Jordan: Some automation. But the key difference: cluster state wasn't defined in Git. Cluster state was what you happened to run last time. If something was wrong, you'd manually fix the cluster. Maybe update a script later to prevent recurrence.

Alex: That doesn't scale?

Jordan: Not at all. With 20 microservices in 3 environments, tracking what's actually running is impossible.

Alex: How did people manage?

Jordan: Poorly. Frequent surprises. 'Wait, is production using v1.2.1 or v1.2.2?' 'I don't know, SSH in and check.' It was chaos.

---

Part 7: GitOps Principles

Alex: Let me understand GitOps deeper. What are the core principles?

Jordan: Four main principles:

Declarative: Infrastructure is declared, not imperative. You say 'run 10 copies', not 'add 5 copies' (imperative). If Git says 10, cluster has 10. Period. Idempotent.

Version Controlled: Every change lives in Git. Full history. Blame tracking. Rollback capability.

Automatically Reconciled: GitOps tool continuously checks: is cluster state matching Git? If not, fix it. Humans deploy by pushing Git. GitOps enforces the state.

Observable: You see what's running, what changed, why. Full visibility. Audit trail.

Alex: These are architectural principles?

Jordan: More philosophical. They guide how you design deployment systems.

---

Part 8: How ArgoCD Works

Alex: Walk me through ArgoCD's flow.

Jordan: Simple:

1. Developer writes Kubernetes manifests (YAML describing desired state). Commits to Git.
2. Developer pushes to Git repository.
3. ArgoCD watches Git repository.
4. ArgoCD detects commit.
5. ArgoCD reads manifests from Git.
6. ArgoCD compares Git state with current cluster state.
7. If different, ArgoCD applies changes (kubectl apply).
8. Cluster now matches Git.
9. Dashboard shows 'Synced'. All green.

Alex: What if ArgoCD fails?

Jordan: Cluster diverges from Git. ArgoCD dashboard shows 'OutOfSync' in red. Someone notices and investigates. Or ArgoCD automatically syncs when it recovers. Worst case, you manually trigger sync.

Alex: Manual recovery is bad?

Jordan: It defeats the purpose. ArgoCD's main value is automation. If you're manually intervening often, you've done something wrong.

---

Part 9: Deployment Strategies

Alex: How do you actually roll out updates?

Jordan: Several strategies:

Blue-Green Deployment: Two identical environments (blue, green). All traffic to blue. Deploy to green. Run tests. Switch traffic to green instantly. If problems, switch back instantly.

Canary Deployment: Deploy new version to small percentage (5%) of traffic. Monitor for errors. If good, gradually increase to 25%, 50%, 100%. If problems, rollback small blast radius.

Rolling Update: Gradually replace old pods with new. One at a time (or in waves). Health checks ensure new is working before moving to next.

Recreate: Stop old version, start new. Downtime. Only acceptable for maintenance windows.

Alex: Which does ArgoCD use?

Jordan: ArgoCD supports all of them. You configure which strategy in manifests. Argo Rollouts (companion tool) provides advanced strategies. But basic rolling updates are built into Kubernetes.

Alex: So deployment strategy is separate from GitOps?

Jordan: Sort of. GitOps is about how you declare desired state. Deployment strategy is how you transition between states. Complementary.

---

Part 10: Applications & Projects in ArgoCD

Alex: What does ArgoCD actually manage?

Jordan: ArgoCD organizes deployments as 'Applications'. Each Application:

• Points to a Git repository and directory
• Specifies a Kubernetes cluster (target)
• Specifies a namespace
• Has sync strategy (automatic or manual)

When you create an Application, ArgoCD:

1. Fetches manifests from Git
2. Deploys to specified cluster/namespace
3. Continuously syncs

Multiple Applications let you manage many services in one ArgoCD instance.

Alex: So ArgoCD is a single control point?

Jordan: Yes. You create Applications for Order Service, Payment Service, User Service. All synced from Git. One dashboard shows all.

Alex: Multi-cluster?

Jordan: ArgoCD can deploy to multiple clusters. Git is source of truth. Prod cluster in us-east, staging in us-west, dev in local. All synced from same Git.

---

Part 11: Helm Integration

Alex: Earlier we mentioned Helm. How does that fit?

Jordan: Helm templates Kubernetes manifests. Instead of writing static YAML, you write templates with variables. Helm renders templates to produce YAML.

Alex: Why?

Jordan: Reusability. One Helm chart works for dev, staging, production with different values. Same application, different configurations.

Alex: How does ArgoCD use this?

Jordan: ArgoCD can deploy Helm charts. Git repository contains Helm chart. Separate values files for dev/staging/prod. ArgoCD fetches chart, renders with values, deploys. We'll discuss Helm more in the next article.

---

Part 12: Secrets Management

Alex: How do you handle sensitive data (passwords, API keys)?

Jordan: Secret problem with GitOps: you can't commit secrets to Git. Anyone with Git access sees them. Insecure.

Alex: So how do you store secrets?

Jordan: Several approaches:

1. Sealed Secrets: Encrypt secrets before committing to Git. Kubernetes Sealed Secrets controller decrypts in cluster.
2. External Secrets: Store secrets in external vault (AWS Secrets Manager, HashiCorp Vault). ArgoCD fetches from vault, doesn't store in Git.
3. Sops: Encrypt specific fields in YAML files before committing.

Alex: Which do we use?

Jordan: Sealed Secrets for simplicity. Secrets are encrypted in Git. Cluster can decrypt but humans can't easily.

Alex: Does ArgoCD handle this?

Jordan: ArgoCD just applies manifests. Kubernetes handles decryption. ArgoCD doesn't need to know about secrets.

---

Part 13: Monitoring & Notifications

Alex: How do you know deployment succeeded?

Jordan: ArgoCD dashboard. Refresh shows sync status. Green = synced (good). Red = out of sync (problem).

Notifications: ArgoCD can send webhooks when sync fails. Triggers Slack notification. Team gets alerted instantly if deployment fails.

Alex: Does it check application health?

Jordan: Yes. Beyond just 'sync', ArgoCD checks health of deployed resources. Pod running? Service has endpoints? All checks pass = healthy. If health checks fail, dashboard shows red.

Alex: So you get visibility?

Jordan: Complete visibility. Git, deployment, and application health all visible in one dashboard.

---

Part 14: Challenges & Best Practices

Alex: What can go wrong?

Jordan: Several pitfalls:

Divergence: Someone manually kubectl applies to cluster, bypassing Git. Cluster diverges. Solution: RBAC (Role-Based Access Control). Only ArgoCD has kubectl access, humans don't.

Git Secrets Exposure: Accidentally commit secrets to Git. Solution: git pre-commit hooks scanning for secrets. Sealed Secrets or External Secrets.

Slow Sync: ArgoCD polls every few seconds. If Git is large, rendering takes time. Solution: webhooks for instant notification.

Conflicting Changes: Multiple teams pushing Git simultaneously. Merge conflicts. Solution: discipline around Git workflow. Feature branches, pull request reviews.

Cascading Failures: Deploying new manifest breaks entire service. Solution: progressive deployment strategies (canary, blue-green). Test in staging first.

Access Control: Who can merge to production Git? Who can trigger manual sync? Solution: RBAC in Git and ArgoCD.

Alex: These are all process issues?

Jordan: Partly. Partly technical. GitOps is a culture. Process and tooling work together.

---

Part 15: ArgoCD in Practice

Alex: Walk me through a typical deployment.

Jordan: 1. Developer writes code. Commits to feature branch. 2. CI pipeline runs tests, builds image, pushes to registry. 3. Developer updates Kubernetes manifest in Git (new image version). 4. Developer creates Pull Request. 5. Team reviews PR. 'Looks good, this uses image v2.0.1, tests passed.' 6. Approval. PR merges to main branch. 7. ArgoCD sees Git change immediately. 8. ArgoCD fetches manifests from main branch. 9. ArgoCD applies to production cluster. 10. Rolling update replaces old pods with new. 11. ArgoCD dashboard shows 'Synced'. 12. Monitoring dashboards show metrics from new version. 13. Health checks pass. All green. 14. Deployment complete.

Alex: Zero human intervention?

Jordan: After merge. Merge itself is human (review), but deployment is automated.

Alex: What if something breaks?

Jordan: ArgoCD health checks fail. Dashboard goes red. Slack notification. Someone investigates. Options:

1. Git revert. ArgoCD reverts deployment.
2. Manual kubectl fix (only if necessary). Then update Git to match.
3. Rollback via Argo Rollouts (advanced).

Alex: Revert is fast?

Jordan: Seconds. Git revert commit. Push. ArgoCD detects. Cluster updated. Very fast.

---

Part 16: When to Use GitOps

Alex: Is GitOps always right?

Jordan: Not for everything. GitOps shines when:

• Multiple services needing coordination
• Frequent deployments (multiple per day)
• Compliance requirements (audit trails)
• Multi-environment (dev/staging/prod)
• Team size large enough to benefit from reviews

Not worth it when:

• Single simple application
• Deployments rare (quarterly)
• Small team (no review process)
• Real-time responsiveness critical (GitOps has slight delay)

Alex: So it's for serious platforms?

Jordan: Yes. Startups might not need it. Enterprise platforms always do.

---

Part 17: Bridging to Helm

Alex: We mention Helm but don't cover it. What's next?

Jordan: Helm in depth. Helm is templating + packaging. Instead of managing hundreds of YAML files, you manage Helm charts. Variables make charts reusable. We manage dependencies between charts. It's application packaging for Kubernetes.

Alex: So flow is: Helm charts define applications, Git stores Helm charts, ArgoCD deploys from Git?

Jordan: Exactly. Helm is packaging. ArgoCD is deployment. They work together.

---

Part 18: The Bottom Line

Alex: If I pitch this to executives?

Jordan: ArgoCD implements GitOps: Git becomes single source of truth for infrastructure. Developers push code, Git is updated, deployment is automatic. Benefits: faster deployments (no manual waiting), safer (reversible, auditable), more reliable (no human error), better compliance (complete audit trail). Teams deploy multiple times per day confidently. A bad deployment is reverted in seconds via Git revert. The entire infrastructure is version controlled, reviewable, and reproducible.

Alex: And the commitment?

Jordan: We're committing to automation. Manual deployments go away. Everyone trusts the system. It requires discipline (Git-based workflow) but payoff is massive.

---

Key Takeaways

Aspect	Details
Core Concept	Git as single source of truth; GitOps tool continuously syncs cluster to match Git
Primary Benefit	Automated, safe, auditable deployments; fast rollbacks; reproducibility
Best For	Multi-service systems, frequent deployments, compliance requirements
Main Challenge	Discipline (no manual kubectl), secrets management, process maturity
Key Principles	Declarative, version controlled, automatically reconciled, observable
Deployment Strategies	Blue-green, canary, rolling updates, recreate
Secret Handling	Sealed Secrets, External Secrets, or Sops encryption
Rollback	Git revert, instant cluster sync, zero downtime