Introduction to GitOps

The 2 AM Production Deployment That Changed Everything

It was 2 AM on a Friday. Our API was down. I was SSH'd into a production Kubernetes node, frantically running kubectl commands:

kubectl get pods -n production
# Some pods running old version, some running new version
# Which deployment updated? Who deployed it? When?

kubectl describe deployment api-service
# Image: api:v2.3.1... or was it v2.3.2?
# I don't remember what we deployed

kubectl logs api-service-7d8f9c-xk2lp
# Error: database connection failed
# Was this deployed with the new config?

kubectl get configmap -n production
# 47 configmaps... which one is current?

I spent 3 hours:

1. Finding what version was "supposed" to be deployed

2. Checking what was actually running

3. Rolling back (maybe?)

4. Fixing broken config

5. Redeploying

6. Hoping I didn't make it worse

The next morning, our CTO asked: "What's deployed in production right now?"

I couldn't answer with confidence.

That's when I discovered GitOps.

What is GitOps?

GitOps is a way of managing infrastructure and applications where Git is the single source of truth.

Instead of running imperative commands (kubectl apply, kubectl set image), you:

1. Declare desired state in Git

2. Let an automated agent sync Git → Cluster

3. Git history = deployment history

4. Want to know what's deployed? Check Git

The Four GitOps Principles

1. Declarative - System described declaratively (YAML manifests)

2. Versioned and Immutable - Stored in Git, every change is a commit

3. Pulled Automatically - Software agents pull desired state from Git

4. Continuously Reconciled - Agents ensure actual state matches desired state

GitOps Mental Model

Flow:

1. Developer commits Kubernetes manifests to Git

2. ArgoCD watches the Git repository

3. ArgoCD detects changes

4. ArgoCD applies changes to cluster

5. ArgoCD reports sync status

Key insight: You never run kubectl apply in production. Git is deployed, not you.

Imperative vs Declarative

Imperative Deployments (The Old Way)

# How to get to desired state
kubectl create deployment nginx --image=nginx:1.19
kubectl scale deployment nginx --replicas=3
kubectl expose deployment nginx --port=80
kubectl set image deployment/nginx nginx=nginx:1.20
kubectl rollout restart deployment/nginx
kubectl annotate deployment nginx deployed-by=alice

# What's deployed? ¯\_(ツ)_/¯
# Who deployed it? Check Slack?
# When was it deployed? No idea
# How to rollback? Hope you have the commands

Problems:

• No source of truth

• No audit trail

• Hard to rollback

• Configuration drift

• "Works on my machine" syndrome

Declarative Deployments (GitOps Way)

# What desired state looks like
# manifests/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  annotations:
    deployed-by: git-commit-sha-abc123
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.20
        ports:
        - containerPort: 80

# Deploy to production
git add manifests/deployment.yaml
git commit -m "Deploy nginx 1.20 with 3 replicas"
git push origin main

# ArgoCD automatically syncs
# What's deployed? Check Git
# Who deployed it? git log
# When was it deployed? git log --date=relative
# How to rollback? git revert

Benefits:

• Git is the source of truth

• Complete audit trail

• Easy rollback (git revert)

• No configuration drift

• Infrastructure as code

Git as Single Source of Truth

This is the most important GitOps principle.

Before GitOps:

Production state = Mystery box
├─ Some kubectl commands
├─ Some Terraform runs
├─ Some manual edits
└─ Some Helm upgrades
   = Who knows what's deployed?

With GitOps:

Production state = Git repository
├─ manifests/api-deployment.yaml (commit abc123)
├─ manifests/api-service.yaml (commit abc123)
├─ manifests/api-configmap.yaml (commit def456)
└─ manifests/api-secret.yaml (sealed)
   = Exact state is in Git

Want to know what's deployed?

# Not this
ssh into production
kubectl get all

# This
git log --oneline
git show HEAD:manifests/

Real Benefits I've Experienced

1. Complete Audit Trail

Before GitOps:

Boss: "Who deployed API v2.3.1 to production?"
Me: "Uh... let me check Slack... maybe Alice? Last Tuesday?"
Boss: "Why was it deployed?"
Me: "Good question..."

With GitOps:

git log --oneline manifests/api-deployment.yaml

abc123 Deploy API v2.3.1 to fix memory leak (Alice, 3 days ago)
def456 Deploy API v2.3.0 with new features (Bob, 1 week ago)

Full context: who, what, when, why.

2. Easy Rollback

Before GitOps:

# Deployment failing, need to rollback
kubectl rollout undo deployment/api
# Which version are we on now? No idea
# Did this rollback the config too? Not sure
# Are we safe? Hope so

With GitOps:

# Deployment failing, need to rollback
git revert HEAD
git push

# ArgoCD syncs previous version
# Everything rolled back: deployment + config + secrets
# Git history shows exactly what changed

3. Disaster Recovery

Our production cluster was accidentally deleted. (Yes, really.)

Before GitOps: Would have taken days to rebuild.

With GitOps:

# Create new cluster
kubectl create namespace production

# Install ArgoCD
kubectl apply -f argocd-install.yaml

# Point ArgoCD at Git repo
kubectl apply -f argocd-application.yaml

# Wait 5 minutes
# Everything back to exact previous state

Full recovery: 30 minutes.

4. Consistency Across Environments

Before GitOps:

Dev: nginx:1.20, 2 replicas, config v1
Staging: nginx:1.19, 3 replicas, config v2
Production: nginx:1.20, 3 replicas, config v1.5

# Why are they different? No idea

With GitOps:

dev/manifests/         (branch: dev)
staging/manifests/     (branch: staging)
production/manifests/  (branch: main)

# Differences visible in Git
# Promote dev → staging → prod via PR
# All environments tracked

5. Eliminates Configuration Drift

Configuration drift = Actual state ≠ Desired state

How it happens:

1. Someone runs kubectl edit deployment api

2. Manual change applied

3. Nobody updates Git

4. Next deployment overwrites manual change

5. Confusion ensues

GitOps prevents drift:

• ArgoCD continuously reconciles

• If actual ≠ desired, ArgoCD fixes it

• Manual changes auto-reverted

• Git always wins

GitOps vs Traditional CD

Traditional CI/CD (Push-based)

Problems:

• CI system needs cluster credentials

• Security risk: credentials in CI

• No visibility into deployed state

• Hard to rollback

• Configuration drift

GitOps (Pull-based)

Benefits:

• Cluster credentials stay in cluster

• ArgoCD has cluster access, CI doesn't

• Clear separation: CI builds, CD deploys

• Git = deployed state

• Auto-sync on Git changes

The GitOps Workflow

Here's how we work with GitOps:

1. Development

# Local development
docker build -t myapp:v1.2.3 .
docker push registry.io/myapp:v1.2.3

2. Update Manifest

# manifests/deployment.yaml
spec:
  template:
    spec:
      containers:
      - name: myapp
        image: registry.io/myapp:v1.2.3  # Changed from v1.2.2

3. Commit and Push

git add manifests/deployment.yaml
git commit -m "Deploy myapp v1.2.3 - Fix critical bug"
git push origin main

4. ArgoCD Syncs

ArgoCD: New commit detected
ArgoCD: Comparing desired (Git) vs actual (cluster)
ArgoCD: Difference found: image v1.2.2 → v1.2.3
ArgoCD: Applying changes...
ArgoCD: Sync successful ✓

5. Verify

# Check ArgoCD UI
# See deployment status
# View sync history
# Check application health

# Or via CLI
argocd app get myapp
# Status: Healthy, Synced

Common GitOps Patterns

Pattern 1: Single Repo, Multiple Environments

my-app-manifests/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
├── overlays/
│   ├── dev/
│   │   └── kustomization.yaml
│   ├── staging/
│   │   └── kustomization.yaml
│   └── production/
│       └── kustomization.yaml

Pattern 2: App Repo + Config Repo

Repo 1: myapp (application code)
├── src/
├── Dockerfile
└── .github/workflows/build.yml

Repo 2: myapp-config (Kubernetes manifests)
├── manifests/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── configmap.yaml

Flow:

1. Developer pushes to myapp repo

2. CI builds Docker image

3. CI updates image tag in myapp-config repo

4. ArgoCD syncs myapp-config → cluster

Pattern 3: Monorepo

my-company/
├── apps/
│   ├── api/
│   ├── web/
│   └── worker/
└── manifests/
    ├── api/
    ├── web/
    └── worker/

All apps and manifests in one repo.

What GitOps Tools Exist?

ArgoCD (This Series)

• Most popular GitOps tool for Kubernetes

• Great UI

• Application management

• Multi-cluster support

• We'll use ArgoCD throughout this series

Flux CD

• Alternative to ArgoCD

• More lightweight

• CLI-focused

• Great for simple use cases

Jenkins X

• Opinionated CI/CD platform

• Includes GitOps

• Full platform vs just GitOps

This series focuses on ArgoCD - it's the most widely adopted and feature-rich.

Is GitOps Right for You?

GitOps is Great For:

✅ Kubernetes deployments ✅ Multiple environments (dev/staging/prod) ✅ Teams > 3 people ✅ Need audit trail ✅ Want disaster recovery ✅ Need consistency ✅ Compliance requirements

GitOps Might Be Overkill For:

❌ Single environment ❌ Solo developer ❌ Rapid experimentation ❌ Not using Kubernetes ❌ Simple static sites

Key Takeaways

1. GitOps = Git as source of truth

   • Desired state lives in Git

   • Actual state matches Git

   • Git history = deployment history

2. Declarative, not imperative

   • Define WHAT you want, not HOW to get there

   • No more kubectl apply in production

   • Manifests in Git, not commands in terminals

3. Pull-based, not push-based

   • ArgoCD pulls from Git

   • Cluster credentials stay in cluster

   • More secure than CI pushing

4. Benefits are real

   • Complete audit trail

   • Easy rollback

   • Disaster recovery

   • No configuration drift

   • Consistency across environments

5. ArgoCD is the tool

   • Most popular GitOps operator

   • Rich UI and CLI

   • Production-ready

   • We'll dive deep in this series

In the next article, we'll explore GitOps core concepts in detail: desired state, reconciliation loops, and how GitOps agents keep your cluster in sync.

---

Previous: GitOps 101 Overview Next: Understanding GitOps Core Concepts