GitOps Best Practices and Production Lessons

What I Learned Running GitOps in Production

After 2 years running ArgoCD in production with 200+ applications across 12 clusters, here's what I learned the hard way.

The Incident That Taught Me About Secrets

3 AM. Slack alert: "Database password exposed in Git!"

Someone committed this:

# DO NOT DO THIS!
apiVersion: v1
kind: Secret
metadata:
  name: db-secret
stringData:
  password: "SuperSecret123!"  # ← Visible in Git history!

Even though we reverted immediately, the secret was in Git history forever. We had to:

1. Rotate all database passwords

2. Audit entire Git history

3. Force-push to remove secret (risky)

4. Implement secret scanning

Cost: 6 hours of downtime, $50k in lost revenue.

Let me show you how to handle secrets properly.

Secret Management

DON'T: Store Secrets in Git

# NEVER do this
apiVersion: v1
kind: Secret
metadata:
  name: db-secret
stringData:
  username: admin
  password: SuperSecret123!  # ✗ EXPOSED IN GIT

DO: Use Sealed Secrets

Sealed Secrets encrypts secrets that can be safely committed to Git.

Install Sealed Secrets Controller

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml

Install kubeseal CLI

# macOS
brew install kubeseal

# Linux
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
tar xfz kubeseal-0.24.0-linux-amd64.tar.gz
sudo mv kubeseal /usr/local/bin/

Create and Seal Secret

# Create regular secret (don't commit!)
kubectl create secret generic db-secret \
  --from-literal=username=admin \
  --from-literal=password=SuperSecret123! \
  --dry-run=client -o yaml > secret.yaml

# Seal the secret
kubeseal -f secret.yaml -w sealed-secret.yaml

# Result: sealed-secret.yaml (safe to commit)
cat sealed-secret.yaml

sealed-secret.yaml:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-secret
  namespace: default
spec:
  encryptedData:
    username: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq...
    password: AgBzbCMyP+f3ClJ8iM0YH7hZSHT3t9nh...

Commit sealed secret to Git:

git add sealed-secret.yaml
git commit -m "Add database sealed secret"
git push

ArgoCD syncs → Controller decrypts → Secret created in cluster

DO: Use External Secrets Operator

For secrets stored in AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, etc.

Install External Secrets Operator

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets-system --create-namespace

Configure SecretStore (AWS Example)

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secretstore
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

Create ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-secret
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretstore
    kind: SecretStore
  target:
    name: db-secret
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: prod/database/password  # AWS Secrets Manager key
  - secretKey: username
    remoteRef:
      key: prod/database/username

Commit to Git (no secrets!):

git add externalsecret.yaml
git commit -m "Add database external secret"
git push

External Secrets Operator fetches from AWS → Creates Secret in cluster

Disaster Recovery

Backup Strategy

What to backup:

1. ArgoCD configuration (ConfigMaps, Secrets)

2. Application CRDs

3. Git repositories (already backed up)

4. Cluster state (optional, Git is source of truth)

Backup ArgoCD Configuration

#!/bin/bash
# backup-argocd.sh

BACKUP_DIR="argocd-backup-$(date +%Y%m%d-%H%M%S)"
mkdir -p $BACKUP_DIR

# Backup ArgoCD namespace
kubectl get all -n argocd -o yaml > $BACKUP_DIR/argocd-resources.yaml

# Backup Applications
kubectl get applications -n argocd -o yaml > $BACKUP_DIR/applications.yaml

# Backup Projects
kubectl get appprojects -n argocd -o yaml > $BACKUP_DIR/projects.yaml

# Backup ConfigMaps
kubectl get configmap -n argocd -o yaml > $BACKUP_DIR/configmaps.yaml

# Backup Secrets (encrypted)
kubectl get secrets -n argocd -o yaml > $BACKUP_DIR/secrets.yaml

# Tar and encrypt
tar czf $BACKUP_DIR.tar.gz $BACKUP_DIR
gpg --symmetric --cipher-algo AES256 $BACKUP_DIR.tar.gz

echo "Backup saved to $BACKUP_DIR.tar.gz.gpg"

Automated Backup with CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: argocd-backup
  namespace: argocd
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: argocd-backup-sa
          containers:
          - name: backup
            image: bitnami/kubectl:latest
            command:
            - /bin/bash
            - -c
            - |
              kubectl get applications -n argocd -o yaml > /backup/applications.yaml
              # Upload to S3
              aws s3 cp /backup/applications.yaml s3://my-backups/argocd/applications-$(date +%Y%m%d).yaml
          restartPolicy: OnFailure

Disaster Recovery Procedure

Scenario: Entire cluster destroyed

# 1. Create new cluster
kind create cluster --name recovery

# 2. Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# 3. Wait for ArgoCD ready
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s

# 4. Restore Applications
kubectl apply -f backup/applications.yaml

# 5. ArgoCD syncs from Git
# All applications restored from Git!

# Total recovery time: ~10 minutes

This is the beauty of GitOps: Git is the disaster recovery.

Monitoring ArgoCD

Prometheus Metrics

ArgoCD exposes Prometheus metrics by default.

Scrape Configuration

# prometheus-scrape-config
scrape_configs:
- job_name: 'argocd-metrics'
  static_configs:
  - targets:
    - argocd-metrics.argocd.svc:8082
- job_name: 'argocd-server-metrics'
  static_configs:
  - targets:
    - argocd-server-metrics.argocd.svc:8083
- job_name: 'argocd-repo-server-metrics'
  static_configs:
  - targets:
    - argocd-repo-server-metrics.argocd.svc:8084

Key Metrics to Monitor

# Application sync status
argocd_app_sync_status{sync_status="OutOfSync"}

# Application health
argocd_app_health_status{health_status="Degraded"}

# Sync operations
rate(argocd_app_sync_total[5m])

# Repo server performance
histogram_quantile(0.99, argocd_git_request_duration_seconds_bucket)

# Controller queue size
argocd_app_reconcile_queue_size

Grafana Dashboard

Import official ArgoCD dashboard: ID 14584

# Or via code
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-cd/master/examples/dashboard.json

Alerting Rules

# prometheus-rules.yaml
groups:
- name: argocd
  rules:
  - alert: ArgoCDAppOutOfSync
    expr: argocd_app_sync_status{sync_status="OutOfSync"} > 0
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Application {{ $labels.name }} out of sync"
      description: "ArgoCD application has been out of sync for 10+ minutes"
  
  - alert: ArgoCDAppUnhealthy
    expr: argocd_app_health_status{health_status!="Healthy"} > 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Application {{ $labels.name }} unhealthy"
      description: "ArgoCD application health status: {{ $labels.health_status }}"
  
  - alert: ArgoCDSyncFailed
    expr: increase(argocd_app_sync_total{phase="Failed"}[5m]) > 3
    labels:
      severity: warning
    annotations:
      summary: "Multiple sync failures for {{ $labels.name }}"

Troubleshooting Common Issues

Issue 1: Application Stuck in Progressing

# Check application status
argocd app get my-app

# Check sync operation
argocd app get my-app --show-operation

# Check individual resources
kubectl get deployment my-app -n production -o yaml

# Common causes:
# - Image pull errors
# - Insufficient resources
# - Health check failures

# Force refresh
argocd app get my-app --refresh --hard-refresh

Issue 2: Sync Takes Too Long

# Check repo server logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server --tail=100

# Common causes:
# - Large Helm charts
# - Complex Kustomize builds
# - Slow Git server

# Solution: Scale repo server
kubectl scale deployment argocd-repo-server -n argocd --replicas=3

Issue 3: Out of Sync but Looks Identical

# Get detailed diff
argocd app diff my-app

# Common causes:
# - Ignored differences not configured
# - Resource modified outside Git
# - Clock skew

# Configure ignored differences
kubectl edit application my-app -n argocd
# Add:
spec:
  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas  # Ignore HPA-managed field

Issue 4: Secret Sync Failures

# Check sealed secrets controller
kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets

# Re-seal secret
kubeseal -f secret.yaml -w sealed-secret.yaml

# Verify controller public key
kubeseal --fetch-cert > pub-cert.pem

Production Checklist

Pre-Production

• ArgoCD installed with resource limits

• RBAC configured, admin account disabled

• SSO integrated

• Sealed Secrets or External Secrets configured

• Git repositories have branch protection

• Prometheus monitoring configured

• Grafana dashboards imported

• Alerting rules configured

• Notifications configured (Slack/email)

• Backup automation configured

• Disaster recovery tested

Application Setup

• Separate app and config repos

• CI/CD pipeline configured

• Image tags use SHA, not latest

• Kustomize overlays or Helm values for each environment

• Sync waves configured for complex apps

• Health checks configured

• Resource requests/limits set

• No secrets in Git

• PreSync/PostSync hooks for migrations/tests

• Rollback procedure documented

Ongoing Operations

• Monitor sync status daily

• Review out-of-sync applications

• Update ArgoCD quarterly

• Rotate sealed secrets controller keys annually

• Audit Git access logs

• Test disaster recovery quarterly

• Review and update RBAC policies

• Monitor resource usage

• Clean up unused applications

• Document GitOps workflow for team

Cost Optimization

ArgoCD Resource Tuning

# Right-size ArgoCD components
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-server
spec:
  replicas: 2  # Start with 2, scale as needed
  template:
    spec:
      containers:
      - name: argocd-server
        resources:
          requests:
            cpu: 50m      # Not 100m
            memory: 64Mi  # Not 128Mi
          limits:
            cpu: 200m
            memory: 256Mi

Application Controller Tuning

# Adjust reconciliation interval
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  timeout.reconciliation: 300s  # Default: 180s (3 min)
  # Longer interval = less CPU, slower drift detection

Repo Server Caching

# Enable aggressive caching
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  repository.credentials: |
    - url: https://github.com/mycompany
      # Cache repo for 24 hours
      insecure: false
      enableLfs: false

Security Best Practices

1. Least Privilege RBAC

# Developers can sync, not delete
p, role:developer, applications, sync, */*, allow
p, role:developer, applications, get, */*, allow
p, role:developer, applications, delete, */*, deny

2. Network Policies

# Restrict ArgoCD egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: argocd-egress
  namespace: argocd
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: argocd
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector: {}
  - to:
    - podSelector: {}
  - ports:
    - port: 443  # Git HTTPS
      protocol: TCP
    - port: 22   # Git SSH
      protocol: TCP
    - port: 53   # DNS
      protocol: UDP

3. Git Repository Protection

• Enable branch protection on main/production branches

• Require PR reviews before merge

• Enable commit signing

• Use separate repos for sensitive environments

• Audit Git access logs

4. Secret Scanning

# .github/workflows/secret-scan.yaml
name: Secret Scanning

on:
  pull_request:

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    
    - name: Run Gitleaks
      uses: gitleaks/gitleaks-action@v2
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Performance Tuning

Large-Scale Deployments (200+ Apps)

# Enable sharding (3 controllers)
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: argocd-application-controller
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: argocd-application-controller
        env:
        - name: ARGOCD_CONTROLLER_REPLICAS
          value: "3"
        resources:
          requests:
            cpu: 1000m
            memory: 2Gi
          limits:
            cpu: 2000m
            memory: 4Gi

# Scale repo server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: argocd-repo-server
        resources:
          requests:
            cpu: 500m
            memory: 512Mi
          limits:
            cpu: 2000m
            memory: 2Gi

Key Takeaways

1. Secret management is critical

   • Never commit secrets to Git

   • Use Sealed Secrets or External Secrets Operator

   • Rotate secrets regularly

   • Audit Git history

2. Disaster recovery is built-in

   • Git is the backup

   • Restore by re-syncing from Git

   • Test recovery procedures

   • Automate backups of ArgoCD config

3. Monitor everything

   • Prometheus metrics

   • Grafana dashboards

   • Alert on out-of-sync and unhealthy apps

   • Track sync performance

4. Production readiness checklist

   • RBAC, SSO, secrets, monitoring, backups

   • CI/CD integration

   • Rollback procedures

   • Team documentation

5. Scale ArgoCD with your needs

   • 1-50 apps: Default setup

   • 50-200 apps: Scale repo server and controller

   • 200+ apps: Enable sharding, external Redis

   • Monitor and tune

Final Thoughts

GitOps with ArgoCD transformed how we deploy applications:

• Before: 2 AM deployments, manual kubectl commands, configuration drift, rollback panic

• After: Git push, automated deployment, zero drift, confident rollbacks

The investment:

• 2 weeks initial setup

• 1 week team training

• Ongoing: ~2 hours/week maintenance

The return:

• 90% reduction in deployment time

• 100% deployment audit trail

• Zero configuration drift incidents

• Disaster recovery in minutes

• Sleep at night

Start small:

1. Install ArgoCD

2. Deploy one app

3. Add second environment

4. Integrate CI/CD

5. Expand to more apps

GitOps is a journey. This series gave you the map. Now go build!

---

Previous: GitOps CI/CD Pipeline Integration Back to: GitOps 101 Overview