Kustomize and Helm with ArgoCD

Managing Dev, Staging, and Production (Without Copy-Paste Hell)

I had three environments: dev, staging, production.

My first approach? Copy the manifests:

manifests/
├── dev/
│   ├── deployment.yaml     # 3 replicas, dev image
│   ├── service.yaml
│   └── configmap.yaml
├── staging/
│   ├── deployment.yaml     # 5 replicas, staging image
│   ├── service.yaml
│   └── configmap.yaml
└── production/
    ├── deployment.yaml     # 10 replicas, prod image
    ├── service.yaml
    └── configmap.yaml

The pain:

Changed service.yaml? Copy to all 3 environments
Fixed a bug in deployment.yaml? Update in 3 places
Added new resource? Copy to 3 directories
Drift between environments constantly

Then I discovered Kustomize and Helm with ArgoCD. Game changer.

The Problem: Environment-Specific Configuration

Same application, different configurations per environment:

Configuration

Dev

Staging

Production

Replicas

Resources

Low

Medium

High

Image tag

latest

v1.2.3

Domain

dev.example.com

staging.example.com

example.com

Database

dev-db

staging-db

prod-db

Secrets

dev-secrets

staging-secrets

prod-secrets

How to manage this without duplication?

Answer: Kustomize or Helm.

Approach 1: Kustomize with ArgoCD

Kustomize is a Kubernetes-native configuration management tool. It uses bases and overlays.

Kustomize Concepts

Repository Structure:
base/              ← Shared configuration (DRY)
├── deployment.yaml
├── service.yaml
└── kustomization.yaml

overlays/          ← Environment-specific patches
├── dev/
│   └── kustomization.yaml
├── staging/
│   └── kustomization.yaml
└── production/
    └── kustomization.yaml

Base = Common resources Overlay = Environment-specific changes

Setting Up Kustomize Structure

# Create repo structure
mkdir -p my-app-kustomize/{base,overlays/{dev,staging,production}}
cd my-app-kustomize

Create Base Manifests

base/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels:
    app: api
spec:
  replicas: 1  # Default, will be overridden
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: myapp/api:latest  # Will be overridden
        ports:
        - containerPort: 8080
        env:
        - name: ENVIRONMENT
          value: "default"  # Will be overridden
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 256Mi

base/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: api
  labels:
    app: api
spec:
  type: ClusterIP
  selector:
    app: api
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP

base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- deployment.yaml
- service.yaml

commonLabels:
  app: api
  managed-by: kustomize

Create Dev Overlay

overlays/dev/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Reference base
bases:
- ../../base

# Environment-specific namespace
namespace: dev

# Common labels for this environment
commonLabels:
  environment: dev

# Patches for dev
patches:
- patch: |-
    - op: replace
      path: /spec/replicas
      value: 1
  target:
    kind: Deployment
    name: api

# Image override
images:
- name: myapp/api
  newTag: dev-latest

# ConfigMap generator
configMapGenerator:
- name: api-config
  literals:
  - ENVIRONMENT=dev
  - LOG_LEVEL=debug
  - DATABASE_URL=postgres://dev-db:5432/myapp

# Name suffix (optional)
nameSuffix: -dev

Create Staging Overlay

overlays/staging/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
- ../../base

namespace: staging

commonLabels:
  environment: staging

patches:
- patch: |-
    - op: replace
      path: /spec/replicas
      value: 3
  target:
    kind: Deployment
    name: api
- patch: |-
    - op: replace
      path: /spec/template/spec/containers/0/resources/requests/cpu
      value: 200m
    - op: replace
      path: /spec/template/spec/containers/0/resources/limits/cpu
      value: 500m
  target:
    kind: Deployment
    name: api

images:
- name: myapp/api
  newTag: v1.2.3

configMapGenerator:
- name: api-config
  literals:
  - ENVIRONMENT=staging
  - LOG_LEVEL=info
  - DATABASE_URL=postgres://staging-db:5432/myapp

nameSuffix: -staging

Create Production Overlay

overlays/production/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
- ../../base

namespace: production

commonLabels:
  environment: production

patches:
- patch: |-
    - op: replace
      path: /spec/replicas
      value: 10
  target:
    kind: Deployment
    name: api
- patch: |-
    - op: replace
      path: /spec/template/spec/containers/0/resources/requests/cpu
      value: 500m
    - op: replace
      path: /spec/template/spec/containers/0/resources/requests/memory
      value: 512Mi
    - op: replace
      path: /spec/template/spec/containers/0/resources/limits/cpu
      value: 1000m
    - op: replace
      path: /spec/template/spec/containers/0/resources/limits/memory
      value: 1Gi
  target:
    kind: Deployment
    name: api

images:
- name: myapp/api
  newTag: v1.2.3  # Pinned version

configMapGenerator:
- name: api-config
  literals:
  - ENVIRONMENT=production
  - LOG_LEVEL=warn
  - DATABASE_URL=postgres://prod-db:5432/myapp

nameSuffix: -prod

Test Locally

# Build dev manifests
kustomize build overlays/dev

# Build production manifests
kustomize build overlays/production

# Compare dev vs production
diff <(kustomize build overlays/dev) <(kustomize build overlays/production)

Commit to Git

git init
git add .
git commit -m "Initial kustomize structure"
git remote add origin https://github.com/yourusername/my-app-kustomize.git
git push -u origin main

Creating ArgoCD Applications for Each Environment

Dev Application

# argocd-apps/dev-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-dev
  namespace: argocd
spec:
  project: default
  
  source:
    repoURL: https://github.com/yourusername/my-app-kustomize.git
    targetRevision: main
    path: overlays/dev  # Point to dev overlay
  
  destination:
    server: https://kubernetes.default.svc
    namespace: dev
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Staging Application

# argocd-apps/staging-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-staging
  namespace: argocd
spec:
  project: default
  
  source:
    repoURL: https://github.com/yourusername/my-app-kustomize.git
    targetRevision: main
    path: overlays/staging  # Point to staging overlay
  
  destination:
    server: https://kubernetes.default.svc
    namespace: staging
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Production Application

# argocd-apps/production-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-production
  namespace: argocd
spec:
  project: default
  
  source:
    repoURL: https://github.com/yourusername/my-app-kustomize.git
    targetRevision: main
    path: overlays/production  # Point to production overlay
  
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  
  syncPolicy:
    automated:
      prune: false  # Manual approval for production
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Deploy All Environments

# Create all applications
kubectl apply -f argocd-apps/dev-app.yaml
kubectl apply -f argocd-apps/staging-app.yaml
kubectl apply -f argocd-apps/production-app.yaml

# Check status
argocd app list
# NAME            CLUSTER                         NAMESPACE    STATUS  HEALTH
# api-dev         https://kubernetes.default.svc  dev          Synced  Healthy
# api-staging     https://kubernetes.default.svc  staging      Synced  Healthy
# api-production  https://kubernetes.default.svc  production   Synced  Healthy

Verify Different Configurations

# Dev: 1 replica
kubectl get deployment -n dev
# NAME       READY   UP-TO-DATE   AVAILABLE
# api-dev    1/1     1            1

# Staging: 3 replicas
kubectl get deployment -n staging
# NAME           READY   UP-TO-DATE   AVAILABLE
# api-staging    3/3     3            3

# Production: 10 replicas
kubectl get deployment -n production
# NAME          READY   UP-TO-DATE   AVAILABLE
# api-prod      10/10   10           10

Making Changes Across Environments

Scenario: Update service port from 80 to 8080.

# Edit base (shared by all environments)
cd my-app-kustomize
vim base/service.yaml
# Change port: 80 → port: 8080

git commit -am "Change service port to 8080"
git push

# ArgoCD auto-syncs ALL environments
# Dev, staging, production all updated!

Result: One change, three environments updated. No copy-paste.

Approach 2: Helm with ArgoCD

Helm is a package manager for Kubernetes. Uses templates with values.

Helm Concepts

Helm Chart Structure:
my-app-chart/
├── Chart.yaml           # Chart metadata
├── values.yaml          # Default values
├── templates/           # Templates
│   ├── deployment.yaml
│   ├── service.yaml
│   └── configmap.yaml
└── values/              # Environment-specific values (custom)
    ├── dev.yaml
    ├── staging.yaml
    └── production.yaml

Template = Manifest with placeholders Values = Data to fill placeholders

Create Helm Chart

# Create chart
helm create my-app-chart
cd my-app-chart

# Clean up generated files
rm -rf templates/tests
rm templates/*.txt

Create Deployment Template

templates/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.name }}
  labels:
    app: {{ .Values.name }}
    environment: {{ .Values.environment }}
spec:
  replicas: {{ .Values.replicas }}
  selector:
    matchLabels:
      app: {{ .Values.name }}
  template:
    metadata:
      labels:
        app: {{ .Values.name }}
        environment: {{ .Values.environment }}
    spec:
      containers:
      - name: {{ .Values.name }}
        image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
        ports:
        - containerPort: {{ .Values.service.targetPort }}
        env:
        {{- range $key, $value := .Values.env }}
        - name: {{ $key }}
          value: {{ $value | quote }}
        {{- end }}
        resources:
          {{- toYaml .Values.resources | nindent 10 }}

Create Service Template

templates/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.name }}
  labels:
    app: {{ .Values.name }}
spec:
  type: {{ .Values.service.type }}
  selector:
    app: {{ .Values.name }}
  ports:
  - port: {{ .Values.service.port }}
    targetPort: {{ .Values.service.targetPort }}
    protocol: TCP

Default Values

values.yaml:

# Default values (used if not overridden)
name: api
environment: default

replicas: 1

image:
  repository: myapp/api
  tag: latest
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 80
  targetPort: 8080

env:
  ENVIRONMENT: default
  LOG_LEVEL: info

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 200m
    memory: 256Mi

Environment-Specific Values

values/dev.yaml:

environment: dev

replicas: 1

image:
  tag: dev-latest

env:
  ENVIRONMENT: dev
  LOG_LEVEL: debug
  DATABASE_URL: postgres://dev-db:5432/myapp

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 200m
    memory: 256Mi

values/staging.yaml:

environment: staging

replicas: 3

image:
  tag: v1.2.3

env:
  ENVIRONMENT: staging
  LOG_LEVEL: info
  DATABASE_URL: postgres://staging-db:5432/myapp

resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

values/production.yaml:

environment: production

replicas: 10

image:
  tag: v1.2.3  # Pinned

env:
  ENVIRONMENT: production
  LOG_LEVEL: warn
  DATABASE_URL: postgres://prod-db:5432/myapp

resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 1000m
    memory: 1Gi

Test Locally

# Render dev
helm template my-app . --values values/dev.yaml

# Render production
helm template my-app . --values values/production.yaml

# Compare
diff <(helm template my-app . --values values/dev.yaml) \
     <(helm template my-app . --values values/production.yaml)

Commit to Git

git init
git add .
git commit -m "Initial Helm chart"
git remote add origin https://github.com/yourusername/my-app-helm.git
git push -u origin main

Creating ArgoCD Applications for Helm

Dev Application

# argocd-apps/helm-dev-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-helm-dev
  namespace: argocd
spec:
  project: default
  
  source:
    repoURL: https://github.com/yourusername/my-app-helm.git
    targetRevision: main
    path: my-app-chart  # Chart directory
    helm:
      valueFiles:
      - values/dev.yaml  # Dev-specific values
  
  destination:
    server: https://kubernetes.default.svc
    namespace: dev
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Staging Application

# argocd-apps/helm-staging-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-helm-staging
  namespace: argocd
spec:
  project: default
  
  source:
    repoURL: https://github.com/yourusername/my-app-helm.git
    targetRevision: main
    path: my-app-chart
    helm:
      valueFiles:
      - values/staging.yaml  # Staging-specific values
  
  destination:
    server: https://kubernetes.default.svc
    namespace: staging
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Production Application with Value Overrides

# argocd-apps/helm-production-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-helm-production
  namespace: argocd
spec:
  project: default
  
  source:
    repoURL: https://github.com/yourusername/my-app-helm.git
    targetRevision: main
    path: my-app-chart
    helm:
      valueFiles:
      - values/production.yaml
      # Additional runtime overrides
      parameters:
      - name: replicas
        value: "15"  # Override production value
      - name: image.tag
        value: v1.2.4  # Hotfix version
  
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  
  syncPolicy:
    automated:
      prune: false  # Manual for production
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Deploy

kubectl apply -f argocd-apps/helm-dev-app.yaml
kubectl apply -f argocd-apps/helm-staging-app.yaml
kubectl apply -f argocd-apps/helm-production-app.yaml

Kustomize vs Helm: Which to Choose?

Feature

Kustomize

Helm

Learning Curve

Low (just patches)

Medium (templating)

Native to K8s

Yes (built into kubectl)

No (separate tool)

Templating

Limited (patches only)

Full (Go templates)

Package Management

Yes (charts, repos)

Community Charts

Yes (huge ecosystem)

Complexity

Simple

Can get complex

ArgoCD Support

Excellent

My recommendation:

Simple apps, single team: Kustomize
Complex apps, many teams: Helm
Using 3rd-party apps: Helm (charts available)
Full control, simplicity: Kustomize

Advanced: Combining Kustomize + Helm

You can use Helm charts with Kustomize post-processing!

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
- name: nginx
  repo: https://charts.bitnami.com/bitnami
  version: 13.2.0
  releaseName: nginx
  namespace: default
  valuesInline:
    replicaCount: 3

# Additional patches with Kustomize
patches:
- patch: |-
    - op: add
      path: /spec/template/spec/containers/0/env
      value:
      - name: CUSTOM_VAR
        value: custom-value
  target:
    kind: Deployment
    name: nginx

ArgoCD Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    path: .
    # ArgoCD runs: kustomize build → helm template

Promoting Changes Across Environments

Workflow:

# 1. Develop in dev
cd my-app-kustomize
vim overlays/dev/kustomization.yaml
# Update image to dev-v1.3.0
git commit -am "Test v1.3.0 in dev"
git push
# ArgoCD syncs to dev

# 2. Test in dev
# Run tests, validate

# 3. Promote to staging
vim overlays/staging/kustomization.yaml
# Update image to v1.3.0
git commit -am "Promote v1.3.0 to staging"
git push
# ArgoCD syncs to staging

# 4. Smoke test in staging

# 5. Promote to production
vim overlays/production/kustomization.yaml
# Update image to v1.3.0
git commit -am "Deploy v1.3.0 to production"
git push
# ArgoCD syncs to production (or manual sync if auto-sync disabled)

Better: Use PRs for production

# Create feature branch
git checkout -b deploy-v1.3.0-prod
vim overlays/production/kustomization.yaml
git commit -am "Deploy v1.3.0 to production"
git push origin deploy-v1.3.0-prod

# Create Pull Request
# Review → Approve → Merge
# ArgoCD auto-syncs after merge

Key Takeaways

Kustomize: Overlays pattern
- Base + environment-specific patches
- DRY, no duplication
- Native to Kubernetes
Helm: Values pattern
- Templates + environment-specific values
- Powerful templating
- Package management
ArgoCD supports both
- Kustomize: source.path points to overlay
- Helm: source.helm.valueFiles specifies values
One repo, multiple environments
- Shared base/templates
- Environment-specific overlays/values
- Single source of truth
Git-driven promotion
- Commit to dev → test → promote to staging → promote to prod
- PR workflow for production changes
- Full audit trail

In the next article, we'll explore advanced ArgoCD features: sync waves, resource hooks, ApplicationSets, multi-cluster deployments, and more.

Previous: Your First ArgoCD Application Next: Advanced ArgoCD Features

PreviousYour First ArgoCD Application NextAdvanced ArgoCD Features

Last updated 1 month ago

hashtagManaging Dev, Staging, and Production (Without Copy-Paste Hell)

hashtagThe Problem: Environment-Specific Configuration

hashtagApproach 1: Kustomize with ArgoCD

hashtagKustomize Concepts

hashtagSetting Up Kustomize Structure

hashtagCreate Base Manifests

hashtagCreate Dev Overlay

hashtagCreate Staging Overlay

hashtagCreate Production Overlay

hashtagTest Locally

hashtagCommit to Git

hashtagCreating ArgoCD Applications for Each Environment

hashtagDev Application

hashtagStaging Application

hashtagProduction Application

hashtagDeploy All Environments

hashtagVerify Different Configurations

hashtagMaking Changes Across Environments

hashtagApproach 2: Helm with ArgoCD

hashtagHelm Concepts

hashtagCreate Helm Chart

hashtagCreate Deployment Template

hashtagCreate Service Template

hashtagDefault Values

hashtagEnvironment-Specific Values

hashtagTest Locally

hashtagCommit to Git

hashtagCreating ArgoCD Applications for Helm

hashtagDev Application

hashtagStaging Application

hashtagProduction Application with Value Overrides

hashtagDeploy

hashtagKustomize vs Helm: Which to Choose?

hashtagAdvanced: Combining Kustomize + Helm

hashtagPromoting Changes Across Environments

hashtagKey Takeaways

Managing Dev, Staging, and Production (Without Copy-Paste Hell)

The Problem: Environment-Specific Configuration

Approach 1: Kustomize with ArgoCD

Kustomize Concepts

Setting Up Kustomize Structure

Create Base Manifests

Create Dev Overlay

Create Staging Overlay

Create Production Overlay

Test Locally

Commit to Git

Creating ArgoCD Applications for Each Environment

Dev Application

Staging Application

Production Application

Deploy All Environments

Verify Different Configurations

Making Changes Across Environments

Approach 2: Helm with ArgoCD

Helm Concepts

Create Helm Chart

Create Deployment Template

Create Service Template

Default Values

Environment-Specific Values

Test Locally

Commit to Git

Creating ArgoCD Applications for Helm

Dev Application

Staging Application

Production Application with Value Overrides

Deploy

Kustomize vs Helm: Which to Choose?

Advanced: Combining Kustomize + Helm

Promoting Changes Across Environments

Key Takeaways