Part 5: Production Best Practices and Security

Series Navigation: ← Part 4: Chart Dependencies and Repositories | Back to Series Overview

Introduction

Throughout this series, we've built Helm Charts from basic to advanced. But getting Charts production-ready requires additional considerations: security, secrets management, testing, CI/CD integration, and operational best practices.

This final part shares the lessons I've learned deploying Charts to production. These patterns have prevented outages, security incidents, and countless debugging sessions.

Secrets Management

Hardcoding secrets in values files is dangerous. Here are secure approaches I use.

Kubernetes Secrets

Generate secrets during installation:

templates/secret.yaml:

{{- if .Values.secrets.create }}
apiVersion: v1
kind: Secret
metadata:
  name: {{ include "typescript-api.fullname" . }}-secrets
  labels:
    {{- include "typescript-api.labels" . | nindent 4 }}
type: Opaque
data:
  {{- if .Values.secrets.databasePassword }}
  database-password: {{ .Values.secrets.databasePassword | b64enc | quote }}
  {{- else }}
  # Generate random password if not provided
  database-password: {{ randAlphaNum 32 | b64enc | quote }}
  {{- end }}
  
  {{- if .Values.secrets.apiKey }}
  api-key: {{ .Values.secrets.apiKey | b64enc | quote }}
  {{- else }}
  api-key: {{ randAlphaNum 64 | b64enc | quote }}
  {{- end }}
  
  {{- if .Values.secrets.jwtSecret }}
  jwt-secret: {{ .Values.secrets.jwtSecret | b64enc | quote }}
  {{- else }}
  jwt-secret: {{ randAlphaNum 128 | b64enc | quote }}
  {{- end }}
{{- end }}

Deployment using secrets:

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: {{ include "typescript-api.fullname" . }}-secrets
        key: database-password
  - name: API_KEY
    valueFrom:
      secretKeyRef:
        name: {{ include "typescript-api.fullname" . }}-secrets
        key: api-key
  - name: JWT_SECRET
    valueFrom:
      secretKeyRef:
        name: {{ include "typescript-api.fullname" . }}-secrets
        key: jwt-secret

Install without exposing secrets:

# Secrets auto-generated
helm install my-api ./typescript-api --set secrets.create=true

# Or provide via stdin
echo "databasePassword=SuperSecurePass123" | helm install my-api ./typescript-api -f -

Helm Secrets Plugin

For encrypted secrets in Git, I use the helm-secrets plugin:

Install plugin:

helm plugin install https://github.com/jkroepke/helm-secrets

Create encrypted secrets file:

# Create secrets.yaml
cat > secrets.yaml <<EOF
secrets:
  databasePassword: "MySecurePassword123"
  apiKey: "abc123xyz789"
  jwtSecret: "super-secret-jwt-key"
EOF

# Encrypt with SOPS
helm secrets encrypt secrets.yaml

# Creates secrets.yaml.enc (encrypted)
# Commit secrets.yaml.enc to Git, delete secrets.yaml

Install with encrypted secrets:

helm secrets install my-api ./typescript-api -f secrets.yaml.enc

External Secrets Operator

For production, I use External Secrets Operator to sync from AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault:

templates/externalsecret.yaml:

{{- if .Values.externalSecrets.enabled }}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: {{ include "typescript-api.fullname" . }}-secrets
  labels:
    {{- include "typescript-api.labels" . | nindent 4 }}
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: {{ .Values.externalSecrets.secretStore }}
    kind: SecretStore
  target:
    name: {{ include "typescript-api.fullname" . }}-secrets
    creationPolicy: Owner
  data:
    - secretKey: database-password
      remoteRef:
        key: {{ .Values.externalSecrets.prefix }}/database-password
    
    - secretKey: api-key
      remoteRef:
        key: {{ .Values.externalSecrets.prefix }}/api-key
    
    - secretKey: jwt-secret
      remoteRef:
        key: {{ .Values.externalSecrets.prefix }}/jwt-secret
{{- end }}

values.yaml:

externalSecrets:
  enabled: true
  secretStore: aws-secrets-manager
  prefix: "production/typescript-api"

Security Best Practices

Security configurations I enforce in all Charts:

Pod Security Context

values.yaml:

podSecurityContext:
  runAsNonRoot: true
  runAsUser: 1000
  runAsGroup: 1000
  fsGroup: 1000
  seccompProfile:
    type: RuntimeDefault

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000
  capabilities:
    drop:
      - ALL

Network Policies

templates/networkpolicy.yaml:

{{- if .Values.networkPolicy.enabled }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: {{ include "typescript-api.fullname" . }}
  labels:
    {{- include "typescript-api.labels" . | nindent 4 }}
spec:
  podSelector:
    matchLabels:
      {{- include "typescript-api.selectorLabels" . | nindent 6 }}
  policyTypes:
    - Ingress
    - Egress
  
  ingress:
    # Allow from ingress controller
    - from:
      {{- range .Values.networkPolicy.ingress.allowedSources }}
        - {{ . | toYaml | nindent 10 }}
      {{- end }}
      ports:
        - protocol: TCP
          port: {{ .Values.service.targetPort }}
  
  egress:
    # DNS
    - to:
        - namespaceSelector:
            matchLabels:
              name: kube-system
      ports:
        - protocol: UDP
          port: 53
    
    # Database
    {{- if .Values.postgresql.enabled }}
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: postgresql
      ports:
        - protocol: TCP
          port: 5432
    {{- end }}
    
    # Redis
    {{- if .Values.redis.enabled }}
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: redis
      ports:
        - protocol: TCP
          port: 6379
    {{- end }}
    
    # HTTPS external
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: TCP
          port: 443
{{- end }}

Service Account with RBAC

templates/rbac.yaml:

{{- if .Values.rbac.create }}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ include "typescript-api.serviceAccountName" . }}
  labels:
    {{- include "typescript-api.labels" . | nindent 4 }}
  {{- with .Values.serviceAccount.annotations }}
  annotations:
    {{- toYaml . | nindent 4 }}
  {{- end }}
---
{{- if .Values.rbac.rules }}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: {{ include "typescript-api.fullname" . }}
  labels:
    {{- include "typescript-api.labels" . | nindent 4 }}
rules:
  {{- toYaml .Values.rbac.rules | nindent 2 }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: {{ include "typescript-api.fullname" . }}
  labels:
    {{- include "typescript-api.labels" . | nindent 4 }}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: {{ include "typescript-api.fullname" . }}
subjects:
  - kind: ServiceAccount
    name: {{ include "typescript-api.serviceAccountName" . }}
    namespace: {{ .Release.Namespace }}
{{- end }}
{{- end }}

Testing Charts

Testing prevents broken deployments.

Helm Lint

# Basic linting
helm lint ./typescript-api

# Lint with values
helm lint ./typescript-api -f values-production.yaml

# Strict mode
helm lint ./typescript-api --strict

Template Validation

# Render and validate with Kubernetes
helm template my-api ./typescript-api | kubectl apply --dry-run=client -f -

# With specific values
helm template my-api ./typescript-api \
  -f values-production.yaml \
  | kubectl apply --dry-run=server -f -

Chart Testing (ct)

I use chart-testing tool in CI/CD:

# Install ct
brew install chart-testing

# Lint charts
ct lint --config ct.yaml --charts charts/typescript-api

# Install and test
ct install --config ct.yaml --charts charts/typescript-api

ct.yaml:

remote: origin
target-branch: main
chart-dirs:
  - charts
chart-repos:
  - bitnami=https://charts.bitnami.com/bitnami
helm-extra-args: --timeout 600s

Unit Testing with Helm-unittest

# Install plugin
helm plugin install https://github.com/helm-unittest/helm-unittest

# Create tests
mkdir -p charts/typescript-api/tests

tests/deployment_test.yaml:

suite: test deployment
templates:
  - deployment.yaml
tests:
  - it: should create deployment with correct replicas
    set:
      replicaCount: 3
    asserts:
      - isKind:
          of: Deployment
      - equal:
          path: spec.replicas
          value: 3
  
  - it: should use correct image
    set:
      image.repository: my-registry/api
      image.tag: "1.2.3"
    asserts:
      - equal:
          path: spec.template.spec.containers[0].image
          value: my-registry/api:1.2.3
  
  - it: should configure environment variables
    set:
      redis.enabled: true
    asserts:
      - contains:
          path: spec.template.spec.containers[0].env
          content:
            name: REDIS_HOST
  
  - it: should set resource limits
    set:
      resources.limits.cpu: "500m"
      resources.limits.memory: "512Mi"
    asserts:
      - equal:
          path: spec.template.spec.containers[0].resources.limits.cpu
          value: "500m"
      - equal:
          path: spec.template.spec.containers[0].resources.limits.memory
          value: "512Mi"

Run tests:

helm unittest charts/typescript-api

Rollback Strategies

Helm makes rollbacks simple, but planning is important.

Release History

# View release history
helm history my-api

# Output shows revisions
REVISION  UPDATED                   STATUS      CHART               DESCRIPTION
1         Mon Feb 3 10:00:00 2026   superseded  typescript-api-1.0.0  Install complete
2         Tue Feb 4 14:30:00 2026   superseded  typescript-api-1.1.0  Upgrade complete
3         Wed Feb 5 09:15:00 2026   deployed    typescript-api-1.2.0  Upgrade complete

Rollback Operations

# Rollback to previous revision
helm rollback my-api

# Rollback to specific revision
helm rollback my-api 2

# Rollback with wait
helm rollback my-api 2 --wait --timeout 5m

# Dry run rollback
helm rollback my-api 2 --dry-run

Pre-Upgrade Hooks

Automate backups before upgrades:

templates/pre-upgrade-job.yaml:

{{- if .Values.backup.preUpgrade }}
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "typescript-api.fullname" . }}-pre-upgrade
  labels:
    {{- include "typescript-api.labels" . | nindent 4 }}
  annotations:
    "helm.sh/hook": pre-upgrade
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": before-hook-creation
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: backup
        image: postgres:15-alpine
        command:
          - sh
          - -c
          - |
            echo "Creating database backup before upgrade..."
            pg_dump -h $DB_HOST -U $DB_USER -d $DB_NAME > /backup/pre-upgrade-$(date +%Y%m%d-%H%M%S).sql
        env:
          - name: DB_HOST
            value: {{ .Release.Name }}-postgresql
          - name: DB_USER
            value: {{ .Values.postgresql.auth.username }}
          - name: DB_NAME
            value: {{ .Values.postgresql.auth.database }}
          - name: PGPASSWORD
            valueFrom:
              secretKeyRef:
                name: {{ .Release.Name }}-postgresql
                key: password
        volumeMounts:
          - name: backup
            mountPath: /backup
      volumes:
        - name: backup
          persistentVolumeClaim:
            claimName: {{ include "typescript-api.fullname" . }}-backup
{{- end }}

CI/CD Integration

Automate Chart deployments with CI/CD.

GitHub Actions Workflow

.github/workflows/helm-deploy.yaml:

name: Deploy Helm Chart

on:
  push:
    branches: [main]
    paths:
      - 'charts/typescript-api/**'
  workflow_dispatch:

env:
  HELM_VERSION: v3.14.0
  CHART_PATH: charts/typescript-api

jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      
      - name: Set up Helm
        uses: azure/setup-helm@v3
        with:
          version: ${{ env.HELM_VERSION }}
      
      - name: Set up chart-testing
        uses: helm/chart-testing-action@v2
      
      - name: Lint chart
        run: |
          helm lint ${{ env.CHART_PATH }}
          ct lint --config ct.yaml --charts ${{ env.CHART_PATH }}
      
      - name: Install unittest plugin
        run: helm plugin install https://github.com/helm-unittest/helm-unittest
      
      - name: Run unit tests
        run: helm unittest ${{ env.CHART_PATH }}
      
      - name: Create kind cluster
        uses: helm/kind-action@v1
      
      - name: Install chart
        run: |
          helm dependency update ${{ env.CHART_PATH }}
          helm install test ${{ env.CHART_PATH }} \
            --wait \
            --timeout 5m \
            --set redis.enabled=false \
            --set postgresql.enabled=false
      
      - name: Test deployment
        run: |
          kubectl get all
          kubectl wait --for=condition=ready pod -l app.kubernetes.io/instance=test --timeout=300s

  publish:
    needs: lint-and-test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      
      - name: Set up Helm
        uses: azure/setup-helm@v3
        with:
          version: ${{ env.HELM_VERSION }}
      
      - name: Login to GitHub Container Registry
        run: |
          echo ${{ secrets.GITHUB_TOKEN }} | helm registry login ghcr.io -u ${{ github.actor }} --password-stdin
      
      - name: Package and push chart
        run: |
          helm package ${{ env.CHART_PATH }}
          helm push typescript-api-*.tgz oci://ghcr.io/${{ github.repository_owner }}/charts
  
  deploy-staging:
    needs: publish
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      
      - name: Set up Helm
        uses: azure/setup-helm@v3
        with:
          version: ${{ env.HELM_VERSION }}
      
      - name: Configure kubectl
        run: |
          echo "${{ secrets.KUBECONFIG_STAGING }}" | base64 -d > kubeconfig
          echo "KUBECONFIG=$(pwd)/kubeconfig" >> $GITHUB_ENV
      
      - name: Deploy to staging
        run: |
          helm upgrade --install staging-api \
            oci://ghcr.io/${{ github.repository_owner }}/charts/typescript-api \
            --namespace staging \
            --create-namespace \
            -f charts/typescript-api/values-staging.yaml \
            --wait \
            --timeout 10m
      
      - name: Verify deployment
        run: |
          kubectl rollout status deployment/staging-api -n staging
          kubectl get pods -n staging
  
  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      
      - name: Set up Helm
        uses: azure/setup-helm@v3
        with:
          version: ${{ env.HELM_VERSION }}
      
      - name: Configure kubectl
        run: |
          echo "${{ secrets.KUBECONFIG_PRODUCTION }}" | base64 -d > kubeconfig
          echo "KUBECONFIG=$(pwd)/kubeconfig" >> $GITHUB_ENV
      
      - name: Deploy to production
        run: |
          helm upgrade --install prod-api \
            oci://ghcr.io/${{ github.repository_owner }}/charts/typescript-api \
            --namespace production \
            --create-namespace \
            -f charts/typescript-api/values-production.yaml \
            --wait \
            --timeout 10m
      
      - name: Verify deployment
        run: |
          kubectl rollout status deployment/prod-api -n production
          kubectl get pods -n production

GitLab CI/CD

.gitlab-ci.yml:

stages:
  - lint
  - test
  - publish
  - deploy

variables:
  HELM_VERSION: "3.14.0"
  CHART_PATH: "charts/typescript-api"

lint:
  stage: lint
  image: alpine/helm:${HELM_VERSION}
  script:
    - helm lint ${CHART_PATH}
    - helm template test ${CHART_PATH} | kubectl apply --dry-run=client -f -

test:
  stage: test
  image: alpine/helm:${HELM_VERSION}
  script:
    - helm plugin install https://github.com/helm-unittest/helm-unittest
    - helm unittest ${CHART_PATH}

publish:
  stage: publish
  image: alpine/helm:${HELM_VERSION}
  only:
    - main
  script:
    - echo ${CI_REGISTRY_PASSWORD} | helm registry login ${CI_REGISTRY} -u ${CI_REGISTRY_USER} --password-stdin
    - helm package ${CHART_PATH}
    - helm push typescript-api-*.tgz oci://${CI_REGISTRY}/${CI_PROJECT_PATH}/charts

deploy:staging:
  stage: deploy
  image: alpine/helm:${HELM_VERSION}
  environment:
    name: staging
  only:
    - main
  script:
    - helm upgrade --install staging-api oci://${CI_REGISTRY}/${CI_PROJECT_PATH}/charts/typescript-api
      --namespace staging
      --create-namespace
      -f ${CHART_PATH}/values-staging.yaml
      --wait

deploy:production:
  stage: deploy
  image: alpine/helm:${HELM_VERSION}
  environment:
    name: production
  when: manual
  only:
    - main
  script:
    - helm upgrade --install prod-api oci://${CI_REGISTRY}/${CI_PROJECT_PATH}/charts/typescript-api
      --namespace production
      --create-namespace
      -f ${CHART_PATH}/values-production.yaml
      --wait

Performance Optimization

Optimize Charts for efficiency.

Resource Management

resources:
  # Always set limits
  limits:
    cpu: 500m
    memory: 512Mi
  # Requests ensure scheduling
  requests:
    cpu: 200m
    memory: 256Mi

# Vertical Pod Autoscaler for recommendations
verticalPodAutoscaler:
  enabled: false
  updateMode: "Off"  # Start with recommendations only

Horizontal Pod Autoscaling

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
        - type: Pods
          value: 4
          periodSeconds: 15
      selectPolicy: Max

Pod Disruption Budgets

# templates/poddisruptionbudget.yaml
{{- if .Values.podDisruptionBudget.enabled }}
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: {{ include "typescript-api.fullname" . }}
  labels:
    {{- include "typescript-api.labels" . | nindent 4 }}
spec:
  {{- if .Values.podDisruptionBudget.minAvailable }}
  minAvailable: {{ .Values.podDisruptionBudget.minAvailable }}
  {{- end }}
  {{- if .Values.podDisruptionBudget.maxUnavailable }}
  maxUnavailable: {{ .Values.podDisruptionBudget.maxUnavailable }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "typescript-api.selectorLabels" . | nindent 6 }}
{{- end }}

values.yaml:

podDisruptionBudget:
  enabled: true
  minAvailable: 1  # or maxUnavailable: 1

Troubleshooting

Common issues and solutions:

Release Stuck

# Check release status
helm status my-api

# View release history
helm history my-api

# Force delete stuck release
kubectl delete secret -l owner=helm,name=my-api

# Reinstall
helm install my-api ./typescript-api

Failed Upgrades

# Check what changed
helm diff upgrade my-api ./typescript-api -f values-production.yaml

# Upgrade with atomic (auto-rollback on failure)
helm upgrade my-api ./typescript-api --atomic --timeout 10m

# Manual rollback if needed
helm rollback my-api

Debug Template Issues

# Render templates with debug
helm template my-api ./typescript-api --debug

# Check specific template
helm template my-api ./typescript-api -s templates/deployment.yaml

# Validate against cluster
helm template my-api ./typescript-api | kubectl apply --dry-run=server -f -

Production Checklist

Before deploying to production:

✅ Security contexts configured (non-root, read-only filesystem)
✅ Resource limits and requests defined
✅ Health probes configured (liveness and readiness)
✅ Secrets managed securely (not in values files)
✅ Network policies implemented
✅ RBAC configured with least privilege
✅ Pod disruption budgets set
✅ Horizontal pod autoscaling configured
✅ Monitoring and logging enabled
✅ Backup and rollback procedures tested
✅ Chart versioned and documented
✅ CI/CD pipeline tested
✅ Values files for each environment
✅ Dependencies pinned to specific versions
✅ Chart linted and unit tested

Conclusion

We've completed the Helm 101 series! From basics to production-ready patterns, you now have the knowledge to:

Install and configure Helm
Create Charts for TypeScript applications
Use advanced templating techniques
Manage dependencies and repositories
Deploy securely to production
Integrate with CI/CD pipelines
Troubleshoot and optimize deployments

Helm has transformed how I manage Kubernetes deployments. What used to require dozens of YAML files and manual coordination now happens with a single command. The power of templating, combined with version control and rollback capabilities, makes Helm indispensable for production Kubernetes workloads.

Additional Resources

Series Navigation: ← Part 4: Chart Dependencies and Repositories | Back to Series Overview

Key Takeaways

✅ Never hardcode secrets—use Kubernetes Secrets, helm-secrets, or External Secrets
✅ Enforce security contexts, network policies, and RBAC
✅ Test charts with lint, unittest, and integration tests
✅ Automate deployments with CI/CD pipelines
✅ Always set resource limits and configure autoscaling
✅ Use pod disruption budgets for high availability
✅ Plan rollback strategies before production deployment
✅ Follow the production checklist for every release

PreviousPart 4: Chart Dependencies and Repositories NextGitOps 101

Last updated 12 days ago

hashtagIntroduction

hashtagSecrets Management

hashtagKubernetes Secrets

hashtagHelm Secrets Plugin

hashtagExternal Secrets Operator

hashtagSecurity Best Practices

hashtagPod Security Context

hashtagNetwork Policies

hashtagService Account with RBAC

hashtagTesting Charts

hashtagHelm Lint

hashtagTemplate Validation

hashtagChart Testing (ct)

hashtagUnit Testing with Helm-unittest

hashtagRollback Strategies

hashtagRelease History

hashtagRollback Operations

hashtagPre-Upgrade Hooks

hashtagCI/CD Integration

hashtagGitHub Actions Workflow

hashtagGitLab CI/CD

hashtagPerformance Optimization

hashtagResource Management

hashtagHorizontal Pod Autoscaling

hashtagPod Disruption Budgets

hashtagTroubleshooting

hashtagRelease Stuck

hashtagFailed Upgrades

hashtagDebug Template Issues

hashtagProduction Checklist

hashtagConclusion

hashtagAdditional Resources

hashtagKey Takeaways

Introduction

Secrets Management

Kubernetes Secrets

Helm Secrets Plugin

External Secrets Operator

Security Best Practices

Pod Security Context

Network Policies

Service Account with RBAC

Testing Charts

Helm Lint

Template Validation

Chart Testing (ct)

Unit Testing with Helm-unittest

Rollback Strategies

Release History

Rollback Operations

Pre-Upgrade Hooks

CI/CD Integration

GitHub Actions Workflow

GitLab CI/CD

Performance Optimization

Resource Management

Horizontal Pod Autoscaling

Pod Disruption Budgets

Troubleshooting

Release Stuck

Failed Upgrades

Debug Template Issues

Production Checklist

Conclusion

Additional Resources

Key Takeaways