Configuration and Secrets

Introduction

Early in my Kubernetes journey, I hard-coded configuration values directly into container images. Database connection strings, API keys, feature flags—all baked into the image. This created a nightmare: every configuration change required rebuilding images, retagging, and redeploying. Different environments needed different images. Security was compromised because sensitive data lived in version control and container registries.

The breakthrough came when I properly understood ConfigMaps and Secrets. Configuration became external to the application, environment-specific values could be injected at runtime, and sensitive data could be managed securely. Through deploying applications across development, staging, and production environments, and integrating with external secret managers like AWS Secrets Manager and HashiCorp Vault, I learned that proper configuration management is foundational to operational excellence.

In this article, I'll share practical knowledge about managing configuration and secrets in Kubernetes, from basic ConfigMaps to enterprise-grade secret management solutions.

Table of Contents

1. Configuration Management Principles

2. ConfigMaps Fundamentals

3. Secrets Fundamentals

4. Consuming ConfigMaps and Secrets

5. Secret Encryption

6. External Secrets Management

7. Configuration Best Practices

8. Troubleshooting Configuration Issues

9. What I Learned

Configuration Management Principles

Effective configuration management follows the Twelve-Factor App methodology: separate configuration from code.

The Problem with Hard-Coded Configuration

# Bad: Configuration baked into image
FROM node:18
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .

# Hard-coded configuration
ENV DATABASE_HOST=postgres.prod.example.com
ENV DATABASE_PORT=5432
ENV API_KEY=sk-1234567890abcdef

CMD ["node", "server.js"]

Problems:

• Same image can't be used across environments

• Configuration changes require rebuilding images

• Secrets exposed in image layers

• Can't rotate credentials without rebuilding

• Violates separation of concerns

The Kubernetes Way

ConfigMaps Fundamentals

ConfigMaps store non-sensitive configuration data as key-value pairs.

Creating ConfigMaps

From Literal Values:

kubectl create configmap app-config \
  --from-literal=APP_ENV=production \
  --from-literal=LOG_LEVEL=info \
  --from-literal=MAX_CONNECTIONS=100

From Files:

# Create from single file
kubectl create configmap nginx-config --from-file=nginx.conf

# Create from multiple files
kubectl create configmap app-configs --from-file=configs/

# Create with specific key names
kubectl create configmap app-config \
  --from-file=database.conf=db-config.txt \
  --from-file=cache.conf=redis-config.txt

From Environment File:

# app.env
DATABASE_HOST=postgres.example.com
DATABASE_PORT=5432
CACHE_ENABLED=true
LOG_LEVEL=info

# Create ConfigMap from env file
kubectl create configmap app-config --from-env-file=app.env

ConfigMap YAML Definitions

Simple Key-Value ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: production
  labels:
    app: web-app
    environment: production
data:
  # Simple key-value pairs
  APP_ENV: "production"
  LOG_LEVEL: "info"
  MAX_CONNECTIONS: "100"
  FEATURE_FLAG_NEW_UI: "true"
  API_TIMEOUT: "30s"

ConfigMap with File Content:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    user nginx;
    worker_processes auto;
    error_log /var/log/nginx/error.log warn;
    pid /var/run/nginx.pid;
    
    events {
        worker_connections 1024;
    }
    
    http {
        include /etc/nginx/mime.types;
        default_type application/octet-stream;
        
        log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
        
        access_log /var/log/nginx/access.log main;
        
        sendfile on;
        tcp_nopush on;
        keepalive_timeout 65;
        gzip on;
        
        include /etc/nginx/conf.d/*.conf;
    }

ConfigMap with Multiple Configuration Files:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-configs
data:
  database.yml: |
    production:
      adapter: postgresql
      host: <%= ENV['DATABASE_HOST'] %>
      port: 5432
      database: myapp_production
      pool: 20
      timeout: 5000
  
  redis.yml: |
    production:
      host: <%= ENV['REDIS_HOST'] %>
      port: 6379
      db: 0
      pool_size: 10
  
  application.yml: |
    production:
      cache_enabled: true
      session_timeout: 3600
      max_upload_size: 10485760
      allowed_origins:
        - https://app.example.com
        - https://www.example.com

Binary Data in ConfigMaps

apiVersion: v1
kind: ConfigMap
metadata:
  name: binary-config
binaryData:
  # Base64 encoded binary data
  logo.png: iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==

Managing ConfigMaps

# List ConfigMaps
kubectl get configmaps
kubectl get cm  # Short form

# Describe ConfigMap
kubectl describe configmap app-config

# View ConfigMap YAML
kubectl get configmap app-config -o yaml

# Edit ConfigMap
kubectl edit configmap app-config

# Delete ConfigMap
kubectl delete configmap app-config

# Replace ConfigMap from file
kubectl replace -f configmap.yaml

# Dry run to test
kubectl create configmap test-config --from-literal=key=value --dry-run=client -o yaml

Secrets Fundamentals

Secrets store sensitive data like passwords, tokens, and keys.

Types of Secrets

Kubernetes supports several secret types:

• Opaque: Generic secret (default)

• kubernetes.io/service-account-token: Service account token

• kubernetes.io/dockercfg: Docker config file

• kubernetes.io/dockerconfigjson: Docker config JSON

• kubernetes.io/basic-auth: Basic authentication

• kubernetes.io/ssh-auth: SSH authentication

• kubernetes.io/tls: TLS certificate and key

• bootstrap.kubernetes.io/token: Bootstrap token

Creating Secrets

From Literal Values:

kubectl create secret generic app-secrets \
  --from-literal=database-password=superSecretP@ssw0rd \
  --from-literal=api-key=sk-1234567890abcdef \
  --from-literal=encryption-key=AES256-KEY-HERE

From Files:

# Create from file
kubectl create secret generic tls-secret \
  --from-file=tls.crt=server.crt \
  --from-file=tls.key=server.key

# Create TLS secret
kubectl create secret tls tls-secret \
  --cert=server.crt \
  --key=server.key

Docker Registry Secret:

kubectl create secret docker-registry regcred \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=myusername \
  --docker-password=mypassword \
  [email protected]

Secret YAML Definitions

Generic Secret:

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
  namespace: production
type: Opaque
stringData:
  # Use stringData for plain text (automatically base64 encoded)
  database-password: "superSecretP@ssw0rd"
  api-key: "sk-1234567890abcdef"
  database-url: "postgresql://user:password@postgres:5432/mydb"

Or with Base64 Encoded Data:

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  # Base64 encoded values
  database-password: c3VwZXJTZWNyZXRQQHNzdzByZA==
  api-key: c2stMTIzNDU2Nzg5MGFiY2RlZg==

# Encode values
echo -n 'superSecretP@ssw0rd' | base64
# Output: c3VwZXJTZWNyZXRQQHNzdzByZA==

# Decode values
echo 'c3VwZXJTZWNyZXRQQHNzdzByZA==' | base64 --decode
# Output: superSecretP@ssw0rd

TLS Secret:

apiVersion: v1
kind: Secret
metadata:
  name: tls-secret
type: kubernetes.io/tls
stringData:
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    MIIDXTCCAkWgAwIBAgIJAKL0UG+mRKe7MA0GCSqGSIb3DQEBCwUAMEUxCzAJBgNV
    ... certificate content ...
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN PRIVATE KEY-----
    MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCx8QKVmKVE5Pqe
    ... private key content ...
    -----END PRIVATE KEY-----

Basic Auth Secret:

apiVersion: v1
kind: Secret
metadata:
  name: basic-auth-secret
type: kubernetes.io/basic-auth
stringData:
  username: admin
  password: t0p-S3cr3t

SSH Auth Secret:

apiVersion: v1
kind: Secret
metadata:
  name: ssh-auth-secret
type: kubernetes.io/ssh-auth
stringData:
  ssh-privatekey: |
    -----BEGIN RSA PRIVATE KEY-----
    ... SSH private key content ...
    -----END RSA PRIVATE KEY-----

Docker Config Secret:

apiVersion: v1
kind: Secret
metadata:
  name: docker-config-secret
type: kubernetes.io/dockerconfigjson
stringData:
  .dockerconfigjson: |
    {
      "auths": {
        "https://index.docker.io/v1/": {
          "username": "myusername",
          "password": "mypassword",
          "email": "[email protected]",
          "auth": "bXl1c2VybmFtZTpteXBhc3N3b3Jk"
        }
      }
    }

Managing Secrets

# List secrets
kubectl get secrets

# Describe secret (doesn't show values)
kubectl describe secret app-secrets

# View secret YAML (shows base64 encoded values)
kubectl get secret app-secrets -o yaml

# Decode specific secret value
kubectl get secret app-secrets -o jsonpath='{.data.database-password}' | base64 --decode

# Edit secret
kubectl edit secret app-secrets

# Delete secret
kubectl delete secret app-secrets

Consuming ConfigMaps and Secrets

There are multiple ways to consume ConfigMaps and Secrets in pods.

As Environment Variables

Single Values:

apiVersion: v1
kind: Pod
metadata:
  name: config-env-pod
spec:
  containers:
  - name: app
    image: myapp:1.0
    env:
    # From ConfigMap
    - name: APP_ENV
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: APP_ENV
    
    - name: LOG_LEVEL
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: LOG_LEVEL
    
    # From Secret
    - name: DATABASE_PASSWORD
      valueFrom:
        secretKeyRef:
          name: app-secrets
          key: database-password
    
    - name: API_KEY
      valueFrom:
        secretKeyRef:
          name: app-secrets
          key: api-key

All Keys as Environment Variables:

apiVersion: v1
kind: Pod
metadata:
  name: config-env-all-pod
spec:
  containers:
  - name: app
    image: myapp:1.0
    # Load all ConfigMap keys
    envFrom:
    - configMapRef:
        name: app-config
    # Load all Secret keys
    - secretRef:
        name: app-secrets
    # Can also add prefix
    - configMapRef:
        name: feature-flags
      prefix: FEATURE_

Combined Environment Variables:

apiVersion: v1
kind: Pod
metadata:
  name: combined-env-pod
spec:
  containers:
  - name: app
    image: myapp:1.0
    env:
    # Static value
    - name: APP_NAME
      value: "my-application"
    
    # From ConfigMap
    - name: LOG_LEVEL
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: LOG_LEVEL
    
    # From Secret
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: app-secrets
          key: database-password
    
    # Constructed from multiple sources
    - name: DATABASE_URL
      value: "postgresql://user:$(DB_PASSWORD)@postgres:5432/mydb"
    
    # From pod metadata
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
    
    # All keys from ConfigMap
    envFrom:
    - configMapRef:
        name: app-config

As Volume Mounts

Mount ConfigMap as Files:

apiVersion: v1
kind: Pod
metadata:
  name: config-volume-pod
spec:
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    # Mount entire ConfigMap
    - name: config-volume
      mountPath: /etc/config
      readOnly: true
    
    # Mount specific keys
    - name: nginx-config
      mountPath: /etc/nginx/nginx.conf
      subPath: nginx.conf
      readOnly: true
    
    # Mount Secret
    - name: secret-volume
      mountPath: /etc/secrets
      readOnly: true
  
  volumes:
  # ConfigMap volume
  - name: config-volume
    configMap:
      name: app-configs
  
  # ConfigMap with specific keys
  - name: nginx-config
    configMap:
      name: nginx-config
      items:
      - key: nginx.conf
        path: nginx.conf
        mode: 0644
  
  # Secret volume
  - name: secret-volume
    secret:
      secretName: app-secrets
      defaultMode: 0400  # Read-only for owner

Selective Key Mounting:

apiVersion: v1
kind: Pod
metadata:
  name: selective-mount-pod
spec:
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - name: config
      mountPath: /etc/config
  
  volumes:
  - name: config
    configMap:
      name: app-configs
      items:
      # Mount only specific keys
      - key: database.yml
        path: database.yml
        mode: 0644
      - key: redis.yml
        path: redis.yml
        mode: 0644
      # Omit application.yml

Projected Volumes (Multiple Sources):

apiVersion: v1
kind: Pod
metadata:
  name: projected-volume-pod
spec:
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - name: all-in-one
      mountPath: /etc/config
      readOnly: true
  
  volumes:
  - name: all-in-one
    projected:
      sources:
      # Include ConfigMap
      - configMap:
          name: app-config
          items:
          - key: APP_ENV
            path: environment
      
      # Include Secret
      - secret:
          name: app-secrets
          items:
          - key: api-key
            path: api/key
      
      # Include ServiceAccount token
      - serviceAccountToken:
          expirationSeconds: 3600
          path: token
      
      # Include downward API
      - downwardAPI:
          items:
          - path: "labels"
            fieldRef:
              fieldPath: metadata.labels
          - path: "annotations"
            fieldRef:
              fieldPath: metadata.annotations

Hot Reloading Configuration

ConfigMaps and Secrets mounted as volumes are automatically updated (with a delay).

apiVersion: v1
kind: Pod
metadata:
  name: hot-reload-pod
spec:
  containers:
  - name: nginx
    image: nginx:1.21
    volumeMounts:
    - name: config
      mountPath: /etc/nginx/nginx.conf
      subPath: nginx.conf
    
    # Sidecar to detect config changes and reload
    command: ["/bin/sh"]
    args:
    - -c
    - |
      # Start nginx in background
      nginx -g 'daemon off;' &
      NGINX_PID=$!
      
      # Watch for config changes
      while true; do
        inotifywait -e modify /etc/nginx/nginx.conf
        echo "Config changed, reloading nginx..."
        nginx -s reload
      done
  
  volumes:
  - name: config
    configMap:
      name: nginx-config

Note: Environment variables from ConfigMaps/Secrets are NOT automatically updated. Pods must be restarted to pick up changes.

Immutable ConfigMaps and Secrets

Prevent accidental modifications:

apiVersion: v1
kind: ConfigMap
metadata:
  name: immutable-config
immutable: true
data:
  APP_ENV: "production"
  LOG_LEVEL: "info"

Once set to immutable, the ConfigMap cannot be modified and must be deleted/recreated to change values.

Secret Encryption

By default, Secrets are stored in etcd as base64-encoded (NOT encrypted). Enable encryption at rest for production.

Encryption Configuration

# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    # Use AES-CBC with PKCS#7 padding
    - aescbc:
        keys:
        - name: key1
          secret: <base64-encoded-32-byte-key>
    # Fallback to unencrypted for migration
    - identity: {}

Generate encryption key:

head -c 32 /dev/urandom | base64

Configure API server:

# /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
spec:
  containers:
  - command:
    - kube-apiserver
    - --encryption-provider-config=/etc/kubernetes/encryption-config.yaml
    volumeMounts:
    - name: encryption-config
      mountPath: /etc/kubernetes/encryption-config.yaml
      readOnly: true
  volumes:
  - name: encryption-config
    hostPath:
      path: /etc/kubernetes/encryption-config.yaml

Encrypt existing secrets:

# Read all secrets and write them back (encrypts with new key)
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Verify Encryption

# Check if secret is encrypted in etcd
ETCDCTL_API=3 etcdctl get /registry/secrets/default/app-secrets \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# If encrypted, you'll see binary data, not base64

External Secrets Management

Integrate with external secret managers for enterprise-grade security.

External Secrets Operator

Install External Secrets Operator:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets-system \
  --create-namespace

AWS Secrets Manager Integration:

# Create SecretStore
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

---
# Create ExternalSecret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  
  target:
    name: database-secret  # Kubernetes Secret to create
    creationPolicy: Owner
  
  data:
  # Simple key mapping
  - secretKey: username
    remoteRef:
      key: production/database
      property: username
  
  - secretKey: password
    remoteRef:
      key: production/database
      property: password
  
  # Get entire JSON secret
  dataFrom:
  - extract:
      key: production/app-config

HashiCorp Vault Integration:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "production-role"
          serviceAccountRef:
            name: external-secrets-sa

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: vault-secrets
  namespace: production
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  
  target:
    name: app-secrets
  
  data:
  - secretKey: api-key
    remoteRef:
      key: production/api-keys
      property: primary-key
  
  - secretKey: database-password
    remoteRef:
      key: production/database
      property: password

Google Secret Manager:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: gcpsm-secret-store
  namespace: production
spec:
  provider:
    gcpsm:
      projectID: "my-gcp-project"
      auth:
        workloadIdentity:
          clusterLocation: us-central1
          clusterName: production-cluster
          serviceAccountRef:
            name: external-secrets-sa

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: gcp-secrets
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcpsm-secret-store
    kind: SecretStore
  
  target:
    name: app-secrets
  
  data:
  - secretKey: api-key
    remoteRef:
      key: projects/12345/secrets/api-key/versions/latest

Sealed Secrets

Encrypt secrets for Git storage:

# Install Sealed Secrets controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml

# Install kubeseal CLI
brew install kubeseal

# Create regular secret
kubectl create secret generic mysecret \
  --from-literal=password=secretvalue \
  --dry-run=client -o yaml > secret.yaml

# Seal the secret
kubeseal -f secret.yaml -w sealed-secret.yaml

# sealed-secret.yaml can be safely committed to Git
# Apply sealed secret
kubectl apply -f sealed-secret.yaml

Sealed Secret Example:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecret
  namespace: production
spec:
  encryptedData:
    password: AgBvPx9jN8Q... # Encrypted value
  template:
    metadata:
      name: mysecret
      namespace: production
    type: Opaque

Configuration Best Practices

Organize by Environment

# Directory structure
configs/
├── base/
│   ├── configmap.yaml
│   └── kustomization.yaml
└── overlays/
    ├── development/
    │   ├── configmap-patch.yaml
    │   └── kustomization.yaml
    ├── staging/
    │   ├── configmap-patch.yaml
    │   └── kustomization.yaml
    └── production/
        ├── configmap-patch.yaml
        ├── secrets.yaml
        └── kustomization.yaml

Base ConfigMap:

# configs/base/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "info"
  CACHE_ENABLED: "true"
  MAX_CONNECTIONS: "100"

Production Overlay:

# configs/overlays/production/configmap-patch.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "warn"
  MAX_CONNECTIONS: "500"
  RATE_LIMIT: "1000"

Use Descriptive Names

# Good: Descriptive and scoped
apiVersion: v1
kind: ConfigMap
metadata:
  name: web-app-nginx-config
  namespace: production
  labels:
    app: web-app
    component: nginx
    environment: production

---
# Bad: Generic names
apiVersion: v1
kind: ConfigMap
metadata:
  name: config

Version Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-v2  # Version in name
  labels:
    app: web-app
    config-version: "2.0"
data:
  VERSION: "2.0"
  # ... configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  template:
    spec:
      containers:
      - name: app
        envFrom:
        - configMapRef:
            name: app-config-v2  # Reference versioned config

Validate Configuration

# Use init container to validate configuration
apiVersion: v1
kind: Pod
metadata:
  name: validated-config-pod
spec:
  initContainers:
  - name: config-validator
    image: myapp:1.0
    command: ['sh', '-c']
    args:
    - |
      # Validate required environment variables
      for var in DATABASE_HOST DATABASE_PORT API_KEY; do
        if [ -z "${!var}" ]; then
          echo "Error: $var not set"
          exit 1
        fi
      done
      
      # Validate configuration file syntax
      /app/bin/validate-config /etc/config/app.yaml
    envFrom:
    - configMapRef:
        name: app-config
    - secretRef:
        name: app-secrets
    volumeMounts:
    - name: config
      mountPath: /etc/config
  
  containers:
  - name: app
    image: myapp:1.0
    # ... configuration
  
  volumes:
  - name: config
    configMap:
      name: app-configs

Troubleshooting Configuration Issues

Debugging Missing Configuration

# Check if ConfigMap exists
kubectl get configmap app-config

# View ConfigMap contents
kubectl get configmap app-config -o yaml

# Check pod environment variables
kubectl exec my-pod -- env | sort

# Check mounted configuration files
kubectl exec my-pod -- ls -la /etc/config
kubectl exec my-pod -- cat /etc/config/app.yaml

# Describe pod to see ConfigMap/Secret references
kubectl describe pod my-pod

Configuration Not Updating

# Check if ConfigMap was updated
kubectl get configmap app-config -o yaml

# For volume mounts, check sync delay (up to kubelet sync period + ttl)
kubectl exec my-pod -- watch -n 1 cat /etc/config/app.yaml

# For environment variables, must restart pod
kubectl rollout restart deployment/web-app

# Force immediate update
kubectl delete pod -l app=web-app

Secret Decoding Issues

# Decode secret value
kubectl get secret app-secrets -o jsonpath='{.data.api-key}' | base64 --decode

# Check for whitespace or encoding issues
kubectl get secret app-secrets -o jsonpath='{.data.api-key}' | base64 --decode | xxd

# Verify secret is being mounted correctly
kubectl exec my-pod -- cat /etc/secrets/api-key | xxd

Permission Issues

# Check volume mount permissions
kubectl exec my-pod -- ls -la /etc/secrets

# Check defaultMode
kubectl get pod my-pod -o jsonpath='{.spec.volumes[?(@.name=="secret-volume")].secret.defaultMode}'

# Fix permissions with initContainer
```yaml
initContainers:
- name: fix-permissions
  image: busybox
  command: ['sh', '-c', 'chmod 600 /etc/secrets/*']
  volumeMounts:
  - name: secrets
    mountPath: /etc/secrets

What I Learned

Mastering configuration management in Kubernetes transformed application deployment and operations:

Separate Configuration from Code: This principle seems obvious but took discipline to implement consistently. Every configuration value—environment-specific settings, feature flags, credentials—should be external to the application image.

Use the Right Tool: ConfigMaps for non-sensitive configuration, Secrets for credentials, and external secret managers for enterprise requirements. Don't store secrets in ConfigMaps or configuration in Secrets.

Environment Variables vs. Files: Both have their place. Environment variables are simple and work everywhere. Files are better for complex configuration, allow hot-reloading, and support tools that expect configuration files.

Encryption Matters: Base64 encoding is not encryption. Enable etcd encryption at rest and use external secret managers for sensitive production data.

Immutability Prevents Accidents: Making ConfigMaps and Secrets immutable prevents accidental modifications in production. When updates are needed, create new versions with different names.

Plan for Configuration Updates: Understand that environment variables require pod restarts while volume mounts auto-update (with delay). Design your update strategy accordingly.

Version Everything: Version your ConfigMaps and Secrets just like application code. This enables rollbacks and makes it clear which configuration belongs to which application version.

Proper configuration management is foundational to running applications successfully in Kubernetes. With ConfigMaps, Secrets, and external secret managers, you can build secure, flexible, environment-agnostic applications. In the next articles, we'll explore persistent storage and how to manage stateful applications effectively.