Pods and Workloads

Introduction

When I first started deploying applications to Kubernetes, I made the mistake of focusing only on pods. I would create individual pod definitions, deploy them, and wonder why Kubernetes wasn't living up to its reputation for resilience and self-healing. A pod would die, and it wouldn't come back. Scale up? I'd have to create more pod definitions manually. Rolling updates? Forget about it.

The breakthrough came when I understood that pods are ephemeral, disposable units—they're not meant to be managed individually in production. Instead, Kubernetes provides higher-level workload resources like Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs that manage pods for you. These controllers implement the patterns you need for real applications: self-healing, scaling, rolling updates, and specialized workload types.

In this article, I'll share what I've learned about pods and the workload resources that manage them, drawing from experience deploying stateless web services, stateful databases, batch processing jobs, and background tasks across development and production environments.

Table of Contents

1. Understanding Pods

2. Pod Lifecycle and States

3. Deployments and ReplicaSets

4. StatefulSets for Stateful Applications

5. DaemonSets for Node-Level Services

6. Jobs for Batch Processing

7. CronJobs for Scheduled Tasks

8. Health Checks and Probes

9. Resource Requests and Limits

10. Pod Disruption Budgets

11. What I Learned

Understanding Pods

A pod is the smallest deployable unit in Kubernetes. It represents a single instance of a running process in your cluster and can contain one or more containers that share network and storage resources.

Pod Anatomy

Simple Pod Definition

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    app: nginx
    environment: development
  annotations:
    description: "Simple nginx web server"
spec:
  containers:
  - name: nginx
    image: nginx:1.21
    ports:
    - containerPort: 80
      name: http
      protocol: TCP
    env:
    - name: ENVIRONMENT
      value: "development"
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

# Create the pod
kubectl apply -f nginx-pod.yaml

# Get pod information
kubectl get pods
kubectl get pod nginx-pod -o wide
kubectl get pod nginx-pod -o yaml

# Describe pod (shows events and details)
kubectl describe pod nginx-pod

# View logs
kubectl logs nginx-pod

# Execute command in pod
kubectl exec nginx-pod -- nginx -v
kubectl exec -it nginx-pod -- /bin/bash

# Delete pod
kubectl delete pod nginx-pod

Multi-Container Pods

Containers in a pod share the same network namespace (IP address and port space) and can communicate via localhost.

apiVersion: v1
kind: Pod
metadata:
  name: multi-container-pod
spec:
  containers:
  # Main application container
  - name: web-app
    image: myapp:1.0
    ports:
    - containerPort: 8080
    volumeMounts:
    - name: shared-data
      mountPath: /app/data
  
  # Sidecar container - log processor
  - name: log-processor
    image: fluentd:latest
    volumeMounts:
    - name: shared-data
      mountPath: /var/log
    env:
    - name: FLUENT_ELASTICSEARCH_HOST
      value: "elasticsearch.logging.svc"
  
  # Sidecar container - metrics exporter
  - name: metrics-exporter
    image: prometheus-exporter:latest
    ports:
    - containerPort: 9090
    env:
    - name: APP_PORT
      value: "8080"
  
  volumes:
  - name: shared-data
    emptyDir: {}

Init Containers

Init containers run before app containers and must complete successfully before the app containers start. They're useful for setup tasks, waiting for services, or populating volumes.

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-init
spec:
  # Init containers run sequentially before main containers
  initContainers:
  # Wait for a service to be available
  - name: wait-for-database
    image: busybox:1.28
    command: ['sh', '-c']
    args:
    - |
      until nslookup postgres.database.svc.cluster.local; do
        echo "Waiting for database service..."
        sleep 2
      done
      echo "Database service is available"
  
  # Download configuration
  - name: fetch-config
    image: curlimages/curl:latest
    command: ['sh', '-c']
    args:
    - |
      curl -o /config/app-config.json \
        http://config-service/config/myapp
    volumeMounts:
    - name: config-volume
      mountPath: /config
  
  # Set permissions
  - name: fix-permissions
    image: busybox:1.28
    command: ['sh', '-c', 'chmod -R 755 /data']
    volumeMounts:
    - name: data-volume
      mountPath: /data
  
  # Main application containers
  containers:
  - name: application
    image: myapp:1.0
    volumeMounts:
    - name: config-volume
      mountPath: /etc/config
      readOnly: true
    - name: data-volume
      mountPath: /var/data
  
  volumes:
  - name: config-volume
    emptyDir: {}
  - name: data-volume
    emptyDir: {}

Pod Networking

Pods get their own IP address and all containers in a pod share that IP:

apiVersion: v1
kind: Pod
metadata:
  name: network-demo
spec:
  containers:
  # Container 1 - web server
  - name: web
    image: nginx:1.21
    ports:
    - containerPort: 80
  
  # Container 2 - can access web via localhost
  - name: client
    image: curlimages/curl:latest
    command: ['sh', '-c']
    args:
    - |
      while true; do
        curl localhost:80
        sleep 10
      done

# Get pod IP
kubectl get pod network-demo -o wide

# Test connectivity from another pod
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl <pod-ip>

Pod Storage

apiVersion: v1
kind: Pod
metadata:
  name: storage-demo
spec:
  containers:
  - name: app
    image: nginx:1.21
    volumeMounts:
    # EmptyDir - temporary storage, deleted with pod
    - name: cache
      mountPath: /cache
    
    # HostPath - node's filesystem
    - name: host-data
      mountPath: /host-data
    
    # ConfigMap - configuration data
    - name: config
      mountPath: /etc/config
      readOnly: true
    
    # Secret - sensitive data
    - name: secrets
      mountPath: /etc/secrets
      readOnly: true
    
    # PersistentVolumeClaim - persistent storage
    - name: persistent-data
      mountPath: /data
  
  volumes:
  - name: cache
    emptyDir:
      sizeLimit: 1Gi
  
  - name: host-data
    hostPath:
      path: /var/data
      type: DirectoryOrCreate
  
  - name: config
    configMap:
      name: app-config
  
  - name: secrets
    secret:
      secretName: app-secrets
  
  - name: persistent-data
    persistentVolumeClaim:
      claimName: app-pvc

Pod Lifecycle and States

Understanding pod lifecycle helps debug issues and design robust applications.

Pod Phase Diagram

Container States

# View container states
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[*].state}'

Waiting:

state:
  waiting:
    reason: ImagePullBackOff
    message: "Back-off pulling image 'myapp:invalid-tag'"

Running:

state:
  running:
    startedAt: "2025-01-15T10:30:00Z"

Terminated:

state:
  terminated:
    exitCode: 0
    reason: Completed
    startedAt: "2025-01-15T10:30:00Z"
    finishedAt: "2025-01-15T10:35:00Z"

Pod Conditions

conditions:
- type: Initialized
  status: "True"
  lastProbeTime: null
  lastTransitionTime: "2025-01-15T10:30:00Z"

- type: Ready
  status: "True"
  lastProbeTime: null
  lastTransitionTime: "2025-01-15T10:30:15Z"

- type: ContainersReady
  status: "True"
  lastProbeTime: null
  lastTransitionTime: "2025-01-15T10:30:15Z"

- type: PodScheduled
  status: "True"
  lastProbeTime: null
  lastTransitionTime: "2025-01-15T10:29:55Z"

Deployments and ReplicaSets

Deployments are the most common way to run stateless applications. They manage ReplicaSets, which in turn manage pods.

Deployment Architecture

Complete Deployment Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  namespace: production
  labels:
    app: web-app
    version: v1
  annotations:
    kubernetes.io/change-cause: "Initial deployment of web-app v1.0"
spec:
  # Number of desired pods
  replicas: 3
  
  # Deployment strategy
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # Max pods above desired during update
      maxUnavailable: 0  # Max pods unavailable during update
  
  # How long to wait before marking deployment as failed
  progressDeadlineSeconds: 600
  
  # Minimum time pod should be ready without crashing
  minReadySeconds: 10
  
  # Number of old ReplicaSets to retain
  revisionHistoryLimit: 10
  
  # Label selector for pods
  selector:
    matchLabels:
      app: web-app
  
  # Pod template
  template:
    metadata:
      labels:
        app: web-app
        version: v1
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      # Service account for pod
      serviceAccountName: web-app
      
      # Security context for pod
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 2000
      
      # Init containers
      initContainers:
      - name: wait-for-db
        image: busybox:1.28
        command: ['sh', '-c']
        args:
        - |
          until nslookup postgres.database.svc.cluster.local; do
            echo "Waiting for database..."
            sleep 2
          done
      
      # Application containers
      containers:
      - name: web-app
        image: myapp:1.0
        imagePullPolicy: IfNotPresent
        
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        - name: metrics
          containerPort: 9090
          protocol: TCP
        
        # Environment variables
        env:
        - name: ENVIRONMENT
          value: "production"
        - name: LOG_LEVEL
          value: "info"
        - name: DATABASE_HOST
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: database.host
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: database-password
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        
        # Resource requirements
        resources:
          requests:
            memory: "256Mi"
            cpu: "500m"
          limits:
            memory: "512Mi"
            cpu: "1000m"
        
        # Liveness probe - restart if unhealthy
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 3
        
        # Readiness probe - remove from service if not ready
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 3
          successThreshold: 1
          failureThreshold: 3
        
        # Startup probe - allow slow-starting apps
        startupProbe:
          httpGet:
            path: /startup
            port: 8080
          initialDelaySeconds: 0
          periodSeconds: 10
          timeoutSeconds: 3
          successThreshold: 1
          failureThreshold: 30  # 300 seconds max startup time
        
        # Volume mounts
        volumeMounts:
        - name: config
          mountPath: /etc/config
          readOnly: true
        - name: cache
          mountPath: /app/cache
        
        # Security context for container
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
            - ALL
      
      # Volumes
      volumes:
      - name: config
        configMap:
          name: app-config
      - name: cache
        emptyDir:
          sizeLimit: 1Gi
      
      # Node selection
      nodeSelector:
        disktype: ssd
        environment: production
      
      # Pod anti-affinity - spread across nodes
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - web-app
              topologyKey: kubernetes.io/hostname
      
      # Tolerations for node taints
      tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "web-app"
        effect: "NoSchedule"

Managing Deployments

# Create deployment
kubectl apply -f deployment.yaml

# Get deployments
kubectl get deployments
kubectl get deploy web-app -o wide

# Describe deployment
kubectl describe deployment web-app

# View rollout status
kubectl rollout status deployment/web-app

# View rollout history
kubectl rollout history deployment/web-app

# View specific revision
kubectl rollout history deployment/web-app --revision=2

# Scale deployment
kubectl scale deployment web-app --replicas=5

# Autoscale deployment
kubectl autoscale deployment web-app --min=3 --max=10 --cpu-percent=80

# Update image
kubectl set image deployment/web-app web-app=myapp:2.0

# Edit deployment
kubectl edit deployment web-app

# Pause rollout
kubectl rollout pause deployment/web-app

# Resume rollout
kubectl rollout resume deployment/web-app

# Rollback to previous version
kubectl rollout undo deployment/web-app

# Rollback to specific revision
kubectl rollout undo deployment/web-app --to-revision=2

# Delete deployment
kubectl delete deployment web-app

Deployment Strategies

Rolling Update (default):

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1        # Can have 4 pods during update (3 + 1)
    maxUnavailable: 0  # All 3 pods must stay running

Recreate:

strategy:
  type: Recreate  # Terminates all pods before creating new ones

Blue-Green Deployment Pattern

# Blue deployment (current)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
      version: blue
  template:
    metadata:
      labels:
        app: web-app
        version: blue
    spec:
      containers:
      - name: web-app
        image: myapp:1.0

---
# Green deployment (new)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
      version: green
  template:
    metadata:
      labels:
        app: web-app
        version: green
    spec:
      containers:
      - name: web-app
        image: myapp:2.0

---
# Service - switch between blue and green
apiVersion: v1
kind: Service
metadata:
  name: web-app
spec:
  selector:
    app: web-app
    version: blue  # Change to 'green' to switch traffic
  ports:
  - port: 80
    targetPort: 8080

Canary Deployment Pattern

# Stable deployment (90% of traffic)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-stable
spec:
  replicas: 9
  selector:
    matchLabels:
      app: web-app
      track: stable
  template:
    metadata:
      labels:
        app: web-app
        track: stable
    spec:
      containers:
      - name: web-app
        image: myapp:1.0

---
# Canary deployment (10% of traffic)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-app
      track: canary
  template:
    metadata:
      labels:
        app: web-app
        track: canary
    spec:
      containers:
      - name: web-app
        image: myapp:2.0

---
# Service selects both stable and canary
apiVersion: v1
kind: Service
metadata:
  name: web-app
spec:
  selector:
    app: web-app  # Matches both stable and canary
  ports:
  - port: 80
    targetPort: 8080

StatefulSets for Stateful Applications

StatefulSets manage stateful applications that require stable network identities, persistent storage, and ordered deployment/scaling.

StatefulSet vs Deployment

StatefulSet Example

apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: postgres
spec:
  ports:
  - port: 5432
    name: postgres
  clusterIP: None  # Headless service for StatefulSet
  selector:
    app: postgres

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres  # Headless service name
  replicas: 3
  
  # Pod management policy
  podManagementPolicy: OrderedReady  # or Parallel
  
  # Update strategy
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0  # Update pods >= this ordinal
  
  selector:
    matchLabels:
      app: postgres
  
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:14
        ports:
        - containerPort: 5432
          name: postgres
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
        
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - pg_isready -U postgres
          initialDelaySeconds: 30
          periodSeconds: 10
        
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - pg_isready -U postgres
          initialDelaySeconds: 10
          periodSeconds: 5
        
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
  
  # Volume claim templates
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "fast-ssd"
      resources:
        requests:
          storage: 10Gi

StatefulSet Features

Stable Network Identity:

# Pods get predictable names
postgres-0.postgres.default.svc.cluster.local
postgres-1.postgres.default.svc.cluster.local
postgres-2.postgres.default.svc.cluster.local

# Test DNS resolution
kubectl run -it --rm debug --image=busybox --restart=Never -- \
  nslookup postgres-0.postgres.default.svc.cluster.local

Ordered Deployment and Scaling:

# Scale up: Creates postgres-3, then postgres-4
kubectl scale statefulset postgres --replicas=5

# Scale down: Deletes postgres-4, then postgres-3
kubectl scale statefulset postgres --replicas=3

Persistent Storage:

# Each pod gets its own PVC
kubectl get pvc
# data-postgres-0
# data-postgres-1
# data-postgres-2

# PVCs are not deleted when StatefulSet is deleted
# Must be manually deleted if needed
kubectl delete pvc data-postgres-0

Managing StatefulSets

# Create StatefulSet
kubectl apply -f statefulset.yaml

# Get StatefulSets
kubectl get statefulsets
kubectl get sts postgres

# Describe StatefulSet
kubectl describe statefulset postgres

# Get pods
kubectl get pods -l app=postgres

# Update StatefulSet
kubectl apply -f statefulset.yaml

# Partition rolling update (update only pods >= partition)
kubectl patch statefulset postgres -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'

# Delete StatefulSet (keeps PVCs)
kubectl delete statefulset postgres

# Delete StatefulSet and pods (keeps PVCs)
kubectl delete statefulset postgres --cascade=orphan

DaemonSets for Node-Level Services

DaemonSets ensure a copy of a pod runs on all (or some) nodes. They're perfect for node monitoring, log collection, and network plugins.

DaemonSet Example

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
  labels:
    app: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  
  # Update strategy
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      # Run on host network
      hostNetwork: true
      hostPID: true
      
      # Tolerate all taints to run on all nodes
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists
      
      containers:
      - name: node-exporter
        image: prom/node-exporter:latest
        args:
        - --path.sysfs=/host/sys
        - --path.rootfs=/rootfs
        - --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)
        ports:
        - containerPort: 9100
          protocol: TCP
          name: metrics
        resources:
          requests:
            memory: "100Mi"
            cpu: "100m"
          limits:
            memory: "200Mi"
            cpu: "200m"
        volumeMounts:
        - name: sys
          mountPath: /host/sys
          mountPropagation: HostToContainer
          readOnly: true
        - name: root
          mountPath: /rootfs
          mountPropagation: HostToContainer
          readOnly: true
      
      volumes:
      - name: sys
        hostPath:
          path: /sys
      - name: root
        hostPath:
          path: /

Node Selection for DaemonSets

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: gpu-monitor
spec:
  selector:
    matchLabels:
      app: gpu-monitor
  template:
    metadata:
      labels:
        app: gpu-monitor
    spec:
      # Only run on nodes with GPUs
      nodeSelector:
        accelerator: nvidia-gpu
      
      # Or use affinity
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: gpu-type
                operator: In
                values:
                - nvidia-t4
                - nvidia-v100
      
      containers:
      - name: gpu-monitor
        image: nvidia/dcgm-exporter:latest

Jobs for Batch Processing

Jobs create one or more pods and ensure they successfully complete. Perfect for batch processing, data migrations, and one-time tasks.

Simple Job

apiVersion: batch/v1
kind: Job
metadata:
  name: data-migration
spec:
  # Number of successful completions needed
  completions: 1
  
  # Number of pods to run in parallel
  parallelism: 1
  
  # Retry failed pods
  backoffLimit: 3
  
  # Maximum time job can run
  activeDeadlineSeconds: 600
  
  # Clean up completed pods automatically
  ttlSecondsAfterFinished: 100
  
  template:
    metadata:
      labels:
        app: data-migration
    spec:
      restartPolicy: Never  # or OnFailure
      
      containers:
      - name: migration
        image: migration-tool:1.0
        command: ["./migrate.sh"]
        env:
        - name: SOURCE_DB
          value: "old-database:5432"
        - name: TARGET_DB
          value: "new-database:5432"
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"

Parallel Jobs

Fixed Completion Count:

apiVersion: batch/v1
kind: Job
metadata:
  name: parallel-processing
spec:
  completions: 10     # Need 10 successful completions
  parallelism: 3      # Run 3 pods at a time
  backoffLimit: 5
  
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: worker
        image: processing-worker:1.0
        command: ["./process.sh"]
        env:
        - name: JOB_COMPLETION_INDEX
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']

Work Queue Pattern:

apiVersion: batch/v1
kind: Job
metadata:
  name: work-queue
spec:
  # No completions specified - workers exit when queue is empty
  parallelism: 5
  
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: worker
        image: queue-worker:1.0
        env:
        - name: QUEUE_URL
          value: "redis://redis:6379"
        command:
        - /bin/sh
        - -c
        - |
          while true; do
            item=$(redis-cli lpop work-queue)
            if [ -z "$item" ]; then
              echo "Queue empty, exiting"
              exit 0
            fi
            process_item "$item"
          done

Managing Jobs

# Create job
kubectl apply -f job.yaml

# Get jobs
kubectl get jobs

# Describe job
kubectl describe job data-migration

# View job pods
kubectl get pods --selector=job-name=data-migration

# View job logs
kubectl logs job/data-migration

# Delete job (and its pods)
kubectl delete job data-migration

# Delete job but keep pods
kubectl delete job data-migration --cascade=orphan

CronJobs for Scheduled Tasks

CronJobs create Jobs on a schedule, similar to cron on Unix systems.

CronJob Example

apiVersion: batch/v1
kind: CronJob
metadata:
  name: database-backup
spec:
  # Schedule in cron format (minute hour day month weekday)
  schedule: "0 2 * * *"  # Run at 2:00 AM every day
  
  # Timezone (Kubernetes 1.25+)
  timeZone: "America/New_York"
  
  # How many successful jobs to keep
  successfulJobsHistoryLimit: 3
  
  # How many failed jobs to keep
  failedJobsHistoryLimit: 1
  
  # Concurrency policy
  concurrencyPolicy: Forbid  # Forbid, Allow, or Replace
  
  # Deadline to start job (seconds after scheduled time)
  startingDeadlineSeconds: 300
  
  # Suspend cron job
  suspend: false
  
  jobTemplate:
    spec:
      ttlSecondsAfterFinished: 86400  # Clean up after 24 hours
      
      template:
        spec:
          restartPolicy: OnFailure
          
          containers:
          - name: backup
            image: postgres:14
            command:
            - /bin/sh
            - -c
            - |
              pg_dump -h $DB_HOST -U $DB_USER $DB_NAME | \
              gzip > /backup/backup-$(date +%Y%m%d-%H%M%S).sql.gz
              
              # Upload to S3
              aws s3 cp /backup/*.sql.gz s3://backups/database/
            
            env:
            - name: DB_HOST
              value: "postgres.database.svc"
            - name: DB_USER
              value: "postgres"
            - name: DB_NAME
              value: "production"
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
            
            volumeMounts:
            - name: backup
              mountPath: /backup
          
          volumes:
          - name: backup
            emptyDir: {}

Common Cron Schedules

# Every minute
schedule: "* * * * *"

# Every hour
schedule: "0 * * * *"

# Every day at 2:30 AM
schedule: "30 2 * * *"

# Every Monday at 9:00 AM
schedule: "0 9 * * 1"

# First day of month at midnight
schedule: "0 0 1 * *"

# Every 15 minutes
schedule: "*/15 * * * *"

# Weekdays at 8:00 AM
schedule: "0 8 * * 1-5"

# Every 6 hours
schedule: "0 */6 * * *"

Managing CronJobs

# Create CronJob
kubectl apply -f cronjob.yaml

# Get CronJobs
kubectl get cronjobs

# Describe CronJob
kubectl describe cronjob database-backup

# View jobs created by CronJob
kubectl get jobs --selector=job-name=database-backup-*

# Manually trigger CronJob
kubectl create job --from=cronjob/database-backup manual-backup-$(date +%s)

# Suspend CronJob
kubectl patch cronjob database-backup -p '{"spec":{"suspend":true}}'

# Resume CronJob
kubectl patch cronjob database-backup -p '{"spec":{"suspend":false}}'

# Delete CronJob
kubectl delete cronjob database-backup

Health Checks and Probes

Kubernetes uses probes to determine container health and readiness.

Probe Types

HTTP Probes

apiVersion: v1
kind: Pod
metadata:
  name: http-probes
spec:
  containers:
  - name: app
    image: myapp:1.0
    
    startupProbe:
      httpGet:
        path: /startup
        port: 8080
        httpHeaders:
        - name: Custom-Header
          value: Health-Check
      initialDelaySeconds: 0
      periodSeconds: 10
      timeoutSeconds: 1
      successThreshold: 1
      failureThreshold: 30  # 300 seconds max startup
    
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 30
      periodSeconds: 10
      timeoutSeconds: 5
      successThreshold: 1
      failureThreshold: 3
    
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 3
      successThreshold: 1
      failureThreshold: 3

TCP Probes

apiVersion: v1
kind: Pod
metadata:
  name: tcp-probes
spec:
  containers:
  - name: database
    image: postgres:14
    
    livenessProbe:
      tcpSocket:
        port: 5432
      initialDelaySeconds: 30
      periodSeconds: 10
    
    readinessProbe:
      tcpSocket:
        port: 5432
      initialDelaySeconds: 10
      periodSeconds: 5

Exec Probes

apiVersion: v1
kind: Pod
metadata:
  name: exec-probes
spec:
  containers:
  - name: database
    image: postgres:14
    
    livenessProbe:
      exec:
        command:
        - sh
        - -c
        - pg_isready -U $POSTGRES_USER
      initialDelaySeconds: 30
      periodSeconds: 10
    
    readinessProbe:
      exec:
        command:
        - sh
        - -c
        - |
          pg_isready -U $POSTGRES_USER && \
          psql -U $POSTGRES_USER -c "SELECT 1" > /dev/null
      initialDelaySeconds: 10
      periodSeconds: 5

gRPC Probes (Kubernetes 1.24+)

apiVersion: v1
kind: Pod
metadata:
  name: grpc-probes
spec:
  containers:
  - name: grpc-service
    image: grpc-service:1.0
    
    livenessProbe:
      grpc:
        port: 9090
        service: myservice.health.v1.Health
      initialDelaySeconds: 10
      periodSeconds: 10
    
    readinessProbe:
      grpc:
        port: 9090
      initialDelaySeconds: 5
      periodSeconds: 5

Resource Requests and Limits

Resource management is crucial for stable clusters.

Resource Types

apiVersion: v1
kind: Pod
metadata:
  name: resource-demo
spec:
  containers:
  - name: app
    image: myapp:1.0
    resources:
      # Requests: Minimum guaranteed resources
      requests:
        memory: "256Mi"   # 256 mebibytes
        cpu: "500m"       # 500 millicores (0.5 CPU)
        ephemeral-storage: "1Gi"
      
      # Limits: Maximum allowed resources
      limits:
        memory: "512Mi"
        cpu: "1000m"      # 1 CPU
        ephemeral-storage: "2Gi"

Resource Units

CPU:

• 1 = 1 vCPU/Core

• 1000m = 1000 millicores = 1 CPU

• 0.5 = 500m = 0.5 CPU

Memory:

• 128Mi = 128 mebibytes (1Mi = 1024 KiB)

• 128M = 128 megabytes (1M = 1000 KB)

• 1Gi = 1 gibibyte = 1024 Mi

• 1G = 1 gigabyte = 1000 M

Quality of Service Classes

Guaranteed QoS:

resources:
  requests:
    memory: "256Mi"
    cpu: "500m"
  limits:
    memory: "256Mi"  # Same as request
    cpu: "500m"      # Same as request

Burstable QoS:

resources:
  requests:
    memory: "256Mi"
    cpu: "500m"
  limits:
    memory: "512Mi"  # Higher than request
    cpu: "1000m"     # Higher than request

BestEffort QoS:

# No requests or limits specified
resources: {}

LimitRange for Namespace Defaults

apiVersion: v1
kind: LimitRange
metadata:
  name: resource-limits
  namespace: development
spec:
  limits:
  # Container limits
  - type: Container
    default:  # Default limits
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:  # Default requests
      cpu: "250m"
      memory: "256Mi"
    max:  # Maximum allowed
      cpu: "2000m"
      memory: "2Gi"
    min:  # Minimum required
      cpu: "100m"
      memory: "64Mi"
    maxLimitRequestRatio:  # Max ratio between limit and request
      cpu: "4"
      memory: "2"
  
  # Pod limits
  - type: Pod
    max:
      cpu: "4000m"
      memory: "4Gi"
  
  # PersistentVolumeClaim limits
  - type: PersistentVolumeClaim
    max:
      storage: "100Gi"
    min:
      storage: "1Gi"

ResourceQuota for Namespace Limits

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: development
spec:
  hard:
    # Compute resources
    requests.cpu: "10"
    requests.memory: "20Gi"
    limits.cpu: "20"
    limits.memory: "40Gi"
    
    # Storage
    requests.storage: "100Gi"
    persistentvolumeclaims: "10"
    
    # Object counts
    pods: "50"
    services: "20"
    secrets: "30"
    configmaps: "30"
    replicationcontrollers: "20"
    
    # Service types
    services.loadbalancers: "2"
    services.nodeports: "5"

Pod Disruption Budgets

Pod Disruption Budgets ensure minimum availability during voluntary disruptions (node maintenance, updates, etc.).

PodDisruptionBudget Example

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-app-pdb
spec:
  # Minimum available pods
  minAvailable: 2
  
  # Or maximum unavailable
  # maxUnavailable: 1
  
  selector:
    matchLabels:
      app: web-app

PDB with Percentage

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-app-pdb
spec:
  minAvailable: 75%  # At least 75% must be available
  selector:
    matchLabels:
      app: web-app

Checking PDB Status

# Get PDBs
kubectl get pdb

# Describe PDB
kubectl describe pdb web-app-pdb

# Check allowed disruptions
kubectl get pdb web-app-pdb -o jsonpath='{.status.disruptionsAllowed}'

What I Learned

Understanding pods and workload resources transformed how I deploy and manage applications in Kubernetes:

Pods Are Ephemeral: I learned early that pods are disposable. Don't treat them like VMs or long-lived servers. Design applications to handle pod restarts gracefully, and use higher-level controllers to manage them.

Choose the Right Workload Type: Deployments for stateless apps, StatefulSets for databases, DaemonSets for node services, Jobs for batch work, and CronJobs for scheduled tasks. Each has specific behaviors optimized for its use case.

Health Checks Are Critical: Implementing proper liveness, readiness, and startup probes dramatically improved application reliability. The time invested in creating accurate health check endpoints pays off immediately.

Resource Requests Matter: Setting appropriate resource requests and limits prevents resource contention and ensures predictable scheduling. I learned to profile applications under load to determine realistic values.

StatefulSets Need Care: StatefulSets have different behaviors than Deployments—ordered scaling, persistent storage, stable network identities. Understanding these differences prevents confusion when things don't work as expected.

PDBs Prevent Outages: Pod Disruption Budgets saved us from accidental outages during node maintenance. Always define PDBs for critical services—they're a simple way to codify availability requirements.

Start Simple, Then Optimize: Begin with basic Deployments and add complexity (affinity rules, init containers, sidecars) only when needed. Premature optimization makes debugging harder.

These workload primitives are the foundation of running applications on Kubernetes. Master them, and you'll be able to deploy virtually any workload type reliably and efficiently. In the next articles, we'll explore how to expose these applications through Services and Ingress, configure them with ConfigMaps and Secrets, and add persistent storage.