Storage and Persistence

Introduction

When I first started working with Kubernetes, I naively thought that storage would work similarly to traditional VM-based deployments. My first production incident taught me otherwise—after a pod restart, all user-uploaded images disappeared because I hadn't configured persistent storage. That painful lesson sent me deep into understanding Kubernetes storage, volumes, and persistence mechanisms.

Over the years, I've designed storage solutions for databases, media processing pipelines, and data analytics platforms. Each use case taught me valuable lessons about volume types, storage classes, and the trade-offs between performance, durability, and cost. In this comprehensive guide, I'll share everything I've learned about managing persistent storage in Kubernetes, from basic concepts to production-ready patterns.

Table of Contents

1. Understanding Kubernetes Storage

2. Volume Basics

3. Persistent Volumes and Claims

4. Storage Classes

5. StatefulSets for Stateful Applications

6. Volume Types and Use Cases

7. Dynamic Provisioning

8. Backup and Disaster Recovery

9. Performance Optimization

10. Security and Access Control

11. Troubleshooting Storage Issues

12. Best Practices

13. What I Learned

Understanding Kubernetes Storage

Kubernetes storage is fundamentally different from traditional storage management. The ephemeral nature of containers and pods means that data written to a container's filesystem disappears when the container restarts. This works fine for stateless applications but poses challenges for databases, file servers, and any application that needs to persist data.

The Storage Challenge

The core challenge in Kubernetes is reconciling the dynamic, ephemeral nature of pods with the need for persistent, durable storage. Pods can be rescheduled to different nodes, scaled up or down, or replaced during updates. Your storage solution must handle these scenarios gracefully.

Storage Layers

Kubernetes storage operates at multiple layers:

1. Container Layer: Ephemeral storage within containers

2. Pod Layer: Volumes shared across containers in a pod

3. Persistent Layer: Durable storage that survives pod restarts

4. Cluster Layer: Storage provisioning and management

Understanding these layers is crucial for designing robust storage solutions.

Volume Basics

Volumes are the fundamental storage abstraction in Kubernetes. They provide a way for containers to access storage, whether ephemeral or persistent.

EmptyDir Volumes

The simplest volume type is emptyDir, created when a pod is assigned to a node and exists as long as the pod runs on that node. All containers in the pod can read and write to it.

apiVersion: v1
kind: Pod
metadata:
  name: shared-volume-example
spec:
  containers:
  - name: nginx
    image: nginx:1.21
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html
  - name: content-generator
    image: busybox:1.35
    command: ["/bin/sh"]
    args:
    - -c
    - |
      while true; do
        echo "Last updated: $(date)" > /data/index.html
        sleep 10
      done
    volumeMounts:
    - name: shared-data
      mountPath: /data
  volumes:
  - name: shared-data
    emptyDir: {}

This example shows two containers sharing an emptyDir volume. The content generator writes HTML files that nginx serves.

Testing the shared volume:

# Create the pod
kubectl apply -f shared-volume-pod.yaml

# Verify pod is running
kubectl get pods shared-volume-example

# Test the nginx server
kubectl port-forward shared-volume-example 8080:80

# In another terminal
curl http://localhost:8080
# Output: Last updated: Tue Dec 31 10:15:23 UTC 2025

EmptyDir with Memory Storage

For high-performance temporary storage, use memory-backed emptyDir:

apiVersion: v1
kind: Pod
metadata:
  name: cache-pod
spec:
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - name: cache-volume
      mountPath: /cache
  volumes:
  - name: cache-volume
    emptyDir:
      medium: Memory
      sizeLimit: 1Gi

This creates a tmpfs (RAM-backed) filesystem, perfect for caching scenarios where speed is critical.

HostPath Volumes

hostPath volumes mount a file or directory from the host node's filesystem into a pod. Use these carefully—they create dependencies on specific nodes.

apiVersion: v1
kind: Pod
metadata:
  name: hostpath-example
spec:
  containers:
  - name: log-processor
    image: log-processor:1.0
    volumeMounts:
    - name: host-logs
      mountPath: /logs
      readOnly: true
  volumes:
  - name: host-logs
    hostPath:
      path: /var/log
      type: Directory

When to use hostPath:

• Accessing node-level logs or metrics

• Running DaemonSets that need host access

• Development and testing (never production for application data)

ConfigMap and Secret Volumes

ConfigMaps and Secrets can be mounted as volumes, making configuration and sensitive data available as files:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  app.properties: |
    database.host=postgres.default.svc.cluster.local
    database.port=5432
    cache.ttl=3600
---
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
stringData:
  database.password: super-secret-password
  api.key: another-secret-key
---
apiVersion: v1
kind: Pod
metadata:
  name: config-example
spec:
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - name: config
      mountPath: /etc/config
      readOnly: true
    - name: secrets
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: config
    configMap:
      name: app-config
  - name: secrets
    secret:
      secretName: app-secrets
      defaultMode: 0400  # Read-only for owner

Accessing mounted configuration:

# Exec into the pod
kubectl exec -it config-example -- /bin/sh

# View configuration files
cat /etc/config/app.properties
cat /etc/secrets/database.password

Persistent Volumes and Claims

Persistent Volumes (PV) and Persistent Volume Claims (PVC) provide a way to provision and consume durable storage independent of pod lifecycle.

The PV/PVC Model

This separation of concerns allows administrators to manage storage infrastructure while developers request storage through claims.

Creating a Persistent Volume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-pv
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: fast-ssd
  hostPath:
    path: /mnt/data/postgres  # For testing only
    type: DirectoryOrCreate

Key PV specifications:

• capacity: Total storage size

• accessModes: How the volume can be mounted

  • ReadWriteOnce (RWO): Single node read-write

  • ReadOnlyMany (ROX): Multiple nodes read-only

  • ReadWriteMany (RWX): Multiple nodes read-write

• persistentVolumeReclaimPolicy: What happens when PVC is deleted

  • Retain: Manual reclamation required

  • Delete: Automatically delete storage

  • Recycle: Basic scrub and make available again (deprecated)

• storageClassName: Links to StorageClass for dynamic provisioning

Creating a Persistent Volume Claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: fast-ssd

Using PVC in a pod:

apiVersion: v1
kind: Pod
metadata:
  name: postgres
spec:
  containers:
  - name: postgres
    image: postgres:14
    env:
    - name: POSTGRES_PASSWORD
      value: mysecretpassword
    - name: PGDATA
      value: /var/lib/postgresql/data/pgdata
    ports:
    - containerPort: 5432
    volumeMounts:
    - name: postgres-storage
      mountPath: /var/lib/postgresql/data
  volumes:
  - name: postgres-storage
    persistentVolumeClaim:
      claimName: postgres-pvc

Managing PVCs:

# Create PV and PVC
kubectl apply -f postgres-pv.yaml
kubectl apply -f postgres-pvc.yaml

# Check PV status
kubectl get pv
# NAME          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM
# postgres-pv   10Gi       RWO            Retain           Bound    default/postgres-pvc

# Check PVC status
kubectl get pvc
# NAME           STATUS   VOLUME        CAPACITY   ACCESS MODES
# postgres-pvc   Bound    postgres-pv   10Gi       RWO

# View detailed PVC information
kubectl describe pvc postgres-pvc

Storage Classes

StorageClasses enable dynamic provisioning of volumes, eliminating the need to pre-create PVs manually.

Dynamic Provisioning Flow

Creating a Storage Class

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs  # AWS example
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
  kmsKeyId: arn:aws:kms:us-east-1:123456789012:key/abcd1234
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

StorageClass parameters vary by provisioner:

AWS EBS:

provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3  # or io1, io2, st1, sc1
  iops: "3000"
  throughput: "125"
  encrypted: "true"

Azure Disk:

provisioner: kubernetes.io/azure-disk
parameters:
  storageaccounttype: Premium_LRS  # or Standard_LRS, StandardSSD_LRS
  kind: Managed
  cachingmode: ReadOnly

GCP Persistent Disk:

provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd  # or pd-standard, pd-balanced
  replication-type: regional-pd

Using Dynamic Provisioning

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dynamic-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: fast-ssd  # References StorageClass
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
      - name: webapp
        image: webapp:1.0
        volumeMounts:
        - name: data
          mountPath: /data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: dynamic-pvc

Testing dynamic provisioning:

# Create PVC
kubectl apply -f dynamic-pvc.yaml

# Watch PVC get bound automatically
kubectl get pvc -w

# Check automatically created PV
kubectl get pv

# The PV was created automatically by the StorageClass provisioner

Volume Expansion

With allowVolumeExpansion: true, you can resize PVCs:

# Edit PVC to increase size
kubectl edit pvc dynamic-pvc
# Change: storage: 20Gi -> storage: 50Gi

# Check expansion status
kubectl describe pvc dynamic-pvc

# For some volume types, you may need to restart pods
kubectl rollout restart deployment webapp

StatefulSets for Stateful Applications

StatefulSets provide stable, persistent identities for pods, essential for stateful applications like databases.

StatefulSet Characteristics

• Stable, unique pod names (podname-0, podname-1, etc.)

• Ordered, graceful deployment and scaling

• Stable network identities via headless service

• Persistent storage that follows the pod

PostgreSQL StatefulSet Example

apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: postgres
spec:
  ports:
  - port: 5432
    name: postgres
  clusterIP: None  # Headless service
  selector:
    app: postgres
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:14
        ports:
        - containerPort: 5432
          name: postgres
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        volumeMounts:
        - name: postgres-storage
          mountPath: /var/lib/postgresql/data
        livenessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - pg_isready -U postgres
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - pg_isready -U postgres
          initialDelaySeconds: 5
          periodSeconds: 5
  volumeClaimTemplates:
  - metadata:
      name: postgres-storage
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 10Gi

Key StatefulSet features:

• volumeClaimTemplates: Automatically creates PVCs for each pod

• serviceName: Links to headless service for stable networking

• Ordered pod names: postgres-0, postgres-1, postgres-2

Managing StatefulSets:

# Create StatefulSet
kubectl apply -f postgres-statefulset.yaml

# Watch pods being created in order
kubectl get pods -l app=postgres -w

# Check PVCs created automatically
kubectl get pvc
# NAME                      STATUS   VOLUME
# postgres-storage-postgres-0   Bound    pvc-xxxxx
# postgres-storage-postgres-1   Bound    pvc-yyyyy
# postgres-storage-postgres-2   Bound    pvc-zzzzz

# Access specific pod
kubectl exec -it postgres-0 -- psql -U postgres

# Scale StatefulSet
kubectl scale statefulset postgres --replicas=5
# Pods will be created: postgres-3, postgres-4

# Scale down (removes from highest ordinal)
kubectl scale statefulset postgres --replicas=2
# Pods postgres-4, postgres-3, postgres-2 will be removed in order

Stable Network Identity

Pods in a StatefulSet get stable DNS names:

<pod-name>.<service-name>.<namespace>.svc.cluster.local

Example DNS names:

postgres-0.postgres.default.svc.cluster.local
postgres-1.postgres.default.svc.cluster.local
postgres-2.postgres.default.svc.cluster.local

Testing DNS resolution:

# From within cluster
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup postgres-0.postgres.default.svc.cluster.local

Volume Types and Use Cases

Local Volumes

Local volumes represent storage devices mounted on nodes. They offer better performance than network storage but lack mobility.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-pv
spec:
  capacity:
    storage: 100Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /mnt/disks/ssd1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node-1

Use cases for local volumes:

• High-performance databases

• Caching layers

• Build agents requiring fast I/O

NFS Volumes

NFS provides shared storage accessible from multiple pods simultaneously:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv
spec:
  capacity:
    storage: 50Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: nfs-server.example.com
    path: /exported/path
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 50Gi

Use cases for NFS:

• Shared configuration files

• Media assets accessed by multiple pods

• Collaborative workspaces

CSI (Container Storage Interface) Volumes

CSI is the modern standard for storage in Kubernetes, supporting many storage providers:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-rbd
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: my-ceph-cluster
  pool: kubernetes
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
  csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - discard

Popular CSI drivers:

• AWS EBS CSI

• Azure Disk CSI

• GCP PD CSI

• Ceph RBD CSI

• Longhorn

• Portworx

Dynamic Provisioning

Setting Default Storage Class

# List storage classes
kubectl get storageclasses

# Set default storage class
kubectl patch storageclass fast-ssd -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

# Verify default
kubectl get storageclass
# NAME                PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE
# fast-ssd (default)  kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer

Volume Binding Modes

Immediate binding:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: immediate-binding
provisioner: kubernetes.io/aws-ebs
volumeBindingMode: Immediate

PV is provisioned immediately when PVC is created.

WaitForFirstConsumer:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: wait-for-consumer
provisioner: kubernetes.io/aws-ebs
volumeBindingMode: WaitForFirstConsumer

PV provisioning is delayed until a pod using the PVC is scheduled. This ensures the volume is created in the same availability zone as the pod.

Volume Snapshots

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-snapclass
driver: ebs.csi.aws.com
deletionPolicy: Delete
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: postgres-snapshot
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: postgres-pvc

Creating and restoring snapshots:

# Create snapshot
kubectl apply -f volume-snapshot.yaml

# Check snapshot status
kubectl get volumesnapshot
# NAME                READYTOUSE   SOURCEPVC      RESTORESIZE
# postgres-snapshot   true         postgres-pvc   10Gi

# Restore from snapshot
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc-restored
spec:
  dataSource:
    name: postgres-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
EOF

Backup and Disaster Recovery

Velero for Backup

Velero is the industry-standard tool for backing up Kubernetes resources and persistent volumes:

# Install Velero CLI
brew install velero  # macOS

# Install Velero in cluster (AWS example)
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.8.0 \
  --bucket my-backup-bucket \
  --secret-file ./credentials-velero \
  --backup-location-config region=us-east-1 \
  --snapshot-location-config region=us-east-1

# Create backup
velero backup create postgres-backup \
  --include-namespaces default \
  --include-resources persistentvolumeclaims,persistentvolumes \
  --selector app=postgres

# Schedule regular backups
velero schedule create daily-backup \
  --schedule="0 2 * * *" \
  --include-namespaces default,production

# List backups
velero backup get

# Restore from backup
velero restore create --from-backup postgres-backup

# Check restore status
velero restore describe postgres-backup-20251231

Manual Backup Strategies

Database-specific backups (PostgreSQL):

apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-backup
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: postgres:14
            command:
            - /bin/sh
            - -c
            - |
              pg_dump -h postgres-0.postgres -U postgres -d mydb | \
              gzip > /backup/backup-$(date +%Y%m%d-%H%M%S).sql.gz
              
              # Keep only last 7 days
              find /backup -name "backup-*.sql.gz" -mtime +7 -delete
            env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
            volumeMounts:
            - name: backup-storage
              mountPath: /backup
          volumes:
          - name: backup-storage
            persistentVolumeClaim:
              claimName: backup-pvc
          restartPolicy: OnFailure

Performance Optimization

Choosing the Right Volume Type

Performance characteristics:

Volume Type	IOPS	Throughput	Latency	Use Case
Local SSD	Very High	Very High	Very Low	Databases, caching
Network SSD (Premium)	High	High	Low	General databases
Network HDD	Low	Medium	Medium	Logs, archives
Object Storage	Medium	Very High	Medium	Media, backups

I/O Performance Testing

# Run fio benchmark
kubectl run fio-test --image=nixery.dev/shell/fio --rm -it -- /bin/sh

# Inside container
fio --name=randwrite --ioengine=libaio --iodepth=16 --rw=randwrite \
    --bs=4k --direct=1 --size=1G --numjobs=4 --runtime=60 \
    --group_reporting --filename=/data/testfile

# Test sequential read
fio --name=seqread --ioengine=libaio --iodepth=16 --rw=read \
    --bs=1M --direct=1 --size=1G --runtime=60 \
    --group_reporting --filename=/data/testfile

Volume Performance Tuning

For AWS EBS:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: high-performance
provisioner: ebs.csi.aws.com
parameters:
  type: io2
  iopsPerGB: "50"  # Up to 64,000 IOPS
  throughput: "1000"  # MB/s
  encrypted: "true"
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

For local volumes with direct I/O:

apiVersion: v1
kind: Pod
metadata:
  name: high-perf-db
spec:
  containers:
  - name: database
    image: postgres:14
    volumeMounts:
    - name: data
      mountPath: /var/lib/postgresql/data
      mountPropagation: HostToContainer
    securityContext:
      capabilities:
        add:
        - SYS_ADMIN  # For O_DIRECT
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: local-ssd-pvc

Security and Access Control

Volume Security Context

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    fsGroup: 2000  # All volumes will be owned by group 2000
    fsGroupChangePolicy: "OnRootMismatch"
  containers:
  - name: app
    image: myapp:1.0
    securityContext:
      runAsUser: 1000
      runAsGroup: 2000
      readOnlyRootFilesystem: true  # Root FS is read-only
    volumeMounts:
    - name: data
      mountPath: /data
    - name: tmp
      mountPath: /tmp
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: app-pvc
  - name: tmp
    emptyDir: {}  # Writable temp storage

Encrypted Volumes

AWS EBS encryption:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: encrypted-storage
provisioner: ebs.csi.aws.com
parameters:
  encrypted: "true"
  kmsKeyId: arn:aws:kms:us-east-1:123456789012:key/abcd-1234

Application-level encryption:

apiVersion: v1
kind: Pod
metadata:
  name: encrypted-app
spec:
  initContainers:
  - name: setup-encryption
    image: alpine:3.18
    command:
    - /bin/sh
    - -c
    - |
      apk add --no-cache cryptsetup
      echo "$ENCRYPTION_KEY" | cryptsetup luksFormat /dev/xvdf -
      echo "$ENCRYPTION_KEY" | cryptsetup luksOpen /dev/xvdf encrypted-vol -
      mkfs.ext4 /dev/mapper/encrypted-vol
    env:
    - name: ENCRYPTION_KEY
      valueFrom:
        secretKeyRef:
          name: encryption-secret
          key: key
    securityContext:
      privileged: true
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - name: encrypted-data
      mountPath: /data
  volumes:
  - name: encrypted-data
    hostPath:
      path: /dev/mapper/encrypted-vol

Read-Only Volumes

apiVersion: v1
kind: Pod
metadata:
  name: readonly-config
spec:
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - name: config
      mountPath: /etc/config
      readOnly: true  # Prevents writes
    - name: secrets
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: config
    configMap:
      name: app-config
  - name: secrets
    secret:
      secretName: app-secrets

Troubleshooting Storage Issues

Common PVC Issues

PVC stuck in Pending:

# Check PVC status
kubectl describe pvc my-pvc

# Common causes:
# 1. No matching PV available
kubectl get pv
# Solution: Create PV or use dynamic provisioning

# 2. Storage class doesn't exist
kubectl get storageclass
# Solution: Create StorageClass or fix name

# 3. Insufficient resources
# Check events
kubectl get events --sort-by='.lastTimestamp'

Volume mount failures:

# Check pod events
kubectl describe pod my-pod

# Common issues:
# - PVC not bound: "persistentvolumeclaim 'my-pvc' not found"
# - Access mode mismatch: "volume already mounted on another node"
# - Permission issues: "permission denied"

# Check logs
kubectl logs my-pod

# Verify volume mounts
kubectl exec my-pod -- df -h
kubectl exec my-pod -- ls -la /mounted/path

Performance Issues

Diagnosing slow I/O:

# Check I/O wait on nodes
kubectl top nodes

# Run performance test inside pod
kubectl exec -it my-pod -- sh
dd if=/dev/zero of=/data/testfile bs=1M count=1000 oflag=direct
# Should show throughput

# Check AWS CloudWatch metrics (for EBS volumes)
# - VolumeReadOps/VolumeWriteOps
# - VolumeThroughputPercentage
# - VolumeQueueLength

Volume Not Detaching

# Force delete stuck pod
kubectl delete pod my-pod --grace-period=0 --force

# If volume still attached to wrong node
# Find node
kubectl get pod my-pod -o wide

# Cordon and drain node
kubectl cordon node-1
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

# Volume should detach automatically

Data Corruption

# For filesystem corruption
kubectl exec -it my-pod -- sh

# Check filesystem
e2fsck -f /dev/xvdf

# For database corruption (PostgreSQL)
kubectl exec -it postgres-0 -- sh
su - postgres
pg_resetwal -f /var/lib/postgresql/data

# Restore from backup
velero restore create --from-backup latest-backup

Best Practices

1. Always Use Persistent Storage for Stateful Apps

# BAD: Using emptyDir for database
volumes:
- name: data
  emptyDir: {}  # Data lost on pod restart!

# GOOD: Using PVC
volumes:
- name: data
  persistentVolumeClaim:
    claimName: postgres-pvc

2. Set Resource Requests and Limits

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sized-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi  # Request exactly what you need
  storageClassName: fast-ssd

3. Use Storage Classes for Dynamic Provisioning

# Avoid manual PV creation
# Use StorageClass instead
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dynamic-pvc
spec:
  storageClassName: fast-ssd  # Let Kubernetes provision
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

4. Implement Backup Strategy

# Regular backups with Velero
velero schedule create daily \
  --schedule="0 2 * * *" \
  --ttl 720h  # Keep for 30 days

# Test restores regularly
velero restore create test-restore \
  --from-backup daily-20251231 \
  --namespace-mappings production:test

5. Monitor Storage Usage

apiVersion: v1
kind: Pod
metadata:
  name: monitoring-pod
spec:
  containers:
  - name: app
    image: myapp:1.0
    resources:
      limits:
        ephemeral-storage: "2Gi"  # Limit ephemeral storage
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: app-pvc

Monitoring with Prometheus:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubelet-metrics
spec:
  endpoints:
  - port: https-metrics
    scheme: https
    path: /metrics
    honorLabels: true
    relabelings:
    - sourceLabels: [__metrics_path__]
      targetLabel: metrics_path
  selector:
    matchLabels:
      k8s-app: kubelet

6. Use Appropriate Access Modes

# Single pod access
accessModes:
- ReadWriteOnce  # RWO

# Multiple pods on same node
accessModes:
- ReadWriteOnce
# Use node affinity to schedule pods together

# Multiple pods across nodes
accessModes:
- ReadWriteMany  # RWX (requires NFS or similar)

7. Set Proper Reclaim Policies

# For production data
persistentVolumeReclaimPolicy: Retain  # Manual cleanup

# For development/test
persistentVolumeReclaimPolicy: Delete  # Auto cleanup

# StorageClass default
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: production-storage
reclaimPolicy: Retain  # Protect data

8. Use Volume Snapshots for Backups

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: daily-snapshot
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: production-pvc

9. Implement Storage Quotas

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-quota
  namespace: development
spec:
  hard:
    requests.storage: "100Gi"  # Max total storage
    persistentvolumeclaims: "10"  # Max number of PVCs

10. Test Disaster Recovery

# Regular DR drills
# 1. Delete pod
kubectl delete pod postgres-0

# 2. Verify data persists
kubectl exec postgres-0 -- psql -U postgres -c "SELECT count(*) FROM users;"

# 3. Test cross-region restore
velero restore create dr-test \
  --from-backup production-backup \
  --namespace-mappings production:dr-environment

What I Learned

After years of managing Kubernetes storage in production, here are my key takeaways:

1. Storage is Different: Kubernetes storage requires a mental shift from traditional infrastructure. Embrace dynamic provisioning, understand the PV/PVC model, and design for pod mobility.

2. Data is Sacred: Implement robust backup and disaster recovery from day one. I've seen too many incidents where proper backups were the only thing preventing catastrophic data loss.

3. Performance Matters: Choose appropriate storage types for your workload. Don't use network HDD for databases or local SSDs for shared media files. Understand IOPS, throughput, and latency characteristics.

4. Test Everything: Test volume provisioning, pod rescheduling with volumes attached, backup and restore procedures, and disaster recovery scenarios. Storage failures in production are stressful—preparation reduces that stress.

5. Monitor Relentlessly: Track storage usage, I/O performance, and volume health. Set alerts for approaching capacity limits and performance degradation.

6. Security First: Use encryption at rest, implement proper access controls with fsGroup and runAsUser, and audit who can create and access volumes.

7. StatefulSets for State: Use StatefulSets for stateful applications. The stable network identities and ordered deployments are essential for databases and similar workloads.

8. Cost Awareness: Storage costs add up quickly. Implement quotas, monitor unused volumes, and choose appropriate storage tiers. That 10TB of provisioned but unused SSD storage gets expensive.

9. Understand Your Provisioner: Each cloud provider's CSI driver has different capabilities and limitations. Read the documentation, understand the parameters, and test thoroughly.

10. Plan for Growth: Design your storage architecture to scale. Use StorageClasses that support volume expansion, implement monitoring to forecast capacity needs, and have procedures for migrating to larger volumes.

The difference between a working storage solution and a production-ready one is preparation, testing, and operational discipline. Storage problems often can't be fixed quickly—proper design and robust procedures are your best defense.

Start with simple patterns, use managed storage services when possible, implement comprehensive monitoring and backups, and evolve your storage strategy as your needs grow. Your future self (and your team) will thank you when that inevitable storage incident occurs.