Container Security and Image Scanning

The $50,000 AWS Bill: When Cryptominers Hijack Your Containers

Monday morning. I grabbed my coffee, opened my laptop, and nearly spat it out when I saw the AWS billing alert: $48,723.42 for the weekend. Our normal weekend bill? Maybe $2,000.

Someone had compromised one of our container images and deployed cryptomining malware across our entire Kubernetes cluster. The attacker exploited a vulnerability in our base image—a three-year-old version of Alpine Linux we'd been using since "the beginning." We'd never scanned it. We'd never updated it. It just... worked. Until it didn't.

The post-mortem revealed 523 critical vulnerabilities in that single base image. The attack vector? A vulnerable version of curl in the base layer, exploited through an exposed debugging endpoint we'd forgotten to remove.

That incident cost us $50,000 in compute charges, a week of incident response, mandatory security audits, and worst of all—customer trust. But it taught me everything I know about container security.

This article covers the container security strategy I built from scratch after that disaster. We went from 500+ vulnerabilities per image to zero critical findings, from ad-hoc image building to a hardened, audited pipeline.

What You'll Learn

• Container security fundamentals and the attack surface

• Image scanning with Trivy, Clair, and Snyk

• Building secure, minimal base images

• Multi-stage builds for security

• Container runtime security

• Kubernetes security contexts and admission controls

• Pod Security Standards implementation

The Container Security Threat Model

Containers introduce unique security challenges across multiple layers:

The container attack surface:

• Application layer: Your code and dependencies

• Image layer: Base images, OS packages

• Build layer: Dockerfile, build process

• Registry layer: Image storage, distribution

• Runtime layer: Container engine, orchestrator

• Host layer: Kernel, system calls

Image Scanning with Trivy

Trivy is my go-to scanner for containers. It's fast, accurate, and finds vulnerabilities in OS packages, application dependencies, and even IaC misconfigurations.

Basic Trivy Scanning

# .gitlab-ci.yml
stages:
  - build
  - scan
  - deploy

variables:
  IMAGE_NAME: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA}
  TRIVY_VERSION: "latest"

build:
  stage: build
  image: docker:24-dind
  services:
    - docker:24-dind
  script:
    - docker build -t ${IMAGE_NAME} .
    - docker push ${IMAGE_NAME}

trivy_scan:
  stage: scan
  image: aquasec/trivy:${TRIVY_VERSION}
  script:
    # Scan for vulnerabilities
    - trivy image --severity CRITICAL,HIGH --exit-code 1 --no-progress ${IMAGE_NAME}
    
    # Generate reports
    - trivy image --format json --output trivy-report.json ${IMAGE_NAME}
    - trivy image --format sarif --output trivy-report.sarif ${IMAGE_NAME}
    
    # Scan for secrets
    - trivy image --scanners secret --severity HIGH,CRITICAL ${IMAGE_NAME}
    
    # Scan for misconfigurations
    - trivy image --scanners config ${IMAGE_NAME}
  artifacts:
    reports:
      container_scanning: trivy-report.json
    paths:
      - trivy-report.json
      - trivy-report.sarif
    expire_in: 30 days
  allow_failure: false

Advanced Trivy Configuration

# trivy.yaml
scan:
  # Skip files/directories
  skip-files:
    - "**/*.md"
    - "**/test/**"
  
  # Skip specific directories
  skip-dirs:
    - "node_modules"
    - "vendor"
  
  # Ignore unfixed vulnerabilities
  ignore-unfixed: false
  
  # Severity levels to detect
  severity:
    - CRITICAL
    - HIGH
    - MEDIUM

vulnerability:
  # VEX (Vulnerability Exploitability eXchange) support
  vex: .trivyignore.vex.json
  
  # Ignore specific vulnerabilities
  ignore:
    - CVE-2023-12345  # False positive
    - CVE-2023-67890  # Mitigated by WAF

# Database configuration
db:
  repository: "ghcr.io/aquasecurity/trivy-db"
  skip-update: false

# Output formatting
format: table
output: trivy-results.txt

Trivy in Pre-commit Hooks

#!/bin/bash
# .git/hooks/pre-commit

echo "🔍 Scanning Docker images for vulnerabilities..."

# Find Dockerfiles changed in this commit
DOCKERFILES=$(git diff --cached --name-only --diff-filter=ACM | grep Dockerfile)

if [ -z "$DOCKERFILES" ]; then
  echo "✅ No Dockerfiles changed"
  exit 0
fi

# Build and scan each Dockerfile
for dockerfile in $DOCKERFILES; do
  IMAGE_TAG="scan-$(basename $(dirname $dockerfile)):$(date +%s)"
  
  echo "Building $dockerfile as $IMAGE_TAG..."
  docker build -f "$dockerfile" -t "$IMAGE_TAG" . || exit 1
  
  echo "Scanning $IMAGE_TAG..."
  trivy image \
    --severity CRITICAL,HIGH \
    --exit-code 1 \
    --no-progress \
    "$IMAGE_TAG"
  
  SCAN_RESULT=$?
  
  # Cleanup
  docker rmi "$IMAGE_TAG" > /dev/null 2>&1
  
  if [ $SCAN_RESULT -ne 0 ]; then
    echo "❌ Vulnerabilities found in $dockerfile"
    echo "Fix vulnerabilities before committing"
    exit 1
  fi
done

echo "✅ All Docker images passed security scan"
exit 0

Building Secure Base Images

The foundation of container security is choosing the right base image.

The Base Image Comparison

# ❌ BAD: Large attack surface, many vulnerabilities
FROM ubuntu:20.04
# Image size: 72.9MB
# Vulnerabilities: 50+ (including critical)
# Packages: 100+

# ⚠️ BETTER: Smaller, but still has package manager
FROM alpine:3.18
# Image size: 7.3MB
# Vulnerabilities: 0-2 typically
# Packages: 14

# ✅ BEST: Minimal attack surface
FROM gcr.io/distroless/nodejs18-debian11
# Image size: 120MB (but minimal attack surface)
# Vulnerabilities: <5 typically
# Packages: Only runtime essentials
# No shell, no package manager

# 🚀 OPTIMAL: Scratch for static binaries
FROM scratch
# Image size: Binary size only
# Vulnerabilities: 0 (no OS)
# Packages: 0

Multi-Stage Build for Security

# Dockerfile - Multi-stage build
# Stage 1: Build environment (large, with tools)
FROM node:18-alpine AS builder

WORKDIR /app

# Install build dependencies
RUN apk add --no-cache python3 make g++

# Copy dependency definitions
COPY package*.json ./

# Install ALL dependencies (including dev)
RUN npm ci

# Copy source code
COPY . .

# Build application
RUN npm run build

# Run tests
RUN npm run test

# Stage 2: Production (minimal, secure)
FROM gcr.io/distroless/nodejs18-debian11

WORKDIR /app

# Copy only production dependencies
COPY --from=builder /app/node_modules ./node_modules

# Copy built application
COPY --from=builder /app/dist ./dist

# Copy only necessary files
COPY package.json ./

# Run as non-root user
USER nonroot:nonroot

# No shell available in distroless
# Healthcheck using node
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD ["node", "-e", "require('http').get('http://localhost:3000/health', (r) => process.exit(r.statusCode === 200 ? 0 : 1))"]

# Start application
CMD ["dist/index.js"]

# Labels for tracking
LABEL org.opencontainers.image.source="https://github.com/myorg/myapp"
LABEL org.opencontainers.image.description="Payment Service"
LABEL org.opencontainers.image.licenses="MIT"

Security-Hardened Dockerfile Best Practices

# Dockerfile - Security hardened
FROM node:18-alpine AS deps

# Create app user
RUN addgroup -g 1001 -S appgroup && \
    adduser -S appuser -u 1001 -G appgroup

WORKDIR /app

# Install only production dependencies
COPY package*.json ./
RUN npm ci --only=production && \
    npm cache clean --force

# ============================================
# Build stage
# ============================================
FROM node:18-alpine AS builder

WORKDIR /app

COPY package*.json ./
RUN npm ci

COPY . .
RUN npm run build && \
    npm run test

# ============================================
# Runtime stage
# ============================================
FROM node:18-alpine

# Install security updates
RUN apk upgrade --no-cache && \
    apk add --no-cache dumb-init

# Create non-root user
RUN addgroup -g 1001 -S appgroup && \
    adduser -S appuser -u 1001 -G appgroup

WORKDIR /app

# Copy dependencies from deps stage
COPY --from=deps --chown=appuser:appgroup /app/node_modules ./node_modules

# Copy built app from builder stage  
COPY --from=builder --chown=appuser:appgroup /app/dist ./dist
COPY --from=builder --chown=appuser:appgroup /app/package.json ./

# Remove unnecessary files
RUN rm -rf /tmp/* /var/cache/apk/*

# Switch to non-root user
USER appuser

# Use dumb-init to handle signals properly
ENTRYPOINT ["dumb-init", "--"]

# Start application
CMD ["node", "dist/index.js"]

# Security labels
LABEL security.scan="trivy"
LABEL security.policy="hardened"

Container Runtime Security

Scanning images isn't enough—you need runtime protection too.

Kubernetes Security Contexts

# deployment.yaml - Secure pod configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payment-service
  template:
    metadata:
      labels:
        app: payment-service
    spec:
      # Pod-level security context
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001
        seccompProfile:
          type: RuntimeDefault
        supplementalGroups: [1001]
      
      # Service account with minimal permissions
      serviceAccountName: payment-service
      automountServiceAccountToken: false
      
      containers:
      - name: payment-service
        image: myregistry/payment-service:v1.2.3
        
        # Container-level security context
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1001
          capabilities:
            drop:
              - ALL
          seccompProfile:
            type: RuntimeDefault
        
        # Resource limits (prevent resource exhaustion attacks)
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        
        # Readiness/liveness probes
        livenessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 10
        
        readinessProbe:
          httpGet:
            path: /ready
            port: 3000
          initialDelaySeconds: 5
          periodSeconds: 5
        
        # Environment variables from secrets
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: payment-db-secret
              key: url
        
        # Volume mounts (read-only where possible)
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: cache
          mountPath: /app/.cache
        - name: config
          mountPath: /app/config
          readOnly: true
      
      # Volumes
      volumes:
      - name: tmp
        emptyDir: {}
      - name: cache
        emptyDir: {}
      - name: config
        configMap:
          name: payment-config

Pod Security Standards

# namespace.yaml - Enforce Pod Security Standards
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    # Enforce restricted policy
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    
    # Audit violations of baseline
    pod-security.kubernetes.io/audit: baseline
    pod-security.kubernetes.io/audit-version: latest
    
    # Warn on violations of baseline
    pod-security.kubernetes.io/warn: baseline
    pod-security.kubernetes.io/warn-version: latest

Network Policies

# networkpolicy.yaml - Restrict network access
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payment-service-netpol
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: payment-service
  
  policyTypes:
    - Ingress
    - Egress
  
  # Ingress: Only from API gateway
  ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            name: gateway
      - podSelector:
          matchLabels:
            app: api-gateway
      ports:
        - protocol: TCP
          port: 3000
  
  # Egress: Only to database and external payment APIs
  egress:
    # Allow DNS
    - to:
      - namespaceSelector:
          matchLabels:
            name: kube-system
      ports:
        - protocol: UDP
          port: 53
    
    # Allow database
    - to:
      - podSelector:
          matchLabels:
            app: postgres
      ports:
        - protocol: TCP
          port: 5432
    
    # Allow external payment gateway
    - to:
      - ipBlock:
          cidr: 203.0.113.0/24  # Stripe IP range
      ports:
        - protocol: TCP
          port: 443

Admission Controllers

Prevent vulnerable containers from being deployed in the first place.

OPA Gatekeeper Policies

# constraint-template.yaml
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8scontainerimagescanning
spec:
  crd:
    spec:
      names:
        kind: K8sContainerImageScanning
      validation:
        openAPIV3Schema:
          properties:
            maxCritical:
              type: integer
            maxHigh:
              type: integer
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8scontainerimagescanning
        
        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          not image_scanned(container.image)
          msg := sprintf("Container image %v has not been scanned", [container.image])
        }
        
        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          vulnerabilities := get_vulnerabilities(container.image)
          vulnerabilities.critical > input.parameters.maxCritical
          msg := sprintf("Container image %v has %v critical vulnerabilities (max: %v)", 
            [container.image, vulnerabilities.critical, input.parameters.maxCritical])
        }
        
        image_scanned(image) {
          # Check if image has scan results in your registry/database
          # This would integrate with your scanning infrastructure
          true
        }
        
        get_vulnerabilities(image) = result {
          # Query vulnerability database
          # Return {critical: N, high: M}
          result := {"critical": 0, "high": 0}
        }
---
# constraint.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sContainerImageScanning
metadata:
  name: container-image-must-be-scanned
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces:
      - production
      - staging
  parameters:
    maxCritical: 0
    maxHigh: 5

Kyverno Policy

# kyverno-policy.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: container-security-policy
spec:
  validationFailureAction: enforce
  background: true
  rules:
    # Require non-root containers
    - name: require-non-root
      match:
        resources:
          kinds:
            - Pod
      validate:
        message: "Containers must run as non-root user"
        pattern:
          spec:
            securityContext:
              runAsNonRoot: true
            containers:
              - securityContext:
                  runAsNonRoot: true
    
    # Require read-only root filesystem
    - name: require-readonly-rootfs
      match:
        resources:
          kinds:
            - Pod
      validate:
        message: "Containers must have read-only root filesystem"
        pattern:
          spec:
            containers:
              - securityContext:
                  readOnlyRootFilesystem: true
    
    # Block privileged containers
    - name: block-privileged
      match:
        resources:
          kinds:
            - Pod
      validate:
        message: "Privileged containers are not allowed"
        pattern:
          spec:
            containers:
              - securityContext:
                  privileged: false
    
    # Require security scanning label
    - name: require-scan-label
      match:
        resources:
          kinds:
            - Pod
      validate:
        message: "Pod must have 'security.scan.passed: true' label"
        pattern:
          metadata:
            labels:
              security.scan.passed: "true"
    
    # Block latest tag
    - name: block-latest-tag
      match:
        resources:
          kinds:
            - Pod
      validate:
        message: "Using 'latest' tag is not allowed"
        pattern:
          spec:
            containers:
              - image: "!*:latest"

Continuous Image Scanning in Registry

Don't just scan on build—scan continuously as new vulnerabilities are discovered.

Harbor Registry with Trivy

# docker-compose.yml - Harbor with Trivy scanner
version: '3.8'
services:
  harbor-core:
    image: goharbor/harbor-core:v2.9.0
    environment:
      - SCANNER_TRIVY_URL=http://trivy:8080
    depends_on:
      - trivy
  
  trivy:
    image: aquasec/harbor-scanner-trivy:latest
    environment:
      SCANNER_LOG_LEVEL: info
      SCANNER_TRIVY_CACHE_DIR: /home/scanner/.cache/trivy
      SCANNER_TRIVY_REPORTS_DIR: /home/scanner/.cache/reports
      SCANNER_TRIVY_DEBUG_MODE: "false"
      SCANNER_TRIVY_VULN_TYPE: "os,library"
      SCANNER_TRIVY_SEVERITY: "CRITICAL,HIGH,MEDIUM"
      SCANNER_TRIVY_TIMEOUT: 5m0s
    volumes:
      - trivy-cache:/home/scanner/.cache
  
  # Automated scheduled scanning
  scanner-cron:
    image: alpine:latest
    command: >
      sh -c "
      while true; do
        echo 'Running scheduled scan...';
        curl -X POST http://harbor-core:8080/api/v2.0/system/scanAll/schedule;
        sleep 21600;  # Every 6 hours
      done
      "
    depends_on:
      - harbor-core

volumes:
  trivy-cache:

Runtime Threat Detection

Falco for Runtime Security

# falco-rules.yaml
- rule: Unexpected outbound connection
  desc: Detect suspicious outbound network connections
  condition: >
    outbound and
    container and
    not k8s_service_lb and
    not user_known_network_destination
  output: >
    Unexpected outbound connection
    (container=%container.name
    user=%user.name
    connection=%fd.name)
  priority: WARNING

- rule: Write below binary dir
  desc: Detect writing to binary directories
  condition: >
    write and
    container and
    bin_dir and
    not package_mgmt_procs
  output: >
    File written below binary directory
    (file=%fd.name
    proc=%proc.name
    container=%container.name)
  priority: CRITICAL

- rule: Read sensitive file
  desc: Detect reading of sensitive files
  condition: >
    open_read and
    container and
    sensitive_files
  output: >
    Sensitive file accessed
    (file=%fd.name
    proc=%proc.name
    container=%container.name)
  priority: WARNING

Best Practices

1. Minimize Base Images

Smaller images = smaller attack surface

# Use distroless when possible
FROM gcr.io/distroless/static-debian11

# Or Alpine if you need a shell
FROM alpine:3.18
RUN apk add --no-cache ca-certificates

2. Scan Everything, Everywhere

# Scan on:
- push to registry
- pull from registry  
- scheduled (daily)
- before deployment
- in production (continuous)

3. Never Run as Root

USER 1001:1001

4. Use Read-Only Filesystems

securityContext:
  readOnlyRootFilesystem: true

5. Set Resource Limits

resources:
  limits:
    memory: "512Mi"
    cpu: "500m"

Key Takeaways

✅ Scan images continuously - Not just at build time ✅ Minimize base images - Distroless or Alpine preferred ✅ Multi-stage builds - Keep build tools out of production ✅ Run as non-root - Always use unprivileged users ✅ Read-only filesystems - Prevent runtime modifications ✅ Admission controls - Block vulnerable pods before deployment ✅ Runtime monitoring - Detect threats in running containers

What's Next

Container images are built from code and dependencies. In the next article, we'll cover Software Bill of Materials (SBOM)—creating comprehensive inventories of everything in your software supply chain for compliance and security.

---

Next Article: Software Bill of Materials (SBOM) →

---

Part of the DevSecOps 101 Series