Article 12: Production Readiness

Introduction

Writing working code is just the beginning—getting it running reliably in production is where the real engineering happens. Through deploying and maintaining production systems, I've learned that production readiness isn't about perfection; it's about predictability, observability, and resilience.

This final article covers containerization with Docker, configuration management, and the monitoring fundamentals you need to run software in production.

What is Production Ready?

Production Readiness Checklist

Category	Requirement
Deployment	Containerized, reproducible builds
Configuration	Environment-based, no hardcoded secrets
Logging	Structured logs, appropriate levels
Monitoring	Health checks, key metrics
Error Handling	Graceful degradation, proper error responses
Documentation	Runbooks, architecture docs

Containerization with Docker

Why Containers?

Dockerfile Basics

# Dockerfile
# Use official Python image as base
FROM python:3.11-slim

# Set working directory
WORKDIR /app

# Copy dependency files first (for layer caching)
COPY pyproject.toml .
COPY requirements.txt .

# Install dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY src/ src/

# Set environment variables
ENV PYTHONPATH=/app
ENV PYTHONUNBUFFERED=1

# Create non-root user
RUN useradd --create-home appuser
USER appuser

# Expose port
EXPOSE 8000

# Run application
CMD ["python", "-m", "src.main"]

Multi-Stage Builds

# Dockerfile with multi-stage build
# Stage 1: Build
FROM python:3.11-slim AS builder

WORKDIR /app

# Install build dependencies
RUN pip install --upgrade pip build

# Copy and build the package
COPY pyproject.toml .
COPY src/ src/
RUN python -m build --wheel

# Stage 2: Runtime
FROM python:3.11-slim AS runtime

WORKDIR /app

# Copy only the built wheel from builder
COPY --from=builder /app/dist/*.whl .

# Install the package
RUN pip install --no-cache-dir *.whl && rm *.whl

# Create non-root user
RUN useradd --create-home appuser
USER appuser

EXPOSE 8000
CMD ["python", "-m", "myapp"]

Docker Compose for Development

# docker-compose.yml
version: '3.8'

services:
  app:
    build: .
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://user:pass@db:5432/mydb
      - REDIS_URL=redis://redis:6379
    depends_on:
      - db
      - redis
    volumes:
      - ./src:/app/src:ro  # Mount source for development
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

  db:
    image: postgres:15
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=pass
      - POSTGRES_DB=mydb
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

volumes:
  postgres_data:

Docker Best Practices

# ✅ Good Dockerfile practices

# Use specific versions, not 'latest'
FROM python:3.11.4-slim

# Combine RUN commands to reduce layers
RUN apt-get update && \
    apt-get install -y --no-install-recommends gcc && \
    rm -rf /var/lib/apt/lists/*

# Use .dockerignore to exclude files
# .dockerignore:
# __pycache__
# *.pyc
# .git
# .env
# tests/

# Don't run as root
USER appuser

# Use COPY instead of ADD (unless you need tar extraction)
COPY requirements.txt .

# Set proper signals handling
STOPSIGNAL SIGTERM

Configuration Management

The 12-Factor App Config

# config.py
"""Configuration management following 12-Factor App principles."""
import os
from dataclasses import dataclass
from functools import lru_cache


@dataclass
class Settings:
    """Application settings loaded from environment."""
    
    # Database
    database_url: str
    database_pool_size: int = 5
    
    # Redis
    redis_url: str = "redis://localhost:6379"
    
    # Application
    debug: bool = False
    log_level: str = "INFO"
    secret_key: str = ""
    
    # External services
    api_key: str = ""
    
    @classmethod
    def from_env(cls) -> "Settings":
        """Load settings from environment variables."""
        return cls(
            database_url=os.environ["DATABASE_URL"],
            database_pool_size=int(os.getenv("DATABASE_POOL_SIZE", "5")),
            redis_url=os.getenv("REDIS_URL", "redis://localhost:6379"),
            debug=os.getenv("DEBUG", "false").lower() == "true",
            log_level=os.getenv("LOG_LEVEL", "INFO"),
            secret_key=os.environ["SECRET_KEY"],
            api_key=os.getenv("API_KEY", ""),
        )


@lru_cache
def get_settings() -> Settings:
    """Get cached settings instance."""
    return Settings.from_env()

Environment Files

# .env.example (committed to git - template only)
DATABASE_URL=postgresql://user:password@localhost:5432/mydb
REDIS_URL=redis://localhost:6379
DEBUG=false
LOG_LEVEL=INFO
SECRET_KEY=change-me-in-production
API_KEY=

# .env (NOT committed - local development values)
DATABASE_URL=postgresql://dev:dev@localhost:5432/devdb
DEBUG=true
LOG_LEVEL=DEBUG
SECRET_KEY=dev-secret-key

# Load .env file in development
from dotenv import load_dotenv

load_dotenv()  # Load from .env file if it exists
settings = Settings.from_env()

Configuration Validation

# config.py
from dataclasses import dataclass
from typing import ClassVar
import os


@dataclass
class Settings:
    """Validated application settings."""
    
    database_url: str
    secret_key: str
    log_level: str = "INFO"
    
    VALID_LOG_LEVELS: ClassVar[set] = {"DEBUG", "INFO", "WARNING", "ERROR"}
    
    def __post_init__(self) -> None:
        """Validate settings after initialization."""
        self._validate()
    
    def _validate(self) -> None:
        """Validate all settings."""
        errors = []
        
        if not self.database_url:
            errors.append("DATABASE_URL is required")
        
        if not self.secret_key:
            errors.append("SECRET_KEY is required")
        elif len(self.secret_key) < 32:
            errors.append("SECRET_KEY must be at least 32 characters")
        
        if self.log_level not in self.VALID_LOG_LEVELS:
            errors.append(f"LOG_LEVEL must be one of {self.VALID_LOG_LEVELS}")
        
        if errors:
            raise ValueError(f"Configuration errors: {'; '.join(errors)}")

Secrets Management

# Never do this:
# API_KEY = "sk-abc123..."  # ❌ Hardcoded secret

# Do this instead:
# 1. Environment variables
api_key = os.environ["API_KEY"]

# 2. Secret management services (example with AWS)
import boto3

def get_secret(secret_name: str) -> str:
    """Retrieve secret from AWS Secrets Manager."""
    client = boto3.client("secretsmanager")
    response = client.get_secret_value(SecretId=secret_name)
    return response["SecretString"]

# 3. Kubernetes secrets (mounted as files)
def read_secret_file(path: str) -> str:
    """Read secret from mounted file."""
    with open(path) as f:
        return f.read().strip()

database_password = read_secret_file("/run/secrets/db_password")

Health Checks

Health Check Endpoints

# health.py
"""Health check endpoints for production monitoring."""
from dataclasses import dataclass
from datetime import datetime
from enum import Enum
import asyncio


class HealthStatus(str, Enum):
    HEALTHY = "healthy"
    DEGRADED = "degraded"
    UNHEALTHY = "unhealthy"


@dataclass
class HealthCheck:
    name: str
    status: HealthStatus
    message: str = ""
    latency_ms: float = 0


@dataclass
class HealthResponse:
    status: HealthStatus
    timestamp: str
    checks: list[HealthCheck]
    version: str


class HealthService:
    """Service for running health checks."""
    
    def __init__(self, database, redis, version: str):
        self.database = database
        self.redis = redis
        self.version = version
    
    async def check_database(self) -> HealthCheck:
        """Check database connectivity."""
        start = datetime.now()
        try:
            await self.database.execute("SELECT 1")
            latency = (datetime.now() - start).total_seconds() * 1000
            return HealthCheck(
                name="database",
                status=HealthStatus.HEALTHY,
                latency_ms=latency,
            )
        except Exception as e:
            return HealthCheck(
                name="database",
                status=HealthStatus.UNHEALTHY,
                message=str(e),
            )
    
    async def check_redis(self) -> HealthCheck:
        """Check Redis connectivity."""
        start = datetime.now()
        try:
            await self.redis.ping()
            latency = (datetime.now() - start).total_seconds() * 1000
            return HealthCheck(
                name="redis",
                status=HealthStatus.HEALTHY,
                latency_ms=latency,
            )
        except Exception as e:
            return HealthCheck(
                name="redis",
                status=HealthStatus.UNHEALTHY,
                message=str(e),
            )
    
    async def get_health(self) -> HealthResponse:
        """Run all health checks."""
        checks = await asyncio.gather(
            self.check_database(),
            self.check_redis(),
        )
        
        # Determine overall status
        if any(c.status == HealthStatus.UNHEALTHY for c in checks):
            overall = HealthStatus.UNHEALTHY
        elif any(c.status == HealthStatus.DEGRADED for c in checks):
            overall = HealthStatus.DEGRADED
        else:
            overall = HealthStatus.HEALTHY
        
        return HealthResponse(
            status=overall,
            timestamp=datetime.utcnow().isoformat(),
            checks=list(checks),
            version=self.version,
        )

Liveness vs Readiness

# Kubernetes health check configuration
apiVersion: v1
kind: Pod
spec:
  containers:
    - name: app
      # Liveness: Is the application alive?
      # Restart if this fails
      livenessProbe:
        httpGet:
          path: /health/live
          port: 8000
        initialDelaySeconds: 10
        periodSeconds: 10
        failureThreshold: 3
      
      # Readiness: Can the application serve traffic?
      # Remove from load balancer if this fails
      readinessProbe:
        httpGet:
          path: /health/ready
          port: 8000
        initialDelaySeconds: 5
        periodSeconds: 5
        failureThreshold: 3

# Separate endpoints for liveness and readiness
async def liveness() -> dict:
    """Liveness check - is the process running?"""
    return {"status": "alive"}

async def readiness() -> dict:
    """Readiness check - can we serve traffic?"""
    health = await health_service.get_health()
    
    if health.status == HealthStatus.UNHEALTHY:
        raise HTTPException(status_code=503, detail="Not ready")
    
    return {"status": "ready", "checks": health.checks}

Monitoring Basics

Key Metrics to Track

The Four Golden Signals

Signal	Description	Example
Latency	Time to serve requests	p50, p95, p99 response time
Traffic	Demand on the system	Requests per second
Errors	Rate of failed requests	5xx error percentage
Saturation	How full the system is	CPU/memory utilization

Structured Logging for Production

# logging_config.py
"""Production logging configuration."""
import logging
import json
from datetime import datetime


class JSONFormatter(logging.Formatter):
    """Format logs as JSON for log aggregation."""
    
    def format(self, record: logging.LogRecord) -> str:
        log_data = {
            "timestamp": datetime.utcnow().isoformat(),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
            "module": record.module,
            "function": record.funcName,
            "line": record.lineno,
        }
        
        # Add exception info if present
        if record.exc_info:
            log_data["exception"] = self.formatException(record.exc_info)
        
        # Add extra fields
        if hasattr(record, "request_id"):
            log_data["request_id"] = record.request_id
        if hasattr(record, "user_id"):
            log_data["user_id"] = record.user_id
        
        return json.dumps(log_data)


def setup_logging(log_level: str = "INFO") -> None:
    """Configure production logging."""
    handler = logging.StreamHandler()
    handler.setFormatter(JSONFormatter())
    
    root_logger = logging.getLogger()
    root_logger.setLevel(log_level)
    root_logger.addHandler(handler)

Request Tracing

# tracing.py
"""Request tracing for debugging production issues."""
import uuid
from contextvars import ContextVar
from functools import wraps
import logging

# Context variable for request ID
request_id_var: ContextVar[str] = ContextVar("request_id", default="")

logger = logging.getLogger(__name__)


class RequestContext:
    """Context manager for request tracing."""
    
    def __init__(self, request_id: str | None = None):
        self.request_id = request_id or str(uuid.uuid4())[:8]
        self.token = None
    
    def __enter__(self):
        self.token = request_id_var.set(self.request_id)
        return self
    
    def __exit__(self, *args):
        request_id_var.reset(self.token)


class TracingFilter(logging.Filter):
    """Add request ID to all log records."""
    
    def filter(self, record: logging.LogRecord) -> bool:
        record.request_id = request_id_var.get() or "-"
        return True


def traced(func):
    """Decorator to add tracing to functions."""
    @wraps(func)
    async def wrapper(*args, **kwargs):
        request_id = request_id_var.get()
        logger.debug(f"Entering {func.__name__}", extra={"request_id": request_id})
        try:
            result = await func(*args, **kwargs)
            logger.debug(f"Exiting {func.__name__}", extra={"request_id": request_id})
            return result
        except Exception as e:
            logger.error(f"Error in {func.__name__}: {e}", extra={"request_id": request_id})
            raise
    return wrapper

Graceful Shutdown

# shutdown.py
"""Graceful shutdown handling."""
import asyncio
import signal
import logging

logger = logging.getLogger(__name__)


class GracefulShutdown:
    """Handle graceful shutdown of the application."""
    
    def __init__(self):
        self.shutdown_event = asyncio.Event()
        self.tasks: list[asyncio.Task] = []
    
    def setup_signal_handlers(self) -> None:
        """Set up signal handlers for graceful shutdown."""
        loop = asyncio.get_event_loop()
        
        for sig in (signal.SIGTERM, signal.SIGINT):
            loop.add_signal_handler(
                sig,
                lambda s=sig: asyncio.create_task(self.shutdown(s))
            )
    
    async def shutdown(self, sig: signal.Signals) -> None:
        """Handle shutdown signal."""
        logger.info(f"Received signal {sig.name}, shutting down...")
        
        # Stop accepting new work
        self.shutdown_event.set()
        
        # Wait for in-flight requests to complete (with timeout)
        if self.tasks:
            logger.info(f"Waiting for {len(self.tasks)} tasks to complete...")
            await asyncio.wait(self.tasks, timeout=30)
        
        # Clean up resources
        await self.cleanup()
        
        logger.info("Shutdown complete")
    
    async def cleanup(self) -> None:
        """Clean up resources on shutdown."""
        # Close database connections
        # Close cache connections
        # Flush logs
        pass
    
    def register_task(self, task: asyncio.Task) -> None:
        """Register a task for tracking during shutdown."""
        self.tasks.append(task)
        task.add_done_callback(self.tasks.remove)

Production Checklist

Pre-Deployment

## Before Deploying to Production

### Code Quality
- [ ] All tests pass
- [ ] Linting passes
- [ ] Type checking passes
- [ ] Code reviewed and approved

### Configuration
- [ ] Environment variables documented
- [ ] Secrets stored securely
- [ ] No hardcoded credentials
- [ ] Configuration validated on startup

### Docker
- [ ] Multi-stage build
- [ ] Non-root user
- [ ] Health checks configured
- [ ] Proper signal handling

### Logging & Monitoring
- [ ] Structured logging configured
- [ ] Log levels appropriate
- [ ] Health check endpoints
- [ ] Key metrics exposed

### Documentation
- [ ] README updated
- [ ] Runbook available
- [ ] Architecture documented
- [ ] API documentation current

Runtime

## Production Runtime Checklist

### Reliability
- [ ] Health checks passing
- [ ] Graceful shutdown implemented
- [ ] Retry logic for external services
- [ ] Circuit breakers where needed

### Security
- [ ] HTTPS only
- [ ] Security headers configured
- [ ] Input validation
- [ ] Rate limiting

### Performance
- [ ] Response times acceptable
- [ ] Resource usage stable
- [ ] Connection pooling configured
- [ ] Caching where appropriate

### Observability
- [ ] Logs being collected
- [ ] Metrics being scraped
- [ ] Alerts configured
- [ ] Dashboards available

Practical Exercise

Exercise: Containerize a Python Application

Create a complete containerized setup:

# Dockerfile
FROM python:3.11-slim AS builder

WORKDIR /app
COPY pyproject.toml .
RUN pip wheel --no-deps --wheel-dir /wheels -e .

FROM python:3.11-slim

WORKDIR /app

# Install from wheels
COPY --from=builder /wheels /wheels
RUN pip install --no-cache /wheels/* && rm -rf /wheels

# Copy application
COPY src/ src/

# Security
RUN useradd --create-home appuser
USER appuser

# Runtime config
ENV PYTHONUNBUFFERED=1
EXPOSE 8000

HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
    CMD curl -f http://localhost:8000/health || exit 1

CMD ["python", "-m", "src.main"]

# docker-compose.yml
version: '3.8'

services:
  app:
    build: .
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - LOG_LEVEL=INFO
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

Key Takeaways

1. Containerize consistently - Same image everywhere eliminates "works on my machine"

2. Externalize configuration - Environment variables, not hardcoded values

3. Implement health checks - Know when your service is ready and healthy

4. Log structured data - JSON logs are searchable and parseable

5. Plan for failure - Graceful shutdown, retries, and circuit breakers

6. Monitor the golden signals - Latency, traffic, errors, saturation

Series Summary

Congratulations on completing the Software Engineering 101 series! You've learned:

Phase	Topics
Foundation	Version control, development environment, code quality tools
Design & Structure	Clean code, SOLID principles, design patterns
Quality Assurance	Testing fundamentals, advanced testing, documentation, error handling
Professional Practices	CI/CD, code review, production readiness

These fundamentals will serve you throughout your engineering career. Continue practicing, keep learning, and remember: great software is built incrementally by applying these principles consistently.

What's Next?

With these fundamentals in place, consider exploring:

• Database 101 - Data layer patterns and database fundamentals

• Advanced topics like distributed systems, performance optimization, and architecture patterns

---

This article completes the Software Engineering 101 series.