Multi-Environment Management

Article 9 of 12 in the Cloud Landing Zone Series

Introduction

Through incident response and root cause analysis, I've learned that proper environment separation is critical for preventing production issues.

Investigating production incidents revealed common patterns:

• Development code accidentally deployed to production

• Wrong database connection strings in production configuration

• Test data mixing with production systems

• Insufficient access controls between environments

• Lack of clear environment boundaries

These incidents consistently trace back to weak environment separation. When development, staging, and production environments aren't clearly isolated, mistakes cascade into production impact.

This article shares the environment management patterns I've implemented to create strong boundaries between dev, staging, and production - covering account-level isolation, network segmentation, IAM separation, and promotion workflows that prevent environment confusion.

Environment Separation Strategies

Account-Level Isolation (AWS)

Organization Root
├── Production OU
│   ├── prod-payments-account
│   ├── prod-users-account
│   └── prod-data-account
├── Staging OU
│   ├── staging-payments-account
│   ├── staging-users-account
│   └── staging-data-account
└── Development OU
    ├── dev-shared-account
    └── dev-sandbox-accounts

Benefits:

• Complete blast radius isolation

• Different IAM policies per environment

• Separate cost tracking

• Production SCPs prevent accidental changes

Subscription-Level Isolation (Azure)

# Production Management Group
resource "azurerm_management_group" "production" {
  display_name = "Production"
  
  # Strict policies
}

# Production subscriptions
resource "azurerm_subscription" "prod_payments" {
  subscription_name = "prod-payments"
  billing_scope_id  = var.production_billing_scope
}

resource "azurerm_management_group_subscription_association" "prod_payments" {
  management_group_id = azurerm_management_group.production.id
  subscription_id     = azurerm_subscription.prod_payments.id
}

# Apply production-only policies
resource "azurerm_subscription_policy_assignment" "prod_require_encryption" {
  name                 = "require-encryption"
  subscription_id      = azurerm_subscription.prod_payments.id
  policy_definition_id = var.encryption_policy_id
  
  parameters = jsonencode({
    effect = { value = "Deny" }  # Deny in production, Audit in dev
  })
}

Network Isolation Patterns

Dedicated VPCs per Environment

# Production VPC (10.0.0.0/16)
module "vpc_production" {
  source = "./modules/vpc"
  
  vpc_cidr           = "10.0.0.0/16"
  environment        = "production"
  enable_flow_logs   = true
  flow_log_retention = 90
  
  # Production: Private subnets only, no internet gateway
  public_subnets  = []
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
}

# Staging VPC (10.1.0.0/16)
module "vpc_staging" {
  source = "./modules/vpc"
  
  vpc_cidr         = "10.1.0.0/16"
  environment      = "staging"
  enable_flow_logs = true
  
  # Staging: Public + Private
  public_subnets  = ["10.1.0.0/24"]
  private_subnets = ["10.1.1.0/24", "10.1.2.0/24"]
}

# Development VPC (10.2.0.0/16)
module "vpc_dev" {
  source = "./modules/vpc"
  
  vpc_cidr         = "10.2.0.0/16"
  environment      = "development"
  enable_flow_logs = false  # Cost optimization in dev
  
  public_subnets  = ["10.2.0.0/24"]
  private_subnets = ["10.2.1.0/24"]
}

# Transit Gateway attachments with route isolation
resource "aws_ec2_transit_gateway_vpc_attachment" "production" {
  transit_gateway_id = var.transit_gateway_id
  vpc_id             = module.vpc_production.vpc_id
  subnet_ids         = module.vpc_production.private_subnet_ids
  
  transit_gateway_default_route_table_association = false
  transit_gateway_default_route_table_propagation = false
  
  tags = {
    Name        = "production-tgw-attachment"
    Environment = "production"
  }
}

# Production route table (isolated)
resource "aws_ec2_transit_gateway_route_table" "production" {
  transit_gateway_id = var.transit_gateway_id
  
  tags = {
    Name = "production-routes"
  }
}

# Production can ONLY route to on-prem, not to dev/staging
resource "aws_ec2_transit_gateway_route" "production_to_onprem" {
  destination_cidr_block         = "192.168.0.0/16"
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.production.id
  transit_gateway_attachment_id  = var.vpn_attachment_id
}

# Dev/Staging share a route table (can talk to each other)
resource "aws_ec2_transit_gateway_route_table" "nonprod" {
  transit_gateway_id = var.transit_gateway_id
  
  tags = {
    Name = "nonprod-routes"
  }
}

IAM Separation

Environment-Specific Roles

# Production: Read-only access (zero standing write access)
resource "aws_iam_role" "production_readonly" {
  name = "ProductionReadOnly"
  
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        AWS = "arn:aws:iam::${var.identity_account_id}:root"
      }
      Action = "sts:AssumeRole"
      Condition = {
        StringEquals = {
          "sts:ExternalId" = var.production_external_id
        }
        IpAddress = {
          "aws:SourceIp" = var.corporate_ip_ranges
        }
      }
    }]
  })
  
  managed_policy_arns = ["arn:aws:iam::aws:policy/ReadOnlyAccess"]
}

# Production write access requires approval (break-glass)
resource "aws_iam_role" "production_admin" {
  name = "ProductionAdmin"
  
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        AWS = "arn:aws:iam::${var.identity_account_id}:role/BreakGlassApprover"
      }
      Action = "sts:AssumeRole"
      Condition = {
        StringEquals = {
          "sts:ExternalId" = var.production_breakglass_id
        }
        IpAddress = {
          "aws:SourceIp" = var.corporate_ip_ranges
        }
        # Require MFA
        Bool = {
          "aws:MultiFactorAuthPresent" = "true"
        }
      }
    }]
  })
  
  managed_policy_arns = ["arn:aws:iam::aws:policy/AdministratorAccess"]
  
  # Max session duration: 1 hour
  max_session_duration = 3600
}

# Development: Full access for developers
resource "aws_iam_role" "development_admin" {
  name = "DevelopmentAdmin"
  
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        AWS = "arn:aws:iam::${var.identity_account_id}:role/Developers"
      }
      Action = "sts:AssumeRole"
    }]
  })
  
  managed_policy_arns = ["arn:aws:iam::aws:policy/AdministratorAccess"]
  max_session_duration = 43200  # 12 hours
}

Configuration Management

Environment-Specific Configuration

# Terraform workspaces for environment configuration
locals {
  environment_config = {
    production = {
      instance_type         = "m5.xlarge"
      min_instances        = 3
      max_instances        = 20
      enable_deletion_protection = true
      backup_retention_days = 35
      multi_az             = true
      encryption_enabled   = true
      enable_monitoring    = true
      log_retention_days   = 2555  # 7 years
    }
    
    staging = {
      instance_type         = "m5.large"
      min_instances        = 2
      max_instances        = 10
      enable_deletion_protection = false
      backup_retention_days = 7
      multi_az             = true
      encryption_enabled   = true
      enable_monitoring    = true
      log_retention_days   = 90
    }
    
    development = {
      instance_type         = "t3.medium"
      min_instances        = 1
      max_instances        = 3
      enable_deletion_protection = false
      backup_retention_days = 1
      multi_az             = false
      encryption_enabled   = true
      enable_monitoring    = false
      log_retention_days   = 7
    }
  }
  
  config = local.environment_config[var.environment]
}

# Use environment-specific configuration
resource "aws_db_instance" "main" {
  identifier = "${var.environment}-database"
  
  instance_class        = local.config.instance_type
  multi_az             = local.config.multi_az
  deletion_protection  = local.config.enable_deletion_protection
  backup_retention_period = local.config.backup_retention_days
  storage_encrypted    = local.config.encryption_enabled
  
  # Production gets read replicas
  replicate_source_db = var.environment == "production" ? var.primary_db_id : null
}

AWS Systems Manager Parameter Store

# Environment-specific parameters
resource "aws_ssm_parameter" "database_url" {
  name  = "/${var.environment}/database/url"
  type  = "SecureString"
  value = var.environment == "production" ? 
          "prod-db.example.com" : 
          "${var.environment}-db.example.com"
  
  kms_key_id = aws_kms_key.ssm_parameters.id
  
  tags = {
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

# Production secrets have additional protection
resource "aws_ssm_parameter" "api_key" {
  name  = "/${var.environment}/api/key"
  type  = "SecureString"
  value = var.api_key
  
  kms_key_id = aws_kms_key.ssm_parameters.id
  
  # Production: Cannot delete without 30-day recovery window
  overwrite = var.environment == "production" ? false : true
  
  lifecycle {
    prevent_destroy = var.environment == "production" ? true : false
  }
}

Promotion Workflows

CI/CD Pipeline with Environment Promotion

# .github/workflows/deploy.yml
name: Deploy Application

on:
  push:
    branches: [main]

jobs:
  deploy-dev:
    name: Deploy to Development
    runs-on: ubuntu-latest
    environment:
      name: development
      url: https://dev.api.example.com
    steps:
      - uses: actions/checkout@v3
      
      - name: Deploy to Dev
        run: |
          ./deploy.sh development
  
  integration-tests:
    name: Integration Tests
    runs-on: ubuntu-latest
    needs: deploy-dev
    steps:
      - uses: actions/checkout@v3
      
      - name: Run Integration Tests
        run: |
          npm run test:integration
        env:
          API_URL: https://dev.api.example.com
  
  deploy-staging:
    name: Deploy to Staging
    runs-on: ubuntu-latest
    needs: integration-tests
    environment:
      name: staging
      url: https://staging.api.example.com
    steps:
      - uses: actions/checkout@v3
      
      - name: Deploy to Staging
        run: |
          ./deploy.sh staging
  
  smoke-tests:
    name: Smoke Tests
    runs-on: ubuntu-latest
    needs: deploy-staging
    steps:
      - name: Run Smoke Tests
        run: |
          npm run test:smoke
        env:
          API_URL: https://staging.api.example.com
  
  deploy-production:
    name: Deploy to Production
    runs-on: ubuntu-latest
    needs: smoke-tests
    environment:
      name: production
      url: https://api.example.com
    steps:
      - uses: actions/checkout@v3
      
      # Production requires manual approval
      - name: Wait for Approval
        uses: trstringer/manual-approval@v1
        with:
          secret: ${{ secrets.GITHUB_TOKEN }}
          approvers: platform-team
          minimum-approvals: 2
      
      - name: Deploy to Production
        run: |
          ./deploy.sh production
      
      - name: Verify Deployment
        run: |
          npm run test:smoke
        env:
          API_URL: https://api.example.com
      
      - name: Notify Slack
        if: always()
        uses: slackapi/slack-github-action@v1
        with:
          webhook-url: ${{ secrets.SLACK_WEBHOOK }}
          payload: |
            {
              "text": "Production deployment completed",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "✅ *Production Deployment*\n\nVersion: ${{ github.sha }}\nStatus: Success"
                  }
                }
              ]
            }

Blue/Green Deployment for Production

# Blue/Green deployment using Route53 weighted routing
resource "aws_route53_record" "production_blue" {
  zone_id = var.hosted_zone_id
  name    = "api.example.com"
  type    = "A"
  
  alias {
    name                   = aws_lb.blue.dns_name
    zone_id               = aws_lb.blue.zone_id
    evaluate_target_health = true
  }
  
  set_identifier = "blue"
  weighted_routing_policy {
    weight = var.blue_weight  # Start at 100, gradually shift to 0
  }
}

resource "aws_route53_record" "production_green" {
  zone_id = var.hosted_zone_id
  name    = "api.example.com"
  type    = "A"
  
  alias {
    name                   = aws_lb.green.dns_name
    zone_id               = aws_lb.green.zone_id
    evaluate_target_health = true
  }
  
  set_identifier = "green"
  weighted_routing_policy {
    weight = var.green_weight  # Start at 0, gradually shift to 100
  }
}

# Gradual traffic shift: 10% → 50% → 100%
# If errors detected, instant rollback by adjusting weights

Data Management Across Environments

Synthetic Data for Non-Production

# scripts/generate_synthetic_data.py
from faker import Faker
import psycopg2
import os

fake = Faker()

def generate_synthetic_users(count=10000):
    """Generate synthetic user data for dev/staging"""
    
    conn = psycopg2.connect(os.environ['DATABASE_URL'])
    cursor = conn.cursor()
    
    for _ in range(count):
        cursor.execute("""
            INSERT INTO users (
                email, 
                first_name, 
                last_name, 
                phone, 
                address
            ) VALUES (%s, %s, %s, %s, %s)
        """, (
            fake.email(),
            fake.first_name(),
            fake.last_name(),
            fake.phone_number(),
            fake.address()
        ))
    
    conn.commit()
    print(f"Generated {count} synthetic users")

# NEVER copy production data to dev/staging
# Always use synthetic data generation

Production Data Anonymization

# scripts/anonymize_production_snapshot.py
def anonymize_database_snapshot(snapshot_id):
    """
    Anonymize production database snapshot 
    before restoring to staging
    """
    
    # 1. Restore snapshot to temporary instance
    temp_db = restore_snapshot(snapshot_id)
    
    # 2. Anonymize PII
    conn = psycopg2.connect(temp_db.connection_string)
    cursor = conn.cursor()
    
    # Anonymize emails
    cursor.execute("""
        UPDATE users
        SET email = CONCAT('user', id, '@example.com')
    """)
    
    # Anonymize names
    cursor.execute("""
        UPDATE users
        SET first_name = CONCAT('User', id),
            last_name = 'Anonymous'
    """)
    
    # Anonymize phone numbers
    cursor.execute("""
        UPDATE users
        SET phone = CONCAT('555-', LPAD(id::TEXT, 7, '0'))
    """)
    
    # Delete payment information
    cursor.execute("TRUNCATE TABLE payment_methods CASCADE")
    
    conn.commit()
    
    # 3. Create new snapshot from anonymized data
    anonymized_snapshot = create_snapshot(temp_db)
    
    # 4. Delete temporary instance
    delete_instance(temp_db)
    
    return anonymized_snapshot

Cost Optimization Per Environment

# Production: Always-on, highly available
resource "aws_autoscaling_group" "production" {
  min_size         = 3
  max_size         = 20
  desired_capacity = 5
  
  # Never scale to zero
  lifecycle {
    ignore_changes = [desired_capacity]
  }
}

# Staging: Business hours only
resource "aws_autoscaling_schedule" "staging_scale_down" {
  scheduled_action_name  = "staging-scale-down"
  autoscaling_group_name = aws_autoscaling_group.staging.name
  
  # Scale down after hours (7 PM)
  recurrence = "0 19 * * 1-5"  # Mon-Fri 7 PM
  
  min_size         = 0
  max_size         = 10
  desired_capacity = 0
}

resource "aws_autoscaling_schedule" "staging_scale_up" {
  scheduled_action_name  = "staging-scale-up"
  autoscaling_group_name = aws_autoscaling_group.staging.name
  
  # Scale up business hours (7 AM)
  recurrence = "0 7 * * 1-5"  # Mon-Fri 7 AM
  
  min_size         = 2
  max_size         = 10
  desired_capacity = 2
}

# Development: On-demand only
resource "aws_autoscaling_group" "development" {
  min_size         = 0
  max_size         = 3
  desired_capacity = 0  # Default to zero, developers start when needed
}

What I Learned

Lesson 1: Account Isolation Prevents Disasters

Every production incident I've seen could have been prevented with proper account separation.

Action: Production OU with strict SCPs, staging OU, development OU. Never mix.

Lesson 2: Network Isolation is Critical

Dev environments should NEVER have network access to production databases.

Action: Separate VPCs, Transit Gateway route table isolation, no peering between environments.

Lesson 3: IAM Policies Must Differ by Environment

Developers need admin in dev, read-only in production (with break-glass for emergencies).

Action: Environment-specific IAM roles, MFA required for production, approval workflows.

Lesson 4: Configuration as Code Prevents Drift

Environment-specific configuration in code, not manual.

Action: Terraform locals, SSM Parameter Store, environment-specific tfvars files.

Lesson 5: Promotion Workflows Ensure Quality

Code must pass tests in dev → staging before reaching production.

Action: CI/CD pipelines with mandatory environment progression, automated testing, manual approval for production.

Lesson 6: Never Copy Production Data to Non-Production

GDPR, HIPAA, PCI-DSS violations waiting to happen.

Action: Synthetic data for dev/staging, anonymization if production snapshots required.

Lesson 7: Cost Optimization Differs by Environment

Production: always-on. Staging: business hours. Dev: on-demand.

Action: Autoscaling schedules, smaller instance types in non-prod, auto-shutdown after hours.

---

Next: Security Operations and Threat Protection - SOC integration, threat detection, incident response, vulnerability management.