Production Workflows and Team Collaboration

The day we deployed to production without approval taught me that workflows prevent disasters

---

Table of Contents

1. Introduction: The Accidental Production Deploy

2. Why Production Workflows Matter

3. GitOps Principles

4. Pull Request Workflow

5. Automated Plan Comments

6. Approval Gates

7. Deployment Strategies

8. Rollback Procedures

9. Real-World Example: Complete CI/CD Pipeline

10. Environment Promotion

11. Disaster Recovery

12. Monitoring Deployments

13. Common Workflow Patterns

14. Production Best Practices

15. What I Learned About Production Workflows

16. Next Steps

---

Introduction: The Accidental Production Deploy

It was a Friday afternoon. I was testing a change in our development environment. I ran terraform apply and confirmed.

30 seconds later, Slack exploded:

ALERT: Production database deleted ALERT: Production services down ALERT: All production traffic failing

My heart stopped. I looked at my terminal.

I was in the production directory.

I had just destroyed production infrastructure. On a Friday. At 4:45 PM.

The damage:

• 2 hours of complete downtime

• Data loss (fortunately, backups worked)

• $50,000 in revenue loss

• Weekend recovery effort

• Very angry customers

The root cause:

• No workflow guardrails

• Manual terraform commands

• No approval process

• Easy to make mistakes

We spent the next month implementing production workflows:

• GitOps: All changes through Git

• Automation: No manual terraform commands

• Approvals: Required reviews before deploy

• Safeguards: Multiple safety checks

That incident taught me: workflows prevent human error.

This article is everything I learned about building safe, automated production workflows for Terraform.

---

Why Production Workflows Matter

Manual Process Problems

Manual terraform workflow:

# Developer's laptop
cd production/
terraform plan
terraform apply  # 😰 One typo away from disaster

Risks:

• Wrong directory

• Wrong environment

• Wrong credentials

• No peer review

• No audit trail

• No rollback plan

Automated Workflow Benefits

GitOps workflow:

Developer → PR → Automated Tests → Review → Approval → Automated Deploy

Benefits:

• ✅ Peer review required

• ✅ Automated validation

• ✅ Audit trail in Git

• ✅ Consistent process

• ✅ Easy rollback

• ✅ No local terraform commands

Workflow Impact

---

GitOps Principles

Core Principles

1. Git is the source of truth

• All infrastructure defined in Git

• Deployed state matches Git state

• Changes only through Git commits

2. Declarative configuration

• Describe desired state, not steps

• Terraform handles how to get there

3. Automated deployment

• No manual terraform commands

• CI/CD applies changes

• Humans approve, machines execute

4. Continuous reconciliation

• Detect drift

• Auto-remediate

• Alert on discrepancies

GitOps Workflow

Repository Structure

terraform-infrastructure/
├── .github/
│   └── workflows/
│       ├── pr-plan.yml          # PR validation
│       ├── main-apply.yml       # Deploy on merge
│       └── drift-detection.yml  # Detect drift
├── modules/
│   ├── networking/
│   ├── database/
│   └── application/
├── environments/
│   ├── dev/
│   │   ├── backend.tf
│   │   ├── main.tf
│   │   └── terraform.tfvars
│   ├── staging/
│   └── production/
├── scripts/
│   ├── plan.sh
│   ├── apply.sh
│   └── validate.sh
└── README.md

---

Pull Request Workflow

PR Template

.github/pull_request_template.md:

## Description
<!-- Describe what this PR changes and why -->

## Type of Change
- [ ] New resource
- [ ] Resource modification
- [ ] Resource deletion
- [ ] Module update
- [ ] Configuration change

## Environments Affected
- [ ] Development
- [ ] Staging
- [ ] Production

## Checklist
- [ ] Terraform fmt run
- [ ] Terraform validate passed
- [ ] Tests passing
- [ ] Documentation updated
- [ ] Breaking changes documented

## Plan Output
<!-- Automated plan will be posted below -->

## Rollback Plan
<!-- How to rollback if this causes issues -->

PR Validation Workflow

.github/workflows/pr-plan.yml:

name: Terraform PR Plan

on:
  pull_request:
    branches: [main]
    paths:
      - '**.tf'
      - '**.tfvars'
      - '.github/workflows/pr-plan.yml'

env:
  TF_VERSION: '1.6.0'

jobs:
  validate:
    name: Validate Terraform
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: ${{ env.TF_VERSION }}
      
      - name: Terraform Format Check
        run: terraform fmt -check -recursive
      
      - name: Terraform Init
        run: |
          for env in dev staging production; do
            cd environments/$env
            terraform init -backend=false
            cd ../..
          done
      
      - name: Terraform Validate
        run: |
          for env in dev staging production; do
            echo "Validating $env..."
            cd environments/$env
            terraform validate
            cd ../..
          done

  security:
    name: Security Scan
    runs-on: ubuntu-latest
    needs: validate
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      
      - name: Run tfsec
        uses: aquasecurity/[email protected]
        with:
          soft_fail: false
      
      - name: Run Checkov
        uses: bridgecrewio/checkov-action@master
        with:
          directory: .
          framework: terraform

  plan-dev:
    name: Plan - Development
    runs-on: ubuntu-latest
    needs: [validate, security]
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: ${{ env.TF_VERSION }}
          cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}
      
      - name: Terraform Init
        working-directory: environments/dev
        run: terraform init
      
      - name: Terraform Plan
        working-directory: environments/dev
        run: terraform plan -out=tfplan
      
      - name: Save Plan
        uses: actions/upload-artifact@v3
        with:
          name: tfplan-dev
          path: environments/dev/tfplan

  plan-staging:
    name: Plan - Staging
    runs-on: ubuntu-latest
    needs: [validate, security]
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: ${{ env.TF_VERSION }}
          cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}
      
      - name: Terraform Init
        working-directory: environments/staging
        run: terraform init
      
      - name: Terraform Plan
        working-directory: environments/staging
        run: terraform plan -out=tfplan
      
      - name: Save Plan
        uses: actions/upload-artifact@v3
        with:
          name: tfplan-staging
          path: environments/staging/tfplan

  plan-production:
    name: Plan - Production
    runs-on: ubuntu-latest
    needs: [validate, security]
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: ${{ env.TF_VERSION }}
          cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}
      
      - name: Terraform Init
        working-directory: environments/production
        run: terraform init
      
      - name: Terraform Plan
        working-directory: environments/production
        run: terraform plan -out=tfplan
      
      - name: Check for Destructive Changes
        working-directory: environments/production
        run: |
          terraform show -json tfplan > plan.json
          if grep -q '"delete"' plan.json; then
            echo "⚠️ WARNING: Plan contains destructive changes"
            echo "destructive_changes=true" >> $GITHUB_ENV
          fi
      
      - name: Save Plan
        uses: actions/upload-artifact@v3
        with:
          name: tfplan-production
          path: environments/production/tfplan

---

Automated Plan Comments

Post plan output to PR for review.

Plan Comment Action

.github/workflows/post-plan-comment.yml:

name: Post Plan Comment

on:
  workflow_run:
    workflows: ["Terraform PR Plan"]
    types: [completed]

jobs:
  comment:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - name: Download Plan Artifacts
        uses: actions/download-artifact@v3
        with:
          name: tfplan-production
          path: plans/
      
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
      
      - name: Generate Plan Summary
        id: plan
        run: |
          cd plans
          terraform show -no-color tfplan > plan_output.txt
          
          # Create summary
          {
            echo "## Terraform Plan - Production"
            echo ""
            echo "### Changes Summary"
            echo '```'
            terraform show -json tfplan | jq -r '
              .resource_changes | 
              group_by(.change.actions[0]) | 
              map({action: .[0].change.actions[0], count: length}) |
              .[] | 
              "\(.action): \(.count)"
            '
            echo '```'
            echo ""
            echo "### Full Plan Output"
            echo '<details><summary>Click to expand</summary>'
            echo ""
            echo '```terraform'
            cat plan_output.txt
            echo '```'
            echo '</details>'
          } > plan_comment.md
      
      - name: Comment on PR
        uses: actions/github-script@v6
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            const fs = require('fs');
            const planComment = fs.readFileSync('plans/plan_comment.md', 'utf8');
            
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: planComment
            });

Example Plan Comment

## Terraform Plan - Production

### Changes Summary

create: 3 update: 1 delete: 0

### Resources Affected
- ✅ `local_file.web_config` will be created
- ✅ `local_file.api_config` will be created
- ✅ `local_file.worker_config` will be created
- ⚠️ `local_file.database_config` will be updated

### Full Plan Output
<details><summary>Click to expand</summary>

```terraform
Terraform will perform the following actions:

  # local_file.web_config will be created
  + resource "local_file" "web_config" {
      + content = "..."
      + filename = "web.conf"
    }

  ...

⚠️ Review Checklist

• Plan output reviewed

• Changes are intentional

• Rollback plan documented

• Approved for production

---

## Approval Gates

Require approvals before deploying.

### GitHub Branch Protection

**Settings → Branches → Branch protection rules:**
```yaml
Branch name pattern: main

Require pull request reviews before merging: ✅
  Required approvals: 2
  Dismiss stale reviews: ✅
  Require review from code owners: ✅

Require status checks to pass: ✅
  Required checks:
    - validate
    - security
    - plan-dev
    - plan-staging
    - plan-production

Require conversation resolution: ✅
Require signed commits: ✅
Include administrators: ✅

CODEOWNERS File

.github/CODEOWNERS:

# Global owners
* @infrastructure-team

# Environment-specific owners
/environments/dev/ @dev-team
/environments/staging/ @staging-team
/environments/production/ @production-team @infrastructure-lead

# Module owners
/modules/networking/ @network-team
/modules/database/ @database-team
/modules/application/ @app-team

Manual Approval Step

.github/workflows/deploy-production.yml:

name: Deploy to Production

on:
  push:
    branches: [main]
    paths:
      - 'environments/production/**'

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment:
      name: production
      url: https://prod.example.com
    
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
      
      - name: Terraform Apply
        working-directory: environments/production
        run: |
          terraform init
          terraform apply -auto-approve
      
      - name: Notify Success
        uses: 8398a7/action-slack@v3
        with:
          status: ${{ job.status }}
          text: 'Production deployment successful'
          webhook_url: ${{ secrets.SLACK_WEBHOOK }}

GitHub Environment Protection:

• Required reviewers: @infrastructure-lead, @security-team

• Wait timer: 5 minutes (thinking time)

• Deployment branches: main only

---

Deployment Strategies

Strategy 1: Blue-Green Deployment

Deploy new version alongside old, then switch.

variable "active_version" {
  type    = string
  default = "blue"
  
  validation {
    condition     = contains(["blue", "green"], var.active_version)
    error_message = "Active version must be blue or green."
  }
}

# Blue environment
module "blue" {
  source = "../../modules/service"
  
  name    = "service-blue"
  version = "v1.0.0"
  active  = var.active_version == "blue"
}

# Green environment
module "green" {
  source = "../../modules/service"
  
  name    = "service-green"
  version = "v2.0.0"
  active  = var.active_version == "green"
}

# Load balancer points to active
resource "local_file" "load_balancer" {
  filename = "load-balancer.conf"
  content  = <<-EOT
    upstream backend {
      server ${var.active_version == "blue" ? module.blue.endpoint : module.green.endpoint};
    }
  EOT
}

Deployment process:

# Deploy green (v2.0.0) alongside blue
terraform apply -var="active_version=blue"

# Test green
curl http://service-green.internal

# Switch traffic to green
terraform apply -var="active_version=green"

# If issues, rollback to blue
terraform apply -var="active_version=blue"

Strategy 2: Canary Deployment

Gradually shift traffic to new version.

variable "canary_percentage" {
  type    = number
  default = 0
  
  validation {
    condition     = var.canary_percentage >= 0 && var.canary_percentage <= 100
    error_message = "Canary percentage must be between 0 and 100."
  }
}

module "stable" {
  source = "../../modules/service"
  
  name     = "service-stable"
  version  = "v1.0.0"
  replicas = max(1, floor((100 - var.canary_percentage) / 100 * 5))
}

module "canary" {
  source = "../../modules/service"
  
  name     = "service-canary"
  version  = "v2.0.0"
  replicas = floor(var.canary_percentage / 100 * 5)
}

resource "local_file" "load_balancer" {
  filename = "load-balancer.conf"
  content  = <<-EOT
    upstream backend {
      ${join("\n      ", [
        for i in range(module.stable.replicas) : 
        "server stable-${i}.internal;"
      ])}
      ${join("\n      ", [
        for i in range(module.canary.replicas) : 
        "server canary-${i}.internal;"
      ])}
    }
  EOT
}

Gradual rollout:

# 0% canary (all stable)
terraform apply -var="canary_percentage=0"

# 20% canary
terraform apply -var="canary_percentage=20"

# Monitor metrics...

# 50% canary
terraform apply -var="canary_percentage=50"

# 100% canary (all new version)
terraform apply -var="canary_percentage=100"

# Promote canary to stable
# (swap versions and reset to 0%)

Strategy 3: Rolling Deployment

Update instances one at a time.

variable "deployment_batch_size" {
  type    = number
  default = 1
}

variable "current_version" {
  type = string
}

locals {
  instances = range(var.instance_count)
  batches   = chunklist(local.instances, var.deployment_batch_size)
}

resource "local_file" "instance" {
  for_each = toset([for i in local.instances : tostring(i)])
  
  filename = "instance-${each.key}.conf"
  content  = <<-EOT
    version: ${var.current_version}
    instance: ${each.key}
  EOT
  
  lifecycle {
    create_before_destroy = true
  }
}

Strategy 4: Terraform Workspaces

Use workspaces for environment isolation.

# Create workspaces
terraform workspace new dev
terraform workspace new staging
terraform workspace new production

# Switch workspace
terraform workspace select production

# Deploy to current workspace
terraform apply

Workspace-aware configuration:

locals {
  environment = terraform.workspace
  
  config = {
    dev = {
      instance_count = 1
      instance_size  = "small"
    }
    staging = {
      instance_count = 2
      instance_size  = "medium"
    }
    production = {
      instance_count = 5
      instance_size  = "large"
    }
  }
  
  env_config = local.config[local.environment]
}

resource "local_file" "config" {
  filename = "${local.environment}-config.yaml"
  content  = yamlencode(local.env_config)
}

---

Rollback Procedures

Rollback via Git Revert

# Find commit to revert
git log --oneline

# Revert the problematic commit
git revert abc123

# Push revert
git push origin main

# CI/CD automatically applies the revert

Rollback via State

# List state versions
terraform state list

# Pull specific state version
terraform state pull > previous-state.json

# Restore previous state
terraform state push previous-state.json

# Apply to match previous state
terraform apply

Rollback Automation

.github/workflows/rollback.yml:

name: Rollback Production

on:
  workflow_dispatch:
    inputs:
      commit_sha:
        description: 'Commit SHA to rollback to'
        required: true
      environment:
        description: 'Environment to rollback'
        required: true
        type: choice
        options:
          - dev
          - staging
          - production

jobs:
  rollback:
    runs-on: ubuntu-latest
    environment: ${{ github.event.inputs.environment }}
    
    steps:
      - name: Checkout at Commit
        uses: actions/checkout@v3
        with:
          ref: ${{ github.event.inputs.commit_sha }}
      
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
      
      - name: Terraform Apply (Rollback)
        working-directory: environments/${{ github.event.inputs.environment }}
        run: |
          terraform init
          terraform apply -auto-approve
      
      - name: Notify Rollback
        uses: 8398a7/action-slack@v3
        with:
          status: ${{ job.status }}
          text: |
            ⚠️ ROLLBACK executed for ${{ github.event.inputs.environment }}
            Rolled back to commit: ${{ github.event.inputs.commit_sha }}
          webhook_url: ${{ secrets.SLACK_WEBHOOK }}

Trigger rollback:

1. Go to Actions → Rollback Production

2. Click "Run workflow"

3. Enter commit SHA to rollback to

4. Select environment

5. Confirm

---

Real-World Example: Complete CI/CD Pipeline

Full production-ready pipeline.

Repository Structure

terraform-production/
├── .github/
│   ├── workflows/
│   │   ├── pr-plan.yml
│   │   ├── deploy.yml
│   │   ├── drift-detection.yml
│   │   └── rollback.yml
│   ├── CODEOWNERS
│   └── pull_request_template.md
├── modules/
├── environments/
│   ├── dev/
│   ├── staging/
│   └── production/
├── scripts/
│   ├── plan.sh
│   ├── apply.sh
│   ├── validate.sh
│   └── notify.sh
└── docs/
    ├── DEPLOYMENT.md
    └── ROLLBACK.md

Complete Deploy Workflow

.github/workflows/deploy.yml:

name: Deploy Infrastructure

on:
  push:
    branches: [main]
  workflow_dispatch:

env:
  TF_VERSION: '1.6.0'

jobs:
  changes:
    name: Detect Changes
    runs-on: ubuntu-latest
    outputs:
      dev: ${{ steps.filter.outputs.dev }}
      staging: ${{ steps.filter.outputs.staging }}
      production: ${{ steps.filter.outputs.production }}
    steps:
      - uses: actions/checkout@v3
      - uses: dorny/paths-filter@v2
        id: filter
        with:
          filters: |
            dev:
              - 'environments/dev/**'
            staging:
              - 'environments/staging/**'
            production:
              - 'environments/production/**'

  deploy-dev:
    name: Deploy - Development
    runs-on: ubuntu-latest
    needs: changes
    if: needs.changes.outputs.dev == 'true'
    environment:
      name: dev
      url: https://dev.example.com
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: ${{ env.TF_VERSION }}
          cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}
      
      - name: Terraform Init
        working-directory: environments/dev
        run: terraform init
      
      - name: Terraform Apply
        working-directory: environments/dev
        run: terraform apply -auto-approve
      
      - name: Notify Success
        if: success()
        run: |
          ./scripts/notify.sh success dev "${{ github.sha }}"
      
      - name: Notify Failure
        if: failure()
        run: |
          ./scripts/notify.sh failure dev "${{ github.sha }}"

  deploy-staging:
    name: Deploy - Staging
    runs-on: ubuntu-latest
    needs: [changes, deploy-dev]
    if: needs.changes.outputs.staging == 'true'
    environment:
      name: staging
      url: https://staging.example.com
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: ${{ env.TF_VERSION }}
          cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}
      
      - name: Terraform Init
        working-directory: environments/staging
        run: terraform init
      
      - name: Terraform Apply
        working-directory: environments/staging
        run: terraform apply -auto-approve
      
      - name: Run Smoke Tests
        run: |
          ./scripts/smoke-test.sh staging
      
      - name: Notify Success
        if: success()
        run: |
          ./scripts/notify.sh success staging "${{ github.sha }}"
      
      - name: Notify Failure
        if: failure()
        run: |
          ./scripts/notify.sh failure staging "${{ github.sha }}"

  deploy-production:
    name: Deploy - Production
    runs-on: ubuntu-latest
    needs: [changes, deploy-staging]
    if: needs.changes.outputs.production == 'true'
    environment:
      name: production
      url: https://prod.example.com
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: ${{ env.TF_VERSION }}
          cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}
      
      - name: Terraform Init
        working-directory: environments/production
        run: terraform init
      
      - name: Create Backup
        run: |
          ./scripts/backup-state.sh production
      
      - name: Terraform Apply
        working-directory: environments/production
        run: terraform apply -auto-approve
      
      - name: Run Smoke Tests
        run: |
          ./scripts/smoke-test.sh production
      
      - name: Monitor Metrics
        run: |
          ./scripts/monitor-deployment.sh production 300
      
      - name: Notify Success
        if: success()
        run: |
          ./scripts/notify.sh success production "${{ github.sha }}"
      
      - name: Auto Rollback
        if: failure()
        run: |
          echo "Deployment failed, initiating rollback..."
          ./scripts/rollback.sh production
      
      - name: Notify Failure
        if: failure()
        run: |
          ./scripts/notify.sh failure production "${{ github.sha }}"

Drift Detection

.github/workflows/drift-detection.yml:

name: Detect Drift

on:
  schedule:
    - cron: '0 */6 * * *'  # Every 6 hours
  workflow_dispatch:

jobs:
  detect-drift:
    name: Detect Drift - ${{ matrix.environment }}
    runs-on: ubuntu-latest
    strategy:
      matrix:
        environment: [dev, staging, production]
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
      
      - name: Terraform Init
        working-directory: environments/${{ matrix.environment }}
        run: terraform init
      
      - name: Terraform Plan
        id: plan
        working-directory: environments/${{ matrix.environment }}
        run: |
          terraform plan -detailed-exitcode -no-color
        continue-on-error: true
      
      - name: Check for Drift
        if: steps.plan.outputs.exitcode == 2
        run: |
          echo "⚠️ DRIFT DETECTED in ${{ matrix.environment }}"
          
          # Notify team
          curl -X POST ${{ secrets.SLACK_WEBHOOK }} \
            -H 'Content-Type: application/json' \
            -d '{
              "text": "⚠️ Infrastructure drift detected in ${{ matrix.environment }}",
              "blocks": [{
                "type": "section",
                "text": {
                  "type": "mrkdwn",
                  "text": "*Drift Detected*\nEnvironment: ${{ matrix.environment }}\nAction: Review and reconcile"
                }
              }]
            }'

---

Environment Promotion

Promotion Strategy

Promotion Workflow

.github/workflows/promote.yml:

name: Promote to Production

on:
  workflow_dispatch:
    inputs:
      staging_version:
        description: 'Staging version to promote'
        required: true

jobs:
  promote:
    runs-on: ubuntu-latest
    environment: production
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Verify Staging Version
        run: |
          cd environments/staging
          CURRENT=$(terraform output -raw version)
          if [ "$CURRENT" != "${{ github.event.inputs.staging_version }}" ]; then
            echo "❌ Version mismatch"
            exit 1
          fi
      
      - name: Copy Configuration
        run: |
          # Copy staging config to production
          cp environments/staging/main.tf environments/production/main.tf
          
          # Update environment variable
          sed -i 's/staging/production/g' environments/production/main.tf
      
      - name: Create PR
        uses: peter-evans/create-pull-request@v5
        with:
          commit-message: "Promote staging version ${{ github.event.inputs.staging_version }} to production"
          title: "Production Release: v${{ github.event.inputs.staging_version }}"
          body: |
            ## Production Promotion
            
            Promoting staging version ${{ github.event.inputs.staging_version }} to production.
            
            ### Checklist
            - [ ] Staging tests passed
            - [ ] Security scan passed
            - [ ] Performance verified
            - [ ] Rollback plan ready
          branch: promote-${{ github.event.inputs.staging_version }}

---

Disaster Recovery

Backup Strategy

scripts/backup-state.sh:

#!/bin/bash
set -e

ENVIRONMENT=$1
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_DIR="backups/${ENVIRONMENT}"

mkdir -p "$BACKUP_DIR"

cd "environments/${ENVIRONMENT}"

# Pull current state
terraform state pull > "${BACKUP_DIR}/terraform-${TIMESTAMP}.tfstate"

# Compress
gzip "${BACKUP_DIR}/terraform-${TIMESTAMP}.tfstate"

# Upload to S3 (or other backup location)
aws s3 cp \
  "${BACKUP_DIR}/terraform-${TIMESTAMP}.tfstate.gz" \
  "s3://terraform-backups/${ENVIRONMENT}/terraform-${TIMESTAMP}.tfstate.gz"

echo "✅ State backed up to s3://terraform-backups/${ENVIRONMENT}/"

# Keep only last 30 days locally
find "$BACKUP_DIR" -name "*.tfstate.gz" -mtime +30 -delete

Recovery Procedure

scripts/recover-state.sh:

#!/bin/bash
set -e

ENVIRONMENT=$1
BACKUP_FILE=$2

if [ -z "$BACKUP_FILE" ]; then
  echo "Usage: $0 <environment> <backup-file>"
  exit 1
fi

echo "⚠️  WARNING: This will restore state from backup"
echo "Environment: $ENVIRONMENT"
echo "Backup: $BACKUP_FILE"
read -p "Continue? (yes/no): " CONFIRM

if [ "$CONFIRM" != "yes" ]; then
  echo "Aborted"
  exit 1
fi

cd "environments/${ENVIRONMENT}"

# Backup current state first
terraform state pull > "current-state-$(date +%Y%m%d-%H%M%S).tfstate"

# Download backup
aws s3 cp "$BACKUP_FILE" - | gunzip > restored-state.tfstate

# Restore
terraform state push restored-state.tfstate

echo "✅ State restored from $BACKUP_FILE"
echo "⚠️  Run 'terraform plan' to verify"

Automated Backups

.github/workflows/backup-state.yml:

name: Backup State

on:
  schedule:
    - cron: '0 0 * * *'  # Daily at midnight
  workflow_dispatch:

jobs:
  backup:
    name: Backup - ${{ matrix.environment }}
    runs-on: ubuntu-latest
    strategy:
      matrix:
        environment: [dev, staging, production]
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Configure AWS
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      
      - name: Backup State
        run: |
          ./scripts/backup-state.sh ${{ matrix.environment }}

---

Monitoring Deployments

Deployment Metrics

scripts/monitor-deployment.sh:

#!/bin/bash
set -e

ENVIRONMENT=$1
DURATION=${2:-300}  # Monitor for 5 minutes by default

echo "Monitoring $ENVIRONMENT deployment for ${DURATION}s..."

START_TIME=$(date +%s)
END_TIME=$((START_TIME + DURATION))

while [ $(date +%s) -lt $END_TIME ]; do
  # Check error rate
  ERROR_RATE=$(curl -s "http://metrics.${ENVIRONMENT}.internal/error_rate")
  
  if [ "$ERROR_RATE" -gt 5 ]; then
    echo "❌ High error rate detected: ${ERROR_RATE}%"
    exit 1
  fi
  
  # Check response time
  RESPONSE_TIME=$(curl -s "http://metrics.${ENVIRONMENT}.internal/avg_response_time")
  
  if [ "$RESPONSE_TIME" -gt 1000 ]; then
    echo "❌ High response time detected: ${RESPONSE_TIME}ms"
    exit 1
  fi
  
  echo "✓ Metrics OK - Error rate: ${ERROR_RATE}%, Response time: ${RESPONSE_TIME}ms"
  
  sleep 30
done

echo "✅ Deployment monitoring complete - all metrics healthy"

Health Checks

scripts/smoke-test.sh:

#!/bin/bash
set -e

ENVIRONMENT=$1

echo "Running smoke tests for $ENVIRONMENT..."

# Test endpoints
ENDPOINTS=(
  "http://web.${ENVIRONMENT}.internal/health"
  "http://api.${ENVIRONMENT}.internal/health"
)

for endpoint in "${ENDPOINTS[@]}"; do
  echo "Testing $endpoint..."
  
  response=$(curl -s -o /dev/null -w "%{http_code}" "$endpoint")
  
  if [ "$response" != "200" ]; then
    echo "❌ Health check failed: $endpoint (HTTP $response)"
    exit 1
  fi
  
  echo "✅ $endpoint is healthy"
done

echo "✅ All smoke tests passed"

---

Common Workflow Patterns

Pattern 1: Feature Branch Workflow

main (protected)
  ↑
  └── feature/add-monitoring
        ↑
        └── PR (plan, test, review, merge)

Pattern 2: GitFlow

main (production)
  ↑
  develop
    ↑
    └── feature/new-service
          ↑
          └── PR to develop

Pattern 3: Trunk-Based

main
  ↑
  ├── short-lived branches
  ├── frequent merges
  └── feature flags

Pattern 4: Environment Branches

production (main)
staging (auto-merge from dev)
dev (auto-deploy on merge)
  ↑
  └── feature branches

---

Production Best Practices

1. Never Run Terraform Manually

✅ All changes through CI/CD ❌ No terraform apply from laptops

2. Require Reviews

✅ Peer review for all changes ✅ CODEOWNERS for ownership ✅ Approval gates for production

3. Test Before Deploy

✅ Validate in PR ✅ Plan before apply ✅ Test in lower environments

4. Monitor Everything

✅ Deployment metrics ✅ Drift detection ✅ Health checks ✅ Alert on failures

5. Automate Rollback

✅ Fast rollback procedure ✅ Tested rollback path ✅ Automated health checks ✅ Circuit breakers

6. Document Procedures

✅ Deployment runbooks ✅ Rollback procedures ✅ Disaster recovery plans ✅ On-call guides

---

What I Learned About Production Workflows

1. Workflows Prevent Human Error

That Friday afternoon production incident taught me: humans make mistakes, workflows prevent them.

Automation removes risk.

2. Approvals Save Production

Never deploy without approval.

Peer review catches what you missed.

3. Rollback Must Be Fast

When things break, every second counts.

Automate rollback procedures.

4. Testing is Non-Negotiable

Test in dev, staging before production.

Automated tests catch issues early.

5. Monitoring Enables Confidence

You can't deploy what you can't monitor.

Metrics tell you when to rollback.

6. Documentation Saves Time

When production is down, there's no time to figure it out.

Document procedures beforehand.

7. GitOps Provides Audit Trail

Git history shows what changed and why.

Essential for troubleshooting and compliance.

---

Next Steps

Congratulations! You've mastered production workflows:

✅ GitOps principles ✅ Pull request workflow ✅ Automated plan comments ✅ Approval gates ✅ Deployment strategies ✅ Rollback procedures ✅ Complete CI/CD pipeline ✅ Disaster recovery

Practice Exercises

Exercise 1: Build CI/CD Pipeline

1. Set up GitHub Actions
2. Automate plan on PR
3. Require approvals
4. Automate apply on merge
5. Test rollback

Exercise 2: Implement GitOps

1. Remove manual terraform commands
2. All changes through Git
3. Automated deployments
4. Monitor drift

Exercise 3: Test Disaster Recovery

1. Backup state
2. Simulate disaster
3. Restore from backup
4. Verify recovery
5. Document procedure

Coming Up Next

In Article 12: Real-World Production Example - Putting It All Together, we'll build:

• Complete production infrastructure

• All concepts integrated

• Multi-environment setup

• Full CI/CD pipeline

• Monitoring and observability

• Production-ready example

This is where everything comes together!

---

This Week's Challenge: Implement automated Terraform workflows for your projects. Move from manual deploys to GitOps.

See you in the final article! 🚀

---

"That Friday afternoon accident—deploying to production instead of dev—cost us $50k and taught me that workflows aren't overhead, they're essential. Now every change goes through automated CI/CD. Zero manual deploys." - Me, GitOps believer