Part 5: Production Deployment and Best Practices

Introduction

Deploying to production is where GitHub Actions truly shines. I've learned this the hard way after a failed deployment at 2 AM that took down our payment service. Since then, I've refined my deployment workflows to include health checks, rollback mechanisms, and comprehensive monitoring.

In this final part, I'll share production-ready deployment strategies, security best practices, and lessons learned from running GitHub Actions in production.

Production Deployment Workflow

Complete Deployment Pipeline

name: Deploy to Production

on:
  release:
    types: [published]
  workflow_dispatch:
    inputs:
      environment:
        description: 'Environment to deploy'
        required: true
        type: choice
        options:
          - staging
          - production
      version:
        description: 'Version to deploy (leave empty for latest)'
        required: false
        type: string

env:
  NODE_VERSION: '18'
  AWS_REGION: 'us-east-1'

jobs:
  prepare:
    name: Prepare Deployment
    runs-on: ubuntu-latest
    outputs:
      version: ${{ steps.version.outputs.version }}
      environment: ${{ steps.env.outputs.environment }}
      should-deploy: ${{ steps.check.outputs.should-deploy }}
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Determine version
        id: version
        run: |
          if [ -n "${{ inputs.version }}" ]; then
            VERSION="${{ inputs.version }}"
          elif [ "${{ github.event_name }}" == "release" ]; then
            VERSION="${{ github.event.release.tag_name }}"
          else
            VERSION=$(cat package.json | jq -r .version)
          fi
          echo "version=$VERSION" >> $GITHUB_OUTPUT
          echo "🏷️ Deploying version: $VERSION"
      
      - name: Determine environment
        id: env
        run: |
          if [ -n "${{ inputs.environment }}" ]; then
            ENV="${{ inputs.environment }}"
          elif [ "${{ github.event_name }}" == "release" ]; then
            ENV="production"
          else
            ENV="staging"
          fi
          echo "environment=$ENV" >> $GITHUB_OUTPUT
          echo "🌍 Target environment: $ENV"
      
      - name: Check if should deploy
        id: check
        run: |
          # Add your deployment checks here
          # Example: check if version already deployed
          echo "should-deploy=true" >> $GITHUB_OUTPUT
  
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    needs: prepare
    if: needs.prepare.outputs.should-deploy == 'true'
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      
      - name: Login to Amazon ECR
        uses: aws-actions/amazon-ecr-login@v1
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_REGION: ${{ env.AWS_REGION }}
      
      - name: Build and push Docker image
        uses: docker/build-push-action@v4
        with:
          context: ./services/auth
          file: ./services/auth/Dockerfile
          push: true
          tags: |
            ${{ secrets.ECR_REGISTRY }}/auth-service:${{ needs.prepare.outputs.version }}
            ${{ secrets.ECR_REGISTRY }}/auth-service:latest
          cache-from: type=registry,ref=${{ secrets.ECR_REGISTRY }}/auth-service:buildcache
          cache-to: type=registry,ref=${{ secrets.ECR_REGISTRY }}/auth-service:buildcache,mode=max
          build-args: |
            NODE_ENV=production
            VERSION=${{ needs.prepare.outputs.version }}
  
  deploy:
    name: Deploy to ${{ needs.prepare.outputs.environment }}
    runs-on: ubuntu-latest
    needs: [prepare, build]
    environment:
      name: ${{ needs.prepare.outputs.environment }}
      url: ${{ steps.deploy.outputs.url }}
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}
      
      - name: Deploy to ECS
        id: deploy
        run: |
          # Update ECS task definition
          NEW_TASK_DEF=$(aws ecs describe-task-definition \
            --task-definition auth-service-${{ needs.prepare.outputs.environment }} \
            --query 'taskDefinition' | \
            jq --arg IMAGE "${{ secrets.ECR_REGISTRY }}/auth-service:${{ needs.prepare.outputs.version }}" \
            '.containerDefinitions[0].image = $IMAGE | 
             del(.taskDefinitionArn, .revision, .status, .requiresAttributes, .compatibilities, .registeredAt, .registeredBy)')
          
          # Register new task definition
          NEW_TASK_ARN=$(echo $NEW_TASK_DEF | \
            aws ecs register-task-definition --cli-input-json file:///dev/stdin | \
            jq -r '.taskDefinition.taskDefinitionArn')
          
          # Update service
          aws ecs update-service \
            --cluster my-cluster-${{ needs.prepare.outputs.environment }} \
            --service auth-service \
            --task-definition $NEW_TASK_ARN \
            --force-new-deployment
          
          # Wait for deployment to complete
          aws ecs wait services-stable \
            --cluster my-cluster-${{ needs.prepare.outputs.environment }} \
            --services auth-service
          
          # Get service URL
          URL="https://auth.${{ needs.prepare.outputs.environment }}.example.com"
          echo "url=$URL" >> $GITHUB_OUTPUT
          echo "✅ Deployed to: $URL"
      
      - name: Run database migrations
        run: |
          # Run migrations in ECS
          aws ecs run-task \
            --cluster my-cluster-${{ needs.prepare.outputs.environment }} \
            --task-definition auth-migrations \
            --launch-type FARGATE \
            --network-configuration "awsvpcConfiguration={subnets=[${{ secrets.SUBNET_IDS }}],securityGroups=[${{ secrets.SECURITY_GROUP_ID }}],assignPublicIp=ENABLED}"
          
          echo "✅ Database migrations completed"
      
      - name: Health check
        run: |
          echo "🏥 Running health check..."
          
          for i in {1..30}; do
            HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" ${{ steps.deploy.outputs.url }}/health)
            
            if [ $HTTP_CODE -eq 200 ]; then
              echo "✅ Health check passed!"
              exit 0
            fi
            
            echo "Attempt $i/30: HTTP $HTTP_CODE - Waiting 10s..."
            sleep 10
          done
          
          echo "❌ Health check failed after 5 minutes"
          exit 1
      
      - name: Smoke tests
        run: |
          echo "🧪 Running smoke tests..."
          
          # Test critical endpoints
          curl -f ${{ steps.deploy.outputs.url }}/api/v1/auth/health || exit 1
          curl -f ${{ steps.deploy.outputs.url }}/api/v1/auth/version || exit 1
          
          echo "✅ Smoke tests passed!"
  
  notify:
    name: Notify Deployment Status
    runs-on: ubuntu-latest
    needs: [prepare, deploy]
    if: always()
    
    steps:
      - name: Notify Slack
        uses: 8398a7/action-slack@v3
        with:
          status: ${{ needs.deploy.result }}
          text: |
            Deployment to ${{ needs.prepare.outputs.environment }}
            Version: ${{ needs.prepare.outputs.version }}
            Status: ${{ needs.deploy.result }}
          webhook_url: ${{ secrets.SLACK_WEBHOOK }}
      
      - name: Create deployment record
        if: needs.deploy.result == 'success'
        uses: actions/github-script@v6
        with:
          script: |
            await github.rest.repos.createDeployment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              ref: context.sha,
              environment: '${{ needs.prepare.outputs.environment }}',
              description: 'Deployed version ${{ needs.prepare.outputs.version }}',
              auto_merge: false,
              required_contexts: []
            });

Blue-Green Deployment

name: Blue-Green Deploy

on:
  workflow_dispatch:
    inputs:
      environment:
        required: true
        type: choice
        options:
          - staging
          - production

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Configure AWS
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      
      - name: Deploy to green environment
        id: deploy-green
        run: |
          # Deploy new version to green environment
          aws ecs update-service \
            --cluster my-cluster \
            --service auth-service-green \
            --task-definition auth-service:latest \
            --force-new-deployment
          
          # Wait for green to be stable
          aws ecs wait services-stable \
            --cluster my-cluster \
            --services auth-service-green
      
      - name: Health check green
        run: |
          for i in {1..30}; do
            if curl -f https://auth-green.example.com/health; then
              echo "✅ Green environment healthy"
              exit 0
            fi
            sleep 10
          done
          echo "❌ Green environment unhealthy"
          exit 1
      
      - name: Run tests against green
        run: |
          npm run test:e2e
        env:
          API_URL: https://auth-green.example.com
      
      - name: Switch traffic to green
        run: |
          # Update load balancer to point to green
          aws elbv2 modify-listener \
            --listener-arn ${{ secrets.LISTENER_ARN }} \
            --default-actions Type=forward,TargetGroupArn=${{ secrets.GREEN_TARGET_GROUP_ARN }}
          
          echo "✅ Traffic switched to green"
      
      - name: Monitor green for 5 minutes
        run: |
          echo "📊 Monitoring green environment..."
          sleep 300
          
          # Check error rate
          ERROR_RATE=$(aws cloudwatch get-metric-statistics \
            --namespace AWS/ECS \
            --metric-name HTTPCode_Target_5XX_Count \
            --dimensions Name=TargetGroup,Value=${{ secrets.GREEN_TARGET_GROUP_ARN }} \
            --statistics Sum \
            --start-time $(date -u -d '5 minutes ago' +%Y-%m-%dT%H:%M:%S) \
            --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
            --period 300 \
            --query 'Datapoints[0].Sum' \
            --output text)
          
          if [ "$ERROR_RATE" -gt 10 ]; then
            echo "❌ High error rate detected: $ERROR_RATE"
            exit 1
          fi
          
          echo "✅ Green environment stable"
      
      - name: Rollback on failure
        if: failure()
        run: |
          echo "🔄 Rolling back to blue..."
          
          # Switch traffic back to blue
          aws elbv2 modify-listener \
            --listener-arn ${{ secrets.LISTENER_ARN }} \
            --default-actions Type=forward,TargetGroupArn=${{ secrets.BLUE_TARGET_GROUP_ARN }}
          
          echo "✅ Rolled back to blue"
      
      - name: Decommission blue
        if: success()
        run: |
          echo "🗑️ Decommissioning old blue environment..."
          
          # Scale down blue
          aws ecs update-service \
            --cluster my-cluster \
            --service auth-service-blue \
            --desired-count 0

Canary Deployment

name: Canary Deploy

on:
  release:
    types: [published]

jobs:
  deploy-canary:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Deploy canary (10% traffic)
        run: |
          # Deploy new version
          kubectl set image deployment/auth-service \
            auth-service=${{ secrets.ECR_REGISTRY }}/auth-service:${{ github.event.release.tag_name }} \
            -n production
          
          # Update traffic split (90% stable, 10% canary)
          kubectl apply -f - <<EOF
          apiVersion: v1
          kind: Service
          metadata:
            name: auth-service
            namespace: production
          spec:
            selector:
              app: auth-service
            ports:
              - protocol: TCP
                port: 80
                targetPort: 3000
            sessionAffinity: None
          ---
          apiVersion: networking.k8s.io/v1
          kind: Ingress
          metadata:
            name: auth-service
            namespace: production
            annotations:
              nginx.ingress.kubernetes.io/canary: "true"
              nginx.ingress.kubernetes.io/canary-weight: "10"
          spec:
            rules:
              - host: auth.example.com
                http:
                  paths:
                    - path: /
                      pathType: Prefix
                      backend:
                        service:
                          name: auth-service-canary
                          port:
                            number: 80
          EOF
      
      - name: Monitor canary
        run: |
          echo "📊 Monitoring canary for 10 minutes..."
          
          for i in {1..20}; do
            # Check error rate
            ERROR_RATE=$(kubectl top pods -n production -l version=canary --no-headers | awk '{sum+=$3} END {print sum}')
            echo "Canary error rate: $ERROR_RATE%"
            
            if [ "$ERROR_RATE" -gt 5 ]; then
              echo "❌ Canary error rate too high"
              exit 1
            fi
            
            sleep 30
          done
          
          echo "✅ Canary looks good"
      
      - name: Increase canary traffic to 50%
        run: |
          kubectl patch ingress auth-service -n production \
            --type merge \
            -p '{"metadata":{"annotations":{"nginx.ingress.kubernetes.io/canary-weight":"50"}}}'
          
          sleep 300  # Monitor for 5 minutes
      
      - name: Full rollout
        run: |
          echo "🚀 Promoting canary to production"
          
          # Update main deployment
          kubectl set image deployment/auth-service \
            auth-service=${{ secrets.ECR_REGISTRY }}/auth-service:${{ github.event.release.tag_name }} \
            -n production
          
          # Remove canary
          kubectl delete ingress auth-service-canary -n production
      
      - name: Rollback canary
        if: failure()
        run: |
          echo "🔄 Rolling back canary..."
          
          # Remove canary ingress
          kubectl delete ingress auth-service-canary -n production
          
          # Scale down canary deployment
          kubectl scale deployment auth-service-canary -n production --replicas=0

Rollback Strategy

name: Rollback

on:
  workflow_dispatch:
    inputs:
      environment:
        required: true
        type: choice
        options:
          - staging
          - production
      version:
        description: 'Version to rollback to (leave empty for previous)'
        required: false
        type: string

jobs:
  rollback:
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment }}
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Get previous version
        id: previous
        run: |
          if [ -n "${{ inputs.version }}" ]; then
            VERSION="${{ inputs.version }}"
          else
            # Get previous deployment version
            VERSION=$(aws ecs describe-services \
              --cluster my-cluster-${{ inputs.environment }} \
              --services auth-service \
              --query 'services[0].deployments[1].taskDefinition' \
              --output text | \
              grep -oP '(?<=:)[0-9]+$')
          fi
          
          echo "version=$VERSION" >> $GITHUB_OUTPUT
          echo "🔄 Rolling back to version: $VERSION"
      
      - name: Confirm rollback
        run: |
          echo "⚠️ Rolling back ${{ inputs.environment }} to version ${{ steps.previous.outputs.version }}"
          echo "This action cannot be undone automatically."
      
      - name: Execute rollback
        run: |
          # Update ECS service to previous task definition
          aws ecs update-service \
            --cluster my-cluster-${{ inputs.environment }} \
            --service auth-service \
            --task-definition auth-service:${{ steps.previous.outputs.version }} \
            --force-new-deployment
          
          # Wait for rollback to complete
          aws ecs wait services-stable \
            --cluster my-cluster-${{ inputs.environment }} \
            --services auth-service
      
      - name: Verify rollback
        run: |
          # Health check
          URL="https://auth.${{ inputs.environment }}.example.com"
          
          for i in {1..30}; do
            if curl -f $URL/health; then
              echo "✅ Rollback successful"
              exit 0
            fi
            sleep 10
          done
          
          echo "❌ Rollback verification failed"
          exit 1
      
      - name: Notify rollback
        uses: 8398a7/action-slack@v3
        with:
          status: custom
          custom_payload: |
            {
              text: "🔄 Rollback executed on ${{ inputs.environment }}",
              attachments: [{
                color: 'warning',
                text: `Rolled back to version ${{ steps.previous.outputs.version }}\nTriggered by: ${{ github.actor }}`
              }]
            }
          webhook_url: ${{ secrets.SLACK_WEBHOOK }}

Security Best Practices

Secret Scanning

name: Security Scan

on:
  push:
  pull_request:
  schedule:
    - cron: '0 0 * * 0'  # Weekly

jobs:
  secret-scan:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      
      - name: TruffleHog Secret Scan
        uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: ${{ github.event.repository.default_branch }}
          head: HEAD
  
  dependency-scan:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Run Snyk
        uses: snyk/actions/node@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          args: --severity-threshold=high
      
      - name: Upload results to GitHub Security
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: snyk.sarif
  
  container-scan:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Build image
        run: docker build -t auth-service:latest ./services/auth
      
      - name: Scan image
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: auth-service:latest
          format: 'sarif'
          output: 'trivy-results.sarif'
      
      - name: Upload results
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'

OIDC Authentication (No Long-Lived Credentials)

name: Deploy with OIDC

on:
  push:
    branches: [main]

permissions:
  id-token: write  # Required for OIDC
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
          aws-region: us-east-1
      
      - name: Deploy
        run: |
          # No need for access keys!
          aws s3 sync ./dist s3://my-bucket

Monitoring and Observability

name: Deploy with Monitoring

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Deploy application
        run: |
          # Your deployment commands
          npm run deploy
      
      - name: Create Datadog deployment marker
        uses: chime/datadog-deployments-action@v1
        with:
          api-key: ${{ secrets.DATADOG_API_KEY }}
          app-key: ${{ secrets.DATADOG_APP_KEY }}
          service: auth-service
          env: production
          version: ${{ github.sha }}
      
      - name: Create New Relic deployment
        uses: newrelic/deployment-marker-action@v1
        with:
          apiKey: ${{ secrets.NEW_RELIC_API_KEY }}
          accountId: ${{ secrets.NEW_RELIC_ACCOUNT_ID }}
          applicationId: ${{ secrets.NEW_RELIC_APP_ID }}
          revision: ${{ github.sha }}
          description: "Deployed via GitHub Actions"
          user: ${{ github.actor }}
      
      - name: Send metrics to CloudWatch
        run: |
          aws cloudwatch put-metric-data \
            --namespace GitHubActions \
            --metric-name Deployments \
            --value 1 \
            --dimensions Service=auth-service,Environment=production

Best Practices Checklist

Workflow Organization

# ✅ Good: Descriptive workflow name
name: Auth Service - CI/CD Pipeline

# ❌ Bad: Generic name
name: CI

# ✅ Good: Specific triggers
on:
  push:
    branches: [main, develop]
    paths:
      - 'services/auth/**'
  pull_request:
    branches: [main]

# ❌ Bad: Too broad
on: [push, pull_request]

# ✅ Good: Clear job names
jobs:
  lint-and-type-check:
    name: Lint and Type Check

# ❌ Bad: Unclear job names
jobs:
  job1:
    name: Job 1

Environment Protection Rules

In GitHub repository settings:

1. Go to Settings → Environments

2. Create production environment

3. Add protection rules:

   • ✅ Required reviewers (at least 1-2)

   • ✅ Wait timer (5-10 minutes)

   • ✅ Deployment branches (only main)

   • ✅ Environment secrets

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment:
      name: production
      url: https://api.example.com
    
    steps:
      - run: echo "Deploying to production"

Comprehensive Testing Strategy

jobs:
  test-matrix:
    strategy:
      matrix:
        test-type: [unit, integration, e2e]
        node-version: [18, 20]
    
    steps:
      - name: Run ${{ matrix.test-type }} tests
        run: npm run test:${{ matrix.test-type }}

Key Takeaways

1. Always have a rollback plan - automate it!

2. Use environment protection rules for production

3. Implement blue-green or canary deployments for zero-downtime

4. Monitor deployments with health checks and metrics

5. Scan for secrets and vulnerabilities before deploying

6. Use OIDC instead of long-lived credentials when possible

7. Add deployment markers to your observability tools

8. Test in production-like environments first

9. Notify teams of deployment status

10. Document your deployment process in the workflow itself

Final Thoughts

GitHub Actions has transformed how I build and deploy software. What started as simple CI checks has evolved into a comprehensive deployment pipeline that handles everything from code quality to production rollouts.

The key is to start simple and iterate. Don't try to build the perfect pipeline on day one. Start with basic CI, add tests, then gradually introduce deployment automation as you gain confidence.

Remember:

• Workflows are code - version control them, review them, test them

• Security first - never commit secrets, use environment protection

• Monitor everything - you can't improve what you don't measure

• Fail fast - catch issues early in the pipeline

• Keep it maintainable - use reusable workflows and composite actions

Now go build amazing CI/CD pipelines! 🚀