Part 4: Advanced Workflows and Optimization

Introduction

After building basic CI/CD pipelines, I realized I was duplicating a lot of workflow code across my microservices. Each service had nearly identical workflows for linting, testing, and building. This is where reusable workflows and composite actions saved me hundreds of lines of YAML.

In this part, I'll show you advanced patterns that will make your workflows DRY (Don't Repeat Yourself), faster, and more maintainable.

Reusable Workflows

Reusable workflows let you create a workflow template that other workflows can call.

Creating a Reusable Workflow

.github/workflows/reusable-node-ci.yml:

name: Reusable Node.js CI

on:
  workflow_call:
    inputs:
      node-version:
        description: 'Node.js version to use'
        required: false
        type: string
        default: '18'
      service-path:
        description: 'Path to the service'
        required: true
        type: string
      run-integration-tests:
        description: 'Run integration tests'
        required: false
        type: boolean
        default: true
    secrets:
      codecov-token:
        description: 'Codecov token'
        required: false
      docker-username:
        required: false
      docker-password:
        required: false

env:
  NODE_VERSION: ${{ inputs.node-version }}
  SERVICE_PATH: ${{ inputs.service-path }}

jobs:
  lint:
    name: Lint and Type Check
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
          cache-dependency-path: '${{ env.SERVICE_PATH }}/package-lock.json'
      
      - name: Install dependencies
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm ci
      
      - name: Run linter
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm run lint
      
      - name: Type check
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm run type-check
  
  test:
    name: Test
    runs-on: ubuntu-latest
    
    services:
      postgres:
        image: postgres:14
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432
      
      redis:
        image: redis:7-alpine
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 6379:6379
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
          cache-dependency-path: '${{ env.SERVICE_PATH }}/package-lock.json'
      
      - name: Install dependencies
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm ci
      
      - name: Generate Prisma Client
        working-directory: ${{ env.SERVICE_PATH }}
        run: npx prisma generate
      
      - name: Run migrations
        if: inputs.run-integration-tests
        working-directory: ${{ env.SERVICE_PATH }}
        run: npx prisma migrate deploy
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test
      
      - name: Run tests
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm test -- --coverage
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test
          REDIS_URL: redis://localhost:6379
      
      - name: Upload coverage
        if: ${{ secrets.codecov-token != '' }}
        uses: codecov/codecov-action@v3
        with:
          files: ./${{ env.SERVICE_PATH }}/coverage/lcov.info
          token: ${{ secrets.codecov-token }}
  
  build:
    name: Build
    runs-on: ubuntu-latest
    needs: [lint, test]
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
          cache-dependency-path: '${{ env.SERVICE_PATH }}/package-lock.json'
      
      - name: Install dependencies
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm ci
      
      - name: Build
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm run build

Using the Reusable Workflow

.github/workflows/auth-ci.yml:

name: Auth Service CI

on:
  push:
    branches: [main, develop]
    paths:
      - 'services/auth/**'
      - 'packages/**'
  pull_request:
    branches: [main, develop]

jobs:
  ci:
    uses: ./.github/workflows/reusable-node-ci.yml
    with:
      node-version: '18'
      service-path: 'services/auth'
      run-integration-tests: true
    secrets:
      codecov-token: ${{ secrets.CODECOV_TOKEN }}
      docker-username: ${{ secrets.DOCKER_USERNAME }}
      docker-password: ${{ secrets.DOCKER_PASSWORD }}

.github/workflows/payment-ci.yml:

name: Payment Service CI

on:
  push:
    branches: [main, develop]
    paths:
      - 'services/payment/**'
      - 'packages/**'
  pull_request:
    branches: [main, develop]

jobs:
  ci:
    uses: ./.github/workflows/reusable-node-ci.yml
    with:
      node-version: '20'
      service-path: 'services/payment'
      run-integration-tests: true
    secrets: inherit  # Pass all repository secrets

Composite Actions

Composite actions let you bundle multiple steps into a single reusable action.

Creating a Composite Action

.github/actions/setup-node-service/action.yml:

name: 'Setup Node.js Service'
description: 'Setup Node.js, install dependencies, and generate Prisma client'

inputs:
  node-version:
    description: 'Node.js version'
    required: false
    default: '18'
  service-path:
    description: 'Path to the service directory'
    required: true
  skip-prisma:
    description: 'Skip Prisma client generation'
    required: false
    default: 'false'

runs:
  using: 'composite'
  steps:
    - name: Setup Node.js
      uses: actions/setup-node@v3
      with:
        node-version: ${{ inputs.node-version }}
        cache: 'npm'
        cache-dependency-path: '${{ inputs.service-path }}/package-lock.json'
    
    - name: Install dependencies
      shell: bash
      working-directory: ${{ inputs.service-path }}
      run: npm ci
    
    - name: Generate Prisma Client
      if: inputs.skip-prisma != 'true'
      shell: bash
      working-directory: ${{ inputs.service-path }}
      run: npx prisma generate

Using the Composite Action

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup service
        uses: ./.github/actions/setup-node-service
        with:
          node-version: '18'
          service-path: 'services/auth'
      
      - name: Run tests
        working-directory: services/auth
        run: npm test

Advanced Composite Action with Outputs

.github/actions/extract-service-info/action.yml:

name: 'Extract Service Information'
description: 'Extract version, name, and other info from package.json'

inputs:
  service-path:
    description: 'Path to the service directory'
    required: true

outputs:
  version:
    description: 'Service version'
    value: ${{ steps.info.outputs.version }}
  name:
    description: 'Service name'
    value: ${{ steps.info.outputs.name }}
  tag:
    description: 'Docker tag'
    value: ${{ steps.info.outputs.tag }}

runs:
  using: 'composite'
  steps:
    - name: Extract information
      id: info
      shell: bash
      working-directory: ${{ inputs.service-path }}
      run: |
        VERSION=$(cat package.json | jq -r .version)
        NAME=$(cat package.json | jq -r .name)
        TAG="${NAME}:${VERSION}"
        
        echo "version=$VERSION" >> $GITHUB_OUTPUT
        echo "name=$NAME" >> $GITHUB_OUTPUT
        echo "tag=$TAG" >> $GITHUB_OUTPUT
        
        echo "📦 Service: $NAME"
        echo "🏷️  Version: $VERSION"
        echo "🐳 Tag: $TAG"

Using it:

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Get service info
        id: service
        uses: ./.github/actions/extract-service-info
        with:
          service-path: 'services/auth'
      
      - name: Build Docker image
        run: |
          docker build -t ${{ steps.service.outputs.tag }} .
          echo "Built version: ${{ steps.service.outputs.version }}"

Concurrency Control

Prevent multiple workflow runs from interfering with each other:

name: Deploy

on:
  push:
    branches: [main]

# Cancel in-progress runs for the same branch
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - run: npm run deploy

More specific concurrency:

# Only one deployment per environment at a time
concurrency:
  group: deploy-${{ inputs.environment }}
  cancel-in-progress: false  # Don't cancel, wait instead

Per-service concurrency:

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}-${{ matrix.service }}
  cancel-in-progress: true

Artifacts and Caching

Uploading and Downloading Artifacts

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      - run: npm run build
      
      - name: Upload build artifacts
        uses: actions/upload-artifact@v3
        with:
          name: dist-files
          path: |
            dist/
            package.json
          retention-days: 7
  
  deploy:
    runs-on: ubuntu-latest
    needs: build
    
    steps:
      - name: Download build artifacts
        uses: actions/download-artifact@v3
        with:
          name: dist-files
      
      - name: Deploy
        run: |
          ls -la
          npm run deploy

Advanced Caching Strategy

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      # Cache node_modules
      - name: Cache dependencies
        id: cache-deps
        uses: actions/cache@v3
        with:
          path: |
            node_modules
            ~/.npm
          key: deps-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
          restore-keys: |
            deps-${{ runner.os }}-
      
      # Only install if cache miss
      - name: Install dependencies
        if: steps.cache-deps.outputs.cache-hit != 'true'
        run: npm ci
      
      # Cache build output
      - name: Cache build
        uses: actions/cache@v3
        with:
          path: dist
          key: build-${{ github.sha }}
      
      - name: Build
        run: npm run build
      
      # Cache TypeScript build info
      - name: Cache TypeScript
        uses: actions/cache@v3
        with:
          path: '*.tsbuildinfo'
          key: tsbuildinfo-${{ github.sha }}
          restore-keys: tsbuildinfo-

Job Outputs

Pass data between jobs:

jobs:
  prepare:
    runs-on: ubuntu-latest
    outputs:
      version: ${{ steps.version.outputs.version }}
      should-deploy: ${{ steps.check.outputs.should-deploy }}
      services: ${{ steps.detect.outputs.services }}
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Get version
        id: version
        run: |
          VERSION=$(cat package.json | jq -r .version)
          echo "version=$VERSION" >> $GITHUB_OUTPUT
      
      - name: Check if should deploy
        id: check
        run: |
          if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
            echo "should-deploy=true" >> $GITHUB_OUTPUT
          else
            echo "should-deploy=false" >> $GITHUB_OUTPUT
          fi
      
      - name: Detect changed services
        id: detect
        run: |
          SERVICES=$(git diff --name-only ${{ github.event.before }} ${{ github.sha }} | grep '^services/' | cut -d'/' -f2 | sort -u | jq -R -s -c 'split("\n")[:-1]')
          echo "services=$SERVICES" >> $GITHUB_OUTPUT
  
  build:
    runs-on: ubuntu-latest
    needs: prepare
    
    steps:
      - name: Build version
        run: echo "Building version ${{ needs.prepare.outputs.version }}"
  
  deploy:
    runs-on: ubuntu-latest
    needs: [prepare, build]
    if: needs.prepare.outputs.should-deploy == 'true'
    
    strategy:
      matrix:
        service: ${{ fromJSON(needs.prepare.outputs.services) }}
    
    steps:
      - name: Deploy ${{ matrix.service }}
        run: echo "Deploying ${{ matrix.service }} version ${{ needs.prepare.outputs.version }}"

Dynamic Matrix from JSON

jobs:
  prepare:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Set matrix
        id: set-matrix
        run: |
          # Create matrix from config file
          MATRIX=$(cat .github/test-matrix.json | jq -c .)
          echo "matrix=$MATRIX" >> $GITHUB_OUTPUT
  
  test:
    runs-on: ubuntu-latest
    needs: prepare
    strategy:
      matrix: ${{ fromJSON(needs.prepare.outputs.matrix) }}
    
    steps:
      - name: Test ${{ matrix.service }} on Node ${{ matrix.node }}
        run: echo "Testing..."

.github/test-matrix.json:

{
  "include": [
    {
      "service": "auth",
      "node": "18",
      "database": "postgres:14"
    },
    {
      "service": "auth",
      "node": "20",
      "database": "postgres:15"
    },
    {
      "service": "payment",
      "node": "18",
      "database": "postgres:14"
    },
    {
      "service": "payment",
      "node": "20",
      "database": "postgres:15"
    }
  ]
}

Conditional Workflows

Run Different Jobs Based on Conditions

jobs:
  check-changes:
    runs-on: ubuntu-latest
    outputs:
      backend-changed: ${{ steps.filter.outputs.backend }}
      frontend-changed: ${{ steps.filter.outputs.frontend }}
      docs-changed: ${{ steps.filter.outputs.docs }}
    
    steps:
      - uses: actions/checkout@v3
      
      - uses: dorny/paths-filter@v2
        id: filter
        with:
          filters: |
            backend:
              - 'services/**'
              - 'packages/**'
            frontend:
              - 'frontend/**'
            docs:
              - 'docs/**'
              - '**.md'
  
  backend-ci:
    runs-on: ubuntu-latest
    needs: check-changes
    if: needs.check-changes.outputs.backend-changed == 'true'
    
    steps:
      - run: echo "Running backend CI"
  
  frontend-ci:
    runs-on: ubuntu-latest
    needs: check-changes
    if: needs.check-changes.outputs.frontend-changed == 'true'
    
    steps:
      - run: echo "Running frontend CI"
  
  docs-build:
    runs-on: ubuntu-latest
    needs: check-changes
    if: needs.check-changes.outputs.docs-changed == 'true'
    
    steps:
      - run: echo "Building docs"

Performance Optimization

Parallel Jobs

jobs:
  # Run these in parallel
  lint:
    runs-on: ubuntu-latest
    steps:
      - run: npm run lint
  
  test-unit:
    runs-on: ubuntu-latest
    steps:
      - run: npm run test:unit
  
  test-integration:
    runs-on: ubuntu-latest
    steps:
      - run: npm run test:integration
  
  security-scan:
    runs-on: ubuntu-latest
    steps:
      - run: npm audit
  
  # This waits for all parallel jobs
  build:
    needs: [lint, test-unit, test-integration, security-scan]
    runs-on: ubuntu-latest
    steps:
      - run: npm run build

Split Tests Across Multiple Runners

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Run test shard ${{ matrix.shard }}
        run: npm test -- --shard=${{ matrix.shard }}/4

Reduce Checkout Time

steps:
  # Shallow clone (faster)
  - uses: actions/checkout@v3
    with:
      fetch-depth: 1
  
  # Sparse checkout (only specific paths)
  - uses: actions/checkout@v3
    with:
      sparse-checkout: |
        services/auth
        packages/shared

Self-Hosted Runners

For faster builds or special requirements:

jobs:
  build:
    runs-on: [self-hosted, linux, x64, high-memory]
    
    steps:
      - uses: actions/checkout@v3
      - run: npm run build

Setting up a self-hosted runner:

1. Go to repository Settings → Actions → Runners

2. Click "New self-hosted runner"

3. Follow instructions to download and configure

4. Label your runner (e.g., high-memory, gpu, production)

Workflow Visualization

Create status badges for your README:

# My Project

![CI](https://github.com/username/repo/workflows/CI/badge.svg)
![Deploy](https://github.com/username/repo/workflows/Deploy/badge.svg)
[![codecov](https://codecov.io/gh/username/repo/branch/main/graph/badge.svg)](https://codecov.io/gh/username/repo)

Advanced Debugging

Enable Debug Logging

Set repository secrets:

• ACTIONS_STEP_DEBUG: true

• ACTIONS_RUNNER_DEBUG: true

Add Debug Step

steps:
  - name: Debug information
    run: |
      echo "Event: ${{ github.event_name }}"
      echo "Ref: ${{ github.ref }}"
      echo "SHA: ${{ github.sha }}"
      echo "Actor: ${{ github.actor }}"
      echo "Workflow: ${{ github.workflow }}"
      echo "Job: ${{ github.job }}"
      echo "Run ID: ${{ github.run_id }}"
      echo "Run Number: ${{ github.run_number }}"
      
      echo "Environment variables:"
      env | sort
      
      echo "Working directory:"
      pwd
      ls -la

SSH Debugging

- name: Setup tmate session
  if: failure()
  uses: mxschmitt/action-tmate@v3
  timeout-minutes: 30

Key Takeaways

1. Reusable workflows eliminate duplication across similar workflows

2. Composite actions bundle common steps into reusable units

3. Concurrency control prevents race conditions and saves compute time

4. Smart caching dramatically reduces workflow run time

5. Job outputs enable complex, multi-stage workflows

6. Dynamic matrices allow flexible testing strategies

7. Conditional jobs save resources by only running when needed

8. Parallel execution speeds up total workflow time

9. Self-hosted runners provide more control and power

10. Proper debugging tools help diagnose workflow issues

In the next part, we'll cover production deployment strategies, including blue-green deployments, rollback mechanisms, and monitoring.