Part 3: Building CI/CD Pipelines for TypeScript Microservices

Introduction

After learning YAML syntax and workflow basics, it's time to build real CI/CD pipelines. In my microservices architecture, I run separate CI workflows for each service (auth, payment, user, notification, etc.), and I've learned that a good pipeline catches bugs before they reach production.

In this part, I'll show you how to build complete CI/CD pipelines specifically for TypeScript microservices with PostgreSQL databases.

Project Structure

Here's the monorepo structure I'm using:

my-project/
├── services/
│   ├── auth/
│   │   ├── src/
│   │   ├── tests/
│   │   ├── prisma/
│   │   ├── package.json
│   │   └── tsconfig.json
│   ├── payment/
│   ├── user/
│   └── notification/
├── packages/
│   ├── shared-types/
│   ├── shared-utils/
│   └── eslint-config/
├── .github/
│   └── workflows/
│       ├── auth-ci.yml
│       ├── payment-ci.yml
│       └── user-ci.yml
└── package.json

Complete CI Pipeline for TypeScript Service

Here's my complete CI workflow for the auth service:

name: Auth Service CI

on:
  push:
    branches: [main, develop]
    paths:
      - 'services/auth/**'
      - 'packages/**'
      - '.github/workflows/auth-ci.yml'
  pull_request:
    branches: [main, develop]
    paths:
      - 'services/auth/**'
      - 'packages/**'

env:
  NODE_VERSION: '18'
  SERVICE_PATH: services/auth
  POSTGRES_VERSION: '14'

jobs:
  lint-and-type-check:
    name: Lint and Type Check
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
        with:
          fetch-depth: 0  # Fetch all history for better analysis
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
          cache-dependency-path: '${{ env.SERVICE_PATH }}/package-lock.json'
      
      - name: Install dependencies
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm ci
      
      - name: Run ESLint
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm run lint
      
      - name: Run TypeScript type check
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm run type-check
      
      - name: Check code formatting (Prettier)
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm run format:check
  
  test-unit:
    name: Unit Tests
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
          cache-dependency-path: '${{ env.SERVICE_PATH }}/package-lock.json'
      
      - name: Install dependencies
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm ci
      
      - name: Run unit tests
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm run test:unit -- --coverage
      
      - name: Upload unit test coverage
        uses: codecov/codecov-action@v3
        with:
          files: ./${{ env.SERVICE_PATH }}/coverage/lcov.info
          flags: auth-unit
          name: auth-unit-coverage
  
  test-integration:
    name: Integration Tests
    runs-on: ubuntu-latest
    
    services:
      postgres:
        image: postgres:${{ env.POSTGRES_VERSION }}
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: auth_test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432
      
      redis:
        image: redis:7-alpine
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 6379:6379
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
          cache-dependency-path: '${{ env.SERVICE_PATH }}/package-lock.json'
      
      - name: Install dependencies
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm ci
      
      - name: Generate Prisma Client
        working-directory: ${{ env.SERVICE_PATH }}
        run: npx prisma generate
      
      - name: Run database migrations
        working-directory: ${{ env.SERVICE_PATH }}
        run: npx prisma migrate deploy
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/auth_test
      
      - name: Run integration tests
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm run test:integration -- --coverage
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/auth_test
          REDIS_URL: redis://localhost:6379
          JWT_SECRET: test_secret_key_for_ci
      
      - name: Upload integration test coverage
        uses: codecov/codecov-action@v3
        with:
          files: ./${{ env.SERVICE_PATH }}/coverage/lcov.info
          flags: auth-integration
          name: auth-integration-coverage
  
  build:
    name: Build
    runs-on: ubuntu-latest
    needs: [lint-and-type-check, test-unit, test-integration]
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
          cache-dependency-path: '${{ env.SERVICE_PATH }}/package-lock.json'
      
      - name: Install dependencies
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm ci
      
      - name: Generate Prisma Client
        working-directory: ${{ env.SERVICE_PATH }}
        run: npx prisma generate
      
      - name: Build TypeScript
        working-directory: ${{ env.SERVICE_PATH }}
        run: npm run build
      
      - name: Check build output
        working-directory: ${{ env.SERVICE_PATH }}
        run: |
          if [ ! -d "dist" ]; then
            echo "Build failed: dist directory not found"
            exit 1
          fi
          echo "Build successful!"
      
      - name: Upload build artifacts
        uses: actions/upload-artifact@v3
        with:
          name: auth-dist
          path: ${{ env.SERVICE_PATH }}/dist/
          retention-days: 7
  
  docker-build:
    name: Docker Build
    runs-on: ubuntu-latest
    needs: build
    if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/develop')
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      
      - name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}
      
      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v4
        with:
          images: ${{ secrets.DOCKER_USERNAME }}/auth-service
          tags: |
            type=ref,event=branch
            type=sha,prefix={{branch}}-
      
      - name: Build and push Docker image
        uses: docker/build-push-action@v4
        with:
          context: ./services/auth
          file: ./services/auth/Dockerfile
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=registry,ref=${{ secrets.DOCKER_USERNAME }}/auth-service:buildcache
          cache-to: type=registry,ref=${{ secrets.DOCKER_USERNAME }}/auth-service:buildcache,mode=max

Caching Dependencies

Caching significantly speeds up workflows. Here's how I implement it:

NPM Cache

- name: Setup Node.js with cache
  uses: actions/setup-node@v3
  with:
    node-version: '18'
    cache: 'npm'
    cache-dependency-path: 'services/auth/package-lock.json'

Custom Cache for Multiple Paths

- name: Cache dependencies
  uses: actions/cache@v3
  with:
    path: |
      ~/.npm
      node_modules
      services/*/node_modules
    key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-npm-

Prisma Client Cache

- name: Cache Prisma Client
  uses: actions/cache@v3
  with:
    path: services/auth/node_modules/.prisma
    key: prisma-${{ hashFiles('services/auth/prisma/schema.prisma') }}

Matrix Builds

Test across multiple Node versions and database versions:

name: Multi-Version Test

on: [push, pull_request]

jobs:
  test:
    name: Test (Node ${{ matrix.node }}, PG ${{ matrix.postgres }})
    runs-on: ubuntu-latest
    
    strategy:
      matrix:
        node: [16, 18, 20]
        postgres: [13, 14, 15]
      fail-fast: false  # Continue other tests even if one fails
    
    services:
      postgres:
        image: postgres:${{ matrix.postgres }}
        env:
          POSTGRES_PASSWORD: postgres
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js ${{ matrix.node }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node }}
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run tests
        run: npm test
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test

Testing with Different Databases

PostgreSQL with Extensions

services:
  postgres:
    image: postgres:14
    env:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: test_db
    options: >-
      --health-cmd pg_isready
      --health-interval 10s
      --health-timeout 5s
      --health-retries 5
    ports:
      - 5432:5432

steps:
  - name: Install PostgreSQL extensions
    run: |
      PGPASSWORD=postgres psql -h localhost -U postgres -d test_db -c "CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";"
      PGPASSWORD=postgres psql -h localhost -U postgres -d test_db -c "CREATE EXTENSION IF NOT EXISTS \"pg_trgm\";"

Testing with MySQL

services:
  mysql:
    image: mysql:8
    env:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_DATABASE: test_db
    options: >-
      --health-cmd="mysqladmin ping"
      --health-interval=10s
      --health-timeout=5s
      --health-retries=3
    ports:
      - 3306:3306

steps:
  - name: Run tests
    run: npm test
    env:
      DATABASE_URL: mysql://root:root@localhost:3306/test_db

Environment-Specific Testing

jobs:
  test-staging:
    name: Test Against Staging
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/develop'
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run E2E tests against staging
        run: npm run test:e2e
        env:
          API_URL: ${{ secrets.STAGING_API_URL }}
          API_KEY: ${{ secrets.STAGING_API_KEY }}
  
  test-production:
    name: Smoke Test Production
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run smoke tests
        run: npm run test:smoke
        env:
          API_URL: ${{ secrets.PRODUCTION_API_URL }}
          API_KEY: ${{ secrets.PRODUCTION_API_KEY }}

Security Scanning

Dependency Scanning

jobs:
  security:
    name: Security Scan
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Run npm audit
        run: npm audit --audit-level=moderate
        working-directory: services/auth
      
      - name: Check for vulnerabilities
        uses: snyk/actions/node@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          args: --severity-threshold=high

Code Scanning with CodeQL

jobs:
  codeql:
    name: CodeQL Analysis
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Initialize CodeQL
        uses: github/codeql-action/init@v2
        with:
          languages: javascript, typescript
      
      - name: Autobuild
        uses: github/codeql-action/autobuild@v2
      
      - name: Perform CodeQL Analysis
        uses: github/codeql-action/analyze@v2

Docker Build Optimization

Multi-Stage Dockerfile

# services/auth/Dockerfile
FROM node:18-alpine AS builder

WORKDIR /app

# Copy package files
COPY package*.json ./
COPY prisma ./prisma/

# Install dependencies
RUN npm ci

# Copy source code
COPY . .

# Generate Prisma Client
RUN npx prisma generate

# Build TypeScript
RUN npm run build

# Production image
FROM node:18-alpine

WORKDIR /app

# Copy package files
COPY package*.json ./
COPY prisma ./prisma/

# Install production dependencies only
RUN npm ci --only=production

# Copy built files from builder
COPY --from=builder /app/dist ./dist

# Generate Prisma Client
RUN npx prisma generate

# Expose port
EXPOSE 3000

# Start application
CMD ["node", "dist/index.js"]

Workflow for Docker Build

jobs:
  docker:
    name: Build and Push Docker Image
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      
      - name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}
      
      - name: Extract version from package.json
        id: package
        run: |
          VERSION=$(cat services/auth/package.json | jq -r .version)
          echo "version=$VERSION" >> $GITHUB_OUTPUT
      
      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: ./services/auth
          file: ./services/auth/Dockerfile
          push: true
          tags: |
            ${{ secrets.DOCKER_USERNAME }}/auth-service:${{ steps.package.outputs.version }}
            ${{ secrets.DOCKER_USERNAME }}/auth-service:latest
          cache-from: type=registry,ref=${{ secrets.DOCKER_USERNAME }}/auth-service:buildcache
          cache-to: type=registry,ref=${{ secrets.DOCKER_USERNAME }}/auth-service:buildcache,mode=max
          build-args: |
            NODE_ENV=production
            BUILD_DATE=${{ github.event.head_commit.timestamp }}
            VCS_REF=${{ github.sha }}

Monorepo Workflow Optimization

Detect Changed Services

jobs:
  detect-changes:
    name: Detect Changes
    runs-on: ubuntu-latest
    outputs:
      auth: ${{ steps.filter.outputs.auth }}
      payment: ${{ steps.filter.outputs.payment }}
      user: ${{ steps.filter.outputs.user }}
    
    steps:
      - uses: actions/checkout@v3
      
      - uses: dorny/paths-filter@v2
        id: filter
        with:
          filters: |
            auth:
              - 'services/auth/**'
              - 'packages/**'
            payment:
              - 'services/payment/**'
              - 'packages/**'
            user:
              - 'services/user/**'
              - 'packages/**'
  
  test-auth:
    name: Test Auth Service
    runs-on: ubuntu-latest
    needs: detect-changes
    if: needs.detect-changes.outputs.auth == 'true'
    
    steps:
      - uses: actions/checkout@v3
      - run: npm test
        working-directory: services/auth
  
  test-payment:
    name: Test Payment Service
    runs-on: ubuntu-latest
    needs: detect-changes
    if: needs.detect-changes.outputs.payment == 'true'
    
    steps:
      - uses: actions/checkout@v3
      - run: npm test
        working-directory: services/payment

Parallel Jobs

jobs:
  lint:
    name: Lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm run lint
  
  test-unit:
    name: Unit Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm run test:unit
  
  test-integration:
    name: Integration Tests
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:14
    steps:
      - uses: actions/checkout@v3
      - run: npm run test:integration
  
  # All these jobs run in parallel
  # Build only runs after all pass
  build:
    name: Build
    runs-on: ubuntu-latest
    needs: [lint, test-unit, test-integration]
    steps:
      - uses: actions/checkout@v3
      - run: npm run build

Notification on Failure

jobs:
  notify:
    name: Notify on Failure
    runs-on: ubuntu-latest
    if: failure()
    needs: [lint, test, build]
    
    steps:
      - name: Send Slack notification
        uses: 8398a7/action-slack@v3
        with:
          status: ${{ job.status }}
          text: 'CI Pipeline failed on ${{ github.ref }}'
          webhook_url: ${{ secrets.SLACK_WEBHOOK }}
      
      - name: Create GitHub issue
        uses: actions/github-script@v6
        with:
          script: |
            github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `CI failed on ${context.ref}`,
              body: `Workflow failed: ${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`,
              labels: ['ci-failure']
            })

Complete Package.json Scripts

Here's how I set up my package.json for CI:

{
  "name": "auth-service",
  "version": "1.0.0",
  "scripts": {
    "dev": "tsx watch src/index.ts",
    "build": "tsc",
    "start": "node dist/index.js",
    "lint": "eslint src/**/*.ts",
    "lint:fix": "eslint src/**/*.ts --fix",
    "format": "prettier --write \"src/**/*.ts\"",
    "format:check": "prettier --check \"src/**/*.ts\"",
    "type-check": "tsc --noEmit",
    "test": "jest",
    "test:unit": "jest --testPathPattern=unit",
    "test:integration": "jest --testPathPattern=integration",
    "test:e2e": "jest --testPathPattern=e2e",
    "test:coverage": "jest --coverage",
    "test:watch": "jest --watch",
    "prisma:generate": "prisma generate",
    "prisma:migrate": "prisma migrate dev",
    "prisma:deploy": "prisma migrate deploy",
    "prisma:studio": "prisma studio"
  }
}

Key Takeaways

Separate linting, testing, and building into different jobs for faster feedback
Use service containers for integration tests with real databases
Cache dependencies to speed up workflows significantly
Matrix builds help test across different versions
Only run workflows when relevant code changes using path filters
Fail fast when appropriate, but allow matrix builds to continue
Upload artifacts from builds for debugging and deployment
Security scanning should be part of your CI pipeline
Parallel jobs run faster than sequential ones
Monitor and notify on failures

In the next part, we'll explore advanced workflow patterns like reusable workflows, composite actions, and performance optimization.

PreviousPart 2: YAML Syntax and Workflow Fundamentals NextPart 4: Advanced Workflows and Optimization

Last updated 15 hours ago

hashtagIntroduction

hashtagProject Structure

hashtagComplete CI Pipeline for TypeScript Service

hashtagCaching Dependencies

hashtagNPM Cache

hashtagCustom Cache for Multiple Paths

hashtagPrisma Client Cache

hashtagMatrix Builds

hashtagTesting with Different Databases

hashtagPostgreSQL with Extensions

hashtagTesting with MySQL

hashtagEnvironment-Specific Testing

hashtagSecurity Scanning

hashtagDependency Scanning

hashtagCode Scanning with CodeQL

hashtagDocker Build Optimization

hashtagMulti-Stage Dockerfile

hashtagWorkflow for Docker Build

hashtagMonorepo Workflow Optimization

hashtagDetect Changed Services

hashtagParallel Jobs

hashtagNotification on Failure

hashtagComplete Package.json Scripts

hashtagKey Takeaways