Part 7: Testing, Performance, and Production Deployment

The Production Deployment That Went Wrong

My first production deployment: pushed code at 9 AM on a Monday. By 9:30 AM, customer support was flooded with complaints. The API was timing out.

The problem? I'd never tested with production data volumes. My local database had 100 users. Production had 500,000. A query that took 10ms locally took 30 seconds in production—no indexes on the database tables.

Emergency rollback. Spent the day adding indexes and load testing. Redeployed at 5 PM.

Lesson: Testing with realistic data and load conditions isn't optional—it's survival.

This article covers the testing and deployment practices that would have saved me that day.

Testing Strategy

Testing Pyramid

        /\
       /E2E\         10% - End-to-end tests
      /______\
     /  API   \      20% - API/Integration tests
    /__________\
   /    Unit     \   70% - Unit tests
  /________________\

Unit Testing

Test business logic in isolation.

npm install -D jest ts-jest @types/jest
npm install -D supertest @types/supertest

// src/services/__tests__/user-service.test.ts
import { UserService } from '../user-service';
import { UserRepository } from '../../repositories/user-repository';

// Mock repository
jest.mock('../../repositories/user-repository');

describe('UserService', () => {
  let userService: UserService;
  let mockUserRepository: jest.Mocked<UserRepository>;

  beforeEach(() => {
    mockUserRepository = new UserRepository(null as any) as jest.Mocked<UserRepository>;
    userService = new UserService(mockUserRepository);
  });

  describe('createUser', () => {
    it('should create user with valid data', async () => {
      const userData = {
        email: '[email protected]',
        password: 'SecurePass123',
        name: 'John Doe',
      };

      const expectedUser = {
        id: 'user_123',
        email: userData.email,
        name: userData.name,
        createdAt: new Date(),
      };

      mockUserRepository.findByEmail.mockResolvedValue(null);
      mockUserRepository.create.mockResolvedValue(expectedUser);

      const result = await userService.createUser(userData);

      expect(result).toEqual(expectedUser);
      expect(mockUserRepository.findByEmail).toHaveBeenCalledWith(userData.email);
      expect(mockUserRepository.create).toHaveBeenCalled();
    });

    it('should throw error if email already exists', async () => {
      const userData = {
        email: '[email protected]',
        password: 'SecurePass123',
        name: 'John Doe',
      };

      mockUserRepository.findByEmail.mockResolvedValue({
        id: 'user_456',
        email: userData.email,
        name: 'Existing User',
      } as any);

      await expect(userService.createUser(userData)).rejects.toThrow(
        'Email already registered'
      );
    });

    it('should throw error for invalid email', async () => {
      const userData = {
        email: 'invalid-email',
        password: 'SecurePass123',
        name: 'John Doe',
      };

      mockUserRepository.findByEmail.mockResolvedValue(null);

      await expect(userService.createUser(userData)).rejects.toThrow(
        'Invalid email format'
      );
    });

    it('should throw error for weak password', async () => {
      const userData = {
        email: '[email protected]',
        password: 'weak',
        name: 'John Doe',
      };

      mockUserRepository.findByEmail.mockResolvedValue(null);

      await expect(userService.createUser(userData)).rejects.toThrow(
        'Password must be at least 8 characters'
      );
    });
  });

  describe('getUserById', () => {
    it('should return user if found', async () => {
      const userId = 'user_123';
      const expectedUser = {
        id: userId,
        email: '[email protected]',
        name: 'John Doe',
      };

      mockUserRepository.findById.mockResolvedValue(expectedUser as any);

      const result = await userService.getUserById(userId);

      expect(result).toEqual(expectedUser);
      expect(mockUserRepository.findById).toHaveBeenCalledWith(userId);
    });

    it('should throw NotFoundError if user not found', async () => {
      const userId = 'nonexistent';

      mockUserRepository.findById.mockResolvedValue(null);

      await expect(userService.getUserById(userId)).rejects.toThrow('User not found');
    });
  });
});

Integration Testing

Test API endpoints with real database (using Testcontainers).

npm install -D @testcontainers/postgresql

// src/__tests__/integration/user-api.test.ts
import request from 'supertest';
import { App } from '../../app';
import { Database } from '../../config/database';
import { PostgreSqlContainer, StartedPostgreSqlContainer } from '@testcontainers/postgresql';

describe('User API Integration Tests', () => {
  let app: any;
  let container: StartedPostgreSqlContainer;

  beforeAll(async () => {
    // Start PostgreSQL container
    container = await new PostgreSqlContainer('postgres:15').start();

    // Override database config
    process.env.DB_HOST = container.getHost();
    process.env.DB_PORT = container.getPort().toString();
    process.env.DB_NAME = container.getDatabase();
    process.env.DB_USER = container.getUsername();
    process.env.DB_PASSWORD = container.getPassword();

    // Initialize app
    app = new App().getApp();

    // Run migrations
    await Database.getInstance().query(`
      CREATE TABLE IF NOT EXISTS users (
        id VARCHAR(255) PRIMARY KEY,
        email VARCHAR(255) UNIQUE NOT NULL,
        password VARCHAR(255) NOT NULL,
        name VARCHAR(255) NOT NULL,
        role VARCHAR(50) DEFAULT 'user',
        created_at TIMESTAMP DEFAULT NOW(),
        updated_at TIMESTAMP DEFAULT NOW()
      );
    `);
  }, 60000);

  afterAll(async () => {
    await Database.close();
    await container.stop();
  });

  beforeEach(async () => {
    // Clean database before each test
    await Database.getInstance().query('DELETE FROM users');
  });

  describe('POST /api/v1/users', () => {
    it('should create user with valid data', async () => {
      const userData = {
        email: '[email protected]',
        password: 'SecurePass123',
        name: 'John Doe',
      };

      const response = await request(app)
        .post('/api/v1/users')
        .send(userData)
        .expect(201);

      expect(response.body.success).toBe(true);
      expect(response.body.data).toMatchObject({
        email: userData.email,
        name: userData.name,
      });
      expect(response.body.data.id).toBeDefined();
      expect(response.body.data.password).toBeUndefined(); // Password should not be returned
    });

    it('should return 422 for invalid email', async () => {
      const userData = {
        email: 'invalid-email',
        password: 'SecurePass123',
        name: 'John Doe',
      };

      const response = await request(app)
        .post('/api/v1/users')
        .send(userData)
        .expect(422);

      expect(response.body.success).toBe(false);
      expect(response.body.error).toContain('Validation failed');
    });

    it('should return 409 for duplicate email', async () => {
      const userData = {
        email: '[email protected]',
        password: 'SecurePass123',
        name: 'John Doe',
      };

      // Create first user
      await request(app).post('/api/v1/users').send(userData).expect(201);

      // Try to create duplicate
      const response = await request(app)
        .post('/api/v1/users')
        .send(userData)
        .expect(409);

      expect(response.body.success).toBe(false);
      expect(response.body.error).toContain('already registered');
    });
  });

  describe('GET /api/v1/users/:id', () => {
    it('should return user if exists', async () => {
      // Create user
      const createResponse = await request(app)
        .post('/api/v1/users')
        .send({
          email: '[email protected]',
          password: 'SecurePass123',
          name: 'John Doe',
        });

      const userId = createResponse.body.data.id;

      // Get user
      const response = await request(app)
        .get(`/api/v1/users/${userId}`)
        .expect(200);

      expect(response.body.success).toBe(true);
      expect(response.body.data.id).toBe(userId);
    });

    it('should return 404 if user not found', async () => {
      const response = await request(app)
        .get('/api/v1/users/nonexistent')
        .expect(404);

      expect(response.body.success).toBe(false);
      expect(response.body.error).toContain('not found');
    });
  });
});

Load Testing

Test API performance under load using k6.

npm install -D k6

// tests/load/user-api-load.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '30s', target: 50 },   // Ramp up to 50 users
    { duration: '1m', target: 50 },    // Stay at 50 users
    { duration: '30s', target: 100 },  // Ramp up to 100 users
    { duration: '1m', target: 100 },   // Stay at 100 users
    { duration: '30s', target: 0 },    // Ramp down to 0
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'], // 95% of requests must complete within 500ms
    http_req_failed: ['rate<0.01'],   // Error rate must be below 1%
  },
};

const BASE_URL = 'http://localhost:4000/api/v1';

export default function () {
  // Test GET /users
  const getResponse = http.get(`${BASE_URL}/users`);
  check(getResponse, {
    'GET /users status is 200': (r) => r.status === 200,
    'GET /users response time < 500ms': (r) => r.timings.duration < 500,
  });

  sleep(1);

  // Test POST /users
  const userData = {
    email: `user${Date.now()}@example.com`,
    password: 'SecurePass123',
    name: 'Load Test User',
  };

  const postResponse = http.post(
    `${BASE_URL}/users`,
    JSON.stringify(userData),
    {
      headers: { 'Content-Type': 'application/json' },
    }
  );

  check(postResponse, {
    'POST /users status is 201': (r) => r.status === 201,
    'POST /users response time < 1000ms': (r) => r.timings.duration < 1000,
  });

  sleep(1);
}

Run load tests:

k6 run tests/load/user-api-load.js

Performance Optimization

Database Indexing

-- Add indexes for frequently queried fields
CREATE INDEX idx_users_email ON users(email);
CREATE INDEX idx_users_role ON users(role);
CREATE INDEX idx_users_created_at ON users(created_at);

-- Composite index for common query patterns
CREATE INDEX idx_users_role_created ON users(role, created_at);

-- Analyze query performance
EXPLAIN ANALYZE SELECT * FROM users WHERE email = '[email protected]';

Query Optimization

// BAD: N+1 query problem
export class OrderService {
  async getOrdersWithUsers(): Promise<any[]> {
    const orders = await this.orderRepository.findAll();
    
    // This fires one query per order!
    for (const order of orders) {
      order.user = await this.userRepository.findById(order.userId);
    }
    
    return orders;
  }
}

// GOOD: Join or batch fetch
export class OrderService {
  async getOrdersWithUsers(): Promise<any[]> {
    // Single query with join
    return this.orderRepository.findAllWithUsers();
  }
}

// In repository
async findAllWithUsers(): Promise<Order[]> {
  const result = await this.db.query(`
    SELECT 
      o.*,
      json_build_object(
        'id', u.id,
        'name', u.name,
        'email', u.email
      ) as user
    FROM orders o
    LEFT JOIN users u ON o.user_id = u.id
  `);
  
  return result.rows;
}

Caching

npm install ioredis

// src/config/redis.ts
import Redis from 'ioredis';

export class RedisClient {
  private static instance: Redis;

  static getInstance(): Redis {
    if (!this.instance) {
      this.instance = new Redis({
        host: process.env.REDIS_HOST || 'localhost',
        port: parseInt(process.env.REDIS_PORT || '6379'),
        password: process.env.REDIS_PASSWORD,
      });
    }
    return this.instance;
  }
}

// src/services/user-service.ts
export class UserService {
  private readonly redis: Redis;

  constructor(
    private readonly userRepository: UserRepository
  ) {
    this.redis = RedisClient.getInstance();
  }

  async getUserById(id: string): Promise<User> {
    // Check cache first
    const cached = await this.redis.get(`user:${id}`);
    if (cached) {
      return JSON.parse(cached);
    }

    // Fetch from database
    const user = await this.userRepository.findById(id);
    if (!user) {
      throw new NotFoundError('User');
    }

    // Cache for 5 minutes
    await this.redis.setex(`user:${id}`, 300, JSON.stringify(user));

    return user;
  }

  async updateUser(id: string, updates: UpdateUserDto): Promise<User> {
    const user = await this.userRepository.update(id, updates);

    // Invalidate cache
    await this.redis.del(`user:${id}`);

    return user;
  }
}

Production Deployment

Dockerfile

# Build stage
FROM node:20-alpine AS builder

WORKDIR /app

# Copy package files
COPY package*.json ./
COPY tsconfig.json ./

# Install dependencies
RUN npm ci

# Copy source
COPY src ./src

# Build TypeScript
RUN npm run build

# Production stage
FROM node:20-alpine

WORKDIR /app

# Copy package files
COPY package*.json ./

# Install production dependencies only
RUN npm ci --production

# Copy built files from builder
COPY --from=builder /app/dist ./dist

# Create non-root user
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001

USER nodejs

EXPOSE 4000

CMD ["node", "dist/server.js"]

Docker Compose

# docker-compose.yml
version: '3.8'

services:
  api:
    build: .
    ports:
      - "4000:4000"
    environment:
      - NODE_ENV=production
      - PORT=4000
      - DB_HOST=postgres
      - DB_PORT=5432
      - DB_NAME=userdb
      - DB_USER=postgres
      - DB_PASSWORD=password
      - REDIS_HOST=redis
      - REDIS_PORT=6379
    depends_on:
      - postgres
      - redis
    restart: unless-stopped

  postgres:
    image: postgres:15-alpine
    environment:
      - POSTGRES_DB=userdb
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    restart: unless-stopped

volumes:
  postgres_data:

Health Checks

// src/routes/health-routes.ts
import { Router, Request, Response } from 'express';
import { Database } from '../config/database';
import { RedisClient } from '../config/redis';

export class HealthRoutes {
  static setup(): Router {
    const router = Router();

    // Liveness probe - is service running?
    router.get('/health/live', (req: Request, res: Response) => {
      res.json({
        status: 'ok',
        timestamp: new Date().toISOString(),
      });
    });

    // Readiness probe - is service ready to handle requests?
    router.get('/health/ready', async (req: Request, res: Response) => {
      try {
        // Check database
        await Database.getInstance().query('SELECT 1');

        // Check Redis
        await RedisClient.getInstance().ping();

        res.json({
          status: 'ready',
          checks: {
            database: 'ok',
            redis: 'ok',
          },
          timestamp: new Date().toISOString(),
        });
      } catch (error) {
        res.status(503).json({
          status: 'not ready',
          error: error.message,
          timestamp: new Date().toISOString(),
        });
      }
    });

    return router;
  }
}

CI/CD Pipeline (GitHub Actions)

# .github/workflows/deploy.yml
name: Deploy

on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '20'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run unit tests
        run: npm test
      
      - name: Run integration tests
        run: npm run test:integration
  
  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Build Docker image
        run: docker build -t user-service:${{ github.sha }} .
      
      - name: Push to registry
        run: |
          echo ${{ secrets.DOCKER_PASSWORD }} | docker login -u ${{ secrets.DOCKER_USERNAME }} --password-stdin
          docker tag user-service:${{ github.sha }} username/user-service:latest
          docker push username/user-service:latest
  
  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to production
        run: |
          # Deploy to Kubernetes, ECS, or other platform
          kubectl set image deployment/user-service user-service=username/user-service:latest

Monitoring

npm install prom-client

// src/middleware/metrics-middleware.ts
import promClient from 'prom-client';

export class MetricsMiddleware {
  private static register = new promClient.Registry();
  
  private static httpRequestDuration = new promClient.Histogram({
    name: 'http_request_duration_ms',
    help: 'Duration of HTTP requests in ms',
    labelNames: ['method', 'route', 'status_code'],
    buckets: [50, 100, 200, 500, 1000, 2000, 5000],
  });

  private static httpRequestTotal = new promClient.Counter({
    name: 'http_requests_total',
    help: 'Total number of HTTP requests',
    labelNames: ['method', 'route', 'status_code'],
  });

  static init() {
    this.register.registerMetric(this.httpRequestDuration);
    this.register.registerMetric(this.httpRequestTotal);
    promClient.collectDefaultMetrics({ register: this.register });
  }

  static middleware(req: Request, res: Response, next: NextFunction) {
    const start = Date.now();

    res.on('finish', () => {
      const duration = Date.now() - start;
      const route = req.route?.path || req.path;

      this.httpRequestDuration.observe(
        { method: req.method, route, status_code: res.statusCode },
        duration
      );

      this.httpRequestTotal.inc({
        method: req.method,
        route,
        status_code: res.statusCode,
      });
    });

    next();
  }

  static async getMetrics(req: Request, res: Response) {
    res.set('Content-Type', this.register.contentType);
    res.end(await this.register.metrics());
  }
}

Key Takeaways

1. Test pyramid: 70% unit, 20% integration, 10% E2E

2. Load test with realistic data volumes

3. Add database indexes for frequently queried fields

4. Cache frequently accessed data (Redis)

5. Use connection pooling for databases

6. Health checks for liveness and readiness

7. Docker for consistent deployments

8. CI/CD for automated testing and deployment

9. Monitor performance metrics (Prometheus)

10. Zero-downtime deployments with rolling updates

Series complete! You now have everything needed to build production-ready REST APIs.

---

Testing and monitoring aren't afterthoughts—they're what separate hobby projects from production systems.