Automatic Instrumentation

The Magic of Auto-Instrumentation

When I first added OpenTelemetry to my production microservices, I was skeptical about auto-instrumentation. How could a library automatically understand my application's behavior without me writing custom code? But after seeing it trace every database query, Redis operation, and HTTP call automatically, I became a believer.

The reality is that 80% of your observability needs can be satisfied with auto-instrumentation alone. It's only the business-specific logic and critical paths that need custom spans. Let me show you how to leverage this superpower.

Available Auto-Instrumentation Libraries

OpenTelemetry provides instrumentation for virtually every popular Node.js library:

HTTP & Web Frameworks

'@opentelemetry/instrumentation-express'    // Express.js
'@opentelemetry/instrumentation-fastify'    // Fastify
'@opentelemetry/instrumentation-koa'        // Koa
'@opentelemetry/instrumentation-http'       // Native HTTP/HTTPS
'@opentelemetry/instrumentation-fetch'      // Fetch API

Databases

'@opentelemetry/instrumentation-pg'         // PostgreSQL
'@opentelemetry/instrumentation-mysql'      // MySQL
'@opentelemetry/instrumentation-mongodb'    // MongoDB
'@opentelemetry/instrumentation-redis-4'    // Redis 4.x
'@opentelemetry/instrumentation-ioredis'    // ioredis

Message Queues

'@opentelemetry/instrumentation-amqplib'    // RabbitMQ
'@opentelemetry/instrumentation-kafkajs'    // Kafka

Cloud Services

'@opentelemetry/instrumentation-aws-sdk'    // AWS SDK
'@opentelemetry/instrumentation-graphql'    // GraphQL

Setting Up Auto-Instrumentation

Let's enhance our order service with PostgreSQL and Redis.

Install dependencies:

npm install pg ioredis
npm install -D @types/pg

# Install instrumentation packages
npm install @opentelemetry/instrumentation-pg \
  @opentelemetry/instrumentation-ioredis

Update src/instrumentation.ts:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { Resource } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: 'order-service',
    [ATTR_SERVICE_VERSION]: '1.0.0',
  }),
  
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:4318/v1/traces',
  }),
  
  instrumentations: [
    getNodeAutoInstrumentations({
      // Configure specific instrumentations
      '@opentelemetry/instrumentation-express': {
        enabled: true,
      },
      '@opentelemetry/instrumentation-http': {
        enabled: true,
        ignoreIncomingPaths: ['/health'], // Don't trace health checks
      },
      '@opentelemetry/instrumentation-pg': {
        enabled: true,
        enhancedDatabaseReporting: true, // Include query parameters
      },
      '@opentelemetry/instrumentation-ioredis': {
        enabled: true,
        dbStatementSerializer: (cmdName, cmdArgs) => {
          // Sanitize sensitive data from Redis commands
          return cmdName.toUpperCase();
        },
      },
    }),
  ],
});

sdk.start();

process.on('SIGTERM', () => {
  sdk.shutdown()
    .then(() => console.log('SDK shut down'))
    .catch((error) => console.error('Shutdown error', error))
    .finally(() => process.exit(0));
});

console.log('OpenTelemetry with auto-instrumentation configured');

Building a Real Application with PostgreSQL

Start PostgreSQL with Docker:

docker run -d --name postgres \
  -e POSTGRES_PASSWORD=password \
  -e POSTGRES_DB=orders \
  -p 5432:5432 \
  postgres:16

Create database schema (src/db.ts):

import { Pool } from 'pg';

export const pool = new Pool({
  host: process.env.DB_HOST || 'localhost',
  port: parseInt(process.env.DB_PORT || '5432'),
  database: process.env.DB_NAME || 'orders',
  user: process.env.DB_USER || 'postgres',
  password: process.env.DB_PASSWORD || 'password',
  max: 20, // Maximum pool size
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,
});

export async function initializeDatabase() {
  const client = await pool.connect();
  try {
    await client.query(`
      CREATE TABLE IF NOT EXISTS orders (
        id VARCHAR(50) PRIMARY KEY,
        user_id VARCHAR(50) NOT NULL,
        amount DECIMAL(10, 2) NOT NULL,
        status VARCHAR(20) NOT NULL,
        created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
        updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
      )
    `);
    
    await client.query(`
      CREATE INDEX IF NOT EXISTS idx_orders_user_id ON orders(user_id);
      CREATE INDEX IF NOT EXISTS idx_orders_status ON orders(status);
    `);
    
    console.log('Database initialized');
  } finally {
    client.release();
  }
}

export interface Order {
  id: string;
  user_id: string;
  amount: number;
  status: 'pending' | 'processing' | 'completed' | 'failed';
  created_at: Date;
  updated_at: Date;
}

export async function createOrder(userId: string, amount: number): Promise<Order> {
  const orderId = `ORD-${Date.now()}`;
  const result = await pool.query<Order>(
    `INSERT INTO orders (id, user_id, amount, status)
     VALUES ($1, $2, $3, $4)
     RETURNING *`,
    [orderId, userId, amount, 'pending']
  );
  return result.rows[0];
}

export async function findOrderById(id: string): Promise<Order | null> {
  const result = await pool.query<Order>(
    'SELECT * FROM orders WHERE id = $1',
    [id]
  );
  return result.rows[0] || null;
}

export async function updateOrderStatus(id: string, status: Order['status']): Promise<void> {
  await pool.query(
    'UPDATE orders SET status = $1, updated_at = CURRENT_TIMESTAMP WHERE id = $2',
    [status, id]
  );
}

export async function findOrdersByUser(userId: string): Promise<Order[]> {
  const result = await pool.query<Order>(
    'SELECT * FROM orders WHERE user_id = $1 ORDER BY created_at DESC',
    [userId]
  );
  return result.rows;
}

Adding Redis for Caching

Start Redis with Docker:

docker run -d --name redis \
  -p 6379:6379 \
  redis:7-alpine

Create cache layer (src/cache.ts):

import Redis from 'ioredis';

export const redis = new Redis({
  host: process.env.REDIS_HOST || 'localhost',
  port: parseInt(process.env.REDIS_PORT || '6379'),
  retryStrategy: (times) => {
    const delay = Math.min(times * 50, 2000);
    return delay;
  },
});

const CACHE_TTL = 300; // 5 minutes

export async function getCachedOrder(orderId: string): Promise<string | null> {
  return await redis.get(`order:${orderId}`);
}

export async function setCachedOrder(orderId: string, orderData: string): Promise<void> {
  await redis.setex(`order:${orderId}`, CACHE_TTL, orderData);
}

export async function invalidateOrderCache(orderId: string): Promise<void> {
  await redis.del(`order:${orderId}`);
}

export async function incrementOrderCounter(userId: string): Promise<number> {
  return await redis.incr(`user:${userId}:order_count`);
}

Updated Application with Full Auto-Instrumentation

Update src/app.ts:

import express, { Express, Request, Response } from 'express';
import { trace, SpanStatusCode } from '@opentelemetry/api';
import { pool, initializeDatabase, createOrder, findOrderById, updateOrderStatus, findOrdersByUser } from './db';
import { redis, getCachedOrder, setCachedOrder, invalidateOrderCache, incrementOrderCounter } from './cache';

const PORT = parseInt(process.env.PORT || '3000');
const app: Express = express();
const tracer = trace.getTracer('order-service', '1.0.0');

app.use(express.json());

// Initialize database on startup
initializeDatabase().catch(console.error);

// Simulate payment processing
async function processPayment(orderId: string, amount: number): Promise<boolean> {
  return tracer.startActiveSpan('processPayment', async (span) => {
    try {
      span.setAttribute('order.id', orderId);
      span.setAttribute('payment.amount', amount);
      span.setAttribute('payment.currency', 'USD');
      
      // Simulate payment gateway call
      await new Promise(resolve => setTimeout(resolve, 200));
      const success = Math.random() > 0.1;
      
      span.setAttribute('payment.success', success);
      
      if (!success) {
        span.setStatus({
          code: SpanStatusCode.ERROR,
          message: 'Payment processing failed'
        });
      }
      
      return success;
    } finally {
      span.end();
    }
  });
}

// Routes
app.get('/health', (req: Request, res: Response) => {
  res.json({ status: 'healthy' });
});

app.post('/api/orders', async (req: Request, res: Response) => {
  try {
    const { userId, amount } = req.body;
    
    if (!userId || !amount) {
      return res.status(400).json({ error: 'userId and amount required' });
    }

    // Create order in database (auto-traced)
    const order = await createOrder(userId, amount);
    
    // Increment user order counter in Redis (auto-traced)
    await incrementOrderCounter(userId);
    
    // Update status
    await updateOrderStatus(order.id, 'processing');
    
    // Process payment
    const paymentSuccess = await processPayment(order.id, amount);
    
    if (paymentSuccess) {
      await updateOrderStatus(order.id, 'completed');
      
      // Cache the completed order (auto-traced)
      await setCachedOrder(order.id, JSON.stringify({ ...order, status: 'completed' }));
      
      res.status(201).json({ ...order, status: 'completed' });
    } else {
      await updateOrderStatus(order.id, 'failed');
      res.status(402).json({ error: 'Payment failed', order });
    }
  } catch (error) {
    console.error('Error creating order:', error);
    res.status(500).json({ error: 'Internal server error' });
  }
});

app.get('/api/orders/:id', async (req: Request, res: Response) => {
  try {
    const { id } = req.params;
    
    // Try cache first (auto-traced)
    const cached = await getCachedOrder(id);
    if (cached) {
      return res.json({ ...JSON.parse(cached), cached: true });
    }
    
    // Fetch from database (auto-traced)
    const order = await findOrderById(id);
    
    if (!order) {
      return res.status(404).json({ error: 'Order not found' });
    }
    
    // Cache for next time (auto-traced)
    await setCachedOrder(id, JSON.stringify(order));
    
    res.json({ ...order, cached: false });
  } catch (error) {
    console.error('Error fetching order:', error);
    res.status(500).json({ error: 'Internal server error' });
  }
});

app.get('/api/users/:userId/orders', async (req: Request, res: Response) => {
  try {
    const { userId } = req.params;
    
    // Fetch all user orders (auto-traced)
    const orders = await findOrdersByUser(userId);
    
    res.json({ orders, count: orders.length });
  } catch (error) {
    console.error('Error fetching user orders:', error);
    res.status(500).json({ error: 'Internal server error' });
  }
});

app.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
});

What Gets Traced Automatically

With this setup, every single operation is automatically traced:

HTTP Layer:

• Incoming HTTP requests (method, URL, status code, duration)

• Request/response headers

• User agent information

Database Layer (PostgreSQL):

• SQL queries with parameters

• Query duration

• Connection pool operations

• Database name and operation type

Cache Layer (Redis):

• Redis commands (GET, SET, INCR, DEL)

• Command arguments

• Response times

• Connection details

Example Trace in Jaeger:

POST /api/orders [250ms]
├─ INSERT INTO orders [45ms]
│  └─ postgres.query [45ms]
├─ INCR user:123:order_count [5ms]
│  └─ redis.incr [5ms]
├─ UPDATE orders SET status [12ms]
│  └─ postgres.query [12ms]
├─ processPayment [200ms]
├─ UPDATE orders SET status [8ms]
│  └─ postgres.query [8ms]
└─ SETEX order:ORD-123 [3ms]
   └─ redis.setex [3ms]

Configuration Options

Ignoring Specific Operations

instrumentations: [
  getNodeAutoInstrumentations({
    '@opentelemetry/instrumentation-http': {
      ignoreIncomingPaths: [
        '/health',
        '/metrics',
        '/favicon.ico'
      ],
      ignoreOutgoingUrls: [
        /^https:\/\/analytics\.google\.com/
      ]
    },
  }),
]

Enhancing Database Reporting

'@opentelemetry/instrumentation-pg': {
  enabled: true,
  enhancedDatabaseReporting: true, // Include query parameters
  responseHook: (span, result) => {
    // Add custom attributes
    span.setAttribute('db.rows_affected', result.rowCount);
  }
}

Sanitizing Sensitive Data

'@opentelemetry/instrumentation-pg': {
  enabled: true,
  requestHook: (span, queryConfig) => {
    // Remove sensitive data from queries
    if (queryConfig.text.includes('password')) {
      span.setAttribute('db.statement', 'REDACTED');
    }
  }
}

Real Production Learnings

1. Cache Hit Ratio Visibility

Auto-instrumentation revealed that my Redis cache hit rate was only 23%. I thought I had effective caching, but traces showed most requests hit the database. This led me to increase TTL and pre-warm critical data.

2. Connection Pool Exhaustion

During load testing, I noticed database query spans with 500ms wait times before execution. The auto-instrumented PostgreSQL library showed connection pool saturation. I increased the pool size from 10 to 30 connections.

3. Inefficient Queries

Auto-instrumentation showed that SELECT * FROM orders WHERE user_id = $1 was taking 300ms for users with many orders. I added pagination and saw query times drop to 15ms.

Debugging with Auto-Instrumentation

Scenario: Slow Endpoint

User reports: "Order lookup is slow"

1. Find slow traces in Jaeger filtered by /api/orders/:id

2. Analyze span duration:

   • HTTP request: 850ms

   • Redis GET: 2ms (cache miss)

   • PostgreSQL SELECT: 820ms ← Problem!

3. Check query details in span attributes

4. Identify missing index on frequently queried column

5. Add index, see queries drop to 12ms

All of this without writing a single custom trace!

Best Practices

1. Start with Auto-Instrumentation

Don't write custom spans until you've maxed out auto-instrumentation. It covers 80% of your needs.

2. Configure, Don't Disable

Instead of disabling noisy instrumentation, configure it to exclude specific paths or sanitize data.

3. Monitor Overhead

Auto-instrumentation adds minimal overhead (~2-5%), but always measure in your environment.

4. Use Semantic Conventions

Auto-instrumentation follows semantic conventions automatically. When you add custom attributes, follow the same patterns.

5. Test in Staging

Always test auto-instrumentation configuration in staging before production. Some libraries may have compatibility issues.

What's Next

You now have comprehensive automatic tracing for your entire stack. Continue to Manual Instrumentation Deep Dive where you'll learn:

• When to use custom spans vs auto-instrumentation

• Creating nested span hierarchies

• Adding business-specific attributes

• Span events and annotations

• Error handling and exception recording

---

Previous: ← Getting Started with TypeScript | Next: Manual Instrumentation Deep Dive →

Auto-instrumentation gives you the forest view. Manual instrumentation adds the tree details.