Metrics Collection

The Metrics vs Traces Dilemma

When I first started with OpenTelemetry, I tried to answer every question with traces. "How many orders per minute?" I'd count spans. "What's our error rate?" I'd filter failed spans. "Average response time?" Span duration aggregation.

This was a terrible idea. Traces are expensive—you can't keep every single one at high volume. I was sampling 10% of traffic, which meant my "metrics" were statistically wrong.

Then I discovered proper metrics, and everything clicked. Traces are for debugging individual requests. Metrics are for understanding system behavior over time.

The Three Types of Metrics

1. Counter: Counting Events

Counters only go up. They track cumulative totals.

When to use:

• Total requests processed

• Total errors encountered

• Total bytes sent

• Total orders completed

import { metrics } from '@opentelemetry/api';
import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions';

// Set up metrics
const metricExporter = new OTLPMetricExporter({
  url: 'http://localhost:4318/v1/metrics',
});

const meterProvider = new MeterProvider({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: 'order-service',
  }),
  readers: [
    new PeriodicExportingMetricReader({
      exporter: metricExporter,
      exportIntervalMillis: 10000, // Export every 10 seconds
    }),
  ],
});

metrics.setGlobalMeterProvider(meterProvider);

const meter = metrics.getMeter('order-service', '1.0.0');

// Create counters
const orderCounter = meter.createCounter('orders.created', {
  description: 'Total number of orders created',
  unit: '1',
});

const errorCounter = meter.createCounter('orders.errors', {
  description: 'Total number of order processing errors',
  unit: '1',
});

const revenueCounter = meter.createCounter('revenue.total', {
  description: 'Total revenue in USD',
  unit: 'USD',
});

// Using counters
export async function createOrder(userId: string, amount: number, items: any[]): Promise<Order> {
  try {
    const order = await saveOrderToDatabase(userId, amount, items);
    
    // Increment counter with attributes
    orderCounter.add(1, {
      'order.status': 'completed',
      'user.tier': await getUserTier(userId),
      'order.channel': 'web'
    });
    
    // Track revenue
    revenueCounter.add(amount, {
      'currency': 'USD',
      'payment.method': 'credit_card'
    });
    
    return order;
  } catch (error) {
    // Track errors
    errorCounter.add(1, {
      'error.type': (error as Error).name,
      'operation': 'createOrder'
    });
    throw error;
  }
}

async function saveOrderToDatabase(userId: string, amount: number, items: any[]): Promise<Order> {
  // Simulated
  return {
    id: `ORD-${Date.now()}`,
    userId,
    amount,
    items,
    status: 'completed',
    createdAt: new Date()
  };
}

async function getUserTier(userId: string): Promise<string> {
  return 'premium'; // Simulated
}

interface Order {
  id: string;
  userId: string;
  amount: number;
  items: any[];
  status: string;
  createdAt: Date;
}

2. Gauge: Measuring Current State

Gauges represent a value that can go up or down.

When to use:

• Current memory usage

• Active connections

• Queue size

• Items in cart

• Current temperature

const activeOrdersGauge = meter.createObservableGauge('orders.active', {
  description: 'Number of orders currently being processed',
  unit: '1',
});

const memoryUsageGauge = meter.createObservableGauge('process.memory.usage', {
  description: 'Current memory usage in bytes',
  unit: 'bytes',
});

const queueSizeGauge = meter.createObservableGauge('queue.size', {
  description: 'Current number of items in processing queue',
  unit: '1',
});

// Track active orders
let activeOrderCount = 0;

activeOrdersGauge.addCallback((result) => {
  result.observe(activeOrderCount, {
    'order.type': 'standard'
  });
});

// Track memory
memoryUsageGauge.addCallback((result) => {
  const memUsage = process.memoryUsage();
  result.observe(memUsage.heapUsed, { 'memory.type': 'heap' });
  result.observe(memUsage.rss, { 'memory.type': 'rss' });
  result.observe(memUsage.external, { 'memory.type': 'external' });
});

// Track queue size
import { Queue } from 'bull'; // Example queue library

const orderQueue = new Queue('orders', {
  redis: { host: 'localhost', port: 6379 }
});

queueSizeGauge.addCallback(async (result) => {
  const waiting = await orderQueue.getWaitingCount();
  const active = await orderQueue.getActiveCount();
  const delayed = await orderQueue.getDelayedCount();
  
  result.observe(waiting, { 'queue.state': 'waiting' });
  result.observe(active, { 'queue.state': 'active' });
  result.observe(delayed, { 'queue.state': 'delayed' });
});

// Update gauge when processing
export async function processOrder(orderId: string): Promise<void> {
  activeOrderCount++;
  
  try {
    await performOrderProcessing(orderId);
  } finally {
    activeOrderCount--;
  }
}

async function performOrderProcessing(orderId: string): Promise<void> {
  await new Promise(r => setTimeout(r, 1000));
}

3. Histogram: Distribution of Values

Histograms track the distribution of values over time.

When to use:

• Request duration

• Request payload size

• Order value distribution

• Database query duration

const orderDurationHistogram = meter.createHistogram('order.processing.duration', {
  description: 'Order processing duration in milliseconds',
  unit: 'ms',
});

const orderValueHistogram = meter.createHistogram('order.value', {
  description: 'Distribution of order values',
  unit: 'USD',
});

const payloadSizeHistogram = meter.createHistogram('http.request.size', {
  description: 'HTTP request payload size',
  unit: 'bytes',
});

// Record histograms
export async function processOrderWithMetrics(order: Order): Promise<void> {
  const startTime = Date.now();
  
  try {
    // Simulate order processing
    await validateOrder(order);
    await chargePayment(order);
    await updateInventory(order);
    await sendConfirmation(order);
    
    const duration = Date.now() - startTime;
    
    // Record duration
    orderDurationHistogram.record(duration, {
      'order.status': 'success',
      'order.item_count': order.items.length,
      'user.tier': 'premium'
    });
    
    // Record order value
    orderValueHistogram.record(order.amount, {
      'order.currency': 'USD',
      'order.channel': 'web'
    });
    
  } catch (error) {
    const duration = Date.now() - startTime;
    
    orderDurationHistogram.record(duration, {
      'order.status': 'failed',
      'error.type': (error as Error).name
    });
    
    throw error;
  }
}

async function validateOrder(order: Order): Promise<void> {
  await new Promise(r => setTimeout(r, 50));
}

async function chargePayment(order: Order): Promise<void> {
  await new Promise(r => setTimeout(r, 200));
}

async function updateInventory(order: Order): Promise<void> {
  await new Promise(r => setTimeout(r, 100));
}

async function sendConfirmation(order: Order): Promise<void> {
  await new Promise(r => setTimeout(r, 30));
}

Real-World Metrics Dashboard

Here's the complete metrics setup I use in production:

import express from 'express';
import { metrics } from '@opentelemetry/api';
import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PrometheusExporter } from '@opentelemetry/exporter-prometheus';
import { Resource } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';

// Configure metrics with multiple exporters
const prometheusExporter = new PrometheusExporter(
  {
    port: 9464, // Prometheus scrape endpoint
  },
  () => {
    console.log('Prometheus scrape endpoint: http://localhost:9464/metrics');
  }
);

const otlpExporter = new OTLPMetricExporter({
  url: 'http://localhost:4318/v1/metrics',
});

const meterProvider = new MeterProvider({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: 'order-service',
    [ATTR_SERVICE_VERSION]: '1.0.0',
  }),
  readers: [
    prometheusExporter,
    new PeriodicExportingMetricReader({
      exporter: otlpExporter,
      exportIntervalMillis: 10000,
    }),
  ],
});

metrics.setGlobalMeterProvider(meterProvider);
const meter = metrics.getMeter('order-service', '1.0.0');

// Business Metrics
const ordersCreated = meter.createCounter('business.orders.created.total', {
  description: 'Total orders created',
});

const ordersCompleted = meter.createCounter('business.orders.completed.total', {
  description: 'Total orders completed',
});

const ordersFailed = meter.createCounter('business.orders.failed.total', {
  description: 'Total orders that failed',
});

const revenue = meter.createCounter('business.revenue.total', {
  description: 'Total revenue',
  unit: 'USD',
});

// Technical Metrics
const httpRequestDuration = meter.createHistogram('http.server.request.duration', {
  description: 'HTTP request duration',
  unit: 'ms',
});

const httpRequestSize = meter.createHistogram('http.server.request.size', {
  description: 'HTTP request body size',
  unit: 'bytes',
});

const httpResponseSize = meter.createHistogram('http.server.response.size', {
  description: 'HTTP response body size',
  unit: 'bytes',
});

const httpRequestsTotal = meter.createCounter('http.server.requests.total', {
  description: 'Total HTTP requests',
});

// Database Metrics
const dbQueryDuration = meter.createHistogram('db.query.duration', {
  description: 'Database query duration',
  unit: 'ms',
});

const dbConnectionsActive = meter.createObservableGauge('db.connections.active', {
  description: 'Active database connections',
});

const dbQueriesTotal = meter.createCounter('db.queries.total', {
  description: 'Total database queries',
});

// Cache Metrics
const cacheHits = meter.createCounter('cache.hits.total', {
  description: 'Total cache hits',
});

const cacheMisses = meter.createCounter('cache.misses.total', {
  description: 'Total cache misses',
});

// Express middleware for HTTP metrics
function metricsMiddleware(req: express.Request, res: express.Response, next: express.NextFunction) {
  const startTime = Date.now();
  
  // Track request size
  const requestSize = parseInt(req.headers['content-length'] || '0');
  httpRequestSize.record(requestSize, {
    'http.method': req.method,
    'http.route': req.route?.path || req.path,
  });
  
  // Capture response
  const originalSend = res.send;
  res.send = function(data: any) {
    const responseSize = Buffer.byteLength(JSON.stringify(data));
    httpResponseSize.record(responseSize, {
      'http.method': req.method,
      'http.route': req.route?.path || req.path,
      'http.status_code': res.statusCode.toString(),
    });
    
    return originalSend.call(this, data);
  };
  
  res.on('finish', () => {
    const duration = Date.now() - startTime;
    
    // Record duration
    httpRequestDuration.record(duration, {
      'http.method': req.method,
      'http.route': req.route?.path || req.path,
      'http.status_code': res.statusCode.toString(),
    });
    
    // Count requests
    httpRequestsTotal.add(1, {
      'http.method': req.method,
      'http.route': req.route?.path || req.path,
      'http.status_code': res.statusCode.toString(),
    });
  });
  
  next();
}

// Application with metrics
const app = express();
app.use(express.json());
app.use(metricsMiddleware);

// Database connection pool tracking
import { Pool } from 'pg';

const pool = new Pool({
  host: 'localhost',
  port: 5432,
  database: 'orders',
  user: 'postgres',
  password: 'password',
  max: 20,
});

dbConnectionsActive.addCallback((result) => {
  result.observe(pool.totalCount, { 'connection.state': 'total' });
  result.observe(pool.idleCount, { 'connection.state': 'idle' });
  result.observe(pool.waitingCount, { 'connection.state': 'waiting' });
});

// Helper: Tracked database query
async function trackedQuery<T>(query: string, params: any[]): Promise<T[]> {
  const startTime = Date.now();
  
  try {
    const result = await pool.query<T>(query, params);
    
    const duration = Date.now() - startTime;
    dbQueryDuration.record(duration, {
      'db.operation': query.split(' ')[0].toUpperCase(), // SELECT, INSERT, etc.
      'db.success': 'true',
    });
    
    dbQueriesTotal.add(1, {
      'db.operation': query.split(' ')[0].toUpperCase(),
    });
    
    return result.rows;
  } catch (error) {
    const duration = Date.now() - startTime;
    dbQueryDuration.record(duration, {
      'db.operation': query.split(' ')[0].toUpperCase(),
      'db.success': 'false',
      'error.type': (error as Error).name,
    });
    
    throw error;
  }
}

// Helper: Tracked cache operations
import Redis from 'ioredis';

const redis = new Redis({
  host: 'localhost',
  port: 6379,
});

async function getCached(key: string): Promise<string | null> {
  const value = await redis.get(key);
  
  if (value !== null) {
    cacheHits.add(1, { 'cache.key_prefix': key.split(':')[0] });
  } else {
    cacheMisses.add(1, { 'cache.key_prefix': key.split(':')[0] });
  }
  
  return value;
}

// Business logic with metrics
app.post('/api/orders', async (req, res) => {
  try {
    const { userId, amount, items } = req.body;
    
    ordersCreated.add(1, {
      'order.channel': 'api',
      'user.tier': 'premium',
    });
    
    // Check cache
    const cachedUser = await getCached(`user:${userId}`);
    
    // Create order
    const orders = await trackedQuery<Order>(
      'INSERT INTO orders (id, user_id, amount, status) VALUES ($1, $2, $3, $4) RETURNING *',
      [`ORD-${Date.now()}`, userId, amount, 'pending']
    );
    
    const order = orders[0];
    
    // Process payment
    const paymentSuccess = Math.random() > 0.1;
    
    if (paymentSuccess) {
      await trackedQuery(
        'UPDATE orders SET status = $1 WHERE id = $2',
        ['completed', order.id]
      );
      
      ordersCompleted.add(1, {
        'order.channel': 'api',
        'payment.method': 'credit_card',
      });
      
      revenue.add(amount, {
        'currency': 'USD',
        'payment.method': 'credit_card',
      });
      
      res.status(201).json({ ...order, status: 'completed' });
    } else {
      await trackedQuery(
        'UPDATE orders SET status = $1 WHERE id = $2',
        ['failed', order.id]
      );
      
      ordersFailed.add(1, {
        'order.channel': 'api',
        'failure.reason': 'payment_declined',
      });
      
      res.status(402).json({ error: 'Payment failed' });
    }
  } catch (error) {
    ordersFailed.add(1, {
      'order.channel': 'api',
      'failure.reason': 'internal_error',
    });
    
    res.status(500).json({ error: 'Internal server error' });
  }
});

app.get('/api/orders/:id', async (req, res) => {
  try {
    const { id } = req.params;
    
    // Try cache
    const cached = await getCached(`order:${id}`);
    if (cached) {
      return res.json({ ...JSON.parse(cached), cached: true });
    }
    
    // Query database
    const orders = await trackedQuery<Order>(
      'SELECT * FROM orders WHERE id = $1',
      [id]
    );
    
    if (orders.length === 0) {
      return res.status(404).json({ error: 'Order not found' });
    }
    
    // Cache result
    await redis.setex(`order:${id}`, 300, JSON.stringify(orders[0]));
    
    res.json({ ...orders[0], cached: false });
  } catch (error) {
    res.status(500).json({ error: 'Internal server error' });
  }
});

app.listen(3000, () => {
  console.log('Server running on http://localhost:3000');
  console.log('Metrics available at http://localhost:9464/metrics');
});

Visualizing Metrics with Prometheus

Start Prometheus with Docker:

# Create prometheus.yml
cat > prometheus.yml <<EOF
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'order-service'
    static_configs:
      - targets: ['host.docker.internal:9464']
EOF

# Run Prometheus
docker run -d --name prometheus \
  -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

Visit http://localhost:9090 and query:

# Request rate
rate(http_server_requests_total[1m])

# Error rate
rate(business_orders_failed_total[1m]) / rate(business_orders_created_total[1m])

# P95 latency
histogram_quantile(0.95, rate(http_server_request_duration_bucket[5m]))

# Cache hit ratio
rate(cache_hits_total[1m]) / (rate(cache_hits_total[1m]) + rate(cache_misses_total[1m]))

# Revenue per minute
rate(business_revenue_total[1m])

Production Learnings: The Metrics That Mattered

1. Error Budget Monitoring

I track error budgets using metrics, not traces:

const errorBudgetGauge = meter.createObservableGauge('slo.error_budget.remaining', {
  description: 'Remaining error budget percentage',
  unit: '%',
});

let totalRequests = 0;
let failedRequests = 0;
const SLO_TARGET = 0.999; // 99.9% success rate

errorBudgetGauge.addCallback((result) => {
  const actualSLO = (totalRequests - failedRequests) / totalRequests;
  const errorBudget = ((actualSLO - SLO_TARGET) / (1 - SLO_TARGET)) * 100;
  
  result.observe(Math.max(0, errorBudget));
});

2. Capacity Planning

Metrics revealed we were hitting PostgreSQL connection limits at 1000 req/s:

dbConnectionsActive.addCallback((result) => {
  const utilization = (pool.totalCount / pool.options.max!) * 100;
  result.observe(utilization, { 'metric.type': 'utilization_percent' });
});

Alert when utilization > 80% → time to scale!

3. Business KPIs

Technical metrics don't tell the full story. Business metrics do:

// Average order value
const avgOrderValue = meter.createObservableGauge('business.order.average_value', {
  description: 'Average order value over last hour',
  unit: 'USD',
});

let orderValues: number[] = [];

avgOrderValue.addCallback((result) => {
  if (orderValues.length > 0) {
    const avg = orderValues.reduce((a, b) => a + b, 0) / orderValues.length;
    result.observe(avg);
  }
});

// Reset hourly
setInterval(() => { orderValues = []; }, 3600000);

Best Practices

1. Use Proper Metric Types

// ❌ Wrong: Using counter for current value
const activeUsers = meter.createCounter('users.active'); // NO!

// ✅ Right: Use gauge for current value
const activeUsers = meter.createObservableGauge('users.active');

2. Keep Cardinality Low

// ❌ Wrong: High cardinality
counter.add(1, {
  'user.id': userId, // Millions of users!
  'order.id': orderId, // Infinite values!
});

// ✅ Right: Low cardinality
counter.add(1, {
  'user.tier': 'premium', // Limited values
  'order.status': 'completed', // Limited values
});

3. Namespace Your Metrics

// Good metric naming
'business.orders.created.total'
'business.revenue.total'
'http.server.request.duration'
'db.query.duration'
'cache.hits.total'

4. Export to Multiple Backends

Production systems need both Prometheus (alerting) and cloud backends (long-term storage):

const meterProvider = new MeterProvider({
  readers: [
    prometheusExporter,           // For alerting
    new PeriodicExportingMetricReader({
      exporter: otlpExporter,      // For correlation with traces
    }),
  ],
});

What's Next

Continue to Distributed Tracing where you'll learn:

• Context propagation across services

• Trace correlation in microservices

• Debugging distributed systems

• Trace sampling strategies

---

Previous: ← Manual Instrumentation | Next: Distributed Tracing →

Metrics tell you what's happening. Traces tell you why.